[pymvpa] high prediction rate in a permutation test

[Vadim Axel]

Thank you a lot for your advices!
I have one more related question:
How reliable in your opinion would be to test significance of the
classification using  t-test vs.0.5, where my vector of classification
results contain subjects results? In other words, subject's A prediction was
0.6, B - 0.52, C - 0.55 etc. I take all those values as an input to t-test.
The values are independent and the normality condition is also fulfilled (I
can check it using Lilie-test).


On Wed, May 18, 2011 at 10:54 PM, Yaroslav Halchenko
<debian at onerussian.com>wrote:

>
> On Wed, 18 May 2011, J.A. Etzel wrote:
> > I agree; I would be worried if the *middle* of the permutation
> > distribution was around 0.6, but a wide distribution such that 0.6 is in
> > the top 0.05 can happen.
>
> yeap -- and even could be heavy tails at 0.7 and 0.8 and even 0.9 --
> everything depends, especially on number of trials ;)
>
> > >permute truly independent (must be in the correct design) items:
> > >sequences of trials across runs: i.e. take sequence of labels from
> > >run 1, and place it into run X, and so across all runs.  That should
> > >account for possible inter-trial dependencies within runs, and thus I
> > >would expect that distribution would get even slightly wider (than if
> > >permuted within each run)
> > Not sure I follow ... you mean taking the order of trials from one
> > run and copying it to another, then partitioning on the runs?
>
> I guess "yes", if "partitioning on the runs" means "splitting into
> training and testing sets for cross-validation".
>
> > >please correct me if I am wrong -- under permutation of samples
> > >labels, those must differ regardless of block structure, simple due
> > >to the change of number of trials (just compare binomial
> > >distributions for 2 trials vs 4 ;) )
> > Yes, the change in the variance of the permutation distribution
> > could be just from the smaller number of samples. But I can imagine
> > setting up dodgy classifications of individual trials from block
> > designs that could also make the permutation distributions change
> > (not that Vadim did that!), so wanted to mention double-checking the
> > not-averaged partitioning scheme.
>
> yeap ;)
>
> --
> =------------------------------------------------------------------=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110518/fff26f80/attachment-0001.htm>