[pymvpa] high prediction rate in a permutation test

[Yaroslav Halchenko]

On Wed, 18 May 2011, J.A. Etzel wrote:
> I agree; I would be worried if the *middle* of the permutation
> distribution was around 0.6, but a wide distribution such that 0.6 is in
> the top 0.05 can happen.

yeap -- and even could be heavy tails at 0.7 and 0.8 and even 0.9 --
everything depends, especially on number of trials ;)

> >permute truly independent (must be in the correct design) items:
> >sequences of trials across runs: i.e. take sequence of labels from
> >run 1, and place it into run X, and so across all runs.  That should
> >account for possible inter-trial dependencies within runs, and thus I
> >would expect that distribution would get even slightly wider (than if
> >permuted within each run)
> Not sure I follow ... you mean taking the order of trials from one
> run and copying it to another, then partitioning on the runs?

I guess "yes", if "partitioning on the runs" means "splitting into
training and testing sets for cross-validation".

> >please correct me if I am wrong -- under permutation of samples
> >labels, those must differ regardless of block structure, simple due
> >to the change of number of trials (just compare binomial
> >distributions for 2 trials vs 4 ;) )
> Yes, the change in the variance of the permutation distribution
> could be just from the smaller number of samples. But I can imagine
> setting up dodgy classifications of individual trials from block
> designs that could also make the permutation distributions change
> (not that Vadim did that!), so wanted to mention double-checking the
> not-averaged partitioning scheme.

yeap ;)  

-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic