[pymvpa] high prediction rate in a permutation test

[J.A. Etzel]

On 5/18/2011 3:54 PM, Yaroslav Halchenko wrote:
>>> permute truly independent (must be in the correct design) items:
>>> sequences of trials across runs: i.e. take sequence of labels from
>>> run 1, and place it into run X, and so across all runs.  That should
>>> account for possible inter-trial dependencies within runs, and thus I
>>> would expect that distribution would get even slightly wider (than if
>>> permuted within each run)
>> Not sure I follow ... you mean taking the order of trials from one
>> run and copying it to another, then partitioning on the runs?
>
> I guess "yes", if "partitioning on the runs" means "splitting into
> training and testing sets for cross-validation".

That is what I meant; too many terms for the same sorts of cross-validation!

But would this give you enough permutations for a decent distribution? I 
usually like at least 1000 if possible, but there are usually only a 
handful of runs. Isn't what you're describing (run-label copying) a 
special case of permuting the labels within each run? If the runs have 
the same number of trials one of the possible orderings would be the one 
from the previous run.

Jo