[pymvpa] high prediction rate in a permutation test

[J.A. Etzel]

I'm not convinced that swapping the labels of entire runs at a time 
would constitute a stricter/more accurate test for overall accuracy than 
swapping labels within a run, even in cases where it's possible (many 
runs, different orderings of events within each run and the same number 
of events in each run).

I can see that the procedure might be useful for some designs, such as 
if there's concern about subtle order effects. But then the 
whole-run-label-swapping would be used to check for order effects, not 
the general accuracy.

I think the two label permutation methods (of trial labels or of entire 
runs of labels) are estimating different null distributions. When it's 
not possible to calculate all the label permutations (because there's 
too many) we pick some permutations at random, hoping to have enough to 
estimate the shape of the distribution if we were to have calculated 
them all. But that's only ok if we choose the labels at random. Swapping 
the labels of entire runs estimates a different distribution; a much 
smaller number of possible label rearrangements.

I can see the logic, in that trials from different runs should be more 
"independent" than trials from within the same run. But even trials from 
different runs are not fully independent: they are from the same person, 
there is a temporal order to the runs (so the person might be getting 
tired as the experiment goes on), etc.

Jo

On 5/20/2011 10:02 AM, Yaroslav Halchenko wrote:
> well, it is a number of combinations but they might be "illegal".
> Non-parametric permutation testing requires the permuted units to be
> independent.  If you believe/assume/guarantee_somehow that your
> samples are independent and free from order/run effects -- then go
> ahead.  If not -- you might take the permutation unit where you can
> guarantee independence and that was my intent for the suggested
> permutation of label sequences across runs.

>> Isn't what you're describing (run-label copying) a special case of
>> permuting the labels within each run?
> yes in the sense that indeed by chance for some permutations you
> might get an order which would match some run's order, but it would
> not be very probable (depending on number of trials in a run of
> cause).