[pymvpa] high prediction rate in a permutation test
jetzel at artsci.wustl.edu
Fri May 20 17:19:02 UTC 2011
I'm not convinced that swapping the labels of entire runs at a time
would constitute a stricter/more accurate test for overall accuracy than
swapping labels within a run, even in cases where it's possible (many
runs, different orderings of events within each run and the same number
of events in each run).
I can see that the procedure might be useful for some designs, such as
if there's concern about subtle order effects. But then the
whole-run-label-swapping would be used to check for order effects, not
the general accuracy.
I think the two label permutation methods (of trial labels or of entire
runs of labels) are estimating different null distributions. When it's
not possible to calculate all the label permutations (because there's
too many) we pick some permutations at random, hoping to have enough to
estimate the shape of the distribution if we were to have calculated
them all. But that's only ok if we choose the labels at random. Swapping
the labels of entire runs estimates a different distribution; a much
smaller number of possible label rearrangements.
I can see the logic, in that trials from different runs should be more
"independent" than trials from within the same run. But even trials from
different runs are not fully independent: they are from the same person,
there is a temporal order to the runs (so the person might be getting
tired as the experiment goes on), etc.
On 5/20/2011 10:02 AM, Yaroslav Halchenko wrote:
> well, it is a number of combinations but they might be "illegal".
> Non-parametric permutation testing requires the permuted units to be
> independent. If you believe/assume/guarantee_somehow that your
> samples are independent and free from order/run effects -- then go
> ahead. If not -- you might take the permutation unit where you can
> guarantee independence and that was my intent for the suggested
> permutation of label sequences across runs.
>> Isn't what you're describing (run-label copying) a special case of
>> permuting the labels within each run?
> yes in the sense that indeed by chance for some permutations you
> might get an order which would match some run's order, but it would
> not be very probable (depending on number of trials in a run of
More information about the Pkg-ExpPsy-PyMVPA