[pymvpa] high prediction rate in a permutation test

[Yaroslav Halchenko]

d'oh -- just now recalled that I have this email in draft:

eh, picture (histogram) would have been useful:

> I wanted to ask your opinion about some weird result that I get.
> To establish the significance I randomly permute my labels and I get a
> prediction rate of 0.6 and even above it (p-value=0.05). In other words 5%
> of of permuted samples result in 0.6+ prediction rate. The training/test
> samples are independent and ROI size is small (no overfitting).

just to make sure:  0.6 is not a mean-chance performance across the
permutations.  You just worry that the distribution of chance
performances is so wide that the right 5% tail is above 0.6 accuracy.

if that is the case, it is indeed a good example case ;)

> Interestingly, the described result I get when I average trials within block
> (use one data-point per block; ~25 blocks in total). When I run the

so it is 25 blocks for 2 conditions? which one has more? ;)

> classification on raw trials, my permutation threshold becomes ~0.55. In
> both cases for non-permuted labels the prediction is around significance
> level.
> How should I treat such a result? What might have gone wrong?

I guess nothing went wrong and everything is logical.  With of
random chance performances distribution is conditioned on many factors,
such as independence of samples, presence of order effects, how
permutation is done (disregarding dependence  of samples or not) etc.

So, to troubleshoot we could start with:

* histogram
* what kind of permutation testing have you done? (i.e. what was
  permutted exactly? was testing set labels permutted?)
  have you seen recently improved
  http://www.pymvpa.org/examples/permutation_test.html
  ? ;)

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]