[pymvpa] high prediction rate in a permutation test

[Vadim Axel]

Hi,

Thank you both for the answers!


1. The mean chance is a perfect 0.5. The 0.6 is a tail.

2. I have 6 trials per block and 25 blocks for each condition in total. So,
in one scenario I average the trials within block and make classification
based on 25 data points per condition. There I get 0.6 permutated prediction
in the tail. In other case I do not average and run classification based on
25x6=150 data points per condition. There I get ~0.55 permutated prediction
in the tail. I attach the histograms for mean and for non-mean permutation.
For raw data definitely looks more normal.

3. I did a manual permutation by reshuffling the labels. In particular, I
have a matrix of data values [trials X voxels] and a vector of correct
labels [correct labels x 1].  For each permutation test I randomize the
order of correct labels vector. Makes sense? As far as I understand the
Monte-Carlo simulation works for artificially generated data values. But I
am using my original data labels.

BTW, I did not use for this analysis PyMVPA, so you have no reason to worry
about potential bug in your code :)


Thanks again,
Vadim



On Mon, May 16, 2011 at 7:53 PM, Yaroslav Halchenko
<debian at onerussian.com>wrote:

> d'oh -- just now recalled that I have this email in draft:
>
> eh, picture (histogram) would have been useful:
>
> > I wanted to ask your opinion about some weird result that I get.
> > To establish the significance I randomly permute my labels and I get a
> > prediction rate of 0.6 and even above it (p-value=0.05). In other words
> 5%
> > of of permuted samples result in 0.6+ prediction rate. The training/test
> > samples are independent and ROI size is small (no overfitting).
>
> just to make sure:  0.6 is not a mean-chance performance across the
> permutations.  You just worry that the distribution of chance
> performances is so wide that the right 5% tail is above 0.6 accuracy.
>
> if that is the case, it is indeed a good example case ;)
>
> > Interestingly, the described result I get when I average trials within
> block
> > (use one data-point per block; ~25 blocks in total). When I run the
>
> so it is 25 blocks for 2 conditions? which one has more? ;)
>
> > classification on raw trials, my permutation threshold becomes ~0.55. In
> > both cases for non-permuted labels the prediction is around significance
> > level.
> > How should I treat such a result? What might have gone wrong?
>
> I guess nothing went wrong and everything is logical.  With of
> random chance performances distribution is conditioned on many factors,
> such as independence of samples, presence of order effects, how
> permutation is done (disregarding dependence  of samples or not) etc.
>
> So, to troubleshoot we could start with:
>
> * histogram
> * what kind of permutation testing have you done? (i.e. what was
>  permutted exactly? was testing set labels permutted?)
>  have you seen recently improved
>  http://www.pymvpa.org/examples/permutation_test.html
>  ? ;)
>
> --
>                                  .-.
> =------------------------------   /v\  ----------------------------=
> Keep in touch                    // \\     (yoh@|www.)onerussian.com
> Yaroslav Halchenko              /(   )\               ICQ#: 60653192
>                   Linux User    ^^-^^    [175555]
>
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110518/145ba0c6/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vk_non_mean_vs_mean.JPG
Type: image/jpeg
Size: 39369 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110518/145ba0c6/attachment-0001.jpeg>