[pymvpa] How you run permutation test for search-light analysis?
J.A. Etzel
jetzel at artsci.wustl.edu
Mon Mar 26 21:35:29 UTC 2012
If I read that section correctly, they ran the complete searchlight
analysis 10 times, then pooled the calculated correlation values across
voxels to determine the significance.
This strikes me potentially troublesome if you want to use the test to
make strong conclusions about *which* voxels have significant
searchlights. In other words, if (in your case) you want to make a claim
that some of the 3000 searchlights classify significantly and others
don't, I wouldn't suggest pooling the permutation distributions over
searchlights. It is very possible (even likely) that different
searchlights have drastically different null distributions.
In my opinion proper statistical control (and interpretation) of
searchlight analyses is still an open question. Parametric statistics
(e.g. t-test for accuracy above chance) are not perfect, but at least
are fast. If some of your 3000 searchlights are truly of more interest
than others you could perhaps run the permutation test for those, then
compare those p-values with the parametric ones to give you some
confidence in the parametric ones. Something like a Mann-Whitney U test
might also be useful if you want to avoid too much outlier instability.
Out of curiosity, why are you running non-overlapping searchlights? Are
you testing brain parcellation schemes?
Jo
On 3/25/2012 3:56 AM, Vadim Axel wrote:
> Hi,
>
> When I run a whole-brain search-light analysis with non-overlaping
> lights (each voxel participates only in one light) I get about 3000
> lights (half a hour classification). Running 1000 times such permutation
> analysis would obviously take weeks. And this is a limited
> non-overlapping lights case. Is there any more realistic approach? I
> found here that people ran permutation analysis only 10 times while they
> stack the predictions of all voxels together to have their chance
> distribution:
> http://cercor.oxfordjournals.org/content/early/2011/12/20/cercor.bhr357.full
> (see "Searchlight analysis" section second paragraph). Does it make
> sense for you?
>
> Thanks,
> Vadim
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
More information about the Pkg-ExpPsy-PyMVPA
mailing list