[pymvpa] How you run permutation test for search-light analysis?

Francisco Pereira francisco.pereira at gmail.com
Tue Mar 27 00:41:13 UTC 2012

I tend to go for something like one of the two procedures below
(Procedure A is analytical, Procedure B is nonparametric). I would
love to hear comments, objections or to be told I'm completely wrong
and should do otherwise :)


Procedure A:

Part 1) within-subject:

1) get p-values for each searchlight (analytical or, preferably,
   (if a gaussian naive bayes classifier is fine, then PyMVPA or
Searchmight can be used to generate permutation p-values very quickly)

2) threshold using FDR (let's say 0.01), yielding a binary image --
"significance map", say -- with 1 for voxels where accuracy was

Part 2) between-subjects

1) if subjects have been registered to a template, add their binary
images to produce an image of counts in each voxel (if not, you can
register all the binary images much more cheaply than the entire
datasets), let's say a "count map". One  could stop here, in my
opinion, given that the previous images have been tested and multiple
comparison correction performed.

2) if you really want p-values at the group level, you could reason as follows:

- if you had a single subject, you would know that out of the 1s in
its significance map q% will be false discoveries on average
- if you had two subjects, you can have these counts at a given voxel
  - 0 (no 1s from either subject)
  - 1 (a 1 from subject 1 or a 1 from subject 2)
  - 2 (1s from both subjects)

The issue here is what null hypothesis you want to test, I think. I
would like to be certain that I can say the following statements in
face of the respective counts
  - 2 - the probability that there are two false positives is 0.01 x 0.01
  - 1 - the probability that there is one false positive is 0.01 (even
though you have two subjects)

but I'm not entirely sure...

Procedure B

This is conceptually simpler. As above, you will be

- obtaining a significance map per subject
- adding them up across subjects

but you will do these many times, bootstrapping one dataset per
subject in each iteration. This gives you a distribution over the
values of the counts in a count map, which you can use as you see fit
(e.g. compute a confidence envelope across the image).

On Mon, Mar 26, 2012 at 8:09 PM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
> sounds like a reasonable conservative approach indeed.  The question is
> when to stop searching for the most worst performer ;)
> On Mon, 26 Mar 2012, Jonas Kaplan wrote:
>> I too, wish we had a better solution to this problem.
>> What we have done recently is to do permutations for a bunch of different voxels around the brain, and upon finding that the null distributions were not different from each other for those voxels, we used the threshold from the most conservative of those distributions (corrected for multiple comparisons) across the brain.
>> -Jonas
> --
> =------------------------------------------------------------------=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

More information about the Pkg-ExpPsy-PyMVPA mailing list