[pymvpa] Searchlight and permutation tests.

[Jo Etzel]

On 4/26/2016 8:41 AM, Roberto Guidotti wrote:
> The across-subject average map has some suspicious results, the accuracy
> histogram is not peaked at chance level (0.5) but is peaked at
> 0.55-0.56, so the majority of voxels has that range of value. Do you
> think is it reasonable? Or it depends on some cross-validation scheme,
> beta issue, or who knows?
 >
If I understand, you ran a whole-brain searchlight analysis, then made a 
histogram of the group-level accuracies, and are wondering what that 
histogram should look like?

My advice is actually not to use histograms to evaluate the results: 
location matters a lot in searchlight analysis. My approach is to look 
at the single-subject and group-level results as brains: do the peak 
areas appear in clusters? (They should for searchlight analysis, since 
adjacent searchlights overlap so much.) Do many of the individual people 
have the same peaks as appear in the group level? Do control analyses 
look reasonable? etc.

It very well could be that a large number of searchlights have 
accuracies just a bit above chance, particularly if your conditions 
differ in effort or something else that makes widespread differences. My 
usual practice when starting a new analysis is to set the threshold for 
plotting accuracy (two class, balanced) at 0.6 for visualization 
purposes, figuring accuracies below that are unlikely to be meaningful. 
But 0.6 is strictly a guideline, not a universal rule or statistical test!

>
> To validate that result I ran a exploratory permutation test (n=100) on
> a single subject to look at accuracy distribution, in that subject, the
> histogram after permutation test is correctly peaked at chance level
> (the map with correct labeling is peaked at 0.57). I don't know if I'm
> correct but this should validate the hypothesis that the map histogram
> peaked at 0.55-0.56 is reasonable!
>
I generally see pretty normal null distributions centered at chance, 
with the variance of the distributions changing between analyses more 
than the shape. (And I think normal null distributions should occur.) 
But the permutation scheme can have a big impact, and schemes other than 
dataset-wise can make very non-normal null distributions.

Jo

>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>

-- 
Joset A. Etzel, Ph.D.
Research Analyst
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
http://mvpa.blogspot.com/