[pymvpa] Searchlight and permutation tests.
jetzel at wustl.edu
Tue Apr 26 14:00:17 UTC 2016
On 4/26/2016 8:41 AM, Roberto Guidotti wrote:
> The across-subject average map has some suspicious results, the accuracy
> histogram is not peaked at chance level (0.5) but is peaked at
> 0.55-0.56, so the majority of voxels has that range of value. Do you
> think is it reasonable? Or it depends on some cross-validation scheme,
> beta issue, or who knows?
If I understand, you ran a whole-brain searchlight analysis, then made a
histogram of the group-level accuracies, and are wondering what that
histogram should look like?
My advice is actually not to use histograms to evaluate the results:
location matters a lot in searchlight analysis. My approach is to look
at the single-subject and group-level results as brains: do the peak
areas appear in clusters? (They should for searchlight analysis, since
adjacent searchlights overlap so much.) Do many of the individual people
have the same peaks as appear in the group level? Do control analyses
look reasonable? etc.
It very well could be that a large number of searchlights have
accuracies just a bit above chance, particularly if your conditions
differ in effort or something else that makes widespread differences. My
usual practice when starting a new analysis is to set the threshold for
plotting accuracy (two class, balanced) at 0.6 for visualization
purposes, figuring accuracies below that are unlikely to be meaningful.
But 0.6 is strictly a guideline, not a universal rule or statistical test!
> To validate that result I ran a exploratory permutation test (n=100) on
> a single subject to look at accuracy distribution, in that subject, the
> histogram after permutation test is correctly peaked at chance level
> (the map with correct labeling is peaked at 0.57). I don't know if I'm
> correct but this should validate the hypothesis that the map histogram
> peaked at 0.55-0.56 is reasonable!
I generally see pretty normal null distributions centered at chance,
with the variance of the distributions changing between analyses more
than the shape. (And I think normal null distributions should occur.)
But the permutation scheme can have a big impact, and schemes other than
dataset-wise can make very non-normal null distributions.
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
Joset A. Etzel, Ph.D.
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
More information about the Pkg-ExpPsy-PyMVPA