[pymvpa] Mixed effects and searchlight group significance thresholding

[Mike E. Klein]

Thanks for the advice!

Best,
Mike

On Monday, July 9, 2012, J.A. Etzel wrote:

> Group analysis for searchlight results is unfortunately not
> straightforward or agreed-upon. But a few thoughts, anyway.
>
> First, I would not do small-volume corrections. This is mixing whole-brain
> (the searchlight) and ROI-based analyses/hypotheses. If you have a
> ROI-based hypothesis you should do ROI-based analyses (test the region
> directly); otherwise it's too easy to draw the ROIs around the blobs and
> create positive results out of anything.
>
> Smoothing the single-subject maps then doing a 'normal' mass-univariate
> analysis in spm is a safer strategy, though as you point out, information
> maps are definitely not activation maps. I'd suggest trying something like
> whole-brain FDR or FWE with a reasonable cluster size threshold. You might
> consider thresholding at p < .1 or something if p < .05 is too restrictive;
> justifiable in my opinion, given how different searchlight data is from
> 'normal' fMRI data.
>
> Given how few subjects you have, I'd also present the single-subject maps;
> obvious results in each subject makes the group results convincing even if
> the group-level p-values are less significant than you might hope.
>
> good luck,
> Jo
>
>
> On 7/5/2012 12:21 PM, Mike E. Klein wrote:
>
>> Hi everyone,
>>
>> I'm attempting to threshold group data for a searchlight-based MVPA . I
>> am performing the group-wise stats via a standard top-level analysis in
>> SPM (using single-subject searchlight accuracy maps as inputs). I am
>> having difficulty figuring out where to set significance thresholds. SPM
>> is using a purely random-effects calculation on the data (n=9, so df=8),
>> leading to enormous t-thresholds (~16), which are impossible to reach
>> and seem way too conservative. If I do small-volume corrections on our
>> a-priori regions of interest, the effects are significant (but not by
>> much...t-thresholds in the 8s), so this seems less than ideal (and
>> ignores most of the brain).
>>
>> Typically, for GLM analyses, we use a "mixed effects" model, which
>> incorporates both within- and between-subjects statistics, yielding an
>> "effective" degree of freedom (which is much higher than the number of
>> study participants, though much lower than the total number of trials in
>> the experiment). However, I am not sure (a) how to calculate this for an
>> MVPA study or (b) if the same set of assumptions hold. Our nine subjects
>> each underwent 9 functional runs (used for 8 -> 1 leave one out cross
>> validation). So each subjects searchlight map was reflecting an average
>> of these nine folds. We used 81 total examples per condition (9 per
>> run), which were temporally averaged, leaving 27 examples per condition
>> that were fed into an SVM. Single-subject results were warped into
>> standard space and also explicitly smoothed with a 7mm gaussian kernel,
>> before being fed into SPM.
>>
>> We have strong results, so really I'm looking for the "most proper" way
>> to perform searchlight group significance testing. Because we're doing
>> 35,000-45,000 spheres per subject, I don't think permutation testing is
>> feasible. There's also the option of reporting p<0.05 FWE stats for the
>> pre-defined ROIs, and p<0.001 (uncorrected) for the rest of the brain,
>> for completeness sake.
>>
>> Any advice is greatly appreciated!
>>
>> Best,
>> Mike
>>
>>
>> ______________________________**_________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**
>> pkg-exppsy-pymvpa<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>>
>>
> --
> Joset A. Etzel, Ph.D.
> Research Analyst
> Cognitive Control & Psychopathology Lab
> Washington University in St. Louis
> http://mvpa.blogspot.com/
>
>
>
> ______________________________**_________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**
> pkg-exppsy-pymvpa<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120727/c5adc832/attachment.html>