[pymvpa] null classification performance in the presence of strong univariate signal??
jetzel at artsci.wustl.edu
Mon Sep 8 14:02:05 UTC 2014
On 9/8/2014 4:31 AM, Nick Oosterhof wrote:
> On Sep 8, 2014, at 11:21 AM, David Soto <d.soto.b at gmail.com> wrote:
> If I understand correctly, you have two samples per subject (one for
> each condition), and each value for a chunk corresponds to one
> subject. With those parameters you would be doing between-subject
> classification. Are you sure that is what you want?
> I'm asking, because /almost/ all MVPA (hyperalignment (TM) being an
> exception) are doing within-subject analysis. If you have not done so
> yet, I suggest strongly to do within-subject analysis first before
> trying between-subject analysis.
I disagree a bit about needing to start with within-subjects analysis:
this strikes me as a situation in which the between-subjects analysis
should work quite well. The strong mass-univariate results suggests
there is a consistent difference in the estimates across the people
(higher values in condition A than B), and the between-subjects analysis
should pick this up; particularly a searchlight analysis (particularly
since it only uses a few voxels and so is in some ways closer to a
mass-univariate analysis), and particularly with a linear classifier
(which is sensitive to consistent differences in individual voxels).
I wouldn't bother with the kNN classifier during the troubleshooting,
but just stick with the linear SVM (kNN tend to be unpredictably bad
with fMRI data). I'm out of practice reading pyMVPA code, but assume you
set it up as leave-one-subject-out cross-validation (train on 18 people,
test on 1)? Since each person contributes 1 parameter estimate image to
the dataset, the classes should be perfectly balanced in all training
and testing sets (which is good).
What sort of pattern are you seeing after the searchlight analysis?
Everything more-or-less at chance, or something else?
My first guess is that there is some sort of bug in the analysis code or
parameter estimate image labeling/extracting; my second guess is that
the normalizing went wrong, or removed your signal (do you see rings or
'doughnuts' around GLM peak areas in the searchlight maps?).
For troubleshooting, you could try picking a few voxels with very strong
(and not) effects in the GLM, then plotting the values from the
parameter estimate images for those same voxels. You should see a strong
difference between the two classes in the plots for the 'good' (strong
GLM effect) voxels, and less of a difference at the 'bad' (weak GLM
effect) voxels. Depending on how everything was set up, you could fit a
linear model on the individual voxels to match against the original GLM,
and run linear SVM on those individual voxels as well: the accuracies
should be consistent with the GLM values (ie better accuracy on 'good'
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
Joset A. Etzel, Ph.D.
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
More information about the Pkg-ExpPsy-PyMVPA