[pymvpa] null classification performance in the presence of strong univariate signal??

[J.A. Etzel]

On 9/8/2014 4:31 AM, Nick Oosterhof wrote:
> On Sep 8, 2014, at 11:21 AM, David Soto <d.soto.b at gmail.com> wrote:
> If I understand correctly, you have two samples per subject (one for
> each condition), and each value for a chunk corresponds to one
> subject. With those parameters you would be doing between-subject
> classification. Are you sure that is what you want?
>
> I'm asking, because /almost/ all MVPA (hyperalignment (TM) being an
> exception) are doing within-subject analysis. If you have not done so
> yet, I suggest strongly to do within-subject analysis first before
> trying between-subject analysis.

I disagree a bit about needing to start with within-subjects analysis: 
this strikes me as a situation in which the between-subjects analysis 
should work quite well. The strong mass-univariate results suggests 
there is a consistent difference in the estimates across the people 
(higher values in condition A than B), and the between-subjects analysis 
should pick this up; particularly a searchlight analysis (particularly 
since it only uses a few voxels and so is in some ways closer to a 
mass-univariate analysis), and particularly with a linear classifier 
(which is sensitive to consistent differences in individual voxels).

I wouldn't bother with the kNN classifier during the troubleshooting, 
but just stick with the linear SVM (kNN tend to be unpredictably bad 
with fMRI data). I'm out of practice reading pyMVPA code, but assume you 
set it up as leave-one-subject-out cross-validation (train on 18 people, 
test on 1)? Since each person contributes 1 parameter estimate image to 
the dataset, the classes should be perfectly balanced in all training 
and testing sets (which is good).

What sort of pattern are you seeing after the searchlight analysis? 
Everything more-or-less at chance, or something else?

My first guess is that there is some sort of bug in the analysis code or 
parameter estimate image labeling/extracting; my second guess is that 
the normalizing went wrong, or removed your signal (do you see rings or 
'doughnuts' around GLM peak areas in the searchlight maps?).

For troubleshooting, you could try picking a few voxels with very strong 
(and not) effects in the GLM, then plotting the values from the 
parameter estimate images for those same voxels. You should see a strong 
difference between the two classes in the plots for the 'good' (strong 
GLM effect) voxels, and less of a difference at the 'bad' (weak GLM 
effect) voxels. Depending on how everything was set up, you could fit a 
linear model on the individual voxels to match against the original GLM, 
and run linear SVM on those individual voxels as well: the accuracies 
should be consistent with the GLM values (ie better accuracy on 'good' 
voxels).

good luck,
Jo


> _______________________________________________
 > Pkg-ExpPsy-PyMVPA mailing list
 > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>
-- 
Joset A. Etzel, Ph.D.
Research Analyst
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
http://mvpa.blogspot.com/