[pymvpa] mysterious whole-brain decoding

[yoh at onerussian.com]

On Tue, 21 Jan 2020, Floris van Vugt wrote:
>...
> **Shortest version**
> What may I be doing wrong if I my group-level significance maps for
> searchlight decoding cover the entire brain?

> **Slightly longer version**
> I ran a movement study in fMRI and I am decoding on a subject-level basis
> which of a set of 4 movements was performed on each trial.

and that is where IMHO the likely source of your trouble is: a physical
movement was performed by subjects.  Depending on the type of movement
(one thing is finger tapping; moving different arms or legs would
be a different story), such movement is likely to be reflected in the
"head movement".   Movement introduces non-linear effects into your
data.  "Motion correction" is doing its best to eliminate some effects
of motion (ideally not correlated with the experiment, which is the case
here), but simply cannot remove it entirely for a variety of reasons.
So, motion corrected data will reflect motion.  That is why
resting-state data analysis typically includes "scrubbing", where
periods of notable motion are simply "removed" or some other way
"imputed".

So what the physical motion in your experiment?

Was motion correlated with the design?:
Before anything, I would have just trained a  classifier on the motion
estimate parameters (translation, rotation, may be with mean displaement
added as yet another featyre) instead of fmri data itself.  If,
across subjects, you have "significant" decoding performance -- it would
mean that your fmri data would have those confounds in it as well, may be
represented more weekly after motion correction, but still there.  That
could explain your "whole brain" decoding.  If you fail to achieve
decoding on motion correction parameters,  it is a good sign but not a 
guarantee that data is not reflecting actual physical motion.

Now some comments inline:

> decode which movement they did. I use off-the-shelf cross-validated
> spherical searchlight decoding with SVM on the subject level, yielding maps
> that to me look pretty decent, with decoding close to theoretical chance
> level. 

if there is consistent (across subjects) skew in errors to "good
side", then those little effects would be your 'significant whole brain
cluster'.

> Then I generate group-level significance levels using the Stelzer
> method (GroupClusterThreshold), i.e. I re-run decoding on subject-level
> datasets with permuted labels, and then sample from those to create a

here also -- permutation could be of a tricky business depending on your
design.  What was it -- block vs single trial?  completely randomized across
subjects or used some pregenerated sequences and swapped them across
runs?

> the entire brain. That is, each and every voxel is deemed to significantly
> decode the movement that subjects make.
> See the linked glass brain image, in which the decoding accuracy is shown
> for each voxel: https://drive.google.com/file/d/1Lb6e1qZlKDcmu5WDAV84McRKDi1ITL4P/view?usp=sharing

would be interesting to see a colorbar and/or distribution of values.  I
guess chance level should be 0.25 (since 4 types of movements)

> I can also submit scripts but it's a bit complex because it's part of a
> large pipeline.
> I felt maybe some of you off the top of your head can already indicate what
> might be going wrong here or maybe somebody has experienced something
> similar.

see above

> If you want an actual reproducible pipeline then I can generate that, I just
> wanted to see if I could spare myself that effort :)

well... if you can achieve that -- it would be of benefit to yourself,
and your future publication!

NB shameless plug -- you might like to familiarize yourself with another
project which we have been developing for the past decade pretty much --
http://datalad.org . datalad run   would be your friend here to "commit"
a record of your reproducible step.  datalad-container extension could
also come handy to guarantee that you have the computing environment
anywhere you go now, or come back to it all later on ;)

> So at this point, feel free to respond: "need more details, send code" and I
> will do that.

"need more details", see above questions.  others might like to see code
as well, but I am afraid the devil is in the data.

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik