[pymvpa] Interpreting mis-classification

[Yaroslav Halchenko]

Hi Matthew again,

On Fri, 11 Sep 2009, Matthew Cieslak wrote:
>    I have a non-software-related question for you all.
lets hope that is indeed non-software related (ie not related to any bug
within PyMVPA :-P )

>    voxels. If there are only two categories you are trying to classify,
>    how would interpret large spatially contiguous clusters of voxels that
>    are performing significantly below chance (20% correct or so) appearing
>    in the results?
;-) You hit the nail with such a question: what is actually
chance-performance distribution of generalization errors given
a. your data (how samples are composed, preprocessed, etc)
b. your design (labels, their temporal relations, etc)
c. your choice of the classifier
d. your error metric (with NFold -- how many folds)
e. your 'searchlight' model (ok, it is sphere, then just how large it
   is)

Spatially contiguous clusters shouldn't be anything surprising given
the nature of the searchlight, especially with growing size of the
sphere and preprocessing (spatial filtering either explicit, or just due
to interpolation during motion correction).  So, (a) and (e) are imho
primary reasons, but there could be also all those 'veins' etc effects
which would contribute as a part of (a).

Now, why 20%?...

> Could it be that there is something changing in the
>    data in relation to the number of times the subject has seen examples
>    of a category?

could be... then it is indeed 'science/research'... More abstract but
somewhat studied effect in machine learning could be 

An Analysis of the Anti-Learning Phenomenon
for the Class Symmetric Polyhedron
Adam Kowalczyk and Olivier Chapelle
http://eprints.pascal-network.org/archive/00001154/01/alt_05.pdf

or it could be simply due to the broaden by-chance distribution due to
(a-e) + multiple model comparisons inherent in the searchlight procedure

> If mis-classification could be caused by an across run
>    repetition-suppression type effect, would re-running the searchlights
>    with an odd-even split and seeing if these voxels are at chance be a
>    legitimate way to show there is meaning in the mis-classification of my
>    svm's?

I just would suggest to assess the true 'by-chance' distribution. I bet
you have more of fMRI data where design (labels/chunks) was not the same
as in your current study.  Then just use similar preprocessing, and run exactly
the same analysis (just change input file name ;)) with your labels for the
given subject -- and look at the distribution of performances/anti-learning
(20%) blobs (if present), before jumping into making psychological conclusions ;)

>    I haven't been able to find any neuroimaging papers that address or
>    report below-chance performance, does anyone know of one? Maybe it
>    would be better to search the machine-learning literature?

Shame on us I guess -- but let me reveal that just now we are working on smth
like you need (i.e. paper).  As for model comparisons and generalization,
Pereira's paper provides some of the reasoning on how not to do evil or why it
matters:

Pereira, F., Mitchell, T. & Botvinick, M. Machine learning classifiers
and fMRI: A tutorial overview. Neuroimage.
DOI: http://dx.doi.org/10.1016/j.neuroimage.2008.11.007 

                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]