[pymvpa] Interpreting mis-classification

[Jo Etzel]

This is a familiar nightmare to me, but unfortunately I have no good 
solution. I have not run many searchlights, but have found significant 
(with permutation tests) below-chance classification in several datasets 
using anatomical ROIs.

I have found severely below-chance classification in data due to an 
error (mislabeling); rechecking the entire data processing pipeline is a 
good idea.

In the other cases I have not been able to find any errors (though of 
course they may still exist!). I sometimes find classifications in the 
range of 30-45% (balanced two-class) in some subjects, while other 
subjects are classifying above chance. I have tried various types of 
scaling, partitioning, and classifiers, but have not had much luck; 
often accuracies stay below chance regardless.

I have wondered if some of my datasets are showing anti-learning (ala 
Adam Kowalczyk), but have not investigated this closely. It seems clear 
that information is being learned if a classifier performs at 20% 
(assuming no labeling or other data errors).

Jo Etzel




Matthew Cieslak wrote:
> Hi fellow PyMVPA users,
> 
> I have a non-software-related question for you all. Imagine a scenario 
> where there are N runs in a scanning session and a searchlight is used 
> to compute transfer error from an N-fold cross validation over all 
> voxels. If there are only two categories you are trying to classify, how 
> would interpret large spatially contiguous clusters of voxels that are 
> performing significantly below chance (20% correct or so) appearing in 
> the results? Could it be that there is something changing in the data in 
> relation to the number of times the subject has seen examples of a 
> category? If mis-classification could be caused by an across run 
> repetition-suppression type effect, would re-running the searchlights 
> with an odd-even split and seeing if these voxels are at chance be a 
> legitimate way to show there is meaning in the mis-classification of my 
> svm's?
> 
> I haven't been able to find any neuroimaging papers that address or 
> report below-chance performance, does anyone know of one? Maybe it would 
> be better to search the machine-learning literature?
> 
> I hope to hear your thoughts on this
> 
> Thanks,
> Matt
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa