[pymvpa] General Question about Evaluating Results form a Multiclass Problem

[Nick Oosterhof]

On Aug 14, 2014, at 12:45 AM, Hanson, Gavin Keith <ghanson0 at ku.edu> wrote:

> I’m just wondering if anyone has any advice on some ways to deal with evaluating classifier performance on a 4-way problem.
> I’ve been using the BayesConfusionHypothesis tool which works quite well, but I just wondered what else was out there by way of quantitative evidence to insure that classifier accuracy isn’t being driven by perfect classification between 2 labels, and confusion between the other 2, or whatever. Just glancing at the confusion matrices can give us a good idea about what ROIs are confusing certain conditions, but a more objective solution would be nice. 

I don't think there's much else out there besides the BCH tool.

If you have a specific hypothesis to test, e.g. effects of classification of classes w, x, y, z is driven by perfect classification between w and x only, you can do a contrast at each subject. If A(p,q) is the classification accuracy between classes p and q, then compute, for example, c=A(w,x) - (A(w,y)+A(w,z)+A(x,y)+A(x,z)+A(y,z))/5. You can use standard group-analysis (random effect) analyses to see if this is reliable across participants.