[pymvpa] comparing accuracies of a 3-way classifier and a 2-way classifier

[Nick Oosterhof]

Hi Michael,

On Fri, 21 Sep 2018 at 18:28, Michael Bannert <mbannert at tuebingen.mpg.de>
wrote:

>
> i have predictions from a 3-way classification and a 2-way
> classification that i would like to compare with one another. how could
> i do this?
>
> 1) i could subtract the chance level from each accuracy score, i.e.,
> subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way
> classification. not ideal because percentage changes above chance are
> not directly comparable anymore. but the approach is pretty intuitive
> and permutation inference against chance levels would still be valid.
>

You may want to take into account differences in number of predictions for
the 2-way versus 3-way, if any, because expected variance in accuracy
depends on the number of samples (and number of classes). This in turn
would affect the relative impact of the 2-way versus 3-way classification.
Maybe convert the accuracies to z-scores based on a binomial distribution,
and look at the difference between 2-way and 3-way classification z-scores
at the participant-level? Note that the z-scores at the participant level
may not be completely valid due to non-independence of predictions, but I
don't expect this to be a concern at the group-analysis level.
In any case using permutation testing sounds like a good idea.


>
> 2) use a different performance metric like (adjusted) mutual information
> maybe? methodologically more appropriate probably but maybe confusing
> for the readers.
>
> 3) but perhaps there are even better ways to do this. for example
> examine the 3-by-3 and 2-by-2 confusion matrices and compare
> main-diagonal with off-diagonal entries?
>

This might work but I cannot provide good advice on this. I would suggest
to keep it simple enough that results can be explained and interpreted
easily.

(on a personal note: it's been a while since Princeton times!)

best,
Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/a702e3ee/attachment.html>