[pymvpa] comparing accuracies of a 3-way classifier and a 2-way classifier

Richard Dinga dinga92 at gmail.com
Sat Sep 22 14:50:01 BST 2018

Are your 3 classes ordered?

On Fri, Sep 21, 2018, 18:28 Michael Bannert <mbannert at tuebingen.mpg.de>

> dear pymvpa users,
> i have predictions from a 3-way classification and a 2-way
> classification that i would like to compare with one another. how could
> i do this?
> 1) i could subtract the chance level from each accuracy score, i.e.,
> subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way
> classification. not ideal because percentage changes above chance are
> not directly comparable anymore. but the approach is pretty intuitive
> and permutation inference against chance levels would still be valid.
> 2) use a different performance metric like (adjusted) mutual information
> maybe? methodologically more appropriate probably but maybe confusing
> for the readers.
> 3) but perhaps there are even better ways to do this. for example
> examine the 3-by-3 and 2-by-2 confusion matrices and compare
> main-diagonal with off-diagonal entries?
> any other ideas?
> thank you,
> michael
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment.html>

More information about the Pkg-ExpPsy-PyMVPA mailing list