[pymvpa] comparing accuracies of a 3-way classifier and a 2-way classifier

[Richard Dinga]

Are your 3 classes ordered?

On Fri, Sep 21, 2018, 18:28 Michael Bannert <mbannert at tuebingen.mpg.de>
wrote:

> dear pymvpa users,
>
> i have predictions from a 3-way classification and a 2-way
> classification that i would like to compare with one another. how could
> i do this?
>
> 1) i could subtract the chance level from each accuracy score, i.e.,
> subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way
> classification. not ideal because percentage changes above chance are
> not directly comparable anymore. but the approach is pretty intuitive
> and permutation inference against chance levels would still be valid.
>
> 2) use a different performance metric like (adjusted) mutual information
> maybe? methodologically more appropriate probably but maybe confusing
> for the readers.
>
> 3) but perhaps there are even better ways to do this. for example
> examine the 3-by-3 and 2-by-2 confusion matrices and compare
> main-diagonal with off-diagonal entries?
>
> any other ideas?
>
> thank you,
> michael
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment.html>