[pymvpa] comparing accuracies of a 3-way classifier and a 2-way classifier

[Michael Bannert]

dear all,

thanks for your replies, danilo, richard, and nick. and sorry for the 
slow reply.

@danilo: i'm using lda so it directly performs three-way classification 
(i guess it's very much like nn classification on mahalanobis 
distances). svm could use one of your approaches. this gave me the idea 
that i could turn the three-way classification problem into 3 pair-wise 
classification problems and then take the average. chance would be 50 % 
again and i could compare the results accuracy in a binary classification.

@richard: ordered in the sense of scale of measurement? no, they're nominal.

@nick: yes, that is almost forever ago. still very good times though in 
my opinion :)
the number of predictions are identical between the two classifications. 
hm, the conversion into z-scores sounds like a good idea. so for each 
participant i would convert the accuracay to a p value using number of 
bernoulli trials and chance level. and the p value would give me a z 
score, which i would then analyze at the group level. that does sound 
straightforward.

thanks for your suggestions!

best,
michael


On 23.09.18 13:32, Danilo Bzdok RWTH wrote:
> Common approaches are:
> 1) One-versus-rest: gives as many weight sets as classes and one overall 
> accuracy
> 2) One-versus-one: gives as many weight sets as possible pairs and one 
> overall accuracy
> 
> In both appeoaches, the binary classifier is applied internally to 
> obtain three-way classification outcomes
> 
> 
> Whether a classifier with native capacity to distinguish three classes 
> is „better“ than the above schemes with a two-class-only classifier is 
> an epistemologically challenging question that may be hard to decide 
> without overfitting the dataset at hand.
> 
> Cheers,
> Danilo
> 
> 
> 
> On Sun 23. Sep 2018 at 13:00, 
> <pkg-exppsy-pymvpa-request at alioth-lists.debian.net 
> <mailto:pkg-exppsy-pymvpa-request at alioth-lists.debian.net>> wrote:
> 
>     Send Pkg-ExpPsy-PyMVPA mailing list submissions to
>     pkg-exppsy-pymvpa at alioth-lists.debian.net
>     <mailto:pkg-exppsy-pymvpa at alioth-lists.debian.net>
> 
>     To subscribe or unsubscribe via the World Wide Web, visit
>     https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
>     or, via email, send a message with subject or body 'help' to
>     pkg-exppsy-pymvpa-request at alioth-lists.debian.net
>     <mailto:pkg-exppsy-pymvpa-request at alioth-lists.debian.net>
> 
>     You can reach the person managing the list at
>     pkg-exppsy-pymvpa-owner at alioth-lists.debian.net
>     <mailto:pkg-exppsy-pymvpa-owner at alioth-lists.debian.net>
> 
>     When replying, please edit your Subject line so it is more specific
>     than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."
> 
> 
>     Today's Topics:
> 
>         1. Re: comparing accuracies of a 3-way classifier and a 2-way
>            classifier (Richard Dinga)
> 
> 
>     ----------------------------------------------------------------------
> 
>     Message: 1
>     Date: Sat, 22 Sep 2018 15:50:01 +0200
>     From: Richard Dinga <dinga92 at gmail.com <mailto:dinga92 at gmail.com>>
>     To: Development and support of PyMVPA
>              <pkg-exppsy-pymvpa at alioth-lists.debian.net
>     <mailto:pkg-exppsy-pymvpa at alioth-lists.debian.net>>
>     Subject: Re: [pymvpa] comparing accuracies of a 3-way classifier and a
>              2-way classifier
>     Message-ID:
>             
>     <CABbjURB=bdovErFzZWpHTo3o7uPu9-i1Lg0ay2LWzKoFS4XRSA at mail.gmail.com
>     <mailto:bdovErFzZWpHTo3o7uPu9-i1Lg0ay2LWzKoFS4XRSA at mail.gmail.com>>
>     Content-Type: text/plain; charset="utf-8"
> 
>     Are your 3 classes ordered?
> 
>     On Fri, Sep 21, 2018, 18:28 Michael Bannert
>     <mbannert at tuebingen.mpg.de <mailto:mbannert at tuebingen.mpg.de>>
>     wrote:
> 
>      > dear pymvpa users,
>      >
>      > i have predictions from a 3-way classification and a 2-way
>      > classification that i would like to compare with one another. how
>     could
>      > i do this?
>      >
>      > 1) i could subtract the chance level from each accuracy score, i.e.,
>      > subtract 1/3 from the 3-way classification accuracy and 1/2 from
>     2-way
>      > classification. not ideal because percentage changes above chance are
>      > not directly comparable anymore. but the approach is pretty intuitive
>      > and permutation inference against chance levels would still be valid.
>      >
>      > 2) use a different performance metric like (adjusted) mutual
>     information
>      > maybe? methodologically more appropriate probably but maybe confusing
>      > for the readers.
>      >
>      > 3) but perhaps there are even better ways to do this. for example
>      > examine the 3-by-3 and 2-by-2 confusion matrices and compare
>      > main-diagonal with off-diagonal entries?
>      >
>      > any other ideas?
>      >
>      > thank you,
>      > michael
>      >
>      > _______________________________________________
>      > Pkg-ExpPsy-PyMVPA mailing list
>      > Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>      >
>     https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>     -------------- next part --------------
>     An HTML attachment was scrubbed...
>     URL:
>     <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html>
> 
>     ------------------------------
> 
>     Subject: Digest Footer
> 
>     _______________________________________________
>     Pkg-ExpPsy-PyMVPA mailing list
>     Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>     https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
>     ------------------------------
> 
>     End of Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3
>     *************************************************
> 
> 
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>