<div><div dir="auto">Common approaches are:</div></div><div dir="auto">1) One-versus-rest: gives as many weight sets as classes and one overall accuracy</div><div dir="auto">2) One-versus-one: gives as many weight sets as possible pairs and one overall accuracy</div><div dir="auto"><br></div><div dir="auto">In both appeoaches, the binary classifier is applied internally to obtain three-way classification outcomes</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Whether a classifier with native capacity to distinguish three classes is „better“ than the above schemes with a two-class-only classifier is an epistemologically challenging question that may be hard to decide without overfitting the dataset at hand.</div><div dir="auto"><br></div><div dir="auto">Cheers,</div><div dir="auto">Danilo</div><div dir="auto"><br></div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Sun 23. Sep 2018 at 13:00, <<a href="mailto:pkg-exppsy-pymvpa-request@alioth-lists.debian.net">pkg-exppsy-pymvpa-request@alioth-lists.debian.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Pkg-ExpPsy-PyMVPA mailing list submissions to<br>
<a href="mailto:pkg-exppsy-pymvpa@alioth-lists.debian.net" target="_blank">pkg-exppsy-pymvpa@alioth-lists.debian.net</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:pkg-exppsy-pymvpa-request@alioth-lists.debian.net" target="_blank">pkg-exppsy-pymvpa-request@alioth-lists.debian.net</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:pkg-exppsy-pymvpa-owner@alioth-lists.debian.net" target="_blank">pkg-exppsy-pymvpa-owner@alioth-lists.debian.net</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: comparing accuracies of a 3-way classifier and a 2-way<br>
classifier (Richard Dinga)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 22 Sep 2018 15:50:01 +0200<br>
From: Richard Dinga <<a href="mailto:dinga92@gmail.com" target="_blank">dinga92@gmail.com</a>><br>
To: Development and support of PyMVPA<br>
<<a href="mailto:pkg-exppsy-pymvpa@alioth-lists.debian.net" target="_blank">pkg-exppsy-pymvpa@alioth-lists.debian.net</a>><br>
Subject: Re: [pymvpa] comparing accuracies of a 3-way classifier and a<br>
2-way classifier<br>
Message-ID:<br>
<CABbjURB=<a href="mailto:bdovErFzZWpHTo3o7uPu9-i1Lg0ay2LWzKoFS4XRSA@mail.gmail.com" target="_blank">bdovErFzZWpHTo3o7uPu9-i1Lg0ay2LWzKoFS4XRSA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Are your 3 classes ordered?<br>
<br>
On Fri, Sep 21, 2018, 18:28 Michael Bannert <<a href="mailto:mbannert@tuebingen.mpg.de" target="_blank">mbannert@tuebingen.mpg.de</a>><br>
wrote:<br>
<br>
> dear pymvpa users,<br>
><br>
> i have predictions from a 3-way classification and a 2-way<br>
> classification that i would like to compare with one another. how could<br>
> i do this?<br>
><br>
> 1) i could subtract the chance level from each accuracy score, i.e.,<br>
> subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way<br>
> classification. not ideal because percentage changes above chance are<br>
> not directly comparable anymore. but the approach is pretty intuitive<br>
> and permutation inference against chance levels would still be valid.<br>
><br>
> 2) use a different performance metric like (adjusted) mutual information<br>
> maybe? methodologically more appropriate probably but maybe confusing<br>
> for the readers.<br>
><br>
> 3) but perhaps there are even better ways to do this. for example<br>
> examine the 3-by-3 and 2-by-2 confusion matrices and compare<br>
> main-diagonal with off-diagonal entries?<br>
><br>
> any other ideas?<br>
><br>
> thank you,<br>
> michael<br>
><br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html" rel="noreferrer" target="_blank">http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
<a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br>
------------------------------<br>
<br>
End of Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3<br>
*************************************************<br>
</blockquote></div></div>