[pymvpa] Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3

[Danilo Bzdok RWTH]

Common approaches are:
1) One-versus-rest: gives as many weight sets as classes and one overall
accuracy
2) One-versus-one: gives as many weight sets as possible pairs and one
overall accuracy

In both appeoaches, the binary classifier is applied internally to obtain
three-way classification outcomes


Whether a classifier with native capacity to distinguish three classes is
„better“ than the above schemes with a two-class-only classifier is an
epistemologically challenging question that may be hard to decide without
overfitting the dataset at hand.

Cheers,
Danilo



On Sun 23. Sep 2018 at 13:00, <
pkg-exppsy-pymvpa-request at alioth-lists.debian.net> wrote:

> Send Pkg-ExpPsy-PyMVPA mailing list submissions to
>         pkg-exppsy-pymvpa at alioth-lists.debian.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
> or, via email, send a message with subject or body 'help' to
>         pkg-exppsy-pymvpa-request at alioth-lists.debian.net
>
> You can reach the person managing the list at
>         pkg-exppsy-pymvpa-owner at alioth-lists.debian.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."
>
>
> Today's Topics:
>
>    1. Re: comparing accuracies of a 3-way classifier and a 2-way
>       classifier (Richard Dinga)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 22 Sep 2018 15:50:01 +0200
> From: Richard Dinga <dinga92 at gmail.com>
> To: Development and support of PyMVPA
>         <pkg-exppsy-pymvpa at alioth-lists.debian.net>
> Subject: Re: [pymvpa] comparing accuracies of a 3-way classifier and a
>         2-way classifier
> Message-ID:
>         <CABbjURB=
> bdovErFzZWpHTo3o7uPu9-i1Lg0ay2LWzKoFS4XRSA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Are your 3 classes ordered?
>
> On Fri, Sep 21, 2018, 18:28 Michael Bannert <mbannert at tuebingen.mpg.de>
> wrote:
>
> > dear pymvpa users,
> >
> > i have predictions from a 3-way classification and a 2-way
> > classification that i would like to compare with one another. how could
> > i do this?
> >
> > 1) i could subtract the chance level from each accuracy score, i.e.,
> > subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way
> > classification. not ideal because percentage changes above chance are
> > not directly comparable anymore. but the approach is pretty intuitive
> > and permutation inference against chance levels would still be valid.
> >
> > 2) use a different performance metric like (adjusted) mutual information
> > maybe? methodologically more appropriate probably but maybe confusing
> > for the readers.
> >
> > 3) but perhaps there are even better ways to do this. for example
> > examine the 3-by-3 and 2-by-2 confusion matrices and compare
> > main-diagonal with off-diagonal entries?
> >
> > any other ideas?
> >
> > thank you,
> > michael
> >
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
> ------------------------------
>
> End of Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3
> *************************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180923/8d4a336f/attachment.html>