[pymvpa] significance assessment: yet another issue...
J.A. Etzel
jetzel at artsci.wustl.edu
Wed May 25 13:24:03 UTC 2011
I'm missing something here: where are the "class 1" and "class 2"
numbers for each person coming from?
I agree that describing the results of a multiclass classification is
tricky, especially when using something naturally two-class like svms.
Reporting the multiclass accuracy rate, then the pairwise ones seems
most prudent.
Jo
On 5/24/2011 2:09 AM, Vadim Axel wrote:
> Attached an output of one such pathological case (completely real data).
> I even do not talk about splits. It's sufficient that for half subjects
> you have 0.8/0.4 prediction and for other half you have 0.4/0.8...
>
> For more than two classes it really becomes hardly maintainable. I
> recently had 5 classes experiment and I ended up reporting each classes
> separately.
>
> On Mon, May 23, 2011 at 11:01 PM, Yaroslav Halchenko
> <debian at onerussian.com <mailto:debian at onerussian.com>> wrote:
>
> it is a good thesis indeed, especially for the case of multiclass
> classification where people make claims about unraveling complex
> categorical structure, whenever it is only few categories which get
> "significantly" well classified.
>
> And your illustration goes even further than your verbal description --
> at first I thought that there is an error, since I expected at least one
> class to be significant when "average" accuracy becomes significant.
> But indeed it might be not the case, e.g. if a classifier favors one
> class over another across splits, thus none of the classes come out with
> a consistently "significant" performance while mean accuracy does (could
> you check if that is indeed the case by looking on per split
> diagonals?). Cool. I always thought that digging in the mud is
> very entertaining ;)
>
> On Mon, 23 May 2011, Vadim Axel wrote:
>
> > Attached an illustration for my thesis.
> > The average classification rate can be considered significant,
> while we
> > clearly see that it is not exactly true...
>
More information about the Pkg-ExpPsy-PyMVPA
mailing list