[pymvpa] significance assessment: yet another issue...

[Vadim Axel]

Attached an output of one such pathological case (completely real data).
I even do not talk about splits. It's sufficient that for half subjects you
have 0.8/0.4 prediction and for other half you have 0.4/0.8...

For more than two classes it really becomes hardly maintainable. I recently
had 5 classes experiment and I ended up reporting each classes separately.

On Mon, May 23, 2011 at 11:01 PM, Yaroslav Halchenko
<debian at onerussian.com>wrote:

> it is a good thesis indeed, especially for the case of multiclass
> classification where people make claims about unraveling complex
> categorical structure, whenever it is only few categories which get
> "significantly" well classified.
>
> And your illustration goes even further than your verbal description --
> at first I thought that there is an error, since I expected at least one
> class to be significant when "average" accuracy becomes significant.
> But indeed it might be not the case, e.g. if a classifier favors one
> class over another across splits, thus none of the classes come out with
> a consistently "significant" performance while mean accuracy does (could
> you check if that is indeed the case by looking on per split
> diagonals?). Cool.  I always thought that digging in the mud is
> very entertaining ;)
>
> On Mon, 23 May 2011, Vadim Axel wrote:
>
> > Attached an illustration for my thesis.
> > The average classification rate can be considered significant, while we
> > clearly see that it is not exactly true...
>
> --
>                                  .-.
> =------------------------------   /v\  ----------------------------=
> Keep in touch                    // \\     (yoh@|www.)onerussian.com
> Yaroslav Halchenko              /(   )\               ICQ#: 60653192
>                   Linux User    ^^-^^    [175555]
>
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110524/bb23f33e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: yaroslav.xls
Type: application/vnd.ms-excel
Size: 13824 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110524/bb23f33e/attachment-0001.xls>