[pymvpa] significance assessment: yet another issue...

[Yaroslav Halchenko]

it is a good thesis indeed, especially for the case of multiclass
classification where people make claims about unraveling complex
categorical structure, whenever it is only few categories which get
"significantly" well classified.

And your illustration goes even further than your verbal description --
at first I thought that there is an error, since I expected at least one
class to be significant when "average" accuracy becomes significant.
But indeed it might be not the case, e.g. if a classifier favors one
class over another across splits, thus none of the classes come out with
a consistently "significant" performance while mean accuracy does (could
you check if that is indeed the case by looking on per split
diagonals?). Cool.  I always thought that digging in the mud is
very entertaining ;)

On Mon, 23 May 2011, Vadim Axel wrote:

> Attached an illustration for my thesis.
> The average classification rate can be considered significant, while we
> clearly see that it is not exactly true...

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]