[pymvpa] significance assessment: yet another issue...

[Vadim Axel]

There are six subjects (rows)
Class 1 and Class 2 are correct predictions of each class (two values on the
diagonal of the confusion matrix). The Total column is the average between
them.

On Wed, May 25, 2011 at 3:24 PM, J.A. Etzel <jetzel at artsci.wustl.edu> wrote:

> I'm missing something here: where are the "class 1" and "class 2" numbers
> for each person coming from?
>
> I agree that describing the results of a multiclass classification is
> tricky, especially when using something naturally two-class like svms.
> Reporting the multiclass accuracy rate, then the pairwise ones seems most
> prudent.
>
> Jo
>
>
>
> On 5/24/2011 2:09 AM, Vadim Axel wrote:
>
>> Attached an output of one such pathological case (completely real data).
>> I even do not talk about splits. It's sufficient that for half subjects
>> you have 0.8/0.4 prediction and for other half you have 0.4/0.8...
>>
>> For more than two classes it really becomes hardly maintainable. I
>> recently had 5 classes experiment and I ended up reporting each classes
>> separately.
>>
>> On Mon, May 23, 2011 at 11:01 PM, Yaroslav Halchenko
>> <debian at onerussian.com <mailto:debian at onerussian.com>> wrote:
>>
>>    it is a good thesis indeed, especially for the case of multiclass
>>    classification where people make claims about unraveling complex
>>    categorical structure, whenever it is only few categories which get
>>    "significantly" well classified.
>>
>>    And your illustration goes even further than your verbal description --
>>    at first I thought that there is an error, since I expected at least
>> one
>>    class to be significant when "average" accuracy becomes significant.
>>    But indeed it might be not the case, e.g. if a classifier favors one
>>    class over another across splits, thus none of the classes come out
>> with
>>    a consistently "significant" performance while mean accuracy does
>> (could
>>    you check if that is indeed the case by looking on per split
>>    diagonals?). Cool.  I always thought that digging in the mud is
>>    very entertaining ;)
>>
>>    On Mon, 23 May 2011, Vadim Axel wrote:
>>
>>     > Attached an illustration for my thesis.
>>     > The average classification rate can be considered significant,
>>    while we
>>     > clearly see that it is not exactly true...
>>
>>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110525/dbe9a2a6/attachment.html>