[pymvpa] multi-class major voting scheme paradox?

Fri May 14 20:39:27 UTC 2010

On Fri, May 14, 2010 at 4:00 PM, Vadim Axel <axel.vadim at gmail.com> wrote:
> Thank you.
> Consider the simplest case, that I do not have other dimensions confounds
> (like a color, contrast etc). I agree with you that all it's about
> distinguishing the information from data. However, if my classifier confused
> significantly more times middle with large, then small with large, then from
> its point of view the middle one looks more close to large, than the small
> does. Although I can not know for 100% that it 's a stimulus size which was
> the root cause of this effect, I can suppose so.  In contrast, if I pick
> randomly three objects and classify them, although my hit rate might be
> beyond the chance for all of them, the trend I am talking about may differ
> from subject to subject.

My intuition is mainly from multinomial logit, where there is no
exante ordering of alternatives.

If there is a metric or ordered characteristic like size, then it
basically means things that are further apart are easier to
distinguish than those close to each other. If the characteristics are
categorical, there is no clear trend.

More my kind of examples: it's easier to predict the size of an
apartment or house from the income of a household, than whether they
have a Toyota or a Honda, or buy Yoplait or Danone.

Josef

>
> On Fri, May 14, 2010 at 9:07 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Fri, May 14, 2010 at 2:27 PM, Vadim Axel <axel.vadim at gmail.com> wrote:
>> > Fine, thanks.
>> > So, then, the interpretation of mutli-class confusion can be misleading.
>> > I have one more related question:
>> > Does it make sense to interpret the confusion matrix off-diagonal values
>> > as
>> > a tuning? For example, if my three classes are the circles of three
>> > different size. For the row of the large circle I get 0.6 classified as
>> > large (the correct one), 0.3 as a medium and 0.1 as a small. Can I say
>> > that
>> > I get sort of tuning to circle size?
>>
>> I'm not sure I understand the term tuning in this context
>> (I'm not a classification expert, but I like puzzles.)
>>
>> To me, these are just probabilities that depend on the distinguishing
>> information that is in the data, and how well the classifier is able
>> to distinguish the separating characteristics.
>>
>> If your circles also have color, then a color sensitive classifier
>> might end up with a different confusion matrix than a size (or shape)
>> sensitive classifier. In that sense the classifier in your example is
>> "atuned" or sensitive to size (and size is a distinguishing
>> characteristic).
>>
>> But as in your initial paradox, I could imagine other cases, e.g. the
>> middle sized circle is further away and clearly defined, while middle
>> and large (1 and 3) are fuzzy and close to each other, then the large
>> and small circles would be more often confused with each other, than
>> with 2.
>>
>> I hope this makes some kind of sense.
>>
>> Josef
>>
>>
>> >
>> > Thanks again.
>> >
>> >
>> > On Fri, May 14, 2010 at 4:29 PM, <josef.pktd at gmail.com> wrote:
>> >>
>> >> On Fri, May 14, 2010 at 9:20 AM, Vadim Axel <axel.vadim at gmail.com>
>> >> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I apply multi-class major voting scheme for three classes (all pairs
>> >> > classification). I try to understand how the confusion matrix should
>> >> > look
>> >> > like when two classes in a pair classification are not discriminated
>> >> > (chance
>> >> > level). Consider pathological case where classes 1,2 and 2,3 are
>> >> > classified
>> >> > with 100% and 1,3 are at chance level (50%). The confusion matrix I
>> >> > which
>> >> > get looks like:
>> >> > 0.584    0.083    0.333
>> >> > 0    1    0
>> >> > 0.327    0.071    0.602
>> >> >
>> >> > So, all of sudden it seems that classes 1 and 3 are discriminated.
>> >> > Isn't
>> >> > it
>> >> > paradoxical?
>> >> >
>> >> > When I checked out how I get this result, I have found that it indeed
>> >> > makes
>> >> > sense. Consider class 1 as a correct label:
>> >> > pair 1: the classification of classes 1,2 always results in '1' (we
>> >> > are
>> >> > at
>> >> > 100%, by definition)
>> >> > pair 2: the classification of classes 1,3 results in half trials in
>> >> > '1'
>> >> > and
>> >> > other half in '3' (we are at chance by definition).
>> >> > pair 3: the classification of classes 2,3 results in half trials in
>> >> > '2'
>> >> > and
>> >> > other in '3' (in case that classes are unrelated, the classifier
>> >> > should
>> >> > be
>> >> > at chance here).
>> >> >
>> >> > The bottom line: since all (1) pairs and half (2) pairs results in
>> >> > '1',
>> >> > I am
>> >> > already at 50% hit rate for correct class.
>> >> >
>> >> > What do you think about all this? Is there any flaw in my logic?
>> >> > If someone is interested, I can send my matlab simulation.
>> >>
>> >> looks right to me if the tie-breaker is unbiased
>> >>
>> >> Probs if 1 is tru:
>> >>
>> >> >>> 0.5*0.5 + 0.5*0.5/3.   # 3 wins, by majority and tie-breaker
>> >> 0.33333333333333331
>> >> >>> 0.5*0.5/3.   # 2 wins, by tie-breaker
>> >> 0.083333333333333329
>> >> >>> 1-(0.5*0.5 + 0.5*0.5/3. + 0.5*0.5/3.)  # 1 wins
>> >> 0.58333333333333337
>> >>
>> >> Josef
>> >>
>> >> >
>> >> > Thanks for help,
>> >> > Vadim
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Pkg-ExpPsy-PyMVPA mailing list
>> >> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> >> > http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>> >> >
>> >> >
>> >>
>> >> _______________________________________________
>> >> Pkg-ExpPsy-PyMVPA mailing list
>> >> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> >> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> > http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
>