[pymvpa] significance assessment: yet another issue...

[Vadim Axel]

Attached an illustration for my thesis.
The average classification rate can be considered significant, while we
clearly see that it is not exactly true...

On Mon, May 23, 2011 at 1:26 PM, Vadim Axel <axel.vadim at gmail.com> wrote:

> Hi,
>
> One more issue, which I had never seen to be mentioned in the papers:
> Two classes classification rate is the confusion matrix diagonal average.
> Things are fine as soon as both values are above 0.5. But what if one of
> them is let's say 0.7 and the other one is 0.45? All significance procedures
> (permutation, group t-test whatever) work on average values. Thus, my
> obscure average mean would be treated as a first class citizen and
> contribute to beyond chance prediction. Clearly, one have to check the
> results before averaging, but they are never reported (at I least I have
> never seen). For example, I can require that  p-values for both classes
> should be significant. However, then, I am afraid, I may get a chart with
> two ROIs with similar average prediction rate when one is highly significant
> and another one is completely not. Nobody would understand what is going
> on...
>
> Any suggestions / thoughts?
>
> Thanks,
> Vadim
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110523/42eb5908/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: class_rate.JPG
Type: image/jpeg
Size: 29842 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110523/42eb5908/attachment-0001.jpe>