[pymvpa] significance assessment: yet another issue...

Vadim Axel axel.vadim at gmail.com
Mon May 23 13:25:27 UTC 2011

Attached an illustration for my thesis.
The average classification rate can be considered significant, while we
clearly see that it is not exactly true...

On Mon, May 23, 2011 at 1:26 PM, Vadim Axel <axel.vadim at gmail.com> wrote:

> Hi,
> One more issue, which I had never seen to be mentioned in the papers:
> Two classes classification rate is the confusion matrix diagonal average.
> Things are fine as soon as both values are above 0.5. But what if one of
> them is let's say 0.7 and the other one is 0.45? All significance procedures
> (permutation, group t-test whatever) work on average values. Thus, my
> obscure average mean would be treated as a first class citizen and
> contribute to beyond chance prediction. Clearly, one have to check the
> results before averaging, but they are never reported (at I least I have
> never seen). For example, I can require that  p-values for both classes
> should be significant. However, then, I am afraid, I may get a chart with
> two ROIs with similar average prediction rate when one is highly significant
> and another one is completely not. Nobody would understand what is going
> on...
> Any suggestions / thoughts?
> Thanks,
> Vadim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110523/42eb5908/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: class_rate.JPG
Type: image/jpeg
Size: 29842 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110523/42eb5908/attachment-0001.jpe>

More information about the Pkg-ExpPsy-PyMVPA mailing list