[pymvpa] significance assessment: yet another issue...

[Vadim Axel]

Hi,

One more issue, which I had never seen to be mentioned in the papers:
Two classes classification rate is the confusion matrix diagonal average.
Things are fine as soon as both values are above 0.5. But what if one of
them is let's say 0.7 and the other one is 0.45? All significance procedures
(permutation, group t-test whatever) work on average values. Thus, my
obscure average mean would be treated as a first class citizen and
contribute to beyond chance prediction. Clearly, one have to check the
results before averaging, but they are never reported (at I least I have
never seen). For example, I can require that  p-values for both classes
should be significant. However, then, I am afraid, I may get a chart with
two ROIs with similar average prediction rate when one is highly significant
and another one is completely not. Nobody would understand what is going
on...

Any suggestions / thoughts?

Thanks,
Vadim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110523/d02263ec/attachment.html>