[pymvpa] significance

[Scott Gorlin]

Yaroslav Halchenko wrote:
> Thank you Jonas for raising a very important concern, and thanks Scott
> for sharing... although I am not yet clear on how to use McNemar even if
> I pair the samples... and either it somehow scales to more than 2
> classes, and what it actually means besides possibly present preference
> to one of the classes. I would really appreciate if you could elaborate
> a bit or may be provide some link (pardon my ignorance if it is an
> obvious thing)
>   

For two conditions A and B, and an expected result (+ or -), the McNemar 
test assesses whether p(A+ | B-) <> p(B+ | A-); that is, whether your 
treatment significantly enhances/decreases the probability of a result.

What happens is you make a matrix:

x A+ | A-
B+ | a | b |
B- | c | d |

Starting with a = b = c = d = 0, you take a pair of two samples (A, B) 
from your two conditions.  Then you add 1 to a,b,c,or d depending on 
whether the effect of interest occured in A, B, both, or neither.

For example, in my own research, I want to study the effect of prior 
exposure on classification.  So I would show you the same image twice 
(once is unprimed A, second is primed condition, B).  Now let's say I 
classified the B image correctly but not the A image: I add one to b in 
the matrix.  If I classify both A and B, I add one to a, etc.

The significance is (b-c)^2/(b+c) which is chisquared with 1 dof:

http://en.wikipedia.org/wiki/McNemar-Test

So in my case, if the test results in p<.05 and b>c, I would say priming 
significantly increases the accuracy of classification.  This test is a 
commonly used discrete equivalent of the paired t-test ie in assessing 
whether a drug has an effect, etc, paired across people of like 
demographics.  Intuitively, if you get more positive's in your B 
condition where their pairs in A were negative, this means condition B 
increases the likelihood of a positive result; whereas you discount 
results where your treatment had no effect (both A and B are either 
positive or negative).

They're really nice because it completely discounts the baseline 
performance of, say, different images classification accuracy, bad 
subjects, etc, so long as they're appropriately paired (a and d don't 
affect the test in any way, since performance was the same in each 
condition!).  Also, you can simply add McNemar matrices say from 
multiple related conditions (ie different image sets), multiple 
subjects, multiple days, etc, so long as A and B have the same meaning 
and the pairs are valid.

To compare between three conditions, I'm not sure if the matrix can be 
made 3d, but you can simply do a one-vs-one mcnemar test.  or did you 
mean with different classification classes?  that doesn't matter, since 
you can pair the A and B samples appropriately (iea separate matrix for 
each image class, then simply add them) and the result of whether 
condition B has an effect on your analysis is still legitimate.