[pymvpa] comparing accuracies of a 3-way classifier and a 2-way classifier

[Michael Bannert]

dear pymvpa users,

i have predictions from a 3-way classification and a 2-way 
classification that i would like to compare with one another. how could 
i do this?

1) i could subtract the chance level from each accuracy score, i.e., 
subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way 
classification. not ideal because percentage changes above chance are 
not directly comparable anymore. but the approach is pretty intuitive 
and permutation inference against chance levels would still be valid.

2) use a different performance metric like (adjusted) mutual information 
maybe? methodologically more appropriate probably but maybe confusing 
for the readers.

3) but perhaps there are even better ways to do this. for example 
examine the 3-by-3 and 2-by-2 confusion matrices and compare 
main-diagonal with off-diagonal entries?

any other ideas?

thank you,
michael