[pymvpa] mvpa2.clfs.transerror.chisquare

[Matteo Visconti di Oleggio Castello]

> On May 12, 2017, at 10:58, Yaroslav Halchenko <debian at onerussian.com> wrote:
> 
> just to prevent possible confusion let me first state that chi2 (one way
> or another) is not appropriate for majority of our use cases as a
> criterion to state that we have an effect of interest!  why? because
> 
> In [9]: print chi2_contingency(np.array([[10,0],[0,10]]), correction=False)[1]
> 7.74421643104e-06
> 
> In [10]: print chi2_contingency(np.array([[0,10],[10,0]]), correction=False)[1]
> 7.74421643104e-06
> 
> i.e. it is not representative of accuracy but representative of some
> notion of "information" present in the contingency table, and
> perfect misclassification would be considered as good as perfect correct
> classification.
> 

Agreed that using chi square might not be the best way to test a classifier’s accuracy (for example, it won’t catch biased classifiers, and my intuition is that permutation testing would be better for that). However, wouldn’t perfect misclassification be as informative as perfect classification? Or what does it mean when you have perfect misclassification? When would you expect that such a case would happen with real data (and does it happen)?

> to add more to the confusion, here in the confusion matrix we actually report
>     stats['CHI^2'] = chisquare(self.__matrix, exp='indep_rows')
> ;-)
> 

As Marco pointed out, pymvpa computes goodness-of-fit, and this is the output with your example:

In [53]: mchisquare(np.array([[10, 0], [0, 10]]), exp='indep_rows')
Out[53]: (20.0, 0.00016974243555282632)

In [54]: mchisquare(np.array([[0, 10], [10, 0]]), exp='indep_rows')
Out[54]: (20.0, 0.00016974243555282632)



--
Matteo Visconti di Oleggio Castello
Ph.D. Candidate in Cognitive Neuroscience
Dartmouth College

+1 (603) 646-8665
mvdoc.me || linkedin.com/in/matteovisconti || github.com/mvdoc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170512/6a31d8fd/attachment.html>