[pymvpa] mvpa2.clfs.transerror.chisquare

[Yaroslav Halchenko]

On Fri, 12 May 2017, Matteo Visconti di Oleggio Castello wrote:

>    Hi Marco,
>    you are absolutely right, and I was wrong about the type of chi-square run
>    :-) Thanks for the example! Looking more carefully at the code, especially
>    at this line that indicates the degrees of freedom
>    (https://github.com/PyMVPA/PyMVPA/blob/e57f92af83004d40c0944d75ee4e4c9cae758102/mvpa2/misc/stats.py#L83),
>    it is indeed using a goodness of fit (1-dim) chi-square (where df = ncells
>    - 1; in a test of independence it should be df = (nrow - 1)*(ncol-1)).
>    We should look at the original reference (Yarik, do you remember which one
>    it was?), but after some thought, here’s what I think, and I look forward
>    to hearing your comments. While we do have a contingency table (Predicted
>    vs. Target), we do not necessarily need to run a test of independence,
>    because we are testing the performance of the classifier against chance
>    level (or the confusion matrix generated by chance performance), so we
>    have a precise prediction of what the confusion matrix should look like if
>    it were random. Indeed, with your example confusion matrix, these are the
>    marginals
>    In [46]: mat
>    Out[46]:
>    array([[22, 19],
>           [26, 29]])
>    In [47]: print('Marginal across rows: {0}; Marginal across columns:
>    {1}'.format(mat.sum(axis=0), mat.sum(axi
>        ...: s=1)))
>    Marginal across rows: [48 48]; Marginal across columns: [41 55]
>    so given that this is Predicted (rows) vs Targets (columns), it shows that
>    the classification is balanced (same number of target samples in each
>    class, that is 48). A classifier performing at random would have 50%
>    accuracy, resulting in this confusion matrix
>    [[24, 24], [24, 24]]
>    so we should be testing our confusion matrix against this “chance”
>    confusion matrix.
>    Note that if we instead run a test of independence, the expected confusion
>    matrix would be
>    In [49]: x, p, df, exp = chi2_contingency(mat, correction=False)
>    In [50]: print(exp)
>    [[ 20.5  20.5]
>     [ 27.5  27.5]]
>    which is not correct under the assumption that the classifier is
>    performing at chance level, because it is weighing more one class than the
>    other, based on the predictions.

just to prevent possible confusion let me first state that chi2 (one way
or another) is not appropriate for majority of our use cases as a
criterion to state that we have an effect of interest!  why? because

In [9]: print chi2_contingency(np.array([[10,0],[0,10]]), correction=False)[1]
7.74421643104e-06

In [10]: print chi2_contingency(np.array([[0,10],[10,0]]), correction=False)[1]
7.74421643104e-06

i.e. it is not representative of accuracy but representative of some
notion of "information" present in the contingency table, and
perfect misclassification would be considered as good as perfect correct
classification.

We added it to the confusion matrix stats as just an additional
indicator to keep an eye on, but didn't really use it much beyond that
iirc.

to add more to the confusion, here in the confusion matrix we actually report
     stats['CHI^2'] = chisquare(self.__matrix, exp='indep_rows')
;-)
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik