<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Marco,<div class=""><br class=""></div><div class="">you are absolutely right, and I was wrong about the type of chi-square run :-) Thanks for the example! Looking more carefully at the code, especially at this line that indicates the degrees of freedom (<a href="https://github.com/PyMVPA/PyMVPA/blob/e57f92af83004d40c0944d75ee4e4c9cae758102/mvpa2/misc/stats.py#L83" class="">https://github.com/PyMVPA/PyMVPA/blob/e57f92af83004d40c0944d75ee4e4c9cae758102/mvpa2/misc/stats.py#L83</a>), it is indeed using a goodness of fit (1-dim) chi-square (where df = ncells - 1; in a test of independence it should be df = (nrow - 1)*(ncol-1)).</div><div class=""><br class=""></div><div class="">We should look at the original reference (Yarik, do you remember which one it was?), but after some thought, here’s what I think, and I look forward to hearing your comments. While we do have a contingency table (Predicted vs. Target), we do not necessarily need to run a test of independence, because we are testing the performance of the classifier against chance level (or the confusion matrix generated by chance performance), so we have a precise prediction of what the confusion matrix should look like if it were random. Indeed, with your example confusion matrix, these are the marginals</div><div class=""><br class=""></div><div class=""><div class="">In [46]: mat</div><div class="">Out[46]:</div><div class="">array([[22, 19],</div><div class=""> [26, 29]])</div><div class=""><br class=""></div><div class="">In [47]: print('Marginal across rows: {0}; Marginal across columns: {1}'.format(mat.sum(axis=0), mat.sum(axi</div><div class=""> ...: s=1)))</div><div class="">Marginal across rows: [48 48]; Marginal across columns: [41 55]</div></div><div class=""><br class=""></div><div class="">so given that this is Predicted (rows) vs Targets (columns), it shows that the classification is balanced (same number of target samples in each class, that is 48). A classifier performing at random would have 50% accuracy, resulting in this confusion matrix</div><div class="">[[24, 24], [24, 24]]</div><div class=""><br class=""></div><div class="">so we should be testing our confusion matrix against this “chance” confusion matrix.</div><div class=""><br class=""></div><div class="">Note that if we instead run a test of independence, the expected confusion matrix would be</div><div class=""><br class=""></div><div class=""><div class="">In [49]: x, p, df, exp = chi2_contingency(mat, correction=False)</div><div class="">In [50]: print(exp)</div><div class="">[[ 20.5 20.5]</div><div class=""> [ 27.5 27.5]]</div></div><div class=""><br class=""></div><div class="">which is not correct under the assumption that the classifier is performing at chance level, because it is weighing more one class than the other, based on the predictions.</div><div class=""><br class=""></div><div class="">Hope it makes sense,</div><div class="">Matteo</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 12, 2017, at 01:12, marco tettamanti <<a href="mailto:mrctttmnt@gmail.com" class="">mrctttmnt@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="content-type" content="text/html; charset=utf-8" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
Dear Matteo,<br class="">
thank you for your reply!<br class="">
This was also my understanding when I looked at the
mvpa2.clfs.transerror.chisquare function's help.<br class="">
However, the reality seems to be somewhat different, at least in my
hands.<br class="">
<br class="">
Given any confusion matrices as yielded by the pymvpa2 toolbox:<br class="">
print cvte.ca.stats.matrix<br class="">
[[22 19]<br class="">
[26 29]]<br class="">
<br class="">
The calculated chi square:<br class="">
print 'Chi^2: %.6f (p=%.6f)' % cvte.ca.stats.stats["CHI^2"]<br class="">
Chi^2: 2.416667 (p=0.490540)<br class="">
<br class="">
Produces the same result as the 1-dimensional chisquare test:<br class="">
from scipy.stats import chisquare<br class="">
chisquare([22, 26, 19, 29], f_exp=[24, 24, 24, 24])<br class="">
Power_divergenceResult(statistic=2.416666666666667,
pvalue=0.49053966890385647)<br class="">
<br class="">
help(chisquare)<br class="">
Help on function chisquare in module scipy.stats.stats:<br class="">
chisquare(f_obs, f_exp=None, ddof=0, axis=0) Calculates a
one-way chi square test.<br class="">
<br class="">
And not as would be with the 2-dimensional independence test:<br class="">
from scipy.stats import chi2_contingency<br class="">
obs = np.array([[22, 26], [19, 29]])<br class="">
g, p, dof, expctd = chi2_contingency(obs, correction=False)<br class="">
print g, p, dof, expctd<br class="">
0.383148558758 0.535922979308 1 [[ 20.5 27.5] [ 20.5
27.5]]<br class="">
<br class="">
help(chi2_contingency)<br class="">
Help on function chi2_contingency in module
scipy.stats.contingency:<br class="">
chi2_contingency(observed, correction=True, lambda_=None)<br class="">
Chi-square test of independence of variables in a
contingency table.<br class="">
<br class="">
<br class="">
Am I doing something wrong?<br class="">
Thank you and best wishes,<br class="">
Marco<br class="">
<br class="">
<span style="white-space: pre;" class="">> *Matteo Visconti di Oleggio Castello* matteo.visconti at <a href="http://gmail.com" class="">gmail.com</a>
> <a class="moz-txt-link-rfc2396E" href="mailto:pkg-exppsy-pymvpa%40lists.alioth.debian.org?Subject=Re%3A%20%5Bpymvpa%5D%20mvpa2.clfs.transerror.chisquare&In-Reply-To=%3CEAED4394-634F-4DED-A42F-99FFBCC9E7F5%40gmail.com%3E"><mailto:pkg-exppsy-pymvpa%40lists.alioth.debian.org?Subject=Re%3A%20%5Bpymvpa%5D%20mvpa2.clfs.transerror.chisquare&In-Reply-To=%3CEAED4394-634F-4DED-A42F-99FFBCC9E7F5%40gmail.com%3E></a>
>
> </span><br class="">
/Thu May 11 17:04:46 UTC 2017/<br class="">
<span style="white-space: pre;" class="">> ------------------------- Hi Marco,
>
> looking at the code, the chi-square being run is a test of
> independence, and not goodness-of-fit.The actual confusion matrix is
> tested against the expected values were rows and column independent.
> In the case of balanced classes (i.e., marginal row count is equal to
> marginal column count for each row and column), the expected value
> will be a matrix with identical values (row marginal * column
> marginal / number of observations; or, as it is computed in the code,
> number of observations/number of cells).
>
> Hope this helps, Matteo
>> / On May 11, 2017, at 08:53, marco tettamanti <mrctttmnt at
>> <a href="http://gmail.com" class="">gmail.com</a>
>> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa"><http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa></a>>
>> wrote:
> />/ />/ Dear Yaroslav, />/ thank you for your reply. />/ I might be
> wrong in the specific case of MVPA, but I think the 1-dimension
> Goodness-of-fit test is appropriate in case />/ you have something
> like one dice and you are expecting each of the 6 sides to occur with
> equal frequencies. />/ The N x N confusion matrix rather reflects the
> case in which you can a variable with N classes (targets) and you />/
> measure how frequent these classes distribute across the levels of a
> different variable (predictions). In such a case, />/ a 2-dimension
> Pearson's test seems more appropriate. />/ />/ Best, />/ Marco />/
> />/ On Thu, 11 May 2017, marco tettamanti wrote: />/ />/ > Dear
> all, />/ > I apologize if this has been asked before, or else is
> too trivial. />/ />/ > I have been trying to understand how the
> the pymvpa2 toolbox calculates />/ > the chi-square test of a
> confusion matrix. />/ />/ > In a cross-validation (e.g.,
> cvte.ca.stats), it seems that by default this />/ > is done by
> means of a one-dimensional Goodness-of-fit chi-square test with />/ >
> expected uniform frequency distribution. />/ />/ > I was wondering
> whether the bi-dimensional Pearson's chi square wouldn't />/ > be
> more appropriate, as it seems to me that this would more closely />/
> > reflect the "predictions vs targets N x N" matrix structure. />/
> />/ Hi Marco, />/ />/ might as well be -- I would need to read
> on/check... IIRC we were just />/ following instructions on
> chi-square test to be done on contingency />/ tables. />/ />/ -- />/
> Yaroslav O. Halchenko />/ Center for Open Neuroscience
> <a class="moz-txt-link-freetext" href="http://centerforopenneuroscience.org/">http://centerforopenneuroscience.org</a>
> <a class="moz-txt-link-rfc2396E" href="http://centerforopenneuroscience.org/"><http://centerforopenneuroscience.org/></a> />/ Dartmouth College, 419
> Moore Hall, Hinman Box 6207, Hanover, NH 03755 />/ Phone: +1 (603)
> 646-9834 Fax: +1 (603) 646-1419 />/ WWW:
> <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/yarik">http://www.linkedin.com/in/yarik</a> <a class="moz-txt-link-rfc2396E" href="http://www.linkedin.com/in/yarik"><http://www.linkedin.com/in/yarik></a>
> />/ />/ -- />/ Marco Tettamanti, Ph.D. />/ Nuclear Medicine
> Department & Division of Neuroscience />/ IRCCS San Raffaele
> Scientific Institute />/ Via Olgettina 58 />/ I-20132 Milano, Italy
> />/ Phone ++39-02-26434888 />/ Fax ++39-02-26434892 />/ Email:
> tettamanti.marco at <a href="http://hsr.it" class="">hsr.it</a>
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa"><http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa></a>
> <a class="moz-txt-link-rfc2396E" href="mailto:tettamanti.marcoathsr.it"><mailto:tettamanti.marco at hsr.it
></a> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa"><http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa></a>>
>
> </span><br class="">
/>/ Skype: mtettamanti<br class="">
<span style="white-space: pre;" class="">> />/ _______________________________________________ />/
> Pkg-ExpPsy-PyMVPA mailing list />/ Pkg-ExpPsy-PyMVPA at
> <a href="http://lists.alioth.debian.org" class="">lists.alioth.debian.org</a>
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa"><http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa></a>
>
> </span><br class="">
/>/
<a class="moz-txt-link-freetext" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br class="">
<span style="white-space: pre;" class="">> / -- Matteo Visconti di Oleggio Castello Ph.D. Candidate in Cognitive
> Neuroscience Dartmouth College
>
> +1 (603) 646-8665 <a href="http://mvdoc.me" class="">mvdoc.me</a> || <a href="http://linkedin.com/in/matteovisconti" class="">linkedin.com/in/matteovisconti</a> ||
> <a href="http://github.com/mvdoc" class="">github.com/mvdoc</a>
>
> -------------- next part -------------- An HTML attachment was
> scrubbed... URL:
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170511/8e8c7f55/attachment.html"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170511/8e8c7f55/attachment.html></a>
>
> </span><br class="">
-------------------------<br class="">
<span style="white-space: pre;" class="">>
> * Previous message: [pymvpa] mvpa2.clfs.transerror.chisquare
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/003618.html"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/003618.html></a>
>
> </span><br class="">
* Next message: [pymvpa] mvpa2.clfs.transerror.chisquare
<a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/003620.html"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/003620.html></a><br class="">
<span style="white-space: pre;" class="">> * *Messages sorted by:* [ date ]
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/date.html#3619"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/date.html#3619></a>
> [ thread ]
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/thread.html#3619"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/thread.html#3619></a>
> [ subject ]
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/subject.html#3619"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/subject.html#3619></a>
> [ author ]
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/author.html#3619"><http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2017q2/author.html#3619></a>
>
>
> ------------------------- More information about the
> Pkg-ExpPsy-PyMVPA mailing list
> <a class="moz-txt-link-rfc2396E" href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa"><http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa></a></span><br class="">
</div>
_______________________________________________<br class="">Pkg-ExpPsy-PyMVPA mailing list<br class=""><a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" class="">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br class="">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</div></blockquote></div><br class=""><div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">--</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Matteo Visconti di Oleggio Castello<br class="">Ph.D. Candidate in Cognitive Neuroscience<br class="">Dartmouth College<br class=""><br class="">+1 (603) 646-8665<br class=""><a href="http://mvdoc.me" class="">mvdoc.me</a> || <a href="http://linkedin.com/in/matteovisconti" class="">linkedin.com/in/matteovisconti</a> || <a href="http://github.com/mvdoc" class="">github.com/mvdoc</a><br class=""></div></div>
</div>
<br class=""></div></body></html>