[pymvpa] mvpa2.clfs.transerror.chisquare

[Matteo Visconti di Oleggio Castello]

Quick follow-up: looking at the code of mvpa2.clfs.transerror.chisquare (https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/clfs/transerror.py#L702 <https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/clfs/transerror.py#L702>), the chi-square being computed also takes into account possible unbalanced classes by computing expected values relative to each row (see https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/misc/stats.py#L23 <https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/misc/stats.py#L23>)
> On May 11, 2017, at 10:04, Matteo Visconti di Oleggio Castello <matteo.visconti at gmail.com> wrote:
> 
> Hi Marco,
> 
> looking at the code, the chi-square being run is a test of independence, and not goodness-of-fit.The actual confusion matrix is tested against the expected values were rows and column independent. In the case of balanced classes (i.e., marginal row count is equal to marginal column count for each row and column), the expected value will be a matrix with identical values (row marginal * column marginal / number of observations; or, as it is computed in the code, number of observations/number of cells).
> 
> Hope this helps,
> Matteo
>> On May 11, 2017, at 08:53, marco tettamanti <mrctttmnt at gmail.com <mailto:mrctttmnt at gmail.com>> wrote:
>> 
>> Dear Yaroslav,
>> thank you for your reply.
>> I might be wrong in the specific case of MVPA, but I think the 1-dimension Goodness-of-fit test is appropriate in case
>> you have something like one dice and you are expecting each of the 6 sides to occur with equal frequencies.
>> The N x N confusion matrix rather reflects the case in which you can a variable with N classes (targets) and you 
>> measure how frequent these classes distribute across the levels of a different variable (predictions). In such a case,
>> a 2-dimension Pearson's test seems more appropriate.
>> 
>> Best,
>> Marco
>> 
>> On Thu, 11 May 2017, marco tettamanti wrote:
>> 
>> >    Dear all,
>> >    I apologize if this has been asked before, or else is too trivial.
>> 
>> >    I have been trying to understand how the the pymvpa2 toolbox calculates
>> >    the chi-square test of a confusion matrix.
>> 
>> >    In a cross-validation (e.g., cvte.ca.stats), it seems that by default this
>> >    is done by means of a one-dimensional Goodness-of-fit chi-square test with
>> >    expected uniform frequency distribution.
>> 
>> >    I was wondering whether the bi-dimensional Pearson's chi square wouldn't
>> >    be more appropriate, as it seems to me that this would more closely
>> >    reflect the "predictions vs targets N x N" matrix structure.
>> 
>> Hi Marco,
>> 
>> might as well be -- I would need to read on/check... IIRC we were just
>> following instructions on chi-square test to be done on contingency
>> tables.
>> 
>> -- 
>> Yaroslav O. Halchenko
>> Center for Open Neuroscience     http://centerforopenneuroscience.org <http://centerforopenneuroscience.org/>
>> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
>> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
>> WWW:   http://www.linkedin.com/in/yarik <http://www.linkedin.com/in/yarik>        
>> 
>> -- 
>> Marco Tettamanti, Ph.D.
>> Nuclear Medicine Department & Division of Neuroscience
>> IRCCS San Raffaele Scientific Institute
>> Via Olgettina 58
>> I-20132 Milano, Italy
>> Phone ++39-02-26434888
>> Fax ++39-02-26434892
>> Email: tettamanti.marco at hsr.it <mailto:tettamanti.marco at hsr.it>
>> Skype: mtettamanti
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> --
> Matteo Visconti di Oleggio Castello
> Ph.D. Candidate in Cognitive Neuroscience
> Dartmouth College
> 
> +1 (603) 646-8665
> mvdoc.me <http://mvdoc.me/> || linkedin.com/in/matteovisconti <http://linkedin.com/in/matteovisconti> || github.com/mvdoc <http://github.com/mvdoc>
> 

--
Matteo Visconti di Oleggio Castello
Ph.D. Candidate in Cognitive Neuroscience
Dartmouth College

+1 (603) 646-8665
mvdoc.me || linkedin.com/in/matteovisconti || github.com/mvdoc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170511/2e596c72/attachment.html>