[pymvpa] group level test on decoding accuracy

[J.A. Etzel]

To inject a slightly different topic: while I and many others have done 
t-tests on accuracies (equivalently, error rates) for group-level 
analyses and gotten sensible-looking results, this is probably not 
ideal: accuracies are bounded by 0 and 1, which violates the assumptions 
of t-tests (and similar statistics).

Logit mixed models may be a better parametric test for the group level. 
I've found some readable introductions to these issues in the 
linguistics literature, such as this one:

http://dx.doi.org/10.1016%2Fj.jml.2007.11.007
Categorical Data Analysis: Away from ANOVAs (transformation or not) and 
towards Logit Mixed Models
T. Florian Jaeger   J Mem Lang. 2008 Nov; 59(4): 434–446.

I haven't yet started playing with these for MVPA datasets; anyone tried?

I don't want to imply that logit mixed models (or t-tests) are better 
than permutation methods; permutation-based tests are almost certainly 
preferable, particularly with cross-validated statistics. However, it is 
sometimes useful to have a quick parametric statistic as well; but this 
should perhaps not be a t-test.

Jo



On 5/9/2015 11:49 AM, Jingwen Jin wrote:
>
> Hi MVPA experts,
>
> I have a general question about conducting group-level analysis on the
> subjects' classification accuracy maps. Let's say I am doing a
> one-sample t-test to find the voxels that have high classification
> accuracy across subjects. Essentially, I am doing a t-test on percentage
> numbers (SVM classification accuracy measured as percentage correct).
> Since percentage is highly affected by the testing example numbers, and
> in general would probably not meet the normal distribution assumption
> for t-test.
>
> So my question is if people adjust for testing trial numbers or any sort
> of transformation? For example, I converted each voxel's percentage
> number to a z score at the individual subject's classification map
> level, and then do group-level t-test on these z score maps. I wonder if
> this is valid?
>
> Thank you very much!
>
> Best,
> Frances
> --
> Jingwen Frances Jin
> Department of Psychology, PhD candidate in Clinical area
> Stony Brook University
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>