[pymvpa] group level test on decoding accuracy

Jingwen Jin jinjingwen111 at gmail.com
Mon May 11 14:14:05 UTC 2015


Hi Jo and all other experts,

Thanks for your reply! Indeed that was the concern for me. For my first
try, I simply transformed the accuracy by normalization taking into account
of the trail numbers. For example, if the accuracy is 70%, I would convert
it by *(0.7-0.5)/sqrt(0.5*0.5/n)*, where n = the number of total testing
examples.

I hope this is valid?

I tried the Lee 2012 mean correction method, and yields very similar
results.

Have not tried the Logit mixed models yet, though.

Best,
Frances

On Mon, May 11, 2015 at 10:04 AM, J.A. Etzel <jetzel at wustl.edu> wrote:

> To inject a slightly different topic: while I and many others have done
> t-tests on accuracies (equivalently, error rates) for group-level analyses
> and gotten sensible-looking results, this is probably not ideal: accuracies
> are bounded by 0 and 1, which violates the assumptions of t-tests (and
> similar statistics).
>
> Logit mixed models may be a better parametric test for the group level.
> I've found some readable introductions to these issues in the linguistics
> literature, such as this one:
>
> http://dx.doi.org/10.1016%2Fj.jml.2007.11.007
> Categorical Data Analysis: Away from ANOVAs (transformation or not) and
> towards Logit Mixed Models
> T. Florian Jaeger   J Mem Lang. 2008 Nov; 59(4): 434–446.
>
> I haven't yet started playing with these for MVPA datasets; anyone tried?
>
> I don't want to imply that logit mixed models (or t-tests) are better than
> permutation methods; permutation-based tests are almost certainly
> preferable, particularly with cross-validated statistics. However, it is
> sometimes useful to have a quick parametric statistic as well; but this
> should perhaps not be a t-test.
>
> Jo
>
>
>
>
> On 5/9/2015 11:49 AM, Jingwen Jin wrote:
>
>>
>> Hi MVPA experts,
>>
>> I have a general question about conducting group-level analysis on the
>> subjects' classification accuracy maps. Let's say I am doing a
>> one-sample t-test to find the voxels that have high classification
>> accuracy across subjects. Essentially, I am doing a t-test on percentage
>> numbers (SVM classification accuracy measured as percentage correct).
>> Since percentage is highly affected by the testing example numbers, and
>> in general would probably not meet the normal distribution assumption
>> for t-test.
>>
>> So my question is if people adjust for testing trial numbers or any sort
>> of transformation? For example, I converted each voxel's percentage
>> number to a z score at the individual subject's classification map
>> level, and then do group-level t-test on these z score maps. I wonder if
>> this is valid?
>>
>> Thank you very much!
>>
>> Best,
>> Frances
>> --
>> Jingwen Frances Jin
>> Department of Psychology, PhD candidate in Clinical area
>> Stony Brook University
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>



-- 
Jingwen Frances Jin
Department of Psychology
Stony Brook University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20150511/0957540b/attachment-0001.html>


More information about the Pkg-ExpPsy-PyMVPA mailing list