[pymvpa] How to run a permutation test for group level analysis?

[J.A. Etzel]

You mention already doing a permutation test ... that was for 
significance in individual subjects?

I have a tutorial paper for group-level permutation testing under 
review, but many of the same ideas are in this post: 
http://mvpa.blogspot.com/2012/11/permutation-testing-groups-of-people.html 
and this paper 
http://mvpa.blogspot.com/2013/06/mvpa-permutation-schemes-permutation.html

Some people do use the binomial for MVPA significance 
(http://mvpa.blogspot.com/search?q=binomial), though I tend to prefer 
permutation tests or the one-sample t-test for the group level.

Sorry for all the links, but I think you'll find them helpful.
Jo


On 3/31/2015 12:06 PM, Yi-Shin Sheu wrote:
> I followed the pymvpa tutorial and successfully obtained accuracy and
> p-value for each of my participants using cross-validation and
> permutation test.
>  From here, in order to obtain the group level accuracy, it makes sense
> to average all the accuracies from each participant, but how do I test
> the significance at the group level?  I heard about performing an
> one-sample t-test agaisnt chance (0.5), but I think it produces false
> positives.
> Another thought of mine is to run a binomial test based on the number of
> successful classification cases.  For example, if I have 13 people out
> of 16 people produce signfiicant classification result using permutation
> test, then according to binomial test (see below), the group level
> result should be significant.
>
> Binomial Test at Group Level
> Number of "successes": 13
> Number of trials (or subjects) per experiment: 16
> Sign test. If the probability of "success" in each trial or subject is
> 0.500, then:
>
>   * The one-tail P value is 0.0106
>     This is the chance of observing 13 or more successes in 16 trials.
>   * The two-tail P value is 0.0213
>     This is the chance of observing either 13 or more successes, or 3 or
>     fewer successes, in 16 trials.
>
>
> If the above method is not logical, then I thought about running a
> permutation result based on the group.  First, build a single-subject
> null distribution by performing permutation for all participants.
> Second, randomly select one data point from step 1 for N times to build
> the group-level null distribution with labels shuffled or not.  Third,
> compare these two distributions to get the p value.  Anybody knows how
> to modify the pymvpa tutorial script to run this analysis?  It does not
> seem to be any tutorial on running a permutation test using pymvpa, and
> I am not good with python....
>
> Yishin
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>