[pymvpa] How to run a permutation test for group level analysis?
J.A. Etzel
jetzel at wustl.edu
Tue Mar 31 17:44:21 UTC 2015
You mention already doing a permutation test ... that was for
significance in individual subjects?
I have a tutorial paper for group-level permutation testing under
review, but many of the same ideas are in this post:
http://mvpa.blogspot.com/2012/11/permutation-testing-groups-of-people.html
and this paper
http://mvpa.blogspot.com/2013/06/mvpa-permutation-schemes-permutation.html
Some people do use the binomial for MVPA significance
(http://mvpa.blogspot.com/search?q=binomial), though I tend to prefer
permutation tests or the one-sample t-test for the group level.
Sorry for all the links, but I think you'll find them helpful.
Jo
On 3/31/2015 12:06 PM, Yi-Shin Sheu wrote:
> I followed the pymvpa tutorial and successfully obtained accuracy and
> p-value for each of my participants using cross-validation and
> permutation test.
> From here, in order to obtain the group level accuracy, it makes sense
> to average all the accuracies from each participant, but how do I test
> the significance at the group level? I heard about performing an
> one-sample t-test agaisnt chance (0.5), but I think it produces false
> positives.
> Another thought of mine is to run a binomial test based on the number of
> successful classification cases. For example, if I have 13 people out
> of 16 people produce signfiicant classification result using permutation
> test, then according to binomial test (see below), the group level
> result should be significant.
>
> Binomial Test at Group Level
> Number of "successes": 13
> Number of trials (or subjects) per experiment: 16
> Sign test. If the probability of "success" in each trial or subject is
> 0.500, then:
>
> * The one-tail P value is 0.0106
> This is the chance of observing 13 or more successes in 16 trials.
> * The two-tail P value is 0.0213
> This is the chance of observing either 13 or more successes, or 3 or
> fewer successes, in 16 trials.
>
>
> If the above method is not logical, then I thought about running a
> permutation result based on the group. First, build a single-subject
> null distribution by performing permutation for all participants.
> Second, randomly select one data point from step 1 for N times to build
> the group-level null distribution with labels shuffled or not. Third,
> compare these two distributions to get the p value. Anybody knows how
> to modify the pymvpa tutorial script to run this analysis? It does not
> seem to be any tutorial on running a permutation test using pymvpa, and
> I am not good with python....
>
> Yishin
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
More information about the Pkg-ExpPsy-PyMVPA
mailing list