[pymvpa] How to run a permutation test for group level analysis?

[Yi-Shin Sheu]

I followed the pymvpa tutorial and successfully obtained accuracy and
p-value for each of my participants using cross-validation and permutation
test.
>From here, in order to obtain the group level accuracy, it makes sense to
average all the accuracies from each participant, but how do I test the
significance at the group level?  I heard about performing an one-sample
t-test agaisnt chance (0.5), but I think it produces false positives.
Another thought of mine is to run a binomial test based on the number of
successful classification cases.  For example, if I have 13 people out of
16 people produce signfiicant classification result using permutation test,
then according to binomial test (see below), the group level result should
be significant.

Binomial Test at Group Level
Number of "successes": 13
Number of trials (or subjects) per experiment: 16
Sign test. If the probability of "success" in each trial or subject is
0.500, then:

   - The one-tail P value is 0.0106
   This is the chance of observing 13 or more successes in 16 trials.
   - The two-tail P value is 0.0213
   This is the chance of observing either 13 or more successes, or 3 or
   fewer successes, in 16 trials.


If the above method is not logical, then I thought about running a
permutation result based on the group.  First, build a single-subject null
distribution by performing permutation for all participants.  Second,
randomly select one data point from step 1 for N times to build the
group-level null distribution with labels shuffled or not.  Third, compare
these two distributions to get the p value.  Anybody knows how to modify
the pymvpa tutorial script to run this analysis?  It does not seem to be
any tutorial on running a permutation test using pymvpa, and I am not good
with python....

Yishin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20150331/253d9c83/attachment.html>