[pymvpa] Train and test on different classes from a dataset

[Yaroslav Halchenko]

Hi Francisco,

Great that you followed up -- could you please clarify for me if indeed
in one of your publications you did some power/ROC analysis of such
permutation scheme (keeping testing set labels assignment) against
"classical" (permute all independent assignments)?  I have vague memory
that you did but I could be wrong.

NB A will argue with Michael in reply to his post ;)

On Tue, 05 Feb 2013, Francisco Pereira wrote:
> I'm catching up with this long thread and all I can say is I fully
> concur with Michael, in particular:

> On Tue, Feb 5, 2013 at 3:11 AM, Michael Hanke <mih at debian.org> wrote:
> > Why are we doing permutation analysis? Because we want to know how
> > likely it is to observe a specific prediction performance on a
> > particular dataset under the H0 hypothesis, i.e. how good can a
> > classifier get at predicting our empirical data when the training
> > did not contain the signal of interest -- aka chance performance.

> Permuting the test set might make sense, perhaps, if you wanted to
> make a statement about the result variability over all possible test
> sets of that size if H0 was true.

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik