[pymvpa] Train and test on different classes from a dataset

[J.A. Etzel]

> that example in the documentation permutes the "whole-dataset"
> without any partitioning on the runs.  And yes -- I think it should be
> sufficient to "permute" in the entire dataset if you do permutation
> within runs (i.e. not breaking any balance of labels across runs) AND
> maintaining dependence between those samples in each run if you have
> more than 1 sample of a class per run.

Ah, I see now. I agree that it's (almost) never ok to permute ignoring 
the structure of the dataset - runs, order, separation into subjects, 
etc. I hadn't realized you meant permuting the labels in the entire 
dataset willy-nilly, losing the stratification; that's certainly not a 
good idea.

I should add an example on my blog showing this explicitly: dataset 
structure and interactions really, really matters in fMRI.

Jo



-- 
Joset A. Etzel, Ph.D.
Research Analyst
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
http://mvpa.blogspot.com/