[pymvpa] Train and test on different classes from a dataset

[J.A. Etzel]

This reply caught my eye. My answer would be that a, b, or c could be 
fine. For example, you might want to consider the condition that made 
the A/B or C/D as the 'truth', in which case it might make sense to just 
permute the other labels. And if you want to classify both directions 
(train A/B test C/D and train C/D test A/B) it might be most sensible to 
permute both sets of labels.

Assuming that the label permutations are done sensibly within the 
structure of the data (such as within runs and/or within people), my 
preference is usually to permute both the training AND testing set 
labels (http://mvpa.blogspot.com/2012/12/which-labels-to-permute.html).

Why do you say in the tutorial that "Doing a whole-dataset permutation 
is a common mistake ..." ? I don't see that permuting the test set 
labels hurts the inter-sample dependencies ... won't I still have (say) 
5 A and 5 B in my test set?

Jo


>> Now, I would like to adapt the permutation analysis accordingly. The first thing
>> I'm not sure about is
>> a) whether I should permute only C/D labels (such that the classifier is trained
>> on the real labels, but tested on permuted labels), or
>> b) whether I should permute only A/B labels, or
>> c) both.
>
> I am pretty sure you want to do
>
> d) only permute the training labels within each fold, but leave the
>     testing labels intact.
>
> The reason is that you want to keep all inter-sample dependencies in the
> test set as they are for your actual empirical result. You only want to
> remove the signal of interest from the training portion to see whether
> any noise-fitted model can do as well as your real one.
>
> You can find a demo for this here
>
> http://pymvpa.org/tutorial_significance.html#part-8-the-earth-is-round-significance-testing
>
> but stop reading/doing at "The following content is incomplete and
> experimental"
>
> ;-)
>
>
> Michael
>

-- 
Joset A. Etzel, Ph.D.
Research Analyst
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
http://mvpa.blogspot.com/