[pymvpa] Train and test on different classes from a dataset

[Michael Hanke]

Hi,

[just a quick response -- will look at the code later]

On Thu, Jan 31, 2013 at 11:29:49AM +0000, Jan wrote:
> Thank you very much for your help!!! The first step seems to give plausible
> results (in the sense that the accuracy maps for training and testing on A/B,
> and for training on A/B and testing on C/D look similar, but with the latter
> giving somewhat lower accuracies - exactly what I would expect).

Great!

> Now, I would like to adapt the permutation analysis accordingly. The first thing
> I'm not sure about is
> a) whether I should permute only C/D labels (such that the classifier is trained
> on the real labels, but tested on permuted labels), or 
> b) whether I should permute only A/B labels, or
> c) both.

I am pretty sure you want to do

d) only permute the training labels within each fold, but leave the
   testing labels intact.

The reason is that you want to keep all inter-sample dependencies in the
test set as they are for your actual empirical result. You only want to
remove the signal of interest from the training portion to see whether
any noise-fitted model can do as well as your real one.

You can find a demo for this here 

http://pymvpa.org/tutorial_significance.html#part-8-the-earth-is-round-significance-testing

but stop reading/doing at "The following content is incomplete and
experimental"

;-)


Michael

-- 
Michael Hanke
http://mih.voxindeserto.de