[pymvpa] Train and test on different classes from a dataset

[Yaroslav Halchenko]

and while you are asleep here is 1 more -- that is where maintaining
original order in testing starts to play off -- I have generated a
dataset with 5x dependent samples.  Ignoring any "dependence" with
regular permutation we start getting slightly smaller AUCs now.

And next one will be the permutation via reassignment I was talking
about -- let's see if it improves upon anything ;-)

Cheers

On Mon, 04 Feb 2013, Yaroslav Halchenko wrote:


> On Sat, 02 Feb 2013, Michael Hanke wrote:
>      

> >      now (if you are already asleep, I will try doing it later) - just extend
> >      it into the power (or ROC detection) analysis -- what we are actually
> >      after here.  Add SNR=0.00 -- collect both types of samples and
> >      describe each plot with its power to detect the true signal using both
> >      kinds of permutation.   Ideally your power should grow with SNR ;)

> >    I am not asleep, but I am having a good time doing nothing ;-)

> I have tortured your script a bit to bootstrap multiple cases and add
> plotting ROCs (even cheated and used scikit-learn for that since we
> apparently need overhaul of ROCCurve).  As you see, keeping
> testing  portion intact results in lower detection power
-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
-------------- next part --------------
A non-text attachment was scrubbed...
Name: permutation_test_rocs_nonindep5.png
Type: image/png
Size: 105653 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/6b275870/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: permutation_test_rocs_nonindep5.py
Type: text/x-python
Size: 4992 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/6b275870/attachment-0001.py>