[pymvpa] Train and test on different classes from a dataset

Yaroslav Halchenko debian at onerussian.com
Tue Feb 5 02:20:56 UTC 2013

and while you are asleep here is 1 more -- that is where maintaining
original order in testing starts to play off -- I have generated a
dataset with 5x dependent samples.  Ignoring any "dependence" with
regular permutation we start getting slightly smaller AUCs now.

And next one will be the permutation via reassignment I was talking
about -- let's see if it improves upon anything ;-)


On Mon, 04 Feb 2013, Yaroslav Halchenko wrote:

> On Sat, 02 Feb 2013, Michael Hanke wrote:

> >      now (if you are already asleep, I will try doing it later) - just extend
> >      it into the power (or ROC detection) analysis -- what we are actually
> >      after here.  Add SNR=0.00 -- collect both types of samples and
> >      describe each plot with its power to detect the true signal using both
> >      kinds of permutation.   Ideally your power should grow with SNR ;)

> >    I am not asleep, but I am having a good time doing nothing ;-)

> I have tortured your script a bit to bootstrap multiple cases and add
> plotting ROCs (even cheated and used scikit-learn for that since we
> apparently need overhaul of ROCCurve).  As you see, keeping
> testing  portion intact results in lower detection power
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
-------------- next part --------------
A non-text attachment was scrubbed...
Name: permutation_test_rocs_nonindep5.png
Type: image/png
Size: 105653 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/6b275870/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: permutation_test_rocs_nonindep5.py
Type: text/x-python
Size: 4992 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/6b275870/attachment-0001.py>

More information about the Pkg-ExpPsy-PyMVPA mailing list