[pymvpa] Train and test on different classes from a dataset

Michael Hanke michael.hanke at gmail.com
Sat Feb 2 14:53:48 UTC 2013

On Fri, Feb 1, 2013 at 9:41 PM, Yaroslav Halchenko <debian at onerussian.com>wrote:

> Thanks Michael,
> It is a nice example showing that we better permute  testing set
> as well.  Otherwise, the stronger our signal is -- worse we become in
> detecting it (kinda awkward isn't it?).

That is not the case. For the higher SNR value the accuracy is at 100% and
associated p-value is at the theoretical minimum.

> e.g. with 2 chunks and SNR 30.00 (i.e. where signal is obviously
> there and classification on unpermutted labels probably is stable 100%)
> -- we would not be able to tell if that is "significant" if we maintain
> original order in the testing set while doing permutation testing.

Not the case -- if you run the script and increase the bin size you can see
that the p-value
is a nice one ;-)

> now (if you are already asleep, I will try doing it later) - just extend
> it into the power (or ROC detection) analysis -- what we are actually
> after here.  Add SNR=0.00 -- collect both types of samples and
> describe each plot with its power to detect the true signal using both
> kinds of permutation.   Ideally your power should grow with SNR ;)

I am not asleep, but I am having a good time doing nothing ;-)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130202/4674f37c/attachment.html>

More information about the Pkg-ExpPsy-PyMVPA mailing list