[pymvpa] Train and test on different classes from a dataset
michael.hanke at gmail.com
Sat Feb 2 14:53:48 UTC 2013
On Fri, Feb 1, 2013 at 9:41 PM, Yaroslav Halchenko <debian at onerussian.com>wrote:
> Thanks Michael,
> It is a nice example showing that we better permute testing set
> as well. Otherwise, the stronger our signal is -- worse we become in
> detecting it (kinda awkward isn't it?).
That is not the case. For the higher SNR value the accuracy is at 100% and
associated p-value is at the theoretical minimum.
> e.g. with 2 chunks and SNR 30.00 (i.e. where signal is obviously
> there and classification on unpermutted labels probably is stable 100%)
> -- we would not be able to tell if that is "significant" if we maintain
> original order in the testing set while doing permutation testing.
Not the case -- if you run the script and increase the bin size you can see
that the p-value
is a nice one ;-)
> now (if you are already asleep, I will try doing it later) - just extend
> it into the power (or ROC detection) analysis -- what we are actually
> after here. Add SNR=0.00 -- collect both types of samples and
> describe each plot with its power to detect the true signal using both
> kinds of permutation. Ideally your power should grow with SNR ;)
I am not asleep, but I am having a good time doing nothing ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Pkg-ExpPsy-PyMVPA