<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
Dear Yaroslav,<div><br></div><div>Thanks very much for your help! I wasn't sure if I can use MCNullDist for CrossValidationTransferError in the same way as for TransferError. Now I can make the code much simpler and run much quicker!</div><div><br></div><div>PS, I think the examples in my data should be well independent. It is a slow event-related design. The inter-stimulus interval is at least 9 seconds and only one fMRI volume (acquired at about 9 sec after each stimulus) per stimulus/trial was taken and fed into the classifier, that is, only one example for each stimulus. </div><div>Actually, my purpose of performing a permutation here is to see how the null distribution looks like, i.e., if it is centred at 50%, to confirm that there is no unknown technical problem which shifts the chance level away from 50%. Since my MVPA is run on each subject, the final P value will be determined by performing a t test (against 50%) or non-parametric test (e.g. Wilcoxon signed rank test) on prediction rates across all subjects. Please let me know if there is anything incorrect.</div><div><br></div><div>Best,</div><div>Meng<br><br>> Date: Fri, 11 Mar 2011 22:29:52 -0500<br>> From: debian@onerussian.com<br>> To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> Subject: Re: [pymvpa] Cross validation and permutation test<br>> <br>> > Thanks, Yaroslav, I've had a glance at the tutorial for 0.6 version and<br>> > it looks like I do have to learn twice... Is there a PDF file of the<br>> > tutorial? - it would be easier to read and search for specific<br>> > commands.<br>> <br>> ha -- we just thought noone would be interested in PDF any more , so we<br>> haven't fixed issues emerged while building it... ok -- demand is here, we will<br>> fix it up and let you know<br>> <br>> > For performing permutation test for cross validation on 0.4 version, if<br>> > there isn't an easy way to do the permutation directly<br>> > using CrossValidatedTransferError<br>> <br>> there is -- but then not on per split but on the average error at once:<br>> CrossValidatedTransferError is just yet another Measure so it could take<br>> null_dist as the argument the same as you did for TransferError itself:<br>> <br>> # Lets do the same for CVTE<br>> cvte = CrossValidatedTransferError(<br>> TransferError(clf=l_clf),<br>> OddEvenSplitter(),<br>> null_dist=MCNullDist(permutations=num_perm,<br>> tail='left',<br>> enable_states=['dist_samples']))<br>> cv_err = cvte(train)<br>> <br>> NB this is a snippet from our unittests... for some reason we haven't included<br>> something like that into the examples, e.g.<br>> <br>> http://v04.pymvpa.org/examples/permutation_test.html<br>> <br>> but there are issues with such testing if your samples might not be fully independent...<br>> <br>> in 0.6 we are going to make it more flexible to allow more adequate permutation<br>> testing - starting from permutting only within training split, and then,<br>> if there is a belief that even that is not enough, permutting chunks labeling<br>> at once, instead of independent samples -- that would allow for possibly more<br>> conservative, but more trustful testing.<br>> <br>> watchout for announcements on the updates of <br>> http://www.pymvpa.org/tutorial_significance.html<br>> <br>> >, is the following code (using TransferError) correct?<br>> <br>> > nd =<br>> > MCNullDist(permutations=2500,tail='left',enable_states=['dist_samples']<br>> > )<br>> > terr = TransferError(clf,null_dist=nd)<br>> > cv_index = 0<br>> > splitter = NFoldSplitter(cvtype=1)<br>> > for wdata, vdata in splitter(dataset):<br>> > err = terr(vdata,wdata)<br>> > Error[SubjIndex,cv_index] = err<br>> > Pvalue[SubjIndex,cv_index] = terr.null_prob<br>> > Distribution_normalizedMeanSD = nd.dist_samples<br>> > nd.clean()<br>> <br>> > cv_index += 1<br>> <br>> > #===========<br>> <br>> > In this way, I can get the null distribution for each split (i.e.<br>> > dist_samples). The prediction error of the cross-validation is the<br>> > average of prediction error of all splits, I suppose I will need to<br>> > generate another null distribution for the mean predication error by<br>> > sampling from these null distributions of all split and then<br>> > calculating the mean of the new samples and repeat the procedure for,<br>> > e.g., 1000 times? Is it correct?<br>> <br>> looks and sounds correct. But once again, if you are not really<br>> interested in per-split assessment, just use the construct above ;-)<br>> <br>> -- <br>> =------------------------------------------------------------------=<br>> Keep in touch www.onerussian.com<br>> Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic<br>> <br>> _______________________________________________<br>> Pkg-ExpPsy-PyMVPA mailing list<br>> Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa<br></div> </body>
</html>