[pymvpa] Cross validation and permutation test

[Meng Liang]

Dear Yaroslav,
Thanks again for your detailed explanation!
> >    Thanks very much for your help! I wasn't sure if I can use MCNullDist
> >    for CrossValidationTransferError in the same way as for TransferError.
> >    Now I can make the code much simpler and run much quicker!
> 
> may be I have missed something but I do not think that you would obtain
> any mentionable speed-up since the same total number of calls to
> TransferError would be performed.
What I meant by 'quicker' is that I don't have to perform the permutation for each split separately. Unless the permutation number specified in MCNullDist embedded within CrossValidationTransferError (the code you suggested in the last email) is also for each split, i.e., permutations=1000 for an OddEvenSplit actually means 2000 permutations in total? But the number of distribution samples obtained is still 1000.
> >    PS, I think the examples in my data should be well independent. It is a
> >    slow event-related design. The inter-stimulus interval is at least 9
> >    seconds and only one fMRI volume (acquired at about 9 sec after each
> >    stimulus) per stimulus/trial was taken and fed into the classifier,
> >    that is, only one example for each stimulus.
> 
> Well... it would still depend on many factors including
> trial-order/preprocessing/... effects.   Probably the most conservative
> assumption could be that 'chunks' represent the independent blocks of
> data, and thus instead of permuting labels among samples it then
> sensible to permute labels across chunks (i.e. taking labels sequence
> from one chunk and giving it to another).  It would require though
> chunks of the same number of trials of each condition, and it might
> considerably widen H0 distribution. But it would be the most adequate
> taking into account the slugishness of the BOLD response and possibly
> undiscovered confounds.
> 
> Such permutation testing will come in 0.6 shortly.
0.6 version sounds much better. I'll have to learn it.
> >    Actually, my purpose of performing a permutation here is to see how the
> >    null distribution looks like, i.e., if it is centred at 50%, to confirm
> >    that there is no unknown technical problem which shifts the chance
> >    level away from 50%.
> 
> ;-)  I have heard at least about 1 such "technical problem" 
> 
> >    Since my MVPA is run on each subject, the final P
> >    value will be determined by performing a t test (against 50%) or
> >    non-parametric test (e.g. Wilcoxon signed rank test) on prediction
> >    rates across all subjects. Please let me know if there is anything
> >    incorrect.
> 
> Everything sounds correct in terms of my knowledge of currently accepted
> practices ;-)
Thanks for your confirmation.
Best,Meng 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20110313/8ecd6bd5/attachment.htm>