[pymvpa] Cross validation and permutation test

[Yaroslav Halchenko]

>    Thanks very much for your help! I wasn't sure if I can use MCNullDist
>    for CrossValidationTransferError in the same way as for TransferError.
>    Now I can make the code much simpler and run much quicker!

may be I have missed something but I do not think that you would obtain
any mentionable speed-up since the same total number of calls to
TransferError would be performed.

>    PS, I think the examples in my data should be well independent. It is a
>    slow event-related design. The inter-stimulus interval is at least 9
>    seconds and only one fMRI volume (acquired at about 9 sec after each
>    stimulus) per stimulus/trial was taken and fed into the classifier,
>    that is, only one example for each stimulus.

Well... it would still depend on many factors including
trial-order/preprocessing/... effects.   Probably the most conservative
assumption could be that 'chunks' represent the independent blocks of
data, and thus instead of permuting labels among samples it then
sensible to permute labels across chunks (i.e. taking labels sequence
from one chunk and giving it to another).  It would require though
chunks of the same number of trials of each condition, and it might
considerably widen H0 distribution. But it would be the most adequate
taking into account the slugishness of the BOLD response and possibly
undiscovered confounds.

Such permutation testing will come in 0.6 shortly.

>    Actually, my purpose of performing a permutation here is to see how the
>    null distribution looks like, i.e., if it is centred at 50%, to confirm
>    that there is no unknown technical problem which shifts the chance
>    level away from 50%.

;-)  I have heard at least about 1 such "technical problem" 

>    Since my MVPA is run on each subject, the final P
>    value will be determined by performing a t test (against 50%) or
>    non-parametric test (e.g. Wilcoxon signed rank test) on prediction
>    rates across all subjects. Please let me know if there is anything
>    incorrect.

Everything sounds correct in terms of my knowledge of currently accepted
practices ;-)

-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic