[pymvpa] cross-validation

[Yaroslav Halchenko]

On Mon, 15 Mar 2010, Jonas Kaplan wrote:

>    "We used a standard k-fold cross- validation testing regime [10]
>    wherein k equalled the number of experimental trials, with the data
>    from each trial set aside in turn as the test data, and the remaining
>    data used as the training set (on each fold, the feature selection step
>    was performed using only data from this training set)."

>    In other words, it seems to me that on each cross-validation fold,
>    training was performed on all trials except one, and then the
>    classifier is tested on one single trial.  Does this sort of approach
>    make sense?
I bet they meant that they did such procedure for each trial... thus
they say: "with the data from each trial set aside", so whatever number
of experimental trials they had -- they assessed performance  all the
folds, like you describe below:

>    classifiers.   I suppose the idea is that across all of the folds you
>    get a measure of how the classifiers work in general.

The only problem (while reading the quote  you gave not the paper) I
could see depends on either by trial they meant just 1 sample (bold
volume) or an independent run (chunk in PyMVPA terms)... if it is
just 1 sample of 1 category, they could get quite biased estimate if
they have overall relatively small number of trials and noisy data...
more over, if 'trials' are coming from within the same scanning session
then it might be hard to believe in significance testing requiring
independence of trials (even for permutation testing strictly speaking) 

>    I'd like to know
>    if this is considered a reasonable approach since I have a dataset with
>    a small number of trials that might benefit from maximizing the number
>    of training trials.
how small is small? number of trials/chunks(sessions)/labels?

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]