[pymvpa] Crossvalidation and permutation scheme on one run only

[Richard Dinga]

> If you need to stick to cross-validation within people, I'd prefer
splitting the dataset into two halves.
>It's sometimes surprising how decent performance can be even with fairly
few examples; often classifying with say, only 6 highly temporally
compressed images in the training set will do better than using a few dozen
less compressed images.

Thanks for your suggestions, we tried split half and performance
significantly dropped. All the blobs we saw before disappeared (using GNB
serachlight). Can you elaborate your point about temporal compression? Do
you mean not to model each trial separately, but multiple trials together?
How many trials? should they be consecutive?

On Tue, May 31, 2016 at 3:53 AM, J.A. Etzel <jetzel at wustl.edu> wrote:

> It's sometimes surprising how decent performance can be even with fairly
> few examples; often classifying with say, only 6 highly temporally
> compressed images in the training set will do better than using a few dozen
> less compressed images. Similarly, I suggest not dismissing the possibility
> of across-subjects classification out of hand; it can work quite well.
>
> Correlation-based analyses (which I assume you mean by RSA) may also be
> suitable, depending on your hypotheses.
>
> Jo
>
> On May 30, 2016 4:31:50 PM Richard Dinga <dinga92 at gmail.com> wrote:
>
>> > Do you have to do within-subjects classification, or could you
>> cross-validate on the subjects? (I guess that is option 4.) That would also
>> simplify the permutation testing.
>>
>> yes, I am interested in within subject pattern of activities, possibly RSA
>>
>> > If you need to stick to cross-validation within people, I'd prefer
>> splitting the dataset into two halves (first half of the run, second half
>> of the run, which I assume is your option 1) over option 2: adding more
>> cross-validation folds just compounds the potential confounds, and there's
>> nothing special about 10 folds.
>>
>> 10 folds has almost twice as many training points, therefore I guess it
>> will have higher accuracy, and with split half, there still wouldn't be
>> much to permute
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20160601/be6d7c4f/attachment.html>