[pymvpa] SVM classification of data with temporal correlation

[Francisco Pereira]

Vadim,

On Thu, Dec 3, 2009 at 4:56 PM, Vadim Axel <axel.vadim at gmail.com> wrote:
>
> 2. I understand the concern with regard to binomial accuracy test. I think
> it might be more robust to create scrambled labels distribution by running
> ~1000  times the classification on scrambled data (some sort of
> non-parametric test). Just recently, I have encountered some class which for
> some unknown reason even with scrambled labels was far beyond chance. So,
> the binomial test would have missed this.

This is indeed the other option: if you suspect there will be an
(optimistic) bias in the result, run a permutation test. The only
caveat here is that whatever it is you are permuting over must be
exchangeable; intuitively this means that any order of examples
could have arisen out of the process generating them, which would
certainly not be the case if using successive TRs as examples. If you
are creating examples from blocks that should be fine. What I usually
do is permute labels between examples in the same run, as that ensures
both the number of examples of each label remains the same as when
using the true labels. Another reason would be to make sure there was
no subtle difference between runs that would undermine
exchangeability.

cheers,
Francisco