[pymvpa] high prediction rate in a permutation test

[Yaroslav Halchenko]

On Wed, 18 May 2011, J.A. Etzel wrote:
> A t-test is possible, assuming you're doing a within-subjects
> analysis (classifying every person separately). But it's not what I
> prefer. One reason is that we're often on the edge of what
> parametric tests can handle (number of subjects, distributions,
> dependencies, etc.). Another is that a t-test isn't quite focused on
> what I want to know: I want to know if the average accuracy is
> greater than what we'd get with random images, which is what's
> tested with a well-designed permutation test. For example, imagine
> your subjects had very similar accuracies just above chance (0.52,
> 0.53, etc.). Under the right conditions this could turn out as
> significant with a t-test, but probably shouldn't be considered
> important.

exactly!  additional example to appreciate the topic:

which of the two cases in case of binary classification you would prefer to see
as the "significant" or trustful result? ;)

   0.60000   0.70000   0.80000   0.90000   1.00000

or

   0.51000   0.52000   0.53000   0.54000   0.55000

which, if I didn't get it wrong should have the same t-score against the chance
level of 0.5 ;-)

in other additional words: who said that raw accuracies are normally
distributed? ;)

But since it is a common practice, Vadim please do not take those words
above as the "stop sign".  Just keep in mind the "effect size" ;)

> As a practical matter, I sometimes calculate t-test p-values in the
> early stages of analysis because they're so fast, then calculate
> permutation tests for the final p-values. In some datasets the
> p-values from the two methods are close, in others they've been far
> apart, sometimes with the t-test p-values much less significant.

It is the evening, and we already celebrated the successful  launch of our
neuroscience software survey (I bet all of you participated already, didn't
you?) --- Jo, could you please elaborate a bit more on above "fast t-test
-> permutations -> p-values"? I might be missing something obvious ;)

Thanks in advance!

> ps: always good to plot data to eyeball normality, not just run tests. :)

Good advice, but data is scary -- blobs in the stat plots are eyecandies! ;)

-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic