[pymvpa] high prediction rate in a permutation test
Yaroslav Halchenko
debian at onerussian.com
Wed May 18 22:54:17 UTC 2011
On Wed, 18 May 2011, J.A. Etzel wrote:
> A t-test is possible, assuming you're doing a within-subjects
> analysis (classifying every person separately). But it's not what I
> prefer. One reason is that we're often on the edge of what
> parametric tests can handle (number of subjects, distributions,
> dependencies, etc.). Another is that a t-test isn't quite focused on
> what I want to know: I want to know if the average accuracy is
> greater than what we'd get with random images, which is what's
> tested with a well-designed permutation test. For example, imagine
> your subjects had very similar accuracies just above chance (0.52,
> 0.53, etc.). Under the right conditions this could turn out as
> significant with a t-test, but probably shouldn't be considered
> important.
exactly! additional example to appreciate the topic:
which of the two cases in case of binary classification you would prefer to see
as the "significant" or trustful result? ;)
0.60000 0.70000 0.80000 0.90000 1.00000
or
0.51000 0.52000 0.53000 0.54000 0.55000
which, if I didn't get it wrong should have the same t-score against the chance
level of 0.5 ;-)
in other additional words: who said that raw accuracies are normally
distributed? ;)
But since it is a common practice, Vadim please do not take those words
above as the "stop sign". Just keep in mind the "effect size" ;)
> As a practical matter, I sometimes calculate t-test p-values in the
> early stages of analysis because they're so fast, then calculate
> permutation tests for the final p-values. In some datasets the
> p-values from the two methods are close, in others they've been far
> apart, sometimes with the t-test p-values much less significant.
It is the evening, and we already celebrated the successful launch of our
neuroscience software survey (I bet all of you participated already, didn't
you?) --- Jo, could you please elaborate a bit more on above "fast t-test
-> permutations -> p-values"? I might be missing something obvious ;)
Thanks in advance!
> ps: always good to plot data to eyeball normality, not just run tests. :)
Good advice, but data is scary -- blobs in the stat plots are eyecandies! ;)
--
=------------------------------------------------------------------=
Keep in touch www.onerussian.com
Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
More information about the Pkg-ExpPsy-PyMVPA
mailing list