[pymvpa] high prediction rate in a permutation test
jtkaplan at usc.edu
Thu May 19 17:36:56 UTC 2011
On May 18, 2011, at 3:54 PM, Yaroslav Halchenko wrote:
> exactly! additional example to appreciate the topic:
> which of the two cases in case of binary classification you would prefer to see
> as the "significant" or trustful result? ;)
> 0.60000 0.70000 0.80000 0.90000 1.00000
> 0.51000 0.52000 0.53000 0.54000 0.55000
> which, if I didn't get it wrong should have the same t-score against the chance
> level of 0.5 ;-)
> in other additional words: who said that raw accuracies are normally
> distributed? ;)
> But since it is a common practice, Vadim please do not take those words
> above as the "stop sign". Just keep in mind the "effect size" ;)
This is an issue I have been thinking about quite a bit recently, as we have used t-tests across subjects in the past (after checking for violation of normalcy and also performing arsine transformation), however I'm no longer convinced it's a great idea, and the main reason for me comes down to interpretation. The interesting hypothetical case to my mind is where a t-test is significant across subjects, but no single subject has significant performance according to a within-subject permutation test. How would we interpret such a result?
A related issue is, what does it mean to have prediction performance that is consistently above chance in all subjects, but so small that prediction is still practically speaking pretty bad? What conclusions does that case allow us to draw about the underlying neural representations? Yes, they contain more information about the stimuli than pure noise would... but is that meaningful? The problem is I'm not sure what an alternative criterion would be. The example quoted above appeals to some sense of this... clearly we want the performance numbers to be higher, but what objective standard do we have other than statistical significance?
Just a bit of rambling...
Jonas Kaplan, Ph.D.
Research Assistant Professor
Brain & Creativity Institute
University of Southern California
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Pkg-ExpPsy-PyMVPA