<div dir="ltr">Yes, I agree with you. However, I somehow feel that reporting significance based on permutation values is more cumbersome than t-tests. Consider the case that out of 10 subjects 8 have significant result (based on permutation) and two remaining are not. What should I say in my results? Does the ROI discriminate between two classes? When I use group t-test everything is simple - the result is true or false for the whole group. Now, suppose that I have more than one ROI and I want to compare their results. Though I can show average prediction rate across subjects, I am afraid that when I start to report for each ROI for how many subjects it was significant and for how many not, everybody (including myself) would be confused....<br>
<br>BTW, how you recommend to correct for multiple comparisons? For example I run 100 search lights.Making Bonferoni correction (0.05/100) = 0.0005 results in very high threshold. Consider my case with the mean values, which is based on 1000 tests only. Based on 0.0005 threshold I need to get classification of 0.75+ (!). My data are not that good :( What people are doing for whole brain when the number of search lights is tens of thousands...<br>
<br><br><div class="gmail_quote">On Thu, May 19, 2011 at 1:54 AM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com">debian@onerussian.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im"><br>
On Wed, 18 May 2011, J.A. Etzel wrote:<br>
</div><div class="im">> A t-test is possible, assuming you're doing a within-subjects<br>
> analysis (classifying every person separately). But it's not what I<br>
> prefer. One reason is that we're often on the edge of what<br>
> parametric tests can handle (number of subjects, distributions,<br>
> dependencies, etc.). Another is that a t-test isn't quite focused on<br>
> what I want to know: I want to know if the average accuracy is<br>
> greater than what we'd get with random images, which is what's<br>
> tested with a well-designed permutation test. For example, imagine<br>
> your subjects had very similar accuracies just above chance (0.52,<br>
> 0.53, etc.). Under the right conditions this could turn out as<br>
> significant with a t-test, but probably shouldn't be considered<br>
> important.<br>
<br>
</div>exactly! additional example to appreciate the topic:<br>
<br>
which of the two cases in case of binary classification you would prefer to see<br>
as the "significant" or trustful result? ;)<br>
<br>
0.60000 0.70000 0.80000 0.90000 1.00000<br>
<br>
or<br>
<br>
0.51000 0.52000 0.53000 0.54000 0.55000<br>
<br>
which, if I didn't get it wrong should have the same t-score against the chance<br>
level of 0.5 ;-)<br>
<br>
in other additional words: who said that raw accuracies are normally<br>
distributed? ;)<br>
<br>
But since it is a common practice, Vadim please do not take those words<br>
above as the "stop sign". Just keep in mind the "effect size" ;)<br>
<div class="im"><br>
> As a practical matter, I sometimes calculate t-test p-values in the<br>
> early stages of analysis because they're so fast, then calculate<br>
> permutation tests for the final p-values. In some datasets the<br>
> p-values from the two methods are close, in others they've been far<br>
> apart, sometimes with the t-test p-values much less significant.<br>
<br>
</div>It is the evening, and we already celebrated the successful launch of our<br>
neuroscience software survey (I bet all of you participated already, didn't<br>
you?) --- Jo, could you please elaborate a bit more on above "fast t-test<br>
-> permutations -> p-values"? I might be missing something obvious ;)<br>
<br>
Thanks in advance!<br>
<div class="im"><br>
> ps: always good to plot data to eyeball normality, not just run tests. :)<br>
<br>
</div>Good advice, but data is scary -- blobs in the stat plots are eyecandies! ;)<br>
<font color="#888888"><br>
--<br>
</font><div class="im">=------------------------------------------------------------------=<br>
Keep in touch <a href="http://www.onerussian.com" target="_blank">www.onerussian.com</a><br>
Yaroslav Halchenko <a href="http://www.ohloh.net/accounts/yarikoptic" target="_blank">www.ohloh.net/accounts/yarikoptic</a><br>
<br>
_______________________________________________<br>
</div><div><div></div><div class="h5">Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>