<div dir="ltr">Thank you a lot, Emanuele!<br><br>The definition that permutation test is reliable when "the data distribution is adequately<br>
represented by the sample data" sounds a little bit puzzling for me :) As far as I looked, the authors do not discuss this point too much. How can I know when it is indeed adequate? For example, with small number of data points the permutation chance level is frequently high and can be even 0.6. The distribution of predictions from reshuffling looks like normal, so it is not something completely pathological. Not surprisingly, for some subjects it might be difficult if not impossible to reach significance with such a high chance level. Is permutation test appropriate is such scenario? In other words, can it be that result is non-significant with permutation test, but the information is still present.<br>
<br> <br><br><br><div class="gmail_quote">On Tue, May 8, 2012 at 1:39 PM, Emanuele Olivetti <span dir="ltr"><<a href="mailto:emanuele@relativita.com" target="_blank">emanuele@relativita.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sorry for the late reply. My comments below inline.<div class="im"><br>
<br>
On 05/02/2012 09:21 PM, J.A. Etzel wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 5/1/2012 11:27 AM, Vadim Axel wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi experts,<br>
<br>
I am talking about basic pattern classification (e.g. no feature<br>
selection etc). SVM algorithm (with built-in regularization).<br>
<br>
1. A small number of data points with large dimension (ROI size) can<br>
cause overfitting, which is high prediction on training set and bad<br>
test set. Now, suppose, I have a beyond chance classification on test<br>
set, which was validated using within subject permutation test and<br>
across subjects t-test vs. chance. Can my results be still unreliable?<br>
If so, how can I test it?<br>
</blockquote></blockquote>
<br></div>
Overfitting is a different issue from assessing the accuracy on the test set.<br>
The first one is related to training, the second one to testing. You can do fairly<br>
suboptimal training and still be highly confident in better-than-chance<br>
accuracy from the predictions on the test set. Or you can be very sound<br>
in training and still get chance level estimated accuracy on the test set.<br>
Of course a tragic training has less hope to provide a classifier highly accurate<br>
on the test set :-)<br>
<br>
You have a small test set and claim to be beyond chance both at<br>
subject level (permutation test) and at the group level (t-test). Are<br>
your results reliable?<br>
<br>
* The permutation test is reliable when "the data distribution is adequately<br>
represented by the sample data" (Golland and Fischl, IPMI 2003 - a paper that<br>
promotes the use of the permutation test). Are "a small number of data points"<br>
enough to represent your high-dimensional space? Maybe. Or maybe not. Being<br>
able to check this assumption seems to be an open problem itself.<br>
<br>
* The t-test assumes Gaussian distribution of the statistic of interest,<br>
i.e. the accuracy or error rate of each single subject. Note that<br>
accuracy/error ranges in [0,1] so the Gaussian distribution - which has infinite<br>
support - cannot describe it properly. I wouldn't use it.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2. Practically, is 10 independent data points (averaged block value or<br>
beta values) with the ROI of 100 voxels is safe enough?<br>
</blockquote>
I don't know about "safe", but this is in the range of reasonable things to try. I currently have a dataset that works well with a few hundred voxels and only 6 examples, and others that have more examples and fewer voxels.<br>
</blockquote>
<br></div>
Would you be sufficiently confident that a coin is not fair after tossing it<br>
10 times (or 6 times) and observing always head? I guess even<br>
the binomial test would disagree. In order to assess the evidence<br>
of the data in support to the better-than-chance hypothesis I would<br>
proceed in a different way, like in the reference I mention below.<br>
<br>
Again, the size of the ROI is of little importance here, in my opinion.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
3. Do you know about any imaging papers which tested / discussed this issue?<br>
</blockquote>
Mukherjee, S., Golland, P., Panchenko, D.: Permutation Tests for Classification. AI Memo 2003-019. Massachusetts Institute of Technology Computer Science and Artificial Intelligence Laboratory (2003)<br>
<br>
Klement, S., Madany Mamlouk, A., Martinetz, T., Kurková, V., Neruda, R., Koutník, J.: Reliability of Cross-Validation for SVMs in High-Dimensional, Low Sample Size Scenarios Artificial Neural Networks - ICANN 2008. Vol. 5163. Springer Berlin / Heidelberg (2008) 41-50<br>
<br>
<br>
</blockquote>
<br></div>
I would suggest this one to support my points:<br>
Emanuele Olivetti, Sriharsha Veeramachaneni, Ewa Nowakowska, Bayesian hypothesis testing for pattern discrimination in brain decoding, Pattern Recognition, 45, 2012. <a href="http://dx.doi.org/10.1016/j.patcog.2011.04.025" target="_blank">http://dx.doi.org/10.1016/j.<u></u>patcog.2011.04.025</a><br>
I know self citations suck, still I haven't found a more convincing one.<br>
<br>
Best,<br>
<br>
Emanuele<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.<u></u>alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.<u></u>org/cgi-bin/mailman/listinfo/<u></u>pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>