You all have my gratitude for offering your (precious!) insight.<div><br></div><div>I will try to post on what workaround I come up with.</div><div><br></div><div>Cheers,</div><div>Jacob<br><br><div class="gmail_quote">On Thu, Jan 31, 2013 at 8:44 PM, J.A. Etzel <span dir="ltr"><<a href="mailto:jetzel@artsci.wustl.edu" target="_blank">jetzel@artsci.wustl.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">First, I'll echo Rawi's permutation suggestion. In "typical" fMRI MVPA the permutation distribution should be more-or-less normal in shape and centered on chance. Being centered on chance, some accuracies will be below chance, but I would expect relatively few to be below .4 (or above .6). Assuming the permutation distribution is centered on chance, doing a one-sided permutation test (real accuracy greater than chance) will result in a p-value of MORE than 0.5, so you won't get below-chance accuracies coming out significant, even if the real value is in the extreme left tail of the distribution.<br>
<br>
People do these differently; I showed a bit how I often set things up at <a href="http://mvpa.blogspot.com/2012/09/permutation-testing-one-person-only.html" target="_blank">http://mvpa.blogspot.com/2012/<u></u>09/permutation-testing-one-<u></u>person-only.html</a> , which also has some null distributions.<br>
<br>
My strategy when running into below-chance accuracies (to be clear, I don't worry about accuracies unless they're below .45 or so for two-class classification) is to check for errors (like mislabeling examples or runs), then change the classification parameters (particularly the cross-validation scheme), then perhaps the temporal compression.<br>
<br>
I've always been able to either find an error or alternative scheme that eliminates the troublesome below-chance accuracies. I'm certainly not suggesting that you should try every possible combination - too many experimenter degrees of freedom is a very dangerous thing in terms of false positives. But avoiding extreme imbalance in the datasets can really help (for example, if you have 16 runs but only a few examples in each run you might have much better results by leaving four runs out at a time instead of just one; if you have thousands of voxels but ten examples, using fewer features can help).<br>
<br>
Another question: how many subjects are showing below-chance classification? One or two out of twenty is probably unsurprising (and usually is considered significant at the group level if the other 18 people classify well).<br>
<br>
If most all subjects are below-chance I'd dig hard for errors and imbalance rather than make interpretations. You might have to get creative; sometimes tests like making random number datafiles then running them through your procedure can turn up subtle problems (like if one class is getting scaled differently than the other).<br>
<br>
good luck,<br>
Jo<br>
<br>
PS - If you have a simulation strategy for creating fMRI-like datasets with below-chance accuracy I'd be curious to see it.<div class="im"><br>
<br>
<br>
On 1/31/2013 7:29 AM, Jacob Itzhacki wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Dear Rawi and fellow PyMVPAers,<br>
<br>
Thanks for your prompt response. Apologies once again for the<br>
difficulties I adscribe to finding this counterintuitive.<br>
<br>
That said, I have considered your suggestion and I have a couple of<br>
questions regarding it:<br>
- First off, what to do about about significant (p<0.01) classifications<br>
that hover around chance level? In the case of 4 way cross validations<br>
(25% chance) there is a (seemingly) much improved chance that<br>
significance threshold is reached even as classification hovers or is<br>
exactly chance level.<br>
- Would we be able to treat the differring significance spectrum as<br>
individual datapoints or would it have to be a dicotomic statistic (eg.<br>
p<0.01, yes or no?)?<br>
<br>
<br>
Moreover, going back to the original question, is it safe to say that in<br>
a below chance classification performance, even though the classifier is<br>
seemingly doing the opposite of what we are expecting, it is actually<br>
"learning" and hence there was information to learn from?<br>
<br>
Regards,<br>
<br>
Jacob Itzhacki<br>
<br>
On Thu, Jan 31, 2013 at 10:35 AM, MS Al-Rawi <<a href="mailto:rawi707@yahoo.com" target="_blank">rawi707@yahoo.com</a><br></div><div class="im">
<mailto:<a href="mailto:rawi707@yahoo.com" target="_blank">rawi707@yahoo.com</a>>> wrote:<br>
<br>
I believe below-chance accuracy is a natural phenomenon in<br>
classification theorem. This issue is obvious when one finds the<br>
permutation distribution of some problem, where in a two-category<br>
problem, the distribution has a peek around 50% accuracy, so there<br>
will always be some (or, a lot of) below-chance values. This case is<br>
more likely to happen when the dataset has few samples, and probably<br>
high dimensional data. I am not sure if any procedure to relabel the<br>
data, or any other fine-tuned algorithm would be considered as<br>
'tweaking the results'.<br>
<br>
>My question is this: How much (statistical?) merit would it be to<br>
come with some sort of index to show how much a given classification<br>
accuracy is off from absolute chance for this classification?<br>
<br>
A p-value via permutation testing is the better candidate to answer<br>
this question, eg, p<0.01.<br>
<br>
Regards,<br>
-Rawi<br>
<br>
<br>
>_____________________________<u></u>___<br>
> From: Jacob Itzhacki <<a href="mailto:jitzhacki@gmail.com" target="_blank">jitzhacki@gmail.com</a><br></div>
<mailto:<a href="mailto:jitzhacki@gmail.com" target="_blank">jitzhacki@gmail.com</a>>><br>
>To: <a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org" target="_blank">pkg-exppsy-pymvpa@lists.<u></u>alioth.debian.org</a><br>
<mailto:<a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org" target="_blank">pkg-exppsy-pymvpa@<u></u>lists.alioth.debian.org</a>><div><div class="h5"><br>
>Sent: Thursday, January 31, 2013 9:20 AM<br>
>Subject: [pymvpa] On below-chance classification (Anti-learning,<br>
encore)<br>
><br>
><br>
>Dear all,<br>
><br>
><br>
>First off, pardon me if anything of what I say might already be<br>
described somewhere else, I've done quite a bit of searching and<br>
reading on the subject (eg. including Dr. Kowalczyks lecture) but it<br>
is always possible to have bypassed something in this internet age.<br>
After reading as much as I could about the problem I've noticed that<br>
the workarounds proposed don't really fix the problem, which I am<br>
facing quite a bit, to the point that around 1/3 of classifications<br>
are below classification accuracy (38-42% for 2way or 17%-19% for<br>
4-way). I would like to have some feedback on an idea I've had to<br>
try to still have this data be useful.<br>
><br>
><br>
>My question is this: How much (statistical?) merit would it be to<br>
come with some sort of index to show how much a given classification<br>
accuracy is off from absolute chance for this classification?<br>
><br>
><br>
>Elaborating, it would be displaying the absolute value of the<br>
substraction of the resulting accuracy from chance level. Say, for a<br>
2-way classification (with 50% chance level), in which you obtain<br>
accuracies of 38% and 62% in 2 different instances the difference<br>
from chance for both would be 12% which would make them equivalent.<br>
><br>
><br>
>Please offer as much criticism as you can to this approach.<br>
><br>
><br>
>Thanks in advance,<br>
><br>
><br>
>Jacob<br>
><br>
><br>
><br>
><br>
>PS. For completions sake, I'll first list the things I've tried.<br>
><br>
><br>
>I'm running the classification on fMRI data obtained from a<br>
paradigm that gives the following classification opportunities:<br>
><br>
><br>
>a. 4 categories, with 40 trials each at its fullest use (160 trials)<br>
>b. 2 categories as one yielding a classification of 80 trials for<br>
each, by including two categories as one.<br>
>c. 2 categories, with 40 trials each, by disregarding 2 of the<br>
conditions.<br>
><br>
><br>
>I am also using a total of 8 different ROI.<br>
><br>
><br>
>I have tried reordering the trials on one of the subjects, however<br>
this results in above chance accuracies in one analysis and below in<br>
the other for the same ROI which gets rather frustrating if I wanted<br>
to do some sort of averaging by the end. However, there seems to be<br>
some consistency into which classification moves away from chance<br>
which leads me once again to believe that there is in fact some<br>
learning even in the below-chance classifications but the seeming<br>
anti-learning baffles me. What does it mean?! (And how is it even<br>
possible? O.o)<br>
><br>
><br>
>Thanks again.<br>
</div></div></blockquote>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
Joset A. Etzel, Ph.D.<br>
Research Analyst<br>
Cognitive Control & Psychopathology Lab<br>
Washington University in St. Louis<br>
<a href="http://mvpa.blogspot.com/" target="_blank">http://mvpa.blogspot.com/</a></font></span><div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.<u></u>alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.<u></u>org/cgi-bin/mailman/listinfo/<u></u>pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>