Hi Jonas and Jo,<div><br></div><div>Thanks for helping out with this!</div><div><br></div><div>So:</div><div><br></div><div>(1) I haven't done a permutation test. By "chance distribution" I just meant the bulk of the data points using my real-label-coded data. While I'm obviously hoping for a histogram that contains a positive skew, <i>at worst</i> I'd expect a normal distribution centered around chance. Once I get this error figured out, I will do some permutation testing as well, but at the moment it doesn't seem necessary. (In other words, with real data or fake data, I can't see why I'd ever see a <i>negative</i> skew unless I'm doing something else wrong.)</div>
<div><br></div><div>(2) I've generally been doing 3-to-1 averaging of my trials because of time/processing limitations; because python seems to choke on a searchlight analysis using LinearCSVMC if I don't first perform run-averaging; and because I'm concerned about the cleanliness of my data (sparse-sampling images). I'm re-running one of these analyses using LinearNuSVMC and without any run-averaging, but my hunch is that this isn't the problem. PyMVPA, after run-averaging, is showing 2 labels with 27 examples each, which I what I was expecting. </div>
<div><br></div><div>(3) This seems to be an issue with multiple subjects...it might in fact be a universal problem. I saw it sporadically before, but then it seems my preprocessing was incorrect (I was zscoring with only 11 samples in mini-runs, instead of the 30+ I had intended). </div>
<div><br></div><div>(4) My distribution is centered on 0 and goes from -50 to +50 because, after the searchlight analysis I'm using:</div><div><div><br></div><div><div>#set up searchlight</div><div>sl = sphere_searchlight(cv, radius=3, space='voxel_indices', center_ids=center_ids, postproc=mean_sample())</div>
<div>#run searchlight on ds</div><div>s1_map = sl(dataset)</div></div><div>#convert into percent-wise accuracies centered around zero</div><div>s1_map.samples *= -1</div><div>s1_map.samples += 1</div><div>s1_map.samples *= 100</div>
<div>s1_map.samples -= 50</div><div><br></div><div>-Mike</div><div><br></div><br><div class="gmail_quote">On Tue, Jan 3, 2012 at 1:05 PM, J.A. Etzel <span dir="ltr"><<a href="mailto:jetzel@artsci.wustl.edu" target="_blank">jetzel@artsci.wustl.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 1/2/2012 12:38 PM, Jonas Kaplan wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
1. For this one particular subject, I'm still seeing the strange<br>
negative peak to the chance distribution, even without any<br>
z-scoring. The shape looks remarkably similar with or without<br>
zscoring (whether I use the raw values or the effect sizes as<br>
input). I think my confusion here is, even if I did several things<br>
wrong in my code, I'd expect no worse than a regular-old looking<br>
chance distribution (centered on 0). There's about 40,000 3.5<br>
isotrophic voxels in that subject's brain mask, so plenty of<br>
observations. Just eyeballing, the peak is centered at about -8%,<br>
and the bulk (95%) of observations fall between about -32% and<br>
+22% … so it's a notable shift.<br>
</blockquote>
<br>
I don't know how exactly these permutations are done nowadays, but<br>
couldn't something like this happen if the permutations resulted in<br>
unequal number of trials of each type in the new chunks? I got<br>
distributions like this before I enabled the perchunk option in the old<br>
v.4 permuteLabels function to ensure trials were balanced.<br>
</blockquote>
<br></div>
Something is definitely odd to get such a bias in the permutation distribution. Are you only seeing this in some subjects, or everyone?<br>
<br>
If everyone, I'd be tempted to test the entire processing stream with data guaranteed to be random. Basically, make a set of fake .nii full of random numbers, the same size, number of images, etc. as your real data. If these random images also result in skewed distributions, you'll know something in the processing is introducing the bias (e.g. scaling the classes separately).<br>
<br>
I usually make histograms based on accuracy, so the range is 0 to 1 (centered on 0.5 for 2-class). What exactly did you plot here to get -50 to 50?<br>
<br>
Jo<div><div><br>
<br>
______________________________<u></u>_________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.<u></u>alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.<u></u>org/cgi-bin/mailman/listinfo/<u></u>pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>