[pymvpa] effect size (in lieu of zscore)

[Mike E. Klein]

Hi Jonas and Jo,

Thanks for helping out with this!

So:

(1) I haven't done a permutation test. By "chance distribution" I just
meant the bulk of the data points using my real-label-coded data. While I'm
obviously hoping for a histogram that contains a positive skew, *at worst* I'd
expect a normal distribution centered around chance. Once I get this error
figured out, I will do some permutation testing as well, but at the moment
it doesn't seem necessary. (In other words, with real data or fake data, I
can't see why I'd ever see a *negative* skew unless I'm doing something
else wrong.)

(2) I've generally been doing 3-to-1 averaging of my trials because of
time/processing limitations; because python seems to choke on a searchlight
analysis using LinearCSVMC if I don't first perform run-averaging; and
because I'm concerned about the cleanliness of my data (sparse-sampling
images). I'm re-running one of these analyses using LinearNuSVMC and
without any run-averaging, but my hunch is that this isn't the problem.
PyMVPA, after run-averaging, is showing 2 labels with 27 examples each,
which I what I was expecting.

(3) This seems to be an issue with multiple subjects...it might in fact be
a universal problem. I saw it sporadically before, but then it seems my
preprocessing was incorrect (I was zscoring with only 11 samples in
mini-runs, instead of the 30+ I had intended).

(4) My distribution is centered on 0 and goes from -50 to +50 because,
after the searchlight analysis I'm using:

#set up searchlight
sl = sphere_searchlight(cv, radius=3, space='voxel_indices',
center_ids=center_ids, postproc=mean_sample())
#run searchlight on ds
s1_map = sl(dataset)
#convert into percent-wise accuracies centered around zero
s1_map.samples *= -1
s1_map.samples += 1
s1_map.samples *= 100
s1_map.samples -= 50

-Mike


On Tue, Jan 3, 2012 at 1:05 PM, J.A. Etzel <jetzel at artsci.wustl.edu> wrote:

> On 1/2/2012 12:38 PM, Jonas Kaplan wrote:
>
>>
>>>    1. For this one particular subject, I'm still seeing the strange
>>>    negative peak to the chance distribution, even without any
>>>    z-scoring. The shape looks remarkably similar with or without
>>>    zscoring (whether I use the raw values or the effect sizes as
>>>    input). I think my confusion here is, even if I did several things
>>>    wrong in my code, I'd expect no worse than a regular-old looking
>>>    chance distribution (centered on 0). There's about 40,000 3.5
>>>    isotrophic voxels in that subject's brain mask, so plenty of
>>>    observations. Just eyeballing, the peak is centered at about -8%,
>>>    and the bulk (95%) of observations fall between about -32% and
>>>    +22% … so it's a notable shift.
>>>
>>
>> I don't know how exactly these permutations are done nowadays, but
>> couldn't something like this happen if the permutations resulted in
>> unequal number of trials of each type in the new chunks? I got
>> distributions like this before I enabled the perchunk option in the old
>> v.4 permuteLabels function to ensure trials were balanced.
>>
>
> Something is definitely odd to get such a bias in the permutation
> distribution. Are you only seeing this in some subjects, or everyone?
>
> If everyone, I'd be tempted to test the entire processing stream with data
> guaranteed to be random. Basically, make a set of fake .nii full of random
> numbers, the same size, number of images, etc. as your real data. If these
> random images also result in skewed distributions, you'll know something in
> the processing is introducing the bias (e.g. scaling the classes
> separately).
>
> I usually make histograms based on accuracy, so the range is 0 to 1
> (centered on 0.5 for 2-class). What exactly did you plot here to get -50 to
> 50?
>
> Jo
>
>
> ______________________________**_________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.**alioth.debian.org<Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**
> pkg-exppsy-pymvpa<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120103/4f306ee9/attachment.html>