[pymvpa] effect size (in lieu of zscore)

[Mike E. Klein]

A separate but related issue:

Does anyone know what could cause a negative shift in the searchlight
accuracy distribution for a 2-category Linear SVM classifier of single
subject? I'm seeing this in a couple of my subjects following a switch to
using zscore(dataset, chunks_attr='chunks', dtype='float32') …. so zscoring
against the mean and not against a baseline condition. (I'm actually
zscoring after removing the rest trials from the timeseries.) While the
vast majority of my subjects show a normal looking curve with a peak right
at 50%, a couple of subjects, while having a normal-esque curve, show
accuracy peaks at 40-45%.

Generally speaking, I'm seeing group data accuracy peaks in more-or-less
our hypothesized areas of the brain, but I'm still a bit worried that I've
done something wrong somewhere along the line. I can't figure out why I'm
seeing the negative peak (in a couple of subjects) or why using zscore
against rest has such a detrimental effect on my accuracies…

Thanks and best,
Mike



On Fri, Dec 16, 2011 at 11:53 PM, Mike E. Klein <michaeleklein at gmail.com>wrote:

> Thanks for the response!
>
> I just took a really quick first look (just a single 2-way comparison for
> a single subject):
>
> Looking between my experimental conditions, it looks like setting C=-5
> and/or using *zscore(dataset, chunks_attr='chunks', dtype='float32') *leads
> to accuracies that are a bit higher than before, but still worse than
> without any zscoring at all. (I hope that I've done the zscoring against
> the full time series correctly...)
>
> Looking at sounds vs. silence:
>
> *With old z-scoring, C=-5:*
> Accuracies are around 60% with a heavy selection bias towards the rest
> condition (chooses "rest" correctly 27/27 times, but also chooses rest for
> 21/27 sound conditions).
>
> *With old z-scoring, C=-1:*
> Same as above, except with accuracies around 54%
>
> *With NO z-scoring, C=-5 or C=-1:*
> Accuracies are 98-100%
>
> *With C=-5 (or C=-1) and zscore(dataset, chunks_attr='chunks',
> dtype='float32'):*
> Accuracies are about 98%
>
>
> So it looks like:
>
> (a) Using C=-5 (as opposed to C=-1) helps a little with the zscore against
> rest method. Although it might help across the board, but there's a ceiling
> effect with the other combinations.
> (b) There's a huge difference between whether I zscore against rest or
> with the whole time series. I'm not sure what's up... running sounds >
> silence GLMs in FSL show obvious responses in the expected brain regions.
>
>
> Thanks again,
> Mike
>
>
>
>
>
> On Fri, Dec 16, 2011 at 11:14 PM, Yaroslav Halchenko <
> debian at onerussian.com> wrote:
>
>> before discussion kicks in -- out of curiosity... what happens if you
>> either do nested cross-validation to choose C parameter or just set it
>> a bit higher (e.g. C=-5 to still be scaled according  to the data), what
>> if you do zscoring across full time series (not just baseline condition)
>> -- for both of those there are explanations, I just wondered to get
>> better idea of what might be happening in your case
>>
>>
>> On Fri, 16 Dec 2011, Mike E. Klein wrote:
>> >    where my baseline condition is silence. Without zscoring, SVMs can
>> tell
>> >    any of the sound conditions vs. the silence condition at 98-100%
>> >    accuracy...which makes sense. With zscoring, this drops to the 80-90%
>> >    level. The experiment has a good amount of samples, is well-balanced,
>> >    motion-corrected, etc., so I can't find other obvious confounds.)�
>> --
>> =------------------------------------------------------------------=
>> Keep in touch                                     www.onerussian.com
>> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20111220/cdd1178a/attachment.html>