[pymvpa] effect size (in lieu of zscore)

[Yaroslav Halchenko]

Hi Mike,

First of all thanks for looking into the details and sharing it here --
it is an important and under-explored topic imho.  My comments are below
-- sorry if they are not very well structured

>    With old z-scoring, C=-5:
>    Accuracies are around 60% with a heavy selection bias towards the rest
>    condition (chooses "rest" correctly 27/27 times, but also chooses rest for
>    21/27 sound conditions).

am I taking you right -- you have multi-class classification (multiple
'sound conditions') here? or did you collapse all non-rest conditions
into 1?

by "any of the sound conditions vs. the silence condition at 98-100" did
you mean separate pair-wise classifiers or multi-class?

what is the output of print dataset.summary()?

1. with dominance of rest condition samples (if you haven't
ballanced it out) classifier might be just preferring 'rest' condition
overall

2. with multiclass you might also be hitting here the present problem of
non-arbitrary breakage of the ties - thus leading to collapsing into
'rest' condition.  With just released mvpa2 -- how does it look if you
try SMLR or kNN?


>    With old z-scoring, C=-1:
>    Same as above, except with accuracies around 54%
>    With NO z-scoring, C=-5 or C=-1:
>    Accuracies are 98-100%
>    With C=-5 (or C=-1) and zscore(dataset, chunks_attr='chunks',
>    dtype='float32'):
>    Accuracies are about 98%

>    So it looks like:
>    (a) Using C=-5 (as opposed to C=-1) helps a little with the zscore against
>    rest method. Although it might help across the board, but there's a
>    ceiling effect with the other combinations.

just to make it clear -- by 'old z-scoring' you meant z-scoring against
rest condition, right?  and in new one you just z-scored across all
conditions (including rest) and it lead to good generalization...

question: what was exact line you used to z-score against rest
condition? we might like also to check if everything implemented as it
should but also it might simply be that due to smaller number of trials
for rest condition alone, estimates of mean/variance were not stable
thus leading to noisy 'standardized' samples thus lower performance.

>    (b) There's a huge difference between whether I zscore against rest or
>    with the whole time series. I'm not sure what's up... running sounds >
>    silence GLMs in FSL show obvious responses in the expected brain regions.

so once again knowing number of samples in each chunk and how z-scoring
against was done would help us to get better clue


-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic