[pymvpa] MC-Simulation & sensitivity map

[Matthias Ekman]

Dear Experts,

besides reporting accuracies for whole-brain binary classification I
would like show some 'pattern maps'. One might go with the classical
searchlight approach, but I personally don't like the idea that the
pattern map derived from searchlight is not really driving the
whole-brain classifier accuracy (comparing searchlight maps and
sensitivity maps showed some (more or less) considerable differences
in my case). One big advantage of the searchlight (besides 'smooth'
maps) is the possibility to calculate z-/t-maps and apply
statistical-testing quite easy.

So, the question is how to handle the sensitivity maps. There may be a
lot of different approaches (e.g. bin weights and to calculate overlap
over subjects, bootstrap as suggested by Pereira et al. (2009), etc.).
I would like to go with the following procedure and I would like to
ask your advice if this might be valid. The Idea is to calculate a
sensitivity map plus an amount of sensitivity maps with permuted
labels like in MCS. The p-value would be the number of permutations
where clf weight is greater than or equal to observed clf weight
(respectively = max((number_of samples>alpha)/number_of_permutations,
1/number_of_permutations as Yarik pointed out here:
http://groups.google.com/group/mvpa-toolbox/browse_thread/thread/1059ece7da64e360)
.

My questions are:
1. Would it be valid to use MCS with clf weights?
2. Would this approach be also valid for sparse classifiers (like e.g.
SMLR, ENET etc.)?
Due to the sparsity a lot of features get values of zero which might
be not "strict" enough for H0-testing.
3. Would it be necessary (for any reason I don't see yet) to L1/L2
norm the pattern weights over permutations?
4. Would it be valid to use absolute weights for MCS for clfs where
high positive & low negative weights have huge influence on the
decision (I don't think so)?
5. To deal with the issue of multiple comparison I would go with FDR
since clustering might ignore the nature of this multivariate pattern
approach. Do you agree?

I would really appreciate to hear your thoughts on this,
 Matthias

_____________________________________________

Matthias Ekman, Dipl.-Psych.
Max Planck Institute for Neurological Research
Gleueler Strasse 50
50931 Cologne, Germany

E-mail: Ekman at nf.mpg.de
Phone: +49-(0)221-4726-255
_____________________________________________