[pymvpa] Searchlight + cross validation & significance (permutation test)

[Nynke van der Laan]

Hello all,

I have a question about the following: I am performing a searchlight
analysis (v04) with a crossvalidation (leave one subject out, 20
samples per subject, 2 labels (10 samples label A, 10 samples label
B)).
I have adapted the example scripts to generate a map with the mean
classification error over de NFold crossvalidation. Now, in addition I
would like to be able to say something about the significance of the
accuracies that the analysis yielded.
In the module reference
(http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight)
I noticed the null_prob and null_T are available variables. Null_prob
is enabled as default. However, the analysis now only results in the
searchlight errormap. How can I acces the null_prob information? Is
this averaged over the NFolds?

cv = CrossValidatedTransferError(
         TransferError(LinearCSVMC()),
         NFoldSplitter())
sl = Searchlight(cv, radius=6)
sl_map = sl(dataset_total)

An additional question I have is whether there is any easy way to
combine a searchlight analysis as described above with a permutation
test. In a previous discussion I read about the NullhypothesisTest
(http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html)
but this does not seem to be present in the v.04 version anymore (?).
I predict that this analysis will be very costly in time as the
permutation is done in each searchlight, but I wouldn't know any
good/valid alternatives. Does anyone else have a suggestion?

Thanks a lot!

Best,
Nynke