[pymvpa] Searchlight + cross validation & significance (permutation test)
Nynke van der Laan
nynkevanderlaan at gmail.com
Mon Aug 1 14:34:03 UTC 2011
I have a question about the following: I am performing a searchlight
analysis (v04) with a crossvalidation (leave one subject out, 20
samples per subject, 2 labels (10 samples label A, 10 samples label
I have adapted the example scripts to generate a map with the mean
classification error over de NFold crossvalidation. Now, in addition I
would like to be able to say something about the significance of the
accuracies that the analysis yielded.
In the module reference
I noticed the null_prob and null_T are available variables. Null_prob
is enabled as default. However, the analysis now only results in the
searchlight errormap. How can I acces the null_prob information? Is
this averaged over the NFolds?
cv = CrossValidatedTransferError(
sl = Searchlight(cv, radius=6)
sl_map = sl(dataset_total)
An additional question I have is whether there is any easy way to
combine a searchlight analysis as described above with a permutation
test. In a previous discussion I read about the NullhypothesisTest
but this does not seem to be present in the v.04 version anymore (?).
I predict that this analysis will be very costly in time as the
permutation is done in each searchlight, but I wouldn't know any
good/valid alternatives. Does anyone else have a suggestion?
Thanks a lot!
More information about the Pkg-ExpPsy-PyMVPA