[pymvpa] Searchlight + cross validation & significance (permutation test)

[Michael Hanke]

Hi,

On Tue, Aug 02, 2011 at 11:55:21AM +0200, Nynke van der Laan wrote:
> searchlight is unclear to me. I assume the null_dist = MCNullDist...
> should be included in the TransferError in the script below, but the
> searchlight now only returns the error scores for each voxel and no
> other results. How could I perform the permutation test for each
> searchlight and let the searchlight output also the p values for
> testing against the null distribution?
> 
> cv = CrossValidatedTransferError(
>          TransferError(LinearCSVMC()),
>          NFoldSplitter())
> sl = Searchlight(cv, radius=6)
> sl_map = sl(dataset_total)

I don't think we ever had this properly implemented in the 0.4 series.
Yarik might know a trick but, unfortunately, both of us are essentially
offline for the next few days.

Getting permutation procedures for arbitrary analyses was one of the
main motivations for the new codebase, because everybody felt like you
know ;-)

Given this fact you might be a lot quicker rerunning the whole analysis
with the new code (especially if it is as simple as outlined above),
instead of fiddling to get the permutation to work with v0.4.

Here is how it works in the new code:

  http://www.pymvpa.org/examples/permutation_test.html

If you decide to go for this keep in mind that the searchlight radius in
the new code is in 'elements/voxels' and not in mm.

Michael

-- 
Michael Hanke
http://mih.voxindeserto.de