[pymvpa] Searchlight + cross validation & significance

[Chris Johnson]

Doing a little thread necromancy, I wanted to query you regarding how
to actually do this.

Supposing that my standard searchlight is currently performed so:

cv = CrossValidation(LinearCSVMC(),NFoldPartitioner())
sl = sphere_searchlight(cv, radius=radius)

I want to get a per-voxel null distribution. Here's my first pass, but
I'm not very confident in my ability to combine these structures
properly:

svm = LinearCSVMC()

partitioner = NFoldPartitioner()
repeater = Repeater(count=200)
permutator = AttributePermutator('targets', limit={'partitions': 1}, count=1)

null_cv = CrossValidation(svm, ChainNode([partitioner, permutator],
space=partitioner.get_space()))
null_sl = sphere_searchlight(null_cv, radius=radius)
distr_est = MCNullDist(repeater,tail='left', measure=null_sl,
enable_ca=['dist_samples'])

cv = CrossValidation(svm, partitioner)
sl = sphere_searchlight(cv, radius=radius, null_dist=distr_est)

Thanks for any pointers you can give. In particular, I'm not sure if
the null_dist=distr_est in the searchlight is the right way to go
about it. (I can validate that it gets down to the last three lines,
depending on whether MCNullDist is lazy or not. I reduced the count to
10, but still anticipate it taking 4 hours to see whether this goes
through, and would like to get some feedback, if I've made any obvious
mistakes.)

Cheers,
Chris


On Thu, Apr 12, 2012 at 3:11 PM, Michael Hanke <mih at debian.org> wrote:
> On Thu, Apr 12, 2012 at 12:29:24PM -0400, Yaroslav Halchenko wrote:
>> Searchlight is just a Measure so its chance distribution (per each
>> searchlight) and corresponding p-values could be computed the same
>> fashion as any other measure, e.g. like in
>> http://www.pymvpa.org/examples/permutation_test.html
>> done for CrossValidation
>
> Note, however, if you get into really serious MC testing (i.e. only
> permuting the respective training portion of the data and _not_ the
> testing part) things get a little more complicated.
>
> You basically need to compose a custom MCNullDist estimator that is very
> similar to your original searchlight cross-validation, but takes care of
> doing the desired constraint permutation inside the cross-validation
> inside the searchlight ;-)
>
>
> Michael
>
>
> --
> Michael Hanke
> http://mih.voxindeserto.de
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa