[pymvpa] Searchlight + cross validation & significance

[Chris Johnson]

Just as an update, that last ran into some issues, but this correction
at least gets results when (run with a count of 1, for a quick test):

svm = LinearCSVMC()
partitioner = NFoldPartitioner()
repeater = Repeater(count=200)
permutator = AttributePermutator('targets', limit={'partitions': 1}, count=1)

null_cv = CrossValidation(svm, ChainNode([partitioner,
permutator],space=partitioner.get_space()))
distr_est = MCNullDist(repeater,tail='left', measure=null_cv,
enable_ca=['dist_samples'])

cv = CrossValidation(svm, partitioner, null_dist=distr_est)
sl = sphere_searchlight(cv, radius=radius)

errors = sl(dataset)

The difference is that the distr_est is run on the CrossValidation,
not the searchlight. So given that, I want to post my understanding of
the process, and make sure that the code is doing what I want,
methodologically.

For each neighborhood in the searchlight, the cross-validator is run,
as usual. But for each test point, the training data is permuted, and
this is performed 200 times to get an estimate of the null
distribution for predicting the actual label from randomized training.
So errors, instead of being the average CV training error, is actually
an average p-value at each voxel, estimated from 200 random trainings
at that voxel.

Chris


On Tue, Oct 2, 2012 at 5:22 PM, Chris Johnson <effigies at gmail.com> wrote:
> Doing a little thread necromancy, I wanted to query you regarding how
> to actually do this.
>
> Supposing that my standard searchlight is currently performed so:
>
> cv = CrossValidation(LinearCSVMC(),NFoldPartitioner())
> sl = sphere_searchlight(cv, radius=radius)
>
> I want to get a per-voxel null distribution. Here's my first pass, but
> I'm not very confident in my ability to combine these structures
> properly:
>
> svm = LinearCSVMC()
>
> partitioner = NFoldPartitioner()
> repeater = Repeater(count=200)
> permutator = AttributePermutator('targets', limit={'partitions': 1}, count=1)
>
> null_cv = CrossValidation(svm, ChainNode([partitioner, permutator],
> space=partitioner.get_space()))
> null_sl = sphere_searchlight(null_cv, radius=radius)
> distr_est = MCNullDist(repeater,tail='left', measure=null_sl,
> enable_ca=['dist_samples'])
>
> cv = CrossValidation(svm, partitioner)
> sl = sphere_searchlight(cv, radius=radius, null_dist=distr_est)
>
> Thanks for any pointers you can give. In particular, I'm not sure if
> the null_dist=distr_est in the searchlight is the right way to go
> about it. (I can validate that it gets down to the last three lines,
> depending on whether MCNullDist is lazy or not. I reduced the count to
> 10, but still anticipate it taking 4 hours to see whether this goes
> through, and would like to get some feedback, if I've made any obvious
> mistakes.)
>
> Cheers,
> Chris
>
>
> On Thu, Apr 12, 2012 at 3:11 PM, Michael Hanke <mih at debian.org> wrote:
>> On Thu, Apr 12, 2012 at 12:29:24PM -0400, Yaroslav Halchenko wrote:
>>> Searchlight is just a Measure so its chance distribution (per each
>>> searchlight) and corresponding p-values could be computed the same
>>> fashion as any other measure, e.g. like in
>>> http://www.pymvpa.org/examples/permutation_test.html
>>> done for CrossValidation
>>
>> Note, however, if you get into really serious MC testing (i.e. only
>> permuting the respective training portion of the data and _not_ the
>> testing part) things get a little more complicated.
>>
>> You basically need to compose a custom MCNullDist estimator that is very
>> similar to your original searchlight cross-validation, but takes care of
>> doing the desired constraint permutation inside the cross-validation
>> inside the searchlight ;-)
>>
>>
>> Michael
>>
>>
>> --
>> Michael Hanke
>> http://mih.voxindeserto.de
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa