[pymvpa] Labels from permutation testing

[Yaroslav Halchenko]

On Mon, 20 Jul 2015, Bill Broderick wrote:

> Hi all,

> I feel like this should be relatively simple, but I can't figure out how to
> do it. Is it possible to get at the labels generated by
> AttributePermutator? I would like to see what the individual permutations
> look like, to make sure it's doing what I think it is, but other than
> saving the whole dataset generated by CrossValidation, I can't see a way to
> do it.

> I'm trying to build a null distribution like the following, so I can save
> each permutation, each searchlight separately (with how long the
> permutation testing has been taking, I want to make sure there's constant
> output in case something crashes and so I can monitor its progress, so I'm
> not using MCNullDist).

if you were just want to check "in general" on what permutations permutator
generates, and didn't have limit={'partitions':1}, count=1  you could just

[x.targets for x in permutator.generate(ds)]

then if you preseeded RNG the same way before testing permutator
(e.g. mvpa2.seed(index_of_subject)) you could thus collect all those
generations without running actual analysis pipeline

>     for i in searchlights:
>         for j in permutations:
>             permutator =
> AttributePermutator('targets',limit={'partitions':1},count=1)
>             nf =
> NFoldPartitioner(attr=partition_attr,cvtype=leave_x_out,count=fold_num,selection_strategy=fold_select_strategy)
>             null_cv =
> CrossValidation(clf,ChainNode([nf,permutator],space=nf.get_space()),enable_ca='datasets',pass_attr=[('ca.datasets','fa')])
>             sl_null = sphere_searchlight(null_cv,radius=3,center_ids=[i])
>             null_dist.append(sl_null(ds))
>             null_dist=hstack(null_dist)

why do you want separate permutations per each center_id?
null_dist  here than mixes random results across all the searchlights if
I see it correctly.  IMHO it is better to estimate that distribution per
each searchlight center (what would happen anyways if you didn't do this
manual "for i in searchlights")


once again I feel that trying to keep target labeling in test split
might be complicating things for your more than being of any value (do
you get notably different significance results in comparison to simple
permutations in all runs?).
-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik