[pymvpa] Labels from permutation testing

[Bill Broderick]

> why do you want separate permutations per each center_id?
> null_dist  here than mixes random results across all the searchlights if
> I see it correctly.  IMHO it is better to estimate that distribution per
> each searchlight center (what would happen anyways if you didn't do this
> manual "for i in searchlights")

I actually don't want separate permutations for each center_id. That
was an oversight. But I do want to run the searchlights separately
from each other (so I can parallelize them), which was what I was
trying to do. I think I'll preseed the RNG the same across all of the
searchlights, and that should do what I want.

> once again I feel that trying to keep target labeling in test split
> might be complicating things for your more than being of any value (do
> you get notably different significance results in comparison to simple
> permutations in all runs?).

I was a bit uncomfortable with the idea of doing the simple
permutation, but re-reading the Stelzer et al (2013) paper where they
introduce the cluster thresholding method (which we're planning on
using), it looks like that's what they do as well. I haven't had a
chance to compare them yet, but I felt like there might be some
theoretical issues. Since there doesn't seem to be any consensus (and
there's some argument in favor of the simple permutations), I'll start
with that way.

Thanks,
Bill

On Fri, Aug 14, 2015 at 11:18 PM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
>
>
> On Mon, 20 Jul 2015, Bill Broderick wrote:
>
> > Hi all,
>
> > I feel like this should be relatively simple, but I can't figure out how to
> > do it. Is it possible to get at the labels generated by
> > AttributePermutator? I would like to see what the individual permutations
> > look like, to make sure it's doing what I think it is, but other than
> > saving the whole dataset generated by CrossValidation, I can't see a way to
> > do it.
>
> > I'm trying to build a null distribution like the following, so I can save
> > each permutation, each searchlight separately (with how long the
> > permutation testing has been taking, I want to make sure there's constant
> > output in case something crashes and so I can monitor its progress, so I'm
> > not using MCNullDist).
>
> if you were just want to check "in general" on what permutations permutator
> generates, and didn't have limit={'partitions':1}, count=1  you could just
>
> [x.targets for x in permutator.generate(ds)]
>
> then if you preseeded RNG the same way before testing permutator
> (e.g. mvpa2.seed(index_of_subject)) you could thus collect all those
> generations without running actual analysis pipeline
>
> >     for i in searchlights:
> >         for j in permutations:
> >             permutator =
> > AttributePermutator('targets',limit={'partitions':1},count=1)
> >             nf =
> > NFoldPartitioner(attr=partition_attr,cvtype=leave_x_out,count=fold_num,selection_strategy=fold_select_strategy)
> >             null_cv =
> > CrossValidation(clf,ChainNode([nf,permutator],space=nf.get_space()),enable_ca='datasets',pass_attr=[('ca.datasets','fa')])
> >             sl_null = sphere_searchlight(null_cv,radius=3,center_ids=[i])
> >             null_dist.append(sl_null(ds))
> >             null_dist=hstack(null_dist)
>
> why do you want separate permutations per each center_id?
> null_dist  here than mixes random results across all the searchlights if
> I see it correctly.  IMHO it is better to estimate that distribution per
> each searchlight center (what would happen anyways if you didn't do this
> manual "for i in searchlights")
>
>
> once again I feel that trying to keep target labeling in test split
> might be complicating things for your more than being of any value (do
> you get notably different significance results in comparison to simple
> permutations in all runs?).
> --
> Yaroslav O. Halchenko, Ph.D.
> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
> Research Scientist,            Psychological and Brain Sciences Dept.
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa