[pymvpa] Searchlights and Permutation testing

[Bill Broderick]

A fix for the GNB searchlight so it can use MCNullDist would be great!
I'd prefer to use that, since it's conceptually cleaner. I'm going to
try implementing the two versions I have now and see how long they
take, but if you get a fix going, please let me know!

I'm also not entirely sure I understand AttributePermutator's
strategy. 'simple' is your basic permutation, shuffling targets (in my
case) randomly. 'uattrs' permits target labels within each unique
value? So all trials that have target 1 still end up with target 1 and
all with target 2 still have target 2? And 'chunks' moves around all
values between chunks. So, say chunk 1 has 100 of target 1 and 200 of
target 2 and chunks 2 has 50 of target 1 and 250 of target 2, then the
permutator would just swap them so chunk 1 has 50 target 1s and 250
target 2s and similarly for chunk 2? What am I missing?

I tried calling the uattrs permutator on my dataset to play around
with (using `permutator=AttributePermutator('targets',limit={'partitions':1},count=1,strategy='uattrs')`
and chaining it with an NFoldPartitioner), and the partitions
alternated: no targets different, almost all different, none
different, almost all different... compared with strategy='simple',
when about 1/3 of the targets were different on each partition.

I thought for permutation testing you want to swap the labels of the
targets, because your null hypothesis is that the patterns contain no
information on the labels. How do either of the above permutation
strategies help with that?

Thanks,
Bill

On Wed, Jun 17, 2015 at 2:30 PM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
>
> On Wed, 17 Jun 2015, Yaroslav Halchenko wrote:
>
>> > > I think this should be res.append(sl_gnb(i))?
>
>> > Yes, my bad. That was a typo. It's res.append(sl_gnb(i))
>
>> sorry... coming too late to the party -- busy at OHBM... quick question -- why
>> don't you just use null_dist ?
>
> doh -- just now read back your original post -- that is indeed a
> shortcoming we should fix :-/  If you don't get it going, I will come up
> with a full recipe later today
>
> my comment on strategy still holds ;)
>
> --
> Yaroslav O. Halchenko, Ph.D.
> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
> Research Scientist,            Psychological and Brain Sciences Dept.
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa