[pymvpa] AttributePermutator: Permute within chunks (& subjects) but only training labels

Wed May 24 14:27:37 UTC 2017

Hey Richard,

Thanks for your time. I believe what I want is really basic and I am
somewhat surprised that it isn't covered in the tutorials (correct me if
I'm wrong).

I'm trying to impose two constraints on the way the labels are shuffled:

1. Leave label assignments intact in the test set.
2. Only permute labels within each subject's individual runs.

I know how to define AttributePermutator to implement each constraint
indvidually, for example

ad 1.:
permutator = AttributePermutator(attr='targets', limit={'permutations':
1}, count=n_perm)

This restricts the permutations to the training set only.

ad 2.:
permutator = AttributePermutator(attr='targets', limit=['subject',
'chunks'], count=n_perm)

Labels are now shuffled only within each pair of subject/chunks values.

It seems that the limit argument has two different functions depending
on whether it is a list or a dictionary. If it's a dictionary (case 1),
then each key/value pair determines which labels should be included in
the permutation in the first place. So here it has a selection function.
If it's a list (case 2), it defines subsets of labels within which to
perform permutation. In this case it has a "chunking" function.

I do not see how I can do both - include selection AND chunking.

Since in my code that I sent before, limit is a dictionary, it will only
have the selection function - it defines which labels to include in the
permutation, namely in my example ALL subjects and ALL runs. Silly me -
this basically just tells the permutator to include all samples in the
dataset :)

So to answer your question, no when I restrict the selection to training
set and subjects, which I assume you would imagine to look like this:

limit={'partitions': 1, 'subject': range(n_subj)}

... I get the same result as if I had used limit={'partitions': 1}.

Best,
Michael

On 24/05/17 12:35, Richard Dinga wrote:
> Hi Michael,
> I am sorry, I misunderstood your problem. Your snipet seems good to me.
> What do you mean by it doesn't work? Does it work if you limit only on
> training set and subjects for example?
> 
> On Mon, May 22, 2017 at 6:28 PM, Michael Bannert
> <mbannert at tuebingen.mpg.de <mailto:mbannert at tuebingen.mpg.de>> wrote:
> 
>     Hi Richard,
> 
>     Thanks for your email. Unfortunately, this is not exactly the answer
>     that I'm looking for:
> 
>     In the section you refer to, the only limiting condition is that the
>     test set labels should remain unpermuted. This solves only 50 % of my
>     problem. It has already been explained very well in the worked examples
>     - so no problem there.
> 
>     However, I also want to permute only within runs (and subjects) and do
>     not see how this can be achieved with AttributePermutator (or any other
>     method).
> 
>     Best,
>     Michael
> 
> 
>     On 22/05/17 17:32, Richard Dinga wrote:
>     > Hi,
>     > Does this answer your question?
>     > http://www.pymvpa.org/tutorial_significance.html#avoiding-the-trap-or-advanced-magic-101
>     <http://www.pymvpa.org/tutorial_significance.html#avoiding-the-trap-or-advanced-magic-101>
>     >
>     >
>     > On Fri, May 19, 2017 at 8:18 PM, Michael Bannert
>     > <mbannert at tuebingen.mpg.de <mailto:mbannert at tuebingen.mpg.de>
>     <mailto:mbannert at tuebingen.mpg.de
>     <mailto:mbannert at tuebingen.mpg.de>>> wrote:
>     >
>     >     Dear all,
>     >
>     >     I would like to use permutation testing for spatially aligned
>     >     across-subject decoding. I have one vector of beta estimates
>     per run
>     >     (aka chunks) and per subject. Hence I figured it would be wise to
>     >     permute within subjects and runs.
>     >
>     >     I can achieve this (I think) if I use AttributePermutator in
>     this way:
>     >
>     >     permutator = AttributePermutator(attr='targets', limit=['subject',
>     >     'chunks'], count=n_perm)
>     >
>     >     According to the debugging information provided when setting
>     the 'APERM'
>     >     option, the permutations that are produced look reasonable.
>     >
>     >     However, I would also like to permute only the training data.
>     How can I
>     >     accomplish this?
>     >
>     >     I tried something like this:
>     >
>     >     permutator = AttributePermutator(attr='targets', limit={
>     >              'partitions': 1, 'subject': range(n_subj), 'chunks':
>     >     ['run%02.f' % j for j in range(1, n_runs + 1)]},
>     >               count=1)
>     >
>     >     ... but it doesn't work.
>     >
>     >     I guess I am not very clear on what the documentation of
>     >     AttributePermutator has to say about the limit argument.
>     >
>     >     Could anyone help?
>     >
>     >     Thanks & best,
>     >     Michael
>     >
>     >
>     >
>     >     _______________________________________________
>     >     Pkg-ExpPsy-PyMVPA mailing list
>     >     Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>     <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>     >     <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>     <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>>
>     >   
>      http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>     <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>     >   
>      <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>     <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > Pkg-ExpPsy-PyMVPA mailing list
>     > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>     <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>     >
>     http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>     <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>     >
> 
>     _______________________________________________
>     Pkg-ExpPsy-PyMVPA mailing list
>     Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>     <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>     http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>     <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
> 
> 
> 
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 