[pymvpa] AttributePermutator: Permute within chunks (& subjects) but only training labels
Michael Bannert
mbannert at tuebingen.mpg.de
Thu May 25 11:42:07 UTC 2017
Hi,
I have decided to go with a quick and dirty fix. I added a new
permutation strategy to permutation.py called 'simple_test_intact'. It
checks if the sample ids belong to the test set and just skips the
permutation if they do. So the partitions attribute has to be defined,
i.e. use only in combination with a partitioner. Otherwise it is
identical to 'simple'.
I think this will have to do until I figure out how it is supposed to be
done in PyMVPA.
Cheers,
Michael
On 24/05/17 16:27, Michael Bannert wrote:
> Hey Richard,
>
> Thanks for your time. I believe what I want is really basic and I am
> somewhat surprised that it isn't covered in the tutorials (correct me if
> I'm wrong).
>
> I'm trying to impose two constraints on the way the labels are shuffled:
>
> 1. Leave label assignments intact in the test set.
> 2. Only permute labels within each subject's individual runs.
>
> I know how to define AttributePermutator to implement each constraint
> indvidually, for example
>
> ad 1.:
> permutator = AttributePermutator(attr='targets', limit={'permutations':
> 1}, count=n_perm)
>
> This restricts the permutations to the training set only.
>
> ad 2.:
> permutator = AttributePermutator(attr='targets', limit=['subject',
> 'chunks'], count=n_perm)
>
> Labels are now shuffled only within each pair of subject/chunks values.
>
> It seems that the limit argument has two different functions depending
> on whether it is a list or a dictionary. If it's a dictionary (case 1),
> then each key/value pair determines which labels should be included in
> the permutation in the first place. So here it has a selection function.
> If it's a list (case 2), it defines subsets of labels within which to
> perform permutation. In this case it has a "chunking" function.
>
> I do not see how I can do both - include selection AND chunking.
>
> Since in my code that I sent before, limit is a dictionary, it will only
> have the selection function - it defines which labels to include in the
> permutation, namely in my example ALL subjects and ALL runs. Silly me -
> this basically just tells the permutator to include all samples in the
> dataset :)
>
> So to answer your question, no when I restrict the selection to training
> set and subjects, which I assume you would imagine to look like this:
>
> limit={'partitions': 1, 'subject': range(n_subj)}
>
> ... I get the same result as if I had used limit={'partitions': 1}.
>
> Best,
> Michael
>
>
>
> On 24/05/17 12:35, Richard Dinga wrote:
>> Hi Michael,
>> I am sorry, I misunderstood your problem. Your snipet seems good to me.
>> What do you mean by it doesn't work? Does it work if you limit only on
>> training set and subjects for example?
>>
>> On Mon, May 22, 2017 at 6:28 PM, Michael Bannert
>> <mbannert at tuebingen.mpg.de <mailto:mbannert at tuebingen.mpg.de>> wrote:
>>
>> Hi Richard,
>>
>> Thanks for your email. Unfortunately, this is not exactly the answer
>> that I'm looking for:
>>
>> In the section you refer to, the only limiting condition is that the
>> test set labels should remain unpermuted. This solves only 50 % of my
>> problem. It has already been explained very well in the worked examples
>> - so no problem there.
>>
>> However, I also want to permute only within runs (and subjects) and do
>> not see how this can be achieved with AttributePermutator (or any other
>> method).
>>
>> Best,
>> Michael
>>
>>
>> On 22/05/17 17:32, Richard Dinga wrote:
>> > Hi,
>> > Does this answer your question?
>> > http://www.pymvpa.org/tutorial_significance.html#avoiding-the-trap-or-advanced-magic-101
>> <http://www.pymvpa.org/tutorial_significance.html#avoiding-the-trap-or-advanced-magic-101>
>> >
>> >
>> > On Fri, May 19, 2017 at 8:18 PM, Michael Bannert
>> > <mbannert at tuebingen.mpg.de <mailto:mbannert at tuebingen.mpg.de>
>> <mailto:mbannert at tuebingen.mpg.de
>> <mailto:mbannert at tuebingen.mpg.de>>> wrote:
>> >
>> > Dear all,
>> >
>> > I would like to use permutation testing for spatially aligned
>> > across-subject decoding. I have one vector of beta estimates
>> per run
>> > (aka chunks) and per subject. Hence I figured it would be wise to
>> > permute within subjects and runs.
>> >
>> > I can achieve this (I think) if I use AttributePermutator in
>> this way:
>> >
>> > permutator = AttributePermutator(attr='targets', limit=['subject',
>> > 'chunks'], count=n_perm)
>> >
>> > According to the debugging information provided when setting
>> the 'APERM'
>> > option, the permutations that are produced look reasonable.
>> >
>> > However, I would also like to permute only the training data.
>> How can I
>> > accomplish this?
>> >
>> > I tried something like this:
>> >
>> > permutator = AttributePermutator(attr='targets', limit={
>> > 'partitions': 1, 'subject': range(n_subj), 'chunks':
>> > ['run%02.f' % j for j in range(1, n_runs + 1)]},
>> > count=1)
>> >
>> > ... but it doesn't work.
>> >
>> > I guess I am not very clear on what the documentation of
>> > AttributePermutator has to say about the limit argument.
>> >
>> > Could anyone help?
>> >
>> > Thanks & best,
>> > Michael
>> >
>> >
>> >
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>> > <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>>
>> >
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>> >
>> <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>> >
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>> >
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> <http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>>
>>
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: permutation.py
Type: text/x-python-script
Size: 13935 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170525/87e8ab73/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_simple_test_intact.py
Type: text/x-python-script
Size: 903 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170525/87e8ab73/attachment-0003.bin>
More information about the Pkg-ExpPsy-PyMVPA
mailing list