[pymvpa] Searchlight + cross validation & significance

Thu Apr 12 16:01:44 UTC 2012

Hi Nynke,

Could you help me about the question that you have asked?

“How could I perform the permutation test for each
searchlight and let the searchlight output also the p values for
testing against the null distribution?”

I want to do the same thing and have you solved this?

Best!

Xiangzhen

On Tue, Aug 2, 2011 at 8:00 PM, <
pkg-exppsy-pymvpa-request at lists.alioth.debian.org> wrote:

> Send Pkg-ExpPsy-PyMVPA mailing list submissions to
>        pkg-exppsy-pymvpa at lists.alioth.debian.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
> or, via email, send a message with subject or body 'help' to
>        pkg-exppsy-pymvpa-request at lists.alioth.debian.org
>
> You can reach the person managing the list at
>        pkg-exppsy-pymvpa-owner at lists.alioth.debian.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."
>
>
> Today's Topics:
>
>   1. Searchlight + cross validation & significance (permutation
>      test) (Nynke van der Laan)
>   2. Re: Searchlight + cross validation & significance
>      (permutation test) (Francisco Pereira)
>   3. Re: Searchlight + cross validation & significance
>      (permutation test) (J.A. Etzel)
>   4. Searchlight + cross validation & significance (permutation
>      test) (Nynke van der Laan)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 1 Aug 2011 16:34:03 +0200
> From: Nynke van der Laan <nynkevanderlaan at gmail.com>
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: [pymvpa] Searchlight + cross validation & significance
>        (permutation    test)
> Message-ID:
>        <CADHpHBOUFyc4nfboeYjBOngHry=m8JV5Cmg0w_75CWQ6nCrZxQ at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello all,
>
> I have a question about the following: I am performing a searchlight
> analysis (v04) with a crossvalidation (leave one subject out, 20
> samples per subject, 2 labels (10 samples label A, 10 samples label
> B)).
> I have adapted the example scripts to generate a map with the mean
> classification error over de NFold crossvalidation. Now, in addition I
> would like to be able to say something about the significance of the
> accuracies that the analysis yielded.
> In the module reference
> (
> http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight
> )
> I noticed the null_prob and null_T are available variables. Null_prob
> is enabled as default. However, the analysis now only results in the
> searchlight errormap. How can I acces the null_prob information? Is
> this averaged over the NFolds?
>
> cv = CrossValidatedTransferError(
>         TransferError(LinearCSVMC()),
>         NFoldSplitter())
> sl = Searchlight(cv, radius=6)
> sl_map = sl(dataset_total)
>
> An additional question I have is whether there is any easy way to
> combine a searchlight analysis as described above with a permutation
> test. In a previous discussion I read about the NullhypothesisTest
> (
> http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html
> )
> but this does not seem to be present in the v.04 version anymore (?).
> I predict that this analysis will be very costly in time as the
> permutation is done in each searchlight, but I wouldn't know any
> good/valid alternatives. Does anyone else have a suggestion?
>
> Thanks a lot!
>
> Best,
> Nynke
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 1 Aug 2011 10:47:35 -0400
> From: Francisco Pereira <francisco.pereira at gmail.com>
> To: Development and support pf PyMVPA
>        <pkg-exppsy-pymvpa at lists.alioth.debian.org>
> Subject: Re: [pymvpa] Searchlight + cross validation & significance
>        (permutation test)
> Message-ID:
>        <CAFdubX0HLpLfKFtoQKZCg7ViZjUq0EcAi7HHNpkHXXptwt8scQ at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
> <nynkevanderlaan at gmail.com> wrote:
> >
> > An additional question I have is whether there is any easy way to
> > combine a searchlight analysis as described above with a permutation
> test.
>
> For a particular classifier (GNB) it's possible to do this fast
> enough, if I recall correctly this is the PyMVPA implementation
>
> http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html
>
> cheers,
> Francisco
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 01 Aug 2011 09:58:19 -0500
> From: "J.A. Etzel" <jetzel at artsci.wustl.edu>
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: Re: [pymvpa] Searchlight + cross validation & significance
>        (permutation test)
> Message-ID: <4E36BF0B.9080607 at artsci.wustl.edu>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> My usual strategy is to split up the permutations and run the jobs on a
> cluster; with luck your university will have access to one. Then you can
> have dozens of computers working on the job at the same time, for
> example one node doing permutations 1 to 5, another 6 to 10, etc.
>
> Jo
>
>
>
> On 8/1/2011 9:47 AM, Francisco Pereira wrote:
> > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
> > <nynkevanderlaan at gmail.com>  wrote:
> >>
> >> An additional question I have is whether there is any easy way to
> >> combine a searchlight analysis as described above with a permutation
> test.
> >
> > For a particular classifier (GNB) it's possible to do this fast
> > enough, if I recall correctly this is the PyMVPA implementation
> >
> > http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html
> >
> > cheers,
> > Francisco
> >
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> >
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 2 Aug 2011 11:55:21 +0200
> From: Nynke van der Laan <nynkevanderlaan at gmail.com>
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: [pymvpa] Searchlight + cross validation & significance
>        (permutation    test)
> Message-ID:
>        <CADHpHBPh=6jy0BSYObZi08aVy0i0+iXgWb4btOX7Yf84seNPvQ at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Thanks for your replies!
> Francisco, great suggestion, I will absolutely consider that
> classifier for my next study. For the present study/additional
> analysis though, I would like to stick with the v4.0 version and the
> linear SVM as this showed good results (and I preferably don't switch
> to a different version in between analyses/revisions).
> Jo, using multiple computers is indeed a good option,
>
> However, in the documentation I read that if there is a strong
> assumptiom of the data being normally/gaussian distributed
> (http://v04.pymvpa.org/examples/permutation_test.html), 30
> permutations would also suffice. As my data indeed approaches a normal
> distribution, I would like to try this first. But, unfortunately I did
> not manage to implement this into the searchlight. In the example, it
> is clearly described how this permutation test would be done in a
> 'normal' analysis, but how exactly to combine this with the
> searchlight is unclear to me. I assume the null_dist = MCNullDist...
> should be included in the TransferError in the script below, but the
> searchlight now only returns the error scores for each voxel and no
> other results. How could I perform the permutation test for each
> searchlight and let the searchlight output also the p values for
> testing against the null distribution?
>
> cv = CrossValidatedTransferError(
>         TransferError(LinearCSVMC()),
>         NFoldSplitter())
> sl = Searchlight(cv, radius=6)
> sl_map = sl(dataset_total)
>
>
>
> ------------------------------
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
> End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1
> ************************************************
>

-- 
-----------------------------------------------------------------------------------------
Kong Xiangzhen
State Key Laboratory of Cognitive Neuroscience and Learning,
Beijing Normal University,
Beijing, China, 100875.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120413/380dc534/attachment.html>