[pymvpa] Searchlight + cross validation & significance

Thu Apr 12 16:29:24 UTC 2012

Hi guys,

sorry for not following up with speedy replies ...

Searchlight is just a Measure so its chance distribution (per each
searchlight) and corresponding p-values could be computed the same
fashion as any other measure, e.g. like in
http://www.pymvpa.org/examples/permutation_test.html
done for CrossValidation

and e.g. look into this unittest
https://github.com/PyMVPA/PyMVPA/blob/HEAD/mvpa2/tests/test_searchlight.py#L236

hope this helps

cheers,

On Fri, 13 Apr 2012, Xiangzhen Kong wrote:

>    Hi Nynke,
>    Could you help me about the question that you have asked?
>    �How could I perform the permutation test for each
>    searchlight�and let the�searchlight�output also the p values for
>    testing against the null distribution?�
>    I want to do the same thing and have you solved this?
>    Best!
>    Xiangzhen

>    On Tue, Aug 2, 2011 at 8:00 PM,
>    <[1]pkg-exppsy-pymvpa-request at lists.alioth.debian.org> wrote:

>      Send Pkg-ExpPsy-PyMVPA mailing list submissions to
>      � � � �[2]pkg-exppsy-pymvpa at lists.alioth.debian.org

>      To subscribe or unsubscribe via the World Wide Web, visit
>      � � �
>      �[3]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

>      or, via email, send a message with subject or body 'help' to
>      � � � �[4]pkg-exppsy-pymvpa-request at lists.alioth.debian.org

>      You can reach the person managing the list at
>      � � � �[5]pkg-exppsy-pymvpa-owner at lists.alioth.debian.org

>      When replying, please edit your Subject line so it is more specific
>      than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."

>      Today's Topics:

>      � 1. Searchlight + cross validation & significance (permutation
>      � � �test) (Nynke van der Laan)
>      � 2. Re: Searchlight + cross validation & significance
>      � � �(permutation test) (Francisco Pereira)
>      � 3. Re: Searchlight + cross validation & significance
>      � � �(permutation test) (J.A. Etzel)
>      � 4. Searchlight + cross validation & significance (permutation
>      � � �test) (Nynke van der Laan)

>      ----------------------------------------------------------------------

>      Message: 1
>      Date: Mon, 1 Aug 2011 16:34:03 +0200
>      From: Nynke van der Laan <[6]nynkevanderlaan at gmail.com>
>      To: [7]pkg-exppsy-pymvpa at lists.alioth.debian.org
>      Subject: [pymvpa] Searchlight + cross validation & significance
>      � � � �(permutation � �test)
>      Message-ID:
>      � � �
>      �<CADHpHBOUFyc4nfboeYjBOngHry=[8]m8JV5Cmg0w_75CWQ6nCrZxQ at mail.gmail.com>
>      Content-Type: text/plain; charset=ISO-8859-1

>      Hello all,

>      I have a question about the following: I am performing a searchlight
>      analysis (v04) with a crossvalidation (leave one subject out, 20
>      samples per subject, 2 labels (10 samples label A, 10 samples label
>      B)).
>      I have adapted the example scripts to generate a map with the mean
>      classification error over de NFold crossvalidation. Now, in addition I
>      would like to be able to say something about the significance of the
>      accuracies that the analysis yielded.
>      In the module reference
>      ([9]http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight)
>      I noticed the null_prob and null_T are available variables. Null_prob
>      is enabled as default. However, the analysis now only results in the
>      searchlight errormap. How can I acces the null_prob information? Is
>      this averaged over the NFolds?

>      cv = CrossValidatedTransferError(
>      � � � � TransferError(LinearCSVMC()),
>      � � � � NFoldSplitter())
>      sl = Searchlight(cv, radius=6)
>      sl_map = sl(dataset_total)

>      An additional question I have is whether there is any easy way to
>      combine a searchlight analysis as described above with a permutation
>      test. In a previous discussion I read about the NullhypothesisTest
>      ([10]http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html)
>      but this does not seem to be present in the v.04 version anymore (?).
>      I predict that this analysis will be very costly in time as the
>      permutation is done in each searchlight, but I wouldn't know any
>      good/valid alternatives. Does anyone else have a suggestion?

>      Thanks a lot!

>      Best,
>      Nynke

>      ------------------------------

>      Message: 2
>      Date: Mon, 1 Aug 2011 10:47:35 -0400
>      From: Francisco Pereira <[11]francisco.pereira at gmail.com>
>      To: Development and support pf PyMVPA
>      � � � �<[12]pkg-exppsy-pymvpa at lists.alioth.debian.org>
>      Subject: Re: [pymvpa] Searchlight + cross validation & significance
>      � � � �(permutation test)
>      Message-ID:
>      � � �
>      �<[13]CAFdubX0HLpLfKFtoQKZCg7ViZjUq0EcAi7HHNpkHXXptwt8scQ at mail.gmail.com>
>      Content-Type: text/plain; charset=ISO-8859-1

>      On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
>      <[14]nynkevanderlaan at gmail.com> wrote:

>      > An additional question I have is whether there is any easy way to
>      > combine a searchlight analysis as described above with a permutation
>      test.

>      For a particular classifier (GNB) it's possible to do this fast
>      enough, if I recall correctly this is the PyMVPA implementation

>      [15]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html

>      cheers,
>      Francisco

>      ------------------------------

>      Message: 3
>      Date: Mon, 01 Aug 2011 09:58:19 -0500
>      From: "J.A. Etzel" <[16]jetzel at artsci.wustl.edu>
>      To: [17]pkg-exppsy-pymvpa at lists.alioth.debian.org
>      Subject: Re: [pymvpa] Searchlight + cross validation & significance
>      � � � �(permutation test)
>      Message-ID: <[18]4E36BF0B.9080607 at artsci.wustl.edu>
>      Content-Type: text/plain; charset=ISO-8859-1; format=flowed

>      My usual strategy is to split up the permutations and run the jobs on a
>      cluster; with luck your university will have access to one. Then you can
>      have dozens of computers working on the job at the same time, for
>      example one node doing permutations 1 to 5, another 6 to 10, etc.

>      Jo

>      On 8/1/2011 9:47 AM, Francisco Pereira wrote:
>      > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
>      > <[19]nynkevanderlaan at gmail.com> �wrote:

>      >> An additional question I have is whether there is any easy way to
>      >> combine a searchlight analysis as described above with a permutation
>      test.

>      > For a particular classifier (GNB) it's possible to do this fast
>      > enough, if I recall correctly this is the PyMVPA implementation

>      > [20]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html

>      > cheers,
>      > Francisco

>      > _______________________________________________
>      > Pkg-ExpPsy-PyMVPA mailing list
>      > [21]Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org

>      [22]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

>      ------------------------------

>      Message: 4
>      Date: Tue, 2 Aug 2011 11:55:21 +0200
>      From: Nynke van der Laan <[23]nynkevanderlaan at gmail.com>
>      To: [24]pkg-exppsy-pymvpa at lists.alioth.debian.org
>      Subject: [pymvpa] Searchlight + cross validation & significance
>      � � � �(permutation � �test)
>      Message-ID:
>      � � �
>      �<CADHpHBPh=[25]6jy0BSYObZi08aVy0i0+iXgWb4btOX7Yf84seNPvQ at mail.gmail.com>
>      Content-Type: text/plain; charset=ISO-8859-1

>      Thanks for your replies!
>      Francisco, great suggestion, I will absolutely consider that
>      classifier for my next study. For the present study/additional
>      analysis though, I would like to stick with the v4.0 version and the
>      linear SVM as this showed good results (and I preferably don't switch
>      to a different version in between analyses/revisions).
>      Jo, using multiple computers is indeed a good option,

>      However, in the documentation I read that if there is a strong
>      assumptiom of the data being normally/gaussian distributed
>      ([26]http://v04.pymvpa.org/examples/permutation_test.html), 30
>      permutations would also suffice. As my data indeed approaches a normal
>      distribution, I would like to try this first. But, unfortunately I did
>      not manage to implement this into the searchlight. In the example, it
>      is clearly described how this permutation test would be done in a
>      'normal' analysis, but how exactly to combine this with the
>      searchlight is unclear to me. I assume the null_dist = MCNullDist...
>      should be included in the TransferError in the script below, but the
>      searchlight now only returns the error scores for each voxel and no
>      other results. How could I perform the permutation test for each
>      searchlight and let the searchlight output also the p values for
>      testing against the null distribution?

>      cv = CrossValidatedTransferError(
>      � � � � TransferError(LinearCSVMC()),
>      � � � � NFoldSplitter())
>      sl = Searchlight(cv, radius=6)
>      sl_map = sl(dataset_total)

>      ------------------------------

>      _______________________________________________
>      Pkg-ExpPsy-PyMVPA mailing list
>      [27]Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>      [28]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

>      End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1
>      ************************************************
-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic