[Pkg-exppsy-pymvpa] Getting searchlight to output something else than errors

[Michael Hanke]

Hi Russel,


[ also sending this to the mailing list with your permission, therefore
  full quote here ]

On Thu, May 01, 2008 at 01:58:45PM +0100, Russell Thompson wrote:
> I've been having a play around with pymvpa (great toolbox by the way  
> Michael, its going to be very useful!), and I was wondering how I could  
> change the values output by a searchlight object. I read in the  
> documentation that it's possible to define custom error functions, but  
> I'm not sure how to do that - or whether that would be the right way to  
> go about doing what I was thinking of. In particular, I'd like to run  
> chi-squared tests on the confusion matrices generated by each  
> classification and return the results of those tests. At the moment I  
> can do this with a call to a CrossValidatedTransferError object by  
> enabling the 'confusion' state and getting it to return the whole  
> confusion matrix, but my python programming skills don't stretch to  
> doing this with a searchlight. As far as I can see, the only thing  
> returned by a searchlight procedure is a list containing the overall  
> error scores for each voxel, and there isn't anything equivalent to the  
> 'confusion' available with the CVTE object. Would it be possible to get  
> a searchlight to generate a confusion matrix on each iteration, then  
> return 2 lists, one of error scores, and one of chi2 / p values? I guess  
> it would be possible to run a searchlight "manually" by coding it in my  
> top level script, but that seems a bit inefficient given that an object  
> already exists to do this.
Right. I'd do this by creating a little function that does the test for
each dataset/sphere, like this:

        transerror = TransferError(WhateverClassifier())
        cv = CrossValidatedTransferError(
                transerror,
                NFoldSplitter(cvtype=1),
                enable_states=['confusion'])

        def getconfusion(data):
            cv(data)
            return cv.confusion.matrix

        # contruct radius 1 searchlight
        sl = Searchlight(getconfusion, radius=5)

        results = sl(data)

The 'getconfusion' function runs the crossvalidation for a particular
sphere dataset and return the corresponding confusion matrix. The final
call will return a list of all confusion matrices as 'results'.
Of course you can do the chisquare test simulanously by adding:

    from mvpa.misc.stats import chisquare

and replace the return value in 'getconfusion' with:

    return chisquare(cv.confusion.matrix)[0]

> Also, would it be possible to integrate the permutation tests offered by  
> a NullHypothesisTest object with a searchlight (i.e. a permutation test  
> at each searchlight iteration)? Not sure how many tonnes of carbon that  
> would burn in terms of processor time, but I'd be interested to compare  
> the results of this method with the results of a chi2 at each voxel.
That should work by wrapping the TransferError object with
NullHypothesisTest:

    transerror = NullHypothesisTest(TransferError(WhateverClassifer))

if you plug this into CrossValidatedTransferError it should return the
probabilities instead of errors. However the NullHypothesisTest is not
nicely integrated with the rest, i.e. you cannot enable any of the
confusion states, as they are not passed through from the underlying
TransferError instances. We should improve this.


HTH,

Michael



-- 
GPG key:  1024D/3144BE0F Michael Hanke
http://apsy.gse.uni-magdeburg.de/hanke
ICQ: 48230050