[pymvpa] confusion matrix in searchlight

[Wouter De Baene]

Thank you very much!
It works like a charm.

Best regards,
Wouter De Baene


On 20 Mar 2014, at 13:18, Michael Hanke <michael.hanke at gmail.com<mailto:michael.hanke at gmail.com>> wrote:

Hey,

On Wed, Mar 19, 2014 at 10:44 PM, Wouter De Baene <Wouter.DeBaene at ugent.be<mailto:Wouter.DeBaene at ugent.be>> wrote:
Hi all,

I’m running my first pymvpa-analysis after attending a very nice pymvpa workshop in Delmenhorst.

Glad you liked it!

Although there are a few related questions in the archive, I don’t seem to get this working.
I’m setting up a searchlight analysis with Nfold cross-validation. I'm trying to look at the confusion matrix for each cross-validation for each searchlight. This information seems to be lost when using searchlights.
As I understood there were some plans to aggregate the output per searchlight into the output of the total results. Is there any news about this? OR could I get to this information in another way for now?

The easiest way to aggregate this kind of non-scalar information within a searchlight is to
get the confusion matrix to come out of the CrossValidation as a vector (flattened matrix).
For the first part of a possible solution we have "Confusion", a node that turns raw targets
predictions into a confusion table. Here is a sketch: no error computing in the classifier, return
raw predictions, use Confusion as a post-processing node to yield matrices.

     CrossValidation(some_classifier,
                      some_partitioner,
                      errorfx=None,
                      postproc=Confusion())

Now the last bit is to flatten the confusion matrix into a vector. You could write a simple Node
implementation that does that. Here is some pseudo code:

class MyFlatMatrix(Node):
  def _call(ds):
    return Dataset(ds.samples.ravel())

and now use this class as a post-processing node for Confusion itself.

    CrossValidation(some_classifier,
                      some_partitioner,
                      errorfx=None,
                      postproc=Confusion(postproc=MyFlatMatrix()))

stick this into a searchlight and it should harvest what you want. This is all written as a sketch and wasn't tested...

Instead of a custom Node class you could also use a ChainMapper comprised of a FlattenMapper and a TransposeMapper
to achieve the same thing....

Good luck!

Michael


--
Michael Hanke
http://mih.voxindeserto.de<http://mih.voxindeserto.de/>

_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org<mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20140321/d00041c0/attachment.html>