[pymvpa] Permutation test for each entry of confusion matrix?

[Meng Liang]

Hi Yaroslav,
It seems working but I'll have to wait until the final results come out and see how they look like. 
Thanks!Meng
> Date: Tue, 6 Dec 2011 14:11:05 -0500
> From: debian at onerussian.com
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: Re: [pymvpa] Permutation test for each entry of contribution matrix?
> 
> quick and ugly-ish answer... try it... but I also spotted some nuisance
> in our implementation which we might adjust (that the original error
> sample is "included" while we estimate CDF for its value within H0
> distribution... then we get a problem if classifier during permutation
> was spitting out the same values because there is no signal... then
> CDF(chance)=1.0 since all values are the same and we would have chosen
> low p for the right tail).
> 
> Meanwhile -- try this which I will push into a  unittest.  does it provide
> "feasible" results? ;)   I have used upcoming mvpa2 , but probably it should
> work with 0.6 as well.
> 
>         class ConfusionMatrixError(object):
>             """Custom error "function"
>             """
>             def __init__(self, labels=None):
>                 self.labels = labels
>             def __call__(self, predictions, targets):
>                 cm = ConfusionMatrix(labels=list(self.labels),
>                                      targets=targets, predictions=predictions)
>                 ## print cm.matrix
>                 # We have to add a degenerate leading dimension
>                 # so we could separate them into separate 'samples'
>                 return cm.matrix[None, :]
> 
>         ds = datasets['uni2small'] # choose your data
> 
>         clf = GNB()  
>         num_perm = 10
>         permutator = AttributePermutator('targets',
>                                          limit='chunks',
>                                          count=num_perm)
>         cv = CrossValidation(
>             clf, NFoldPartitioner(),
>             errorfx=ConfusionMatrixError(labels=ds.sa['targets'].unique),
>             postproc=mean_sample(),
>             null_dist=MCNullDist(permutator,
>                                  tail='left',
>                                  enable_ca=['dist_samples']),
>             enable_ca=['stats'])
>         cmatrix = cv(ds)
>         ## print cmatrix.samples
>         cvnp = cv.ca.null_prob.samples
> 
> 
> On Tue, 06 Dec 2011, Meng Liang wrote:
> 
> >    Hi Yaroslav,
> >    Thanks for the quick response! I'm using Pymvpa 0.6. 
> >    Best,
> >    Meng
> 
> >    > Date: Tue, 6 Dec 2011 11:10:15 -0500
> >    > From: debian at onerussian.com
> >    > To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> >    > Subject: Re: [pymvpa] Permutation test for each entry of contribution
> >    matrix?
> 
> >    > Good thinking Meng Liang -- that is what we also encourage people to do
> >    > to look at the actual confusion matrix where possible... to help you
> >    > best, please let us know which version of PyMVPA you are using?
> 
> >    > On Tue, 06 Dec 2011, Meng Liang wrote:
> 
> >    > > Dear all,
> >    > > I am trying to run a permutation test for a four-way classification
> >    and
> >    > > I'm interested in the classification accuracy of each class rather
> >    than
> >    > > the overall accuracy averaged across classes. I tried MCNullDist but
> >    it
> >    > > only gives me the null distribution of the overall accuracy. Is there
> >    a
> >    > > way to get the null distribution of the accuracy of each class? The
> >    best
> >    > > thing would be to get the null distribution of each entry of the
> >    confusion
> >    > > matrix so that I can calculate null distribution for any measure based
> >    on
> >    > > the confusion matrix. 
> >    > > Any suggestions?
> >    > > Best,
> >    > > Meng
> 
> >    > > _______________________________________________
> >    > > Pkg-ExpPsy-PyMVPA mailing list
> >    > > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> 
> >    http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> 
> >    > --
> >    > =------------------------------------------------------------------=
> >    > Keep in touch www.onerussian.com
> >    > Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
> 
> >    > _______________________________________________
> >    > Pkg-ExpPsy-PyMVPA mailing list
> >    > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> 
> >    http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> 
> -- 
> =------------------------------------------------------------------=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20111208/596376f1/attachment.html>