[pymvpa] Permutation test for each entry of contribution matrix?

[Yaroslav Halchenko]

quick and ugly-ish answer... try it... but I also spotted some nuisance
in our implementation which we might adjust (that the original error
sample is "included" while we estimate CDF for its value within H0
distribution... then we get a problem if classifier during permutation
was spitting out the same values because there is no signal... then
CDF(chance)=1.0 since all values are the same and we would have chosen
low p for the right tail).

Meanwhile -- try this which I will push into a  unittest.  does it provide
"feasible" results? ;)   I have used upcoming mvpa2 , but probably it should
work with 0.6 as well.

        class ConfusionMatrixError(object):
            """Custom error "function"
            """
            def __init__(self, labels=None):
                self.labels = labels
            def __call__(self, predictions, targets):
                cm = ConfusionMatrix(labels=list(self.labels),
                                     targets=targets, predictions=predictions)
                ## print cm.matrix
                # We have to add a degenerate leading dimension
                # so we could separate them into separate 'samples'
                return cm.matrix[None, :]

        ds = datasets['uni2small'] # choose your data

        clf = GNB()  
        num_perm = 10
        permutator = AttributePermutator('targets',
                                         limit='chunks',
                                         count=num_perm)
        cv = CrossValidation(
            clf, NFoldPartitioner(),
            errorfx=ConfusionMatrixError(labels=ds.sa['targets'].unique),
            postproc=mean_sample(),
            null_dist=MCNullDist(permutator,
                                 tail='left',
                                 enable_ca=['dist_samples']),
            enable_ca=['stats'])
        cmatrix = cv(ds)
        ## print cmatrix.samples
        cvnp = cv.ca.null_prob.samples


On Tue, 06 Dec 2011, Meng Liang wrote:

>    Hi Yaroslav,
>    Thanks for the quick response! I'm using Pymvpa 0.6. 
>    Best,
>    Meng

>    > Date: Tue, 6 Dec 2011 11:10:15 -0500
>    > From: debian at onerussian.com
>    > To: pkg-exppsy-pymvpa at lists.alioth.debian.org
>    > Subject: Re: [pymvpa] Permutation test for each entry of contribution
>    matrix?

>    > Good thinking Meng Liang -- that is what we also encourage people to do
>    > to look at the actual confusion matrix where possible... to help you
>    > best, please let us know which version of PyMVPA you are using?

>    > On Tue, 06 Dec 2011, Meng Liang wrote:

>    > > Dear all,
>    > > I am trying to run a permutation test for a four-way classification
>    and
>    > > I'm interested in the classification accuracy of each class rather
>    than
>    > > the overall accuracy averaged across classes. I tried MCNullDist but
>    it
>    > > only gives me the null distribution of the overall accuracy. Is there
>    a
>    > > way to get the null distribution of the accuracy of each class? The
>    best
>    > > thing would be to get the null distribution of each entry of the
>    confusion
>    > > matrix so that I can calculate null distribution for any measure based
>    on
>    > > the confusion matrix. 
>    > > Any suggestions?
>    > > Best,
>    > > Meng

>    > > _______________________________________________
>    > > Pkg-ExpPsy-PyMVPA mailing list
>    > > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org

>    http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa


>    > --
>    > =------------------------------------------------------------------=
>    > Keep in touch www.onerussian.com
>    > Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic

>    > _______________________________________________
>    > Pkg-ExpPsy-PyMVPA mailing list
>    > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org

>    http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa


-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic