[pymvpa] Alternatives to Cross-validation

[Yaroslav Halchenko]

Thanks everyone and sorry I haven't managed to understand the goal
myself.

Here is the ultimate example:

    clf.train(ds_train)
    est = clf.predict(ds_test)
    # now you can get the error
    print mean_mismatch_error(est, ds_test.targets)
    # or if you like a full ConfusionMatrix beast
    cm = ConfusionMatrix(targets=ds_test.targets,
                         predictions=est)
    print cm

This, and more of trickeries you can discover if you go through the
tutorial, e.g. in this particular case it was pretty much the beginning:

http://www.pymvpa.org/tutorial_start.html#dealing-with-a-classifier

;-)

Cheers,

On Wed, 24 Oct 2012, Jacob Itzhacki wrote:

>    Thank you for your prompt response.
>    "not sure why cross-validation doesn't fit your needs here. �Could you
>    elaborate a bit more on what you are trying to achieve."
>    I'll put it bluntly. I am trying to script a way for the algorithm to
>    create a comparison pattern based on one set of training-stimuli, then
>    disregard this set of stimuli and compare the pattern formed to a
>    different set of test-stimuli, which will also be of different kind, but
>    should produce similar activation patterns in the ROIs scrutinized.
>    "Do you mean that you want to train with all examples from one dataset,
>    then test with all examples from a different dataset?"
>    This is exactly it.
>    "You could always combine the two datasets as the two cross-validation
>    folds and get per-fold results, no?"
>    This is an option we have considered (and will take upon if we find no
>    better solution), to have all the stimuli fall under the same dataset to
>    make cross-validation viable. However we believe that the predictions
>    given on the confusion tables would not be as ideal as if the comparison
>    could be performed as described above.
>    Thanks again for any help you can offer us.
>    J.

>    On Wed, Oct 24, 2012 at 3:16 PM, Francisco Pereira
>    <[1]francisco.pereira at gmail.com> wrote:

>      You could always combine the two datasets as the two cross-validation
>      folds and get per-fold results, no?
>      Francisco
>      _______________________________________________
>      Pkg-ExpPsy-PyMVPA mailing list
>      [2]Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>      [3]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

> References

>    Visible links
>    1. mailto:francisco.pereira at gmail.com
>    2. mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>    3. http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik