[pymvpa] Training and testing on only 1 run (no cross validation)

[Lynda Lin]

Thanks so much for responding to my question - I used the CustomPartitioner
option and it worked! (I'm so sorry it took so long to reply, wanted to
wait til I ran the searchlight analyses, which I just finished, to have a
more coherent answer). I had 2 follow up questions regarding interpretation
of results...

1) if my searchlight script doesn't include the "sl_map.samples - 1" line,
does that mean that all my values when I visualize them on FSL for example
will be error rates? (instead of accuracy values?) Here's my script:

*results = sl(dataset)*
*niftiresults = map2nifti(results, imghdr=dataset.a.imghdr)*

Do I need to add the following lines to my script to get accuracy rates?
*sphere_errors = results.samples[0]*
*map2nifti(1-sphere_errors)*

2) I ran a one-sample t-test on SPM using the .nii images I got for each
participant from the searchlight analyses (after converting them to
standard space) but I'm not sure how to threshold the image to correct for
multiple comparisons? Or is this something that the program does? For
example, when I open the group contrast (or beta) image after running the
one-sample t-test on FSL and specify min=0.5, max=1, I get several regions
that show up but when I do min=0.51 there are a lot less regions (regions
that I hypothesized would be able to discriminate between trials in my
task) but 0.51 doesn't seem that much different from 0.5 - how can one tell
what the threshold to be different from chance should be?
Similarly when I open the contrast (or beta) image on xjview, if I change
the intensity value from 0.5 to 0.51 I get similar results.

Thank you again for any clarification you might provide!
Lynda

On Fri, Oct 6, 2017 at 5:03 PM, Yaroslav Halchenko <debian at onerussian.com>
wrote:

>
> On Thu, 05 Oct 2017, Lynda Lin wrote:
> >    I've tried it 3 different ways and I'm getting different results for
> each
> >    way so just wanted to know if any of these ways is valid:
>
> sorry... just a quick reply, let's see if it would satisfy ;)
>
> >    1) Using the manual split example from the tutorial and calling the
> >    "training_stats" conditional attribute in the classifier
> >    In the tutorial we can get the individual accuracies for each run
> through
> >    cv_results.samples but I'm interested in the TPR (True Positive Rate)
> for
> >    Ingroup and Outgroup separately so I'm looking to print the confusion
> >    matrix to calculate these numbers
>
> you could access .stats dictionary with all those values, e.g.
> cve.ca.stats.stats["TPR"]
>
> >    ds_split1 = ds[ds.sa.chunks == 1.]
> >    ds_split2 = ds[ds.sa.chunks == 2.]
> >    clf = LinearCSVMC(enable_ca=['training_stats'])
> >    clf.set_postproc(BinaryFxNode(mean_mismatch_error,'targets'))
> >    clf.train(ds_split1)
> >    err = clf(ds_split2)
> >    clf.ca.training_stats.as_string(description=True)
>
> this is training_stats, so how well you fit (you can get it right after
> .train), not as you predict
>
> >    2) Using the HalfPartitioner function's "count" argument
>
> >    Â clf = LinearCSVMC(enable_ca=['training_stats']) #The training_stats
> >    confusion matrix from this method doesn't match the one above
> >    Â hpart = HalfPartitioner(count=1, attr='chunks')
> >    Â cvte = CrossValidation(clf,hpart,errorfx=lambda p,t:
> >    np.mean(p==t),enable_ca=['stats'])
> >    Â cv_results = cvte(ds)
> >    Â cvte.ca.stats.as_string(description=True)
>
> You could also just use CustomPartitioner([([1], [2])])
> which should generate a single split where first partition would have
> only the ones with value 1, and the second with the value 2.
> ca.stats should be appropriate
>
> is your dataset balanced in terms of # of samples per label?
> output of ds.summary()
>
> --
> Yaroslav O. Halchenko
> Center for Open Neuroscience     http://centerforopenneuroscience.org
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20171111/64650e18/attachment-0001.html>