[pymvpa] Consistently bad accuracy?

Tue Nov 27 12:59:19 GMT 2018

I think that it could be a registration problem or a
normalization/detrending problem.
I used to have below-chance accuracy in across-subject analyses, but this
is not your case!

Bests,
R

On Mon, 26 Nov 2018 at 16:29, Raúl Hernández <raul at lafuentelab.org> wrote:

> Thank you for the link, I will look into it carefully.
> Sorry for not being clear, yes I have 4 acquisitions from each
> participant. I calculate an accuracy for each participant by calculating
> the mean across all cross validation folds. Then I take the this calculated
> mean from each participant and run a t test in which each participant
> contributes with a single number.
>
> Regards,
>
> Raul
>
> On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <jetzel at wustl.edu> wrote:
>
>> I agree with Patil that consistent below-chance accuracy is a sign that
>> something is not working properly.
>>
>> I collected some thoughts in
>> http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html
>> (and a few other posts tagged "below-chance").
>>
>> Also, be careful with terminology; I assume by "leave-one-run-out
>> cross-validation on 4 acquisitions" you mean each person completed four
>> scanning runs (each with the same fMRI acquisition parameters)? And a
>> t-test can be fine for a quick significance test, but it should be done
>> at the group level, testing if the subjects' accuracies are above chance
>> (i.e., each person contributing one number to the t-test), not on the
>> cross-validation folds within each person.
>>
>> Jo
>>
>>
>> On 11/26/2018 7:05 AM, Raúl Hernández wrote:
>> > I also consider that option, but when I try the very same thing with a
>> > different, region (not related to the task). I get accuracies of 50%.
>> So
>> > that makes me think that it is affected by the task, but I don't know
>> > what to think of it.
>> >
>> > Regards
>> >
>> > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <
>> kaustubh.patil at gmail.com
>> > <mailto:kaustubh.patil at gmail.com>> wrote:
>> >
>> >     I suspect that there might be something wrong in the code/how the
>> >     data is handled.
>> >
>> >     If you 30% accuracy then that would mean that you will get 70% if
>> >     you use a simple rule to predict the "other class" after your
>> >     classifier. This is a sign that something is not right in data
>> >     handling/evaluation.
>> >
>> >     Best
>> >
>> >     On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <
>> raul at lafuentelab.org
>> >     <mailto:raul at lafuentelab.org>> wrote:
>> >
>> >         No, it is balanced. It has the same number of observations for
>> >         each class.
>> >
>> >         On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil
>> >         <kaustubh.patil at gmail.com <mailto:kaustubh.patil at gmail.com>>
>> wrote:
>> >
>> >             Just for clarification.
>> >
>> >             Is that data imbalanced, i.e. many more observations from
>> >             one class?
>> >
>> >             Best,
>> >             Kaustubh
>> >
>> >             On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández
>> >             <raul at lafuentelab.org <mailto:raul at lafuentelab.org>> wrote:
>> >
>> >                 Dear PyMVPA community,
>> >
>> >                 I'm doing classification in ROI's, I'm performing a
>> >                 simple 2 way classification using LSVM, and a
>> >                 leave-one-run-out cross-validation on 4 acquisitions. On
>> >                 some ROI's, I get a good accuracy for the number of
>> >                 participants (60%), but in others I get consistently bad
>> >                 accuracy (30%). To test whether the performance is above
>> >                 chance, I use a one sample t test (I know that it is not
>> >                 the best test for this type of data, I just use it as
>> >                 quick overview). When I test the bad accuracies, those
>> >                 are also significant.
>> >
>> >                 What does it mean a consistently bad accuracy?
>> >
>> >                 Regards,
>> >
>> >                 Raul
>> >                 _______________________________________________
>> >                 Pkg-ExpPsy-PyMVPA mailing list
>> >                 Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> >                 <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>> >
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >             _______________________________________________
>> >             Pkg-ExpPsy-PyMVPA mailing list
>> >             Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> >             <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>> >
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >         _______________________________________________
>> >         Pkg-ExpPsy-PyMVPA mailing list
>> >         Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> >         <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>> >
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >     _______________________________________________
>> >     Pkg-ExpPsy-PyMVPA mailing list
>> >     Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> >     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>> >
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> >
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

-- 
Ing. Roberto Guidotti, PhD.
PostDoc Fellow
Institute for Advanced Biomedical Technologies - ITAB
Department of Neuroscience and Imaging
University of Chieti "G. D'Annunzio"
Via dei Vestini, 33
66013 Chieti, Italy
tel: +39 0871 3556919
e-mail: r.guidotti at unich.it; rguidotti at acm.org
linkedin: http://it.linkedin.com/in/robertogui/
twitter: @robbisg
github: https://github.com/robbisg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20181127/71f0aa69/attachment.html>