[pymvpa] Consistently bad accuracy?

[Raúl Hernández]

Thank you for the link, I will look into it carefully.
Sorry for not being clear, yes I have 4 acquisitions from each participant.
I calculate an accuracy for each participant by calculating the mean across
all cross validation folds. Then I take the this calculated mean from each
participant and run a t test in which each participant contributes with a
single number.

Regards,

Raul

On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <jetzel at wustl.edu> wrote:

> I agree with Patil that consistent below-chance accuracy is a sign that
> something is not working properly.
>
> I collected some thoughts in
> http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html
> (and a few other posts tagged "below-chance").
>
> Also, be careful with terminology; I assume by "leave-one-run-out
> cross-validation on 4 acquisitions" you mean each person completed four
> scanning runs (each with the same fMRI acquisition parameters)? And a
> t-test can be fine for a quick significance test, but it should be done
> at the group level, testing if the subjects' accuracies are above chance
> (i.e., each person contributing one number to the t-test), not on the
> cross-validation folds within each person.
>
> Jo
>
>
> On 11/26/2018 7:05 AM, Raúl Hernández wrote:
> > I also consider that option, but when I try the very same thing with a
> > different, region (not related to the task). I get accuracies of 50%. So
> > that makes me think that it is affected by the task, but I don't know
> > what to think of it.
> >
> > Regards
> >
> > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <kaustubh.patil at gmail.com
> > <mailto:kaustubh.patil at gmail.com>> wrote:
> >
> >     I suspect that there might be something wrong in the code/how the
> >     data is handled.
> >
> >     If you 30% accuracy then that would mean that you will get 70% if
> >     you use a simple rule to predict the "other class" after your
> >     classifier. This is a sign that something is not right in data
> >     handling/evaluation.
> >
> >     Best
> >
> >     On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <raul at lafuentelab.org
> >     <mailto:raul at lafuentelab.org>> wrote:
> >
> >         No, it is balanced. It has the same number of observations for
> >         each class.
> >
> >         On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil
> >         <kaustubh.patil at gmail.com <mailto:kaustubh.patil at gmail.com>>
> wrote:
> >
> >             Just for clarification.
> >
> >             Is that data imbalanced, i.e. many more observations from
> >             one class?
> >
> >             Best,
> >             Kaustubh
> >
> >             On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández
> >             <raul at lafuentelab.org <mailto:raul at lafuentelab.org>> wrote:
> >
> >                 Dear PyMVPA community,
> >
> >                 I'm doing classification in ROI's, I'm performing a
> >                 simple 2 way classification using LSVM, and a
> >                 leave-one-run-out cross-validation on 4 acquisitions. On
> >                 some ROI's, I get a good accuracy for the number of
> >                 participants (60%), but in others I get consistently bad
> >                 accuracy (30%). To test whether the performance is above
> >                 chance, I use a one sample t test (I know that it is not
> >                 the best test for this type of data, I just use it as
> >                 quick overview). When I test the bad accuracies, those
> >                 are also significant.
> >
> >                 What does it mean a consistently bad accuracy?
> >
> >                 Regards,
> >
> >                 Raul
> >                 _______________________________________________
> >                 Pkg-ExpPsy-PyMVPA mailing list
> >                 Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >                 <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> >
> >             _______________________________________________
> >             Pkg-ExpPsy-PyMVPA mailing list
> >             Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >             <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> >
> >         _______________________________________________
> >         Pkg-ExpPsy-PyMVPA mailing list
> >         Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >         <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> >
> >     _______________________________________________
> >     Pkg-ExpPsy-PyMVPA mailing list
> >     Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> >
> >
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> >
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> >
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20181126/e6529671/attachment-0001.html>