[pymvpa] Consistently bad accuracy?

[Etzel, Jo]

I agree with Patil that consistent below-chance accuracy is a sign that 
something is not working properly.

I collected some thoughts in 
http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html 
(and a few other posts tagged "below-chance").

Also, be careful with terminology; I assume by "leave-one-run-out 
cross-validation on 4 acquisitions" you mean each person completed four 
scanning runs (each with the same fMRI acquisition parameters)? And a 
t-test can be fine for a quick significance test, but it should be done 
at the group level, testing if the subjects' accuracies are above chance 
(i.e., each person contributing one number to the t-test), not on the 
cross-validation folds within each person.

Jo


On 11/26/2018 7:05 AM, Raúl Hernández wrote:
> I also consider that option, but when I try the very same thing with a 
> different, region (not related to the task). I get accuracies of 50%. So 
> that makes me think that it is affected by the task, but I don't know 
> what to think of it.
> 
> Regards
> 
> On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <kaustubh.patil at gmail.com 
> <mailto:kaustubh.patil at gmail.com>> wrote:
> 
>     I suspect that there might be something wrong in the code/how the
>     data is handled.
> 
>     If you 30% accuracy then that would mean that you will get 70% if
>     you use a simple rule to predict the "other class" after your
>     classifier. This is a sign that something is not right in data
>     handling/evaluation.
> 
>     Best
> 
>     On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <raul at lafuentelab.org
>     <mailto:raul at lafuentelab.org>> wrote:
> 
>         No, it is balanced. It has the same number of observations for
>         each class.
> 
>         On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil
>         <kaustubh.patil at gmail.com <mailto:kaustubh.patil at gmail.com>> wrote:
> 
>             Just for clarification.
> 
>             Is that data imbalanced, i.e. many more observations from
>             one class?
> 
>             Best,
>             Kaustubh
> 
>             On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández
>             <raul at lafuentelab.org <mailto:raul at lafuentelab.org>> wrote:
> 
>                 Dear PyMVPA community,
> 
>                 I'm doing classification in ROI's, I'm performing a
>                 simple 2 way classification using LSVM, and a
>                 leave-one-run-out cross-validation on 4 acquisitions. On
>                 some ROI's, I get a good accuracy for the number of
>                 participants (60%), but in others I get consistently bad
>                 accuracy (30%). To test whether the performance is above
>                 chance, I use a one sample t test (I know that it is not
>                 the best test for this type of data, I just use it as
>                 quick overview). When I test the bad accuracies, those
>                 are also significant.
> 
>                 What does it mean a consistently bad accuracy?
> 
>                 Regards,
> 
>                 Raul
>                 _______________________________________________
>                 Pkg-ExpPsy-PyMVPA mailing list
>                 Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>                 <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>                 https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
>             _______________________________________________
>             Pkg-ExpPsy-PyMVPA mailing list
>             Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>             <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>             https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
>         _______________________________________________
>         Pkg-ExpPsy-PyMVPA mailing list
>         Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>         <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>         https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
>     _______________________________________________
>     Pkg-ExpPsy-PyMVPA mailing list
>     Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>     https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>