[pymvpa] Consistently bad accuracy?

Tue Nov 27 13:44:15 GMT 2018

Actually, I'm working with dog fMRI data, and I have a constant issue with
normalization/registration. So you might be right.

Thank you for the input.

Regards

Raúl

On Tue, Nov 27, 2018 at 2:31 PM Roberto Guidotti <robbenson18 at gmail.com>
wrote:

> I think that it could be a registration problem or a
> normalization/detrending problem.
> I used to have below-chance accuracy in across-subject analyses, but this
> is not your case!
>
> Bests,
> R
>
> On Mon, 26 Nov 2018 at 16:29, Raúl Hernández <raul at lafuentelab.org> wrote:
>
>> Thank you for the link, I will look into it carefully.
>> Sorry for not being clear, yes I have 4 acquisitions from each
>> participant. I calculate an accuracy for each participant by calculating
>> the mean across all cross validation folds. Then I take the this calculated
>> mean from each participant and run a t test in which each participant
>> contributes with a single number.
>>
>> Regards,
>>
>> Raul
>>
>> On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <jetzel at wustl.edu> wrote:
>>
>>> I agree with Patil that consistent below-chance accuracy is a sign that
>>> something is not working properly.
>>>
>>> I collected some thoughts in
>>>
>>> http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html
>>> (and a few other posts tagged "below-chance").
>>>
>>> Also, be careful with terminology; I assume by "leave-one-run-out
>>> cross-validation on 4 acquisitions" you mean each person completed four
>>> scanning runs (each with the same fMRI acquisition parameters)? And a
>>> t-test can be fine for a quick significance test, but it should be done
>>> at the group level, testing if the subjects' accuracies are above chance
>>> (i.e., each person contributing one number to the t-test), not on the
>>> cross-validation folds within each person.
>>>
>>> Jo
>>>
>>>
>>> On 11/26/2018 7:05 AM, Raúl Hernández wrote:
>>> > I also consider that option, but when I try the very same thing with a
>>> > different, region (not related to the task). I get accuracies of 50%.
>>> So
>>> > that makes me think that it is affected by the task, but I don't know
>>> > what to think of it.
>>> >
>>> > Regards
>>> >
>>> > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <
>>> kaustubh.patil at gmail.com
>>> > <mailto:kaustubh.patil at gmail.com>> wrote:
>>> >
>>> >     I suspect that there might be something wrong in the code/how the
>>> >     data is handled.
>>> >
>>> >     If you 30% accuracy then that would mean that you will get 70% if
>>> >     you use a simple rule to predict the "other class" after your
>>> >     classifier. This is a sign that something is not right in data
>>> >     handling/evaluation.
>>> >
>>> >     Best
>>> >
>>> >     On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <
>>> raul at lafuentelab.org
>>> >     <mailto:raul at lafuentelab.org>> wrote:
>>> >
>>> >         No, it is balanced. It has the same number of observations for
>>> >         each class.
>>> >
>>> >         On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil
>>> >         <kaustubh.patil at gmail.com <mailto:kaustubh.patil at gmail.com>>
>>> wrote:
>>> >
>>> >             Just for clarification.
>>> >
>>> >             Is that data imbalanced, i.e. many more observations from
>>> >             one class?
>>> >
>>> >             Best,
>>> >             Kaustubh
>>> >
>>> >             On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández
>>> >             <raul at lafuentelab.org <mailto:raul at lafuentelab.org>>
>>> wrote:
>>> >
>>> >                 Dear PyMVPA community,
>>> >
>>> >                 I'm doing classification in ROI's, I'm performing a
>>> >                 simple 2 way classification using LSVM, and a
>>> >                 leave-one-run-out cross-validation on 4 acquisitions.
>>> On
>>> >                 some ROI's, I get a good accuracy for the number of
>>> >                 participants (60%), but in others I get consistently
>>> bad
>>> >                 accuracy (30%). To test whether the performance is
>>> above
>>> >                 chance, I use a one sample t test (I know that it is
>>> not
>>> >                 the best test for this type of data, I just use it as
>>> >                 quick overview). When I test the bad accuracies, those
>>> >                 are also significant.
>>> >
>>> >                 What does it mean a consistently bad accuracy?
>>> >
>>> >                 Regards,
>>> >
>>> >                 Raul
>>> >                 _______________________________________________
>>> >                 Pkg-ExpPsy-PyMVPA mailing list
>>> >                 Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>> >                 <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>>> >
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>> >
>>> >             _______________________________________________
>>> >             Pkg-ExpPsy-PyMVPA mailing list
>>> >             Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>> >             <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>>> >
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>> >
>>> >         _______________________________________________
>>> >         Pkg-ExpPsy-PyMVPA mailing list
>>> >         Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>> >         <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>>> >
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>> >
>>> >     _______________________________________________
>>> >     Pkg-ExpPsy-PyMVPA mailing list
>>> >     Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>> >     <mailto:Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net>
>>> >
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>> >
>>> >
>>> > _______________________________________________
>>> > Pkg-ExpPsy-PyMVPA mailing list
>>> > Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>> >
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>> >
>>> _______________________________________________
>>> Pkg-ExpPsy-PyMVPA mailing list
>>> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>>>
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>
>
> --
> Ing. Roberto Guidotti, PhD.
> PostDoc Fellow
> Institute for Advanced Biomedical Technologies - ITAB
> Department of Neuroscience and Imaging
> University of Chieti "G. D'Annunzio"
> Via dei Vestini, 33
> 66013 Chieti, Italy
> tel: +39 0871 3556919
> e-mail: r.guidotti at unich.it; rguidotti at acm.org
> linkedin: http://it.linkedin.com/in/robertogui/
> twitter: @robbisg
> github: https://github.com/robbisg
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20181127/79dbfdd3/attachment-0001.html>