<div dir="ltr">Thank you for the link, I will look into it carefully. <div>Sorry for not being clear, yes I have 4 acquisitions from each participant. I calculate an accuracy for each participant by calculating the mean across all cross validation folds. Then I take the this calculated mean from each participant and run a t test in which each participant contributes with a single number.</div><div><br></div><div>Regards,</div><div><br></div><div>Raul</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <<a href="mailto:jetzel@wustl.edu">jetzel@wustl.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I agree with Patil that consistent below-chance accuracy is a sign that <br>
something is not working properly.<br>
<br>
I collected some thoughts in <br>
<a href="http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html" rel="noreferrer" target="_blank">http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html</a> <br>
(and a few other posts tagged "below-chance").<br>
<br>
Also, be careful with terminology; I assume by "leave-one-run-out <br>
cross-validation on 4 acquisitions" you mean each person completed four <br>
scanning runs (each with the same fMRI acquisition parameters)? And a <br>
t-test can be fine for a quick significance test, but it should be done <br>
at the group level, testing if the subjects' accuracies are above chance <br>
(i.e., each person contributing one number to the t-test), not on the <br>
cross-validation folds within each person.<br>
<br>
Jo<br>
<br>
<br>
On 11/26/2018 7:05 AM, Raúl Hernández wrote:<br>
> I also consider that option, but when I try the very same thing with a <br>
> different, region (not related to the task). I get accuracies of 50%. So <br>
> that makes me think that it is affected by the task, but I don't know <br>
> what to think of it.<br>
> <br>
> Regards<br>
> <br>
> On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <<a href="mailto:kaustubh.patil@gmail.com" target="_blank">kaustubh.patil@gmail.com</a> <br>
> <mailto:<a href="mailto:kaustubh.patil@gmail.com" target="_blank">kaustubh.patil@gmail.com</a>>> wrote:<br>
> <br>
> I suspect that there might be something wrong in the code/how the<br>
> data is handled.<br>
> <br>
> If you 30% accuracy then that would mean that you will get 70% if<br>
> you use a simple rule to predict the "other class" after your<br>
> classifier. This is a sign that something is not right in data<br>
> handling/evaluation.<br>
> <br>
> Best<br>
> <br>
> On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <<a href="mailto:raul@lafuentelab.org" target="_blank">raul@lafuentelab.org</a><br>
> <mailto:<a href="mailto:raul@lafuentelab.org" target="_blank">raul@lafuentelab.org</a>>> wrote:<br>
> <br>
> No, it is balanced. It has the same number of observations for<br>
> each class.<br>
> <br>
> On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil<br>
> <<a href="mailto:kaustubh.patil@gmail.com" target="_blank">kaustubh.patil@gmail.com</a> <mailto:<a href="mailto:kaustubh.patil@gmail.com" target="_blank">kaustubh.patil@gmail.com</a>>> wrote:<br>
> <br>
> Just for clarification.<br>
> <br>
> Is that data imbalanced, i.e. many more observations from<br>
> one class?<br>
> <br>
> Best,<br>
> Kaustubh<br>
> <br>
> On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández<br>
> <<a href="mailto:raul@lafuentelab.org" target="_blank">raul@lafuentelab.org</a> <mailto:<a href="mailto:raul@lafuentelab.org" target="_blank">raul@lafuentelab.org</a>>> wrote:<br>
> <br>
> Dear PyMVPA community,<br>
> <br>
> I'm doing classification in ROI's, I'm performing a<br>
> simple 2 way classification using LSVM, and a<br>
> leave-one-run-out cross-validation on 4 acquisitions. On<br>
> some ROI's, I get a good accuracy for the number of<br>
> participants (60%), but in others I get consistently bad<br>
> accuracy (30%). To test whether the performance is above<br>
> chance, I use a one sample t test (I know that it is not<br>
> the best test for this type of data, I just use it as<br>
> quick overview). When I test the bad accuracies, those<br>
> are also significant.<br>
> <br>
> What does it mean a consistently bad accuracy?<br>
> <br>
> Regards,<br>
> <br>
> Raul<br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <mailto:<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a>><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
> <br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <mailto:<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a>><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
> <br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <mailto:<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a>><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
> <br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <mailto:<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a>><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
> <br>
> <br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
> <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
> <br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net" target="_blank">Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net</a><br>
<a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a></blockquote></div>