<div dir="ltr"><div>BTW another way to handle imbbalanced data (and perhaps easier to implement and test) could be assign weights in libvsm. This has to be done for each partition separately, any ideas on how this can be done?<br><br></div>Thanks <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 15, 2016 at 11:28 PM, Kaustubh Patil <span dir="ltr"><<a href="mailto:kaustubh.patil@gmail.com" target="_blank">kaustubh.patil@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div>Thanks again Yaroslav.<br><br></div>I agree that the classifier might end up giving 0 or very small balanced accuracy (or micro accuracy) values but I think thats still a better measure than using overall accuracy (or macro accuracy). There are couple of other measures that can be useful for imbalanced datasets:<br><br></div>1. A-mean: arithmetic mean, same as average class-wise accuracy or micro-accuracy<br></div>2. G-mean: geometric mean instead of arithmetic mean above<br></div>3. F-measure<br></div>4. Area under the ROC curve<br><br></div>Of course a better solution would be using a classifier that can handle imbalanced datasets, as you suggested. I have previously used SVMperf that can optimize AU-ROC: <a href="https://www.cs.cornell.edu/people/tj/svm_light/svm_perf.html" target="_blank">https://www.cs.cornell.edu/people/tj/svm_light/svm_perf.html</a><br><br></div>Not sure how easy it is to incorporate new classifiers in PyMVPA but I could give it a try with some guideline.<br><br></div>Best regards,<br></div>Kaustubh<br><br><div><div><div><div><br><div><div><div><div><div><div><br></div></div></div></div></div></div></div></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 15, 2016 at 11:08 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
On Fri, 15 Jan 2016, Kaustubh Patil wrote:<br>
<br>
> Thanks Yaroslav.<br>
<br>
> I tried your solution and it seems to work for this particular dataset but<br>
> unfortunately not for other datasets as the labels cannot be balanced<br>
> easily.<br>
<br>
> Maybe it's possible to directly calculate balanced measures in the CV? I<br>
> guess I will have to change the code to do that, any suggestions where to<br>
> start?<br>
<br>
</span>some toolboxes compute 'mean of within class accuracies' (not mean<br>
overall accuracy) which allows to account for disbalance. I guess we<br>
could code it quite easily if you like<br>
<br>
BUT the problem really would remain: with small number of samples<br>
classifier might just take the "majority" label since it would minimize<br>
error more than low performace decision. So you would hurt yourself<br>
more than help.<br>
<br>
another solution is to try a classifier which provides weighting<br>
to the classes, e.g. as GNB with default prior setting does. you could<br>
try it and see how it goes. It is not the greatest classifier but a<br>
start. then you could add similar class weighting to some other<br>
classifiers supporting that.<br>
<span><br>
> Best regards<br>
> On Sat, Dec 19, 2015 at 3:51 PM, Yaroslav Halchenko<br>
> <<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>> wrote:<br>
<br>
> On Sat, 19 Dec 2015, Kaustubh Patil wrote:<br>
<br>
> > Hi,<br>
<br>
> > I want to use PyMVPA for whole-brain searchlight analysis on some<br>
> existing<br>
> > data. The data has been already preprocessed (skull stripping, motion<br>
> > correction etc.). Each subject data contains 10 runs and each run was<br>
> processed<br>
> > separately, so there is a separate full brain boolean mask for each<br>
> run.<br>
<br>
> > My question is what is the recommended/correct a way to use this data<br>
> to<br>
> > perform run-wise cross-validation searchlight?<br>
<br>
> you have a problem here, since you have done per run preprocessing, in<br>
> particular motion-correction, your volumes are misaligned across runs.<br>
> (used FSL, didn't you? )<br>
<br>
> ideally, you redo preprocessing while motion correcting to the same<br>
</span>> volume across all the runs.A Alternatively, you reslice all the runs<br>
<span>> into the same space (could well be the common space your toolkit used<br>
> for analysis across runs -- common anatomical or MNI) and then do<br>
> analysis there, while again unifying your mask, which must be the same<br>
> across all the runs.<br>
> > As I understand, each run has to be in the same space (same number of<br>
> voxels)<br>
> > so that training and test can be performed, so the whole brain masks<br>
> have to be<br>
> > somehow aligned. How would you recommend doing this?<br>
<br>
> it is not a mere 'number of voxels' problem but rather that you have<br>
</span>> misaligned across runs volumes.A if just voxel number -- choose<br>
<div><div>> intersection of all masks.<br>
--<br>
Yaroslav O. Halchenko<br>
Center for Open Neuroscience <a href="http://centerforopenneuroscience.org" rel="noreferrer" target="_blank">http://centerforopenneuroscience.org</a><br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834" target="_blank">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419" target="_blank">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" rel="noreferrer" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>