<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">Hello, </span></font><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">I'd like to pick the group's brain about an issue related to cross-validation. </span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">I just read a paper in Current Biology (Chadwick et al., 2010, Decoding Individual Episodic Memory Traces in the Hippocampus) in which the following cross-validation procedure was used: </span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">"We used a standard k-fold cross- validation testing regime [10] wherein k equalled the number of experimental trials, with the data from each trial set aside in turn as the test data, and the remaining data used as the training set (on each fold, the feature selection step was performed using only data from this training set)."</span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div>In other words, it seems to me that on each cross-validation fold, training was performed on all trials except one, and then the classifier is tested on one single trial. Does this sort of approach make sense? It seems to me that with only one test for each classifier you are not adequately assessing the performance of any of the classifiers. I suppose the idea is that across all of the folds you get a measure of how the classifiers work in general. I'd like to know if this is considered a reasonable approach since I have a dataset with a small number of trials that might benefit from maximizing the number of training trials. </div><div><br></div><div>Thanks!</div><div><br></div><div>Jonas</div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font><div>
<div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">----</span></font></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">Jonas Kaplan, Ph.D.<br>Research Assistant Professor<br>Brain & Creativity Institute<br>University of Southern California</span></font></div></div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;">
</span></font></div>
<br></div></body></html>