<div dir="ltr"><div>Yes, I'm getting better performance when leaving out only one trial instead of the full run. I guess that this is expected because I have more training samples, but the increase of accuracy is well above what I would expect (testing by runs is around 32% and testing by trial is about 49%).</div>I have 4 different stimulus. Each stimulus is repeated 6 times per run and I have 12 runs. When I test a single trial, I'm using the full database, without balancing for the one I used. </div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 23, 2015 at 4:28 PM, Jo Etzel <span dir="ltr"><<a href="mailto:jetzel@wustl.edu" target="_blank">jetzel@wustl.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Do you mean that you're getting better performance when you're just leaving out one trial instead of one run?<br>
<br>
If so, How many runs? How many examples per run? Is everything fully balanced (equal number of training examples in each class) under each cross-validation scheme?<br>
<br>
Jo<div><div class="h5"><br>
<br>
<br>
On 9/23/2015 3:03 PM, Raúl Hernández wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi, I’m trying to evaluate on trial by trial basis how well a region can<br>
predict the stimulus being presented to compare it with the<br>
participant’s judgment of the stimulus. So I’m training the classifier<br>
with data from all the trials on all the runs except by the one that I<br>
want to predict.<br>
<br>
I’m getting really good classifications better than when I was<br>
predicting one run using all the others. Supposedly it should be a<br>
little better as I’m training with a little more data but I’m worried<br>
I’m doing something wrong.<br>
<br>
<br>
Could anyone let me know if I’m making some sort of mistake?<br>
<br>
<br>
I know that there should be a more efficient way to do it but I wanted<br>
something easy, this is my code:<br>
<br>
<br>
predictions = [] #this is a vector that will contain the predictions of<br>
the classifier<br>
<br>
for i,dsTest in enumerate(ds): #go through all the trials on ds and<br>
separate one to test<br>
<br>
clf = LinearCSVMC()<br>
<br>
fclf = FeatureSelectionClassifier(clf, fsel)<br>
<br>
dsTrain = []<br>
<br>
dsTrain.append(ds[0:i]) #separates the training data<br>
<br>
dsTrain.append(ds[i:-1])<br>
<br>
dsTrain = vstack(dsTrain) #stacks it<br>
<br>
fclf.train(dsTrain)<br>
<br>
predicted = fclf.predict(dsTest) #stores the prediction<br>
<br>
predictions.append(dsTest.targets == predicted) #checks whether the<br>
prediction was correct<br>
<br>
print np.mean(predictions) #checks the mean -accuracy of all predictions<br>
<br>
I would really appreciate any feedback, thanks!<br>
<br>
<br>
<br></div></div>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
Joset A. Etzel, Ph.D.<br>
Research Analyst<br>
Cognitive Control & Psychopathology Lab<br>
Washington University in St. Louis<br>
<a href="http://mvpa.blogspot.com/" rel="noreferrer" target="_blank">http://mvpa.blogspot.com/</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</font></span></blockquote></div><br></div>