[pymvpa] Sanity check

[Raúl Hernández]

Yes, I'm getting better performance when leaving out only one trial instead
of the full run. I guess that this is expected because I have more training
samples, but the increase of accuracy is well above what I would expect
(testing by runs is around 32% and testing by trial is about 49%).
I have 4 different stimulus. Each stimulus is repeated 6 times per run and
I have 12 runs. When I test a single trial, I'm using the full database,
without balancing for the one I used.

On Wed, Sep 23, 2015 at 4:28 PM, Jo Etzel <jetzel at wustl.edu> wrote:

> Do you mean that you're getting better performance when you're just
> leaving out one trial instead of one run?
>
> If so, How many runs? How many examples per run? Is everything fully
> balanced (equal number of training examples in each class) under each
> cross-validation scheme?
>
> Jo
>
>
>
> On 9/23/2015 3:03 PM, Raúl Hernández wrote:
>
>> Hi, I’m trying to evaluate on trial by trial basis how well a region can
>> predict the stimulus being presented to compare it with the
>> participant’s judgment of the stimulus. So I’m training the classifier
>> with data from all the trials on all the runs except by the one that I
>> want to predict.
>>
>> I’m getting really good classifications better than when I was
>> predicting one run using all the others. Supposedly it should be a
>> little better as I’m training with a little more data but I’m worried
>> I’m doing something wrong.
>>
>>
>> Could anyone let me know if I’m making some sort of mistake?
>>
>>
>> I know that there should be a more efficient way to do it but I wanted
>> something easy, this is my code:
>>
>>
>> predictions = [] #this is a vector that will contain the predictions of
>> the classifier
>>
>> for i,dsTest in enumerate(ds): #go through all the trials on ds and
>> separate one to test
>>
>>      clf = LinearCSVMC()
>>
>>      fclf = FeatureSelectionClassifier(clf, fsel)
>>
>>      dsTrain = []
>>
>>      dsTrain.append(ds[0:i]) #separates the training data
>>
>>      dsTrain.append(ds[i:-1])
>>
>>      dsTrain = vstack(dsTrain) #stacks it
>>
>>      fclf.train(dsTrain)
>>
>>      predicted = fclf.predict(dsTest) #stores the prediction
>>
>>      predictions.append(dsTest.targets == predicted) #checks whether the
>> prediction was correct
>>
>> print np.mean(predictions) #checks the mean -accuracy of all predictions
>>
>> I would really appreciate any feedback, thanks!
>>
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>>
> --
> Joset A. Etzel, Ph.D.
> Research Analyst
> Cognitive Control & Psychopathology Lab
> Washington University in St. Louis
> http://mvpa.blogspot.com/
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20150923/110bf8a4/attachment.html>