[pymvpa] Balancing with searchlight and statistical issues.

[Roberto Guidotti]

Thank you for your response Jo.

You don't want to try something like Stelzer's method (or any statistical
> test, really) until you're sure the single subject analyses are sensible.
> Do the actual classification accuracy maps for each person look reasonable?
> If every searchlight is classifying at 100% accuracy, something is
> obviously wrong with the analysis code and you should fix that first.
>

The group mean histogram looks a gaussian peaked at 0.57 accuracy, I think
it is reasonable.
The problem, I think, is the cross-validation scheme since I had 66 values
of accuracy each voxel, when I average withing subjects, across folds, I
have a map that is almost above chance for each voxel, when I perform a
simple t-test vs chance, across-subjects, I have all (or almost all) voxels
significant!
So I would like to know if my cross-validation scheme and approach is good
or try a different statistic, before run Stelzer's method.


> It's usually good to start by classifying something that should be a
> strong signal in easily-predicted areas (e.g., which response button was
> pushed, which should have signal in motor areas). Then you can make sure
> your cross-validation scheme, event coding, etc. is set up properly before
> trying your actual analysis.


> Also, you say the dataset is unbalanced, but has 12 runs, each with 10
> trials, half A and half B. That sounds balanced to me


I'm sorry I explained it badly!
I classified in few subject the motor response with good accuracies, but
now I would like to decode decision, since is a decision task, which is the
main reason why my dataset is unbalanced. Stimuli are balanced, since the
subject views half A and half B, but he has to respond if the stimulus is
either A or B, thus I could have runs with unbalanced condition (e.g. 8 A
vs 2 B, etc.).

Then if it is suggested to run different cv scheme and Stelzer's method, I
will... :)

Thank you
Roberto


> ...
>
> Jo
>
>
>
>
> On 2/25/2016 9:43 AM, Roberto Guidotti wrote:
>
>> Hi all mvpaers,
>>
>> I need some theoretical help!
>>
>> I did some analysis on a unbalanced dataset, 12 runs with 10 trials (5
>> condition A, 5 condition B), so I got 120 samples. Since I had an
>> unbalanced dataset, I could have a run with 7A vs 3B or also a 9A vs 1B
>> samples and/or viceversa.
>> I analyzed the dataset balancing samples in each run, using a leave TWO
>> run cross-validation (L2ROCV) searchlight, in order to have more
>> combination of samples to train the dataset and the same for the testing
>> set, I didn't analyze the dataset using different balancing since the
>> searchlight in a L2ROCV is high time consuming and I had 25 subjects
>> with 3 unbalanced dataset per subject!! :\
>>
>> Now, my questions are these:
>>
>> 1) I used a good approach to analyze the dataset or you suggest a
>> different approach?
>>
>> 2) I did an average map of the 66 cross-validation map I obtained for
>> each subject; to do a first exploratory analysis I did a simple t-test
>> versus chance level (I didn't do the Stelzer's method because of the
>> computational time) and I had almost all voxels significative (not
>> corrected), because of the L2ROCV, I think. So do you think I can do
>> other more robust statistical tests using these maps? Or I have to do
>> the Stelzer's method? Or throw away the searchlight maps?
>>
>> Thank you!!
>> Roberto
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>>
>>
> --
> Joset A. Etzel, Ph.D.
> Research Analyst
> Cognitive Control & Psychopathology Lab
> Washington University in St. Louis
> http://mvpa.blogspot.com/
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20160229/80743893/attachment.html>