[pymvpa] significance

[Jonas Kaplan]

On May 7, 2009, at 3:17 PM, Scott Gorlin wrote:

> Are your 8 chunks just separate scans, or were the conditions  
> different?  If they're just repetitions of the same thing  
> (especially, if chance performance is equal) then there should not  
> be an issue with summing the confusion matrices in each.  In theory  
> the results are independent provided that for each chunk you're  
> training your classifiers on the others.

That's right they are just 8 separate scans, each a repetition of the  
same thing.   Regarding independence, what I was thinking is that the  
results of one cross-validation step are not entirely independent from  
another since the training occurs on 6/7th of the same data points.

> Although, if you can pair up trials across your classes, I prefer  
> the McNemar test to the binomial.  This will also let you handle  
> things like if the baseline chance is expected to be different in  
> each chunk, ie if you use different stimuli.

Thanks, I will look into that.

>
> What do you mean by scrambling the regressors?  the null  
> distribution test only requires that you scramble the labels (ie see  
> how seperable the data are in directions orthogonal to the effect of  
> interest); if you mean GLM regressors then you probably should leave  
> those alone.  It might be interesting to do a ND test by placing the  
> stim regressors randomly throughout the scan but assuming that your  
> data are balanced and there's no bias in the GLM estimation, I don't  
> see there being any benefit to this over simply scrambling the  
> labels.  This would surely be much slower too if you need to go back  
> to SPM/FSL
>

Sorry, I just meant scrambling the labels.


> -Scott
>
> Jonas Kaplan wrote:
>> Hello,
>>
>> I wonder if anyone could help me think through the issue of testing  
>> classifier results for significance and how it relates to cross- 
>> validation.
>>
>> We are running a design with 8 chunks, 27 trials in each chunk  
>> divided into 3 classes.   Let's say we do an eight way (leave one  
>> out) cross-validation.  This results in an accuracy value for each  
>> set of 27 tests... 8 x 27 for a total of 216 trials that were  
>> predicted correctly or incorrectly.
>>
>> Is it wrong to use a binomial test for significance on the total  
>> number of correct predictions out of 216?  Or would that be  
>> inappropriate given that the 8 cross-validation steps are not  
>> really independent from each other and we must test each cross- 
>> validation step separately as a binomial with n=27? This latter  
>> option raises the issue of how to combine across the 8 tests.
>>
>> Alternatively, if we use the Monte Carlo simulation to produce a  
>> null distribution we have the same issue -- we are generating this  
>> null distribution for each cross-validation step -- and therefore  
>> not taking into account the overall success of the cross-validation  
>> routine across all 216 trials.   Would it make sense to generate a  
>> null distribution by scrambling the regressors and generating the  
>> results of an entire cross-validation procedure for scrambled  
>> regressors?  If so, does pymvpa have a routine for doing this?
>>
>> Thanks for any input or corrections of my thought,
>>
>>
>> Jonas
>>
>>
>>
>> P.S. we are using pymvpa for several active projects with much  
>> pleasure and will happily send you posters/papers when our work is  
>> more complete.
>>
>>
>> ----
>> Jonas Kaplan, Ph.D.
>> Research Assistant Professor
>> Brain & Creativity Institute
>> University of Southern California
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa