[pymvpa] same sensitivity values for all dataset splits from cross validation using tutorial script snippet

Meng Liang meng.liang at hotmail.co.uk
Mon Jun 30 13:18:07 UTC 2014


Dear all,
I am trying to obtain the sensitivity values for all splits of the dataset during leave-one-out cross-validation (classification using SVM). I found in the tutorial "Classification Model Parameters – Sensitivity Analysis" ( http://www.pymvpa.org/tutorial_sensitivity.html ) that RepeatedMeasure(sensana, NFoldPartitioner()) should give the sensitivity values for each fold. Here are the code snippet I used in my script slightly adapted from the tutorial:
            clf = LinearNuSVMC()            cv = CrossValidation(clf, NFoldPartitioner(),enable_ca=['stats'])            sensana = clf.get_sensitivity_analyzer()            cv_sensana = RepeatedMeasure(sensana, NFoldPartitioner())            error = cv(ds)            sensmap_cv = cv_sensana(ds)
'print sensmap_cv.shape'    gave me: (14L, 87L). 
I have 14 subjects and I am using leave-one-subject-out cross-validation, and there are 87 features. So the data structure seems correct. However, when I look at the values of this 14x87 array, all the rows in the array contain exactly the same values (i.e., the first row looks the same with all the other rows). 
Is this the correct way to obtain the sensitivity values for each fold of cross-validation classification? If not, any suggestions how to do?
A related question about normalizing the sensitivity values: in the "Closing Words" of the tutorial on the same webpage, it says: "It should also be noted that sensitivities can not be directly compared to each other, even if they stem from the same algorithm and are just computed on different dataset splits. In an analysis one would have to normalize them first." My question is: if we cannot compare the sensitivity values from different data splits without normalizing them first, why can we average them or take the maximum value across data splits without applying any normalization (the example script snippets in the tutorial seem to do so)? I would imagine that the average or the max value would also be affected by the scale of the data. 
Any help would be very much appreciated!
Best,Meng

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20140630/538f4323/attachment.html>


More information about the Pkg-ExpPsy-PyMVPA mailing list