<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Dear Michael,<div><br></div><div>Thanks for the clarification and the modification of the codes - it works now!</div><div><br></div><div>Best,</div><div>Meng</div><div><br><div>> Date: Mon, 30 Jun 2014 21:33:42 +0200<br>> From: mih@debian.org<br>> To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> Subject: Re: [pymvpa] same sensitivity values for all dataset splits from cross validation using tutorial script snippet<br>> <br>> Hi,<br>> <br>> I am sorry you had to go through this...<br>> <br>> On Mon, Jun 30, 2014 at 02:18:07PM +0100, Meng Liang wrote:<br>> > I am trying to obtain the sensitivity values for all splits of the<br>> > dataset during leave-one-out cross-validation (classification using<br>> > SVM). I found in the tutorial "Classification Model Parameters –<br>> > Sensitivity Analysis" (<br>> > http://www.pymvpa.org/tutorial_sensitivity.html ) that<br>> > RepeatedMeasure(sensana, NFoldPartitioner()) should give the<br>> > sensitivity values for each fold. Here are the code snippet I used in<br>> > my script slightly adapted from the tutorial:<br>> ><br>> > clf = LinearNuSVMC()<br>> > cv = CrossValidation(clf, NFoldPartitioner(),enable_ca=['stats'])<br>> > sensana = clf.get_sensitivity_analyzer()<br>> > cv_sensana = RepeatedMeasure(sensana, NFoldPartitioner())<br>> > error = cv(ds)<br>> > sensmap_cv = cv_sensana(ds)<br>> > 'print sensmap_cv.shape'<br>> ><br>> > gave me: (14L, 87L). <br>> ><br>> > I have 14 subjects and I am using leave-one-subject-out<br>> > cross-validation, and there are 87 features. So the data structure<br>> > seems correct. However, when I look at the values of this 14x87 array,<br>> > all the rows in the array contain exactly the same values (i.e., the<br>> > first row looks the same with all the other rows). <br>> <br>> I am afraid you found a bug in the documentation (more specifically a<br>> bit of code that has not been properly adjusted when we switched from<br>> dataset splitters to dataset partitioners -- just mentioning it for<br>> those who have been around for that long...).<br>> <br>> The reason for the behavior you observe is that, in contrast to what is<br>> advertised in the tutorial, RepeatedMeasure does not split any dataset.<br>> It does what it says on the label: it repeats a measure, for whatever<br>> datasets come out of the provided generator -- in your case<br>> NFoldPartitioner. However, partitioners only add a sample attribute to a<br>> dataset that indicate the current partitioning scheme -- they do not<br>> split a dataset -- hence you are actually computing sensitivities,<br>> repeatedly, from the identical dataset.<br>> <br>> If you want to compute the sensitivities on the respective training<br>> samples of each data fold (which I think you do) you need to change that<br>> line to:<br>> <br>> cv_sensana = RepeatedMeasure(sensana,<br>> ChainNode((NFoldPartitioner(),<br>> Splitter('partitions',<br>> attr_values=(1,)))))<br>> <br>> This change amends the partitioner with a splitter that actually takes<br>> out the training samples of each fold and feeds them into the<br>> sensitivity measure.<br>> <br>> > A related question about normalizing the sensitivity values: in the<br>> > "Closing Words" of the tutorial on the same webpage, it says: "It<br>> > should also be noted that sensitivities can not be directly compared<br>> > to each other, even if they stem from the same algorithm and are just<br>> > computed on different dataset splits. In an analysis one would have to<br>> > normalize them first." My question is: if we cannot compare the<br>> > sensitivity values from different data splits without normalizing them<br>> > first, why can we average them or take the maximum value across data<br>> > splits without applying any normalization (the example script snippets<br>> > in the tutorial seem to do so)? I would imagine that the average or<br>> > the max value would also be affected by the scale of the data. <br>> <br>> Yes, you are right: they could be normalized even more (the dataset in<br>> the tutorial, however, is a single subject and it was z-scored upfront.<br>> So it is not that bad...<br>> <br>> Sorry for the bug. I filed a bug report and we'll fix it ASAP.<br>> <br>> Michael<br>> <br>> -- <br>> J.-Prof. Dr. Michael Hanke<br>> Psychoinformatik Labor, Institut für Psychologie II<br>> Otto-von-Guericke-Universität Magdeburg, Universitätsplatz 2, Geb.24<br>> Tel.: +49(0)391-67-18481 Fax: +49(0)391-67-11947 GPG: 4096R/7FFB9E9B<br>> <br>> _______________________________________________<br>> Pkg-ExpPsy-PyMVPA mailing list<br>> Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br></div></div> </div></body>
</html>