<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"><style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Times}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Times; min-height: 14.0px}
p.p3 {margin: 0.0px 0.0px 0.0px 0.0px; line-height: 20.8px; font: 16.0px Arial; -webkit-text-stroke: #000000}
p.p4 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Times; -webkit-text-stroke: #000000; min-height: 14.0px}
span.s1 {font-kerning: none}
span.s2 {font-kerning: none; background-color: #dcf1ff}
span.s3 {font: 16.0px Courier; font-kerning: none}
</style>
<p class="p1"><font face="Arial" size="2">Hello,</font></p>
<p class="p1"><font face="Arial" size="2"><br>
</font></p>
<p class="p1"><font face="Arial" size="2">I'm new to pyMVPA and have read through all of the tutorial and other documentation, but can't seem to figure out what is probably a very simple question: is there a way to get the individual confusion matrices for
each classification (each run) that results from using the halfpartitioner generator? I'm guessing it has something to do with the attributes/parameters of the generator? I'm trying to do this both for a whole brain classification and for a searchlight. I
have 2 runs/chunks (each run has 72 trials and each trial is associated with an Ingroup/Outgroup target). But I only want it to train on run 1 and test on run 2 (I'm not interested in the results from train run 2 and test run 1 and thus would just want to
look at the confusion matrix for train run1, test run2 - ie an Ingroup/Outgroup classification with 72 trials rather than 144 classifications). For the whole brain my goal is to calculate the TPR for Ingroup and Outgroup targets only training on run 1 and
testing on run 2.</font></p>
<p class="p1"><font face="Arial" size="2"><br>
</font></p>
<p class="p1"><font face="Arial" size="2">I've tried it three different ways and I'm getting different results for each way so just wanted to know if any of these ways is valid:</font></p>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2">1) <b>Using the manual split example from the tutorial and calling the "training_stats" conditional attribute in the classifier</b></font></div>
<div><font face="Arial" size="2">In the tutorial we can get the individual accuracies for each run through <span class="n" style="color: rgb(51, 51, 51);">cv_results</span><span class="o" style="color: rgb(102, 102, 102);">.</span><span class="n" style="color: rgb(51, 51, 51);">samples
but I'm interested in the TPR (True Positive Rate) for Ingroup and Outgroup separately so I'm looking to print the confusion matrix to calculate these numbers</span></font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div>
<div><font face="Arial" size="2">ds_split1 = ds[ds.sa.chunks == 1.]</font></div>
<div><font face="Arial" size="2">ds_split2 = ds[ds.sa.chunks == 2.]</font></div>
<div><span style="font-family: Arial; font-size: small;">clf = LinearCSVMC(<b>enable_ca=['training_stats'</b>])</span></div>
</div>
<div>
<div><span style="font-family: Arial; font-size: small;">clf.set_postproc(BinaryFxNode(mean_mismatch_error,'targets'))</span></div>
<div><span style="font-family: Arial; font-size: small;">clf.train(ds_split1)</span></div>
<div><span style="font-family: Arial; font-size: small;">err = clf(ds_split2)</span></div>
</div>
<div><font face="Arial" size="2">clf.ca.training_stats.as_string(description=True)</font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2"><b>2) Using the HalfPartitioner function's "count" argument</b></font></div>
<div><span style="font-family: Arial; font-size: small;"><br>
</span></div>
<div><span style="font-family: Arial; font-size: small;"> clf = LinearCSVMC(enable_ca=['training_stats']) #The training_stats confusion matrix from this method doesn't match the one above </span></div>
<div>
<div><span style="font-family: Arial; font-size: small;"> hpart = HalfPartitioner(<b>count=1</b>, attr='chunks')</span></div>
</div>
<div>
<div><span style="font-family: Arial; font-size: small;"> cvte = CrossValidation(clf,hpart,errorfx=lambda p,t: np.mean(p==t),enable_ca=['stats'])</span></div>
</div>
<div>
<div><span style="font-family: Arial; font-size: small;"> cv_results = cvte(ds)</span></div>
</div>
<div><span style="font-family: Arial; font-size: small;"> cvte.ca.stats.as_string(description=True)</span></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2">3)<b> Doing manual counters of the predicted vs actual targets</b></font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div>
<div><font face="Arial" size="2">ds_split1 = ds[ds.sa.chunks == 1.]</font></div>
<div><font face="Arial" size="2">ds_split2 = ds[ds.sa.chunks == 2.]</font></div>
<div><span style="font-family: Arial; font-size: small;">clf = LinearCSVMC()</span></div>
</div>
<div>
<div><font face="Arial" size="2">clf.train(ds_split1)</font></div>
<div><span style="font-family: Arial; font-size: small;">predictions=clf.predict(ds_split2.samples)</span></div>
<div><font face="Arial" size="2">prediction_values=predictions==ds_split2.sa.targets</font></div>
</div>
<div><span style="font-family: Arial; font-size: small;"><br>
</span></div>
<div><span style="font-family: Arial; font-size: small;">counter=0</span></div>
<div><font face="Arial" size="2">
<div> for stimulus in ds_split2.sa.targets:</div>
<div> current_prediction_value=prediction_values[counter]</div>
<div> print current_prediction_value</div>
<div> if stimulus=='I': #Ingroup</div>
<div> if current_prediction_value==True:</div>
<div> num_correct_ingroup+=1.0</div>
<div> counter+=1</div>
<div> else:</div>
<div> counter+=1</div>
<div> elif stimulus=='O': #Outgroup</div>
<div> if current_prediction_value==True:</div>
<div> num_correct_outgroup+=1.0</div>
<div> counter+=1</div>
<div> else:</div>
<div> counter+=1</div>
<div><br>
</div>
<div> sensitivity_ingroup=float(num_correct_ingroup/36.0)</div>
<div> sensitivity_outgroup=float(num_correct_outgroup/36.0)</div>
</font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2">I'm getting different results (Ingroup/Outgroup TPRs) for each of these methods so just wondering which, if any, of the above mentioned methods would be the correct method for getting the confusion matrices or TPRs for Ingroup/Outgroup
only training on run 1 and testing on run 2? The last method I wouldn't be able to use for a searchlight but might be valid for a wholebrain?</font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2">The confusion matrix that I get from method 1 has highly accurate predictions, which makes me doubt that's the confusion matrix I'm looking for.</font></div>
<div><font face="Arial" size="2"><br>
</font></div>
<div><font face="Arial" size="2">Thank you for your help!</font></div>
<p class="p2"><font face="Arial" size="2"><br>
</font></p>
<p class="p2"><font face="Arial" size="2"><br>
</font></p>
<p class="p1"><font face="Arial" size="2"><br>
</font></p>
<p class="p2"><font face="Arial" size="2"><br>
</font></p>
<p class="p1"><br>
</p>
</div>
</body>
</html>