[pymvpa] Pairwise classification
emanuele at relativita.com
Wed Jan 11 13:38:50 UTC 2012
On 01/11/2012 02:04 PM, Jonas Kubilius wrote:
> I wanted to ask if there was some built-in way to perform a pairwise classification
> similar to how one would do correlations with one_minus_correlation. Suppose I give my
> classifier 4 targets. I want the classifier to take all possible pairs including target
> pairs like (1,1) (the latter is in order to see how noisy my data is). That makes 4*3+4
> pairs or at least 4*3/2+4 due to confusion matrix being symmetric in this case. Then we
> should calculate classification accuracy for each pair, i.e. how many times each of the
> two targets were correctly predicted.
I guess you should generate the pairs yourself.
But pay attention that estimating the accuracy using "all" possible pairs of the
test set incurs in an optimisitc bias. The pairs are not independent since
they sometimes share one element. Having a test set of non-independent
examples leads to optimistic bias. In order not to have that bias you should
use instead the subset of non-overlapping pairs.
More information about the Pkg-ExpPsy-PyMVPA