[pymvpa] ROC

[Yaroslav Halchenko]

Dear All,

I've committed series of changes which now provide AUC computation
support within ConfusionMatrix, as well as basic plotting of ROC curves
for a given label_index (as it is given by dataset.uniquelabels).
Multiclass ROC support is provided just in 1-vs-all scenarios, e.g. like
SMLR provides us (probability of each label given a sample), so all
those multiclass classifiers with 1-vs-1 pairs and maxvote (like libsvm
or MultiClass) will not have AUC and ROC in their ConfusionMatrix.

So -- there is plenty of room for improvement, including more stable
computation in the case of equal labels for different labels, improved
plotting, etc. I am not that much interested in it atm, so I will leave
it open for contributions.

If you are seeking for an interesting article covering 'stable
computation' as well as other aspects (multiclass, plotting, etc), I can
recommend

http://dx.doi.org/10.1016/j.patrec.2005.10.010
An introduction to ROC analysis
Tom Fawcett

We are still missing 1 tiny example (may be we should add it within one
of the existing examples) which plots our confusion matrix, and there we
can add plotting of ROC curve.

Cheers
Yarik

On Wed, 01 Oct 2008, Yaroslav Halchenko wrote:


> I committed and pushed some not-finished yet code... so you could
> torture if you like, but first I want to bring your attention to the
> following misfunction imho:


> In [27]:AUCErrorFx()([0.5, 0.5], [0, 1])
> Out[27]:1.0

> In [28]:AUCErrorFx()([0.5, 0.5, 0.5, 0.5], [0, 1, 0, 1])
> Out[28]:0.75

> In [29]:AUCErrorFx()([0.5, 0.5, 0.5, 0.5, 0.5, 0.5], [0, 1, 0, 1, 0, 1])
> Out[29]:0.666666666667

> shouldn't in all of those cases we get 0.5?

> I know that it is a boundary and artificial case, but!
> So whoever is awake, he is welcome to join the coding ;)
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik