[pymvpa] ROC

[Emanuele Olivetti]

Even though it is not about visualizing performance of classifiers,
a related nice article about (statistical) comparison of classifiers
(that you could be interested in) is this one:

http://jmlr.csail.mit.edu/papers/volume7/demsar06a/demsar06a.pdf

Best,

E.

Yaroslav Halchenko wrote:
> Dear All,
>
> I've committed series of changes which now provide AUC computation
> support within ConfusionMatrix, as well as basic plotting of ROC curves
> for a given label_index (as it is given by dataset.uniquelabels).
> Multiclass ROC support is provided just in 1-vs-all scenarios, e.g. like
> SMLR provides us (probability of each label given a sample), so all
> those multiclass classifiers with 1-vs-1 pairs and maxvote (like libsvm
> or MultiClass) will not have AUC and ROC in their ConfusionMatrix.
>
> So -- there is plenty of room for improvement, including more stable
> computation in the case of equal labels for different labels, improved
> plotting, etc. I am not that much interested in it atm, so I will leave
> it open for contributions.
>
> If you are seeking for an interesting article covering 'stable
> computation' as well as other aspects (multiclass, plotting, etc), I can
> recommend
>
> http://dx.doi.org/10.1016/j.patrec.2005.10.010
> An introduction to ROC analysis
> Tom Fawcett
>
> We are still missing 1 tiny example (may be we should add it within one
> of the existing examples) which plots our confusion matrix, and there we
> can add plotting of ROC curve.
>
> Cheers
> Yarik
>
> On Wed, 01 Oct 2008, Yaroslav Halchenko wrote:
>
>
>   
>> I committed and pushed some not-finished yet code... so you could
>> torture if you like, but first I want to bring your attention to the
>> following misfunction imho:
>>     
>
>
>   
>> In [27]:AUCErrorFx()([0.5, 0.5], [0, 1])
>> Out[27]:1.0
>>     
>
>   
>> In [28]:AUCErrorFx()([0.5, 0.5, 0.5, 0.5], [0, 1, 0, 1])
>> Out[28]:0.75
>>     
>
>   
>> In [29]:AUCErrorFx()([0.5, 0.5, 0.5, 0.5, 0.5, 0.5], [0, 1, 0, 1, 0, 1])
>> Out[29]:0.666666666667
>>     
>
>   
>> shouldn't in all of those cases we get 0.5?
>>     
>
>   
>> I know that it is a boundary and artificial case, but!
>> So whoever is awake, he is welcome to join the coding ;)
>>