Abstract The use of Receiver Operator Characteristic (ROC) analysis for the sake of model selection and threshold optimisation has become a standard practice for the design of two-class pattern recognition systems. Advantages include decision boundary adaptation to imbalanced misallocation costs, the ability to fix some classification errors, and performance evaluation in imprecise, ill-defined conditions where costs, or prior probabilities may vary. Extending this to the multiclass case has recently become a topic of interest. The primary challenge involved is the computational complexity, that increases to the power of the number of classes, rendering many problems intractable. In this paper the multiclass ROC is formalised, and the computational complexities exposed. A pairwise approach is proposed that approximates the multi-dimensional operating characteristic by discounting some interactions, resulting in an algorithm that is tractable, and extensible to large numbers of classes. Two additional multiclass optimisation techniques are also proposed that provide a benchmark for the pairwise algorithm. Experiments compare the various approaches in a variety of practical situations, demonstrating the efficacy of the pairwise approach. Keywords: ROC analysis; Multiclass ROC; Cost sensitive; Threshold optimisation@en

Consider the class of problems in which a target class is well-defined, and an outlier class is ill-defined. In these cases new outlier classes can appear, or the class-conditional distribution of the outlier class itself may be poorly sampled. A strategy to deal with this problem involves a two-stage classifier, in which one stage is designed to perform discrimination between known classes, and the other stage encloses known data to protect against changing conditions. The two stages are, however, interrelated, implying that optimising one may compromise the other. In this paper the relation between the two stages is studied within an ROC analysis framework. We show how the operating characteristics can be used for both model selection, and in aiding in the choice of the reject threshold. An analytic study on a controlled experiment is performed, followed by some experiments on real-world datasets with the distance-based reject-option classifier. Keywords: Ill-defined classification problems; Unseen classes; Reject-option; Model selection; ROC analysis@en

Consideringtheclassi¿cationprobleminwhichclasspriorsormisallocationcostsarenotknownprecisely,receiveroperatorcharacteristic(ROC)analysishasbecomeastandardtoolinpatternrecognitionforobtainingintegratedperformancemeasurestocopewiththeuncertainty.Similarly,insituationsinwhichpriorsmayvaryinapplication,theROCcanbeusedtoinspectperformanceovertheexpectedrangeofvariation.InthispaperwearguethateventhoughmeasuressuchastheareaundertheROC(AUC)areusefulinobtaininganintegratedperformancemeasureindependentofthepriors,itmayalsobeimportanttoincorporatethesensitivityacrosstheexpectedprior-range.Weshowthataclassi¿ermayresultinagoodAUCscore,butapoor(large)priorsensitivity,whichmaybeundesirable.Amethodologyisproposedthatcombinesbothaccuracyandsensitivity,providinganewmodelselectioncriterionthatisrelevanttocertainproblems.Experimentsshowthatincorporatingsensitivityisveryimportantinsomerealisticscenarios,leadingtobettermodelselectioninsomecases@en