Combining accuracy and prior sensitivity for classifier design under prior uncertainty

Conference paper (2006)
More Info
expand_more

Published Date

2006

Language

Undefined/Unknown

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Consideringtheclassi¿cationprobleminwhichclasspriorsormisallocationcostsarenotknownprecisely,receiveroperatorcharacteristic(ROC)analysishasbecomeastandardtoolinpatternrecognitionforobtainingintegratedperformancemeasurestocopewiththeuncertainty.Similarly,insituationsinwhichpriorsmayvaryinapplication,theROCcanbeusedtoinspectperformanceovertheexpectedrangeofvariation.InthispaperwearguethateventhoughmeasuressuchastheareaundertheROC(AUC)areusefulinobtaininganintegratedperformancemeasureindependentofthepriors,itmayalsobeimportanttoincorporatethesensitivityacrosstheexpectedprior-range.Weshowthataclassi¿ermayresultinagoodAUCscore,butapoor(large)priorsensitivity,whichmaybeundesirable.Amethodologyisproposedthatcombinesbothaccuracyandsensitivity,providinganewmodelselectioncriterionthatisrelevanttocertainproblems.Experimentsshowthatincorporatingsensitivityisveryimportantinsomerealisticscenarios,leadingtobettermodelselectioninsomecases