License
When quoting this document, please refer to the following
URN: urn:nbn:de:0030-drops-13828
URL: http://drops.dagstuhl.de/opus/volltexte/2008/1382/
Go to the corresponding Portal


Flach, Peter ; Matsubara, Edson

On classification, ranking, and probability estimation

pdf-format:
Document 1.pdf (286 KB)


Abstract

Given a binary classification task, a ranker is an algorithm that can sort a set of instances from highest to lowest expectation that the instance is positive. In contrast to a classifier, a ranker does not output class predictions – although it can be turned into a classifier with help of an additional procedure to split the ranked list into two. A straightforward way to compute rankings is to train a scoring classifier to assign numerical scores to instances, for example the predicted odds that an instance is positive. However, rankings can be computed without scores, as we demonstrate in this paper. We propose a lexicographic ranker, LexRank , whose rankings are derived not from scores, but from a simple ranking of attribute values obtained from the training data. Although various metrics can be used, we show that by using the odds ratio to rank the attribute values we obtain a ranker that is conceptually close to the naive Bayes classifier, in the sense that for every instance of LexRank there exists an instance of naive Bayes that achieves the same ranking. However, the reverse is not true, which means that LexRank is more biased than naive Bayes. We systematically develop the relationships and differences between classification, ranking, and probability estimation, which leads to a novel connection between the Brier score and ROC curves. Combining LexRank with isotonic regression, which derives probability estimates from the ROC convex hull, results in the lexicographic probability estimator LexProb.

BibTeX - Entry

@InProceedings{flach_et_al:DSP:2008:1382,
  author =	{Peter Flach and Edson Matsubara},
  title =	{On classification, ranking, and probability estimation},
  booktitle =	{Probabilistic, Logical and Relational Learning - A Further Synthesis},
  year =	{2008},
  editor =	{Luc de Raedt and Thomas Dietterich and Lise Getoor and Kristian Kersting and Stephen H. Muggleton},
  number =	{07161},
  series =	{Dagstuhl Seminar Proceedings},
  ISSN =	{1862-4405},
  publisher =	{Internationales Begegnungs- und Forschungszentrum f{\"u}r Informatik (IBFI), Schloss Dagstuhl, Germany},
  address =	{Dagstuhl, Germany},
  URL =		{http://drops.dagstuhl.de/opus/volltexte/2008/1382},
  annote =	{Keywords: Ranking, probability estimation, ROC analysis, calibration}
}

Keywords: Ranking, probability estimation, ROC analysis, calibration
Seminar: 07161 - Probabilistic, Logical and Relational Learning - A Further Synthesis
Issue Date: 2008
Date of publication: 06.03.2008


DROPS-Home | Fulltext Search | Imprint Published by LZI