PaperlayerView exact revision on arXiv

Source-linked AI summary

Accuracy Measures for the Comparison of Classifiers

Vincent Labatut, Hocine Cherifi

arXiv:1207.3790v1cs.LG

TL;DR

Choosing a classifier for a dataset requires deciding how to measure performance, and this paper examines popular accuracy measures for that task. It finds that many measures are equivalent, difficult to interpret, or unsuitable, and recommends overall success rate or marginal rates for this specific purpose.

  • Problem

    Selecting a classifier for a dataset requires choosing among many heterogeneous accuracy measures, whose suitability depends on the classification context and objectives.

    Source: §I · p. 1

  • Method

    The paper reviews accuracy measures and assesses their relevance to empirical classifier selection on a given dataset with discrete classifier outputs.

    Source: §I · p. 1

  • Results

    Measures with monotonic relationships produce the same classifier ordering, while association measures can pair low accuracy with high association and some measures are difficult to interpret.

    Source: A. Nominal Association Measures · p. 2, §V · p. 5

  • Takeaways & Limitations

    For overall accuracy, the authors favor the overall success rate; for a specific class, they recommend TPR and PPV or a meaningful combination such as the F-measure.

    Source: §V · p. 5

    The analysis focuses on comparing classifiers with discrete outputs rather than outputs expressed as real scores or probabilities.

    Source: §I · p. 1

Abstract

from arXiv · show

The selection of the best classification algorithm for a given dataset is a very widespread problem. It is also a complex one, in the sense it requires to make several important methodological choices. Among them, in this work we focus on the measure used to assess the classification performance and rank the algorithms. We present the most popular measures and discuss their properties. Despite the numerous measures proposed over the years, many of them turn out to be equivalent in this specific case, to have interpretation problems, or to be unsuitable for our purpose. Consequently, classic overall success rate or marginal rates should be preferred for this specific task.

I. INTRODUCTION

Comparing classification algorithms requires choices about what performance means, how to measure it, and how to compare the measurements. This paper examines accuracy measures for selecting an algorithm on a given dataset and assesses their relevance to that specific task.

  • Classifier comparison requires defining performance, measuring it appropriately, and choosing a consistent way to compare measured values.Performance may refer to accuracy, speed, cost, or readability.Source: §I · p. 1
  • Accuracy measures vary in purpose, terminology, interpretation, and the situations to which they apply.Some focus on particular aspects of classification results, and some lack a clear interpretation.Source: §I · p. 1
  • No single measure is best in every situation, so the appropriate choice depends on the classification context and objectives.The large, heterogeneous set of measures makes that choice difficult, and selection may follow field tradition rather than suitability.Source: §I · p. 1
  • The paper focuses on empirical selection of a classifier for a given dataset, rather than on methods for comparing performance values.It considers classifiers with discrete outputs and discusses existing accuracy measures in that specific context.Source: §I · p. 1
  • Earlier comparisons include context-independent correlation studies and work in remote sensing, while this paper evaluates measures for its own application context.The introduction notes that prior context-specific studies addressed settings different from this one.Source: §I · pp. 1–2
  • The paper proceeds from notation to a review of classification accuracy measures and then assesses their relevance to the target task.Source: §I · p. 2

II. NOTATIONS AND TERMINOLOGY

The section establishes confusion-matrix notation and distinguishes correct from incorrect classifications, including the four per-class outcome types.

  • The confusion matrix places estimated classes in rows and true classes in columns.Source: §II · p. 2
  • Diagonal entries represent correctly classified instances, while off-diagonal entries represent incorrect classifications.The matrix entries count instances assigned to an estimated class that belong to a particular true class.Source: §II · p. 2
  • For a particular class, TP and FP are instances classified as that class correctly and incorrectly, while TN and FN are instances correctly and incorrectly classified as not that class.Source: §II · p. 2

III. SELECTED MEASURES

The section reviews nominal association measures and explains why association does not necessarily indicate classification accuracy.

A. Nominal Association Measures

  • Association measures quantify the strength or magnitude of a relationship and have been applied to classification accuracy using true- and estimated-class distributions.Examples include chi-square-based, Yule, correlation, error-reduction, and mutual-information measures.Source: A. Nominal Association Measures · p. 2
  • In this setting, association measures compare the estimated-class distribution with the true-class distribution, but measure a more general relationship than class agreement.A high association can mean estimated classes are predictable from true classes without matching them.Source: A. Nominal Association Measures · p. 2
  • A confusion matrix can show low accuracy alongside high association, making association measures unsuitable for assessing accuracy.Source: A. Nominal Association Measures · p. 2

B. Overall Success Rate

The section describes overall success rate and class-specific marginal rates as interpretable measures of classification performance.

B. Overall Success Rate

  • Overall success rate is the confusion-matrix trace divided by the number of classified instances.It is a multiclass, symmetric measure whose value corresponds to the observed proportion of correctly classified instances.Source: B. Overall Success Rate · p. 2

C. Marginal Rates

  • TPR and TNR are reference-oriented rates based on true classes, whereas PPV and NPV are estimation-oriented rates based on predicted classes.TPR and TNR concern type II and type I errors, respectively; PPV and NPV concern type I and type II errors, respectively.Source: C. Marginal Rates · p. 2
  • TPR and TNR describe correct classification among instances belonging or not belonging to a class, while PPV and NPV describe correctness among instances predicted to belong or not belong.The four rates have straightforward interpretations and range from 0 to 1.Source: C. Marginal Rates · p. 3

D. F-measure and Jaccard Coefficient

The F-measure and Jaccard coefficient are class-specific symmetric measures that quantify overlap between estimated and true classes. They are linearly related and differ in their formulation.

D. F-measure and Jaccard Coefficient

E. Classification Success Index

The section describes ICSI and CSI as classification-oriented success measures, then reviews chance-corrected agreement coefficients. The authors caution that agreement corrections may not fit classifier evaluation, where true classes are known and serve as the reference.

E. Classification Success Index

F. Agreement Coefficients

  • Agreement coefficients subtract estimated chance agreement from observed agreement, with different coefficients using different chance-agreement estimates or assumptions.Source: F. Agreement Coefficients · p. 3
  • Inter-rater chance corrections may not suit classifier accuracy because true classes are known and the classifier is evaluated against a reference whose position matters.Source: F. Agreement Coefficients · p. 3

G. Ground Truth Index

The Ground Truth Index models classification as a mixture of always-correct and random predictions, with its estimated coefficient interpreted as the share always classified correctly. Its use is constrained by class count, perfect classification, and a quasi-independence assumption.

G. Ground Truth Index

  • GTI models a classifier as an always-correct component mixed with a random component and interprets its estimate as the proportion always classified correctly.Source: G. Ground Truth Index · p. 3
  • GTI has both overall and class-specific versions, depending on whether the model assumes the always-correct proportion is shared across classes.Source: G. Ground Truth Index · p. 3
  • GTI cannot be used with fewer than three classes or on perfectly classified data, and it relies on quasi-independence, which is rarely met in practice.Source: G. Ground Truth Index · p. 3

IV. DISCUSSION

For classifier ranking, several accuracy measures yield the same ordering because they are monotonically related. The discussion also distinguishes class-focused assessment from overall measures and describes weighting confusion-matrix cells as a flexible way to represent error importance.

A. Class Focus

  • Class-specific measures suit analyses focused on a particular class, while most basic overall measures treat classes as equally important.Source: A. Class Focus · pp. 3–4
  • Averaging class-specific values can make an overall measure hard to interpret, whereas merging other classes to assess one class preserves the overall measure’s interpretation.Source: A. Class Focus · p. 4
  • Weighting confusion-matrix cells within an unweighted overall measure can distinguish the importance of individual classes and types of classification error.Source: A. Class Focus · p. 4

B. Functional Relationships

  • F-measure and Jaccard’s coefficient, and ICSI and Kulczynski’s measure, each produce the same classifier order because each pair is monotonically related.Source: B. Functional Relationships · p. 4
  • Sensitivity-precision product, F-measure, Kulczynski’s measure, Jaccard’s coefficient, and ICSI are equivalent for ranking because they combine the same marginal rates monotonically.Source: B. Functional Relationships · p. 4
  • Measures that combine TPR and TNR also have the same ranking behavior when monotonically related.Source: B. Functional Relationships · p. 4

C. Range

Fixed bounds are often valued because they support comparisons across datasets and absolute interpretation. For ranking classifiers on the same data by relative accuracy, however, the range is unnecessary, and measures differing only by scale are equivalent.

C. Range

  • Perfect classification and perfect misclassification should correspond to the upper and lower bounds of an accuracy measure, respectively.Source: C. Range · p. 4
  • Fixed bounds are generally considered favorable because they enable comparisons across datasets without scale normalization and make measured features easier to interpret absolutely.The bounds may be fixed or depend on the processed data.Source: C. Range · p. 4
  • For classifiers evaluated on the same data, only relative accuracy differences matter, so fixed bounds are not necessary.Measures normalized to different intervals can be rescaled, making some apparently distinct measures equivalent apart from scale.Source: C. Range · p. 4

D. Interpretation

Classifier rankings require accuracy measures that can be interpreted in the relevant context. For this comparison task, the paper discusses why chance correction is needed but finds existing approaches problematic.

D. Interpretation

  • Measured accuracies can rank classifiers on a dataset, but interpreting the criterion matters because PPV and TPR focus on different error types.Source: D. Interpretation · p. 4
  • Accuracy measures vary in interpretability: some have clear analytical meanings, while ad hoc measures or mixtures may lack a direct interpretation.The paper notes that interpretations can be questioned or have alternatives, as with agreement coefficients.Source: D. Interpretation · p. 4

E. Correction for Chance

  • Chance correction seems necessary when generalizing sample accuracy to the population, to distinguish reproducible success from lucky guesses on the test sample.Source: E. Correction for Chance · p. 4
  • Corrections based only on true class distribution preserve OSR ordering, while estimated-class corrections are more relevant but require modeling classifier-specific lucky guesses.Designing a specific model for each classifier seems difficult or even impossible.Source: E. Correction for Chance · pp. 4–5

V. CONCLUSION

For comparing classifiers on a data sample, many differences among accuracy measures do not affect relative rankings. The paper favors simple, interpretable measures for this setting and identifies real-valued outputs as a separate case for future work.

V. CONCLUSION

  • The review concerns comparing algorithms by testing them on a data sample to identify which is best on the sampled population.Source: §V · p. 5
  • Monotonically related measures produce the same algorithm ordering, and measure range matters little when comparing relative values.Source: §V · p. 5
  • Class-weighted measures can be discarded because class importance can instead be distinguished with an unweighted multiclass measure, while combined measures may be difficult or impossible to interpret.Chance correction appears necessary for this purpose, but the paper finds no existing chance estimation relevant.Source: §V · p. 5
  • For overall accuracy assessment, the paper finds OSR most adapted; for a specific class, it recommends TPR and PPV together or a meaningful combination such as F-measure.A weight matrix can specify differences between classes or errors.Source: §V · p. 5
  • The authors plan to extend the work to classifiers with real-valued outputs, which allow additional measures such as AUC and error measures.Source: §V · p. 5
Loading 1207.3790v1…