Source-linked AI summary
A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice
Juri Opitz
TL;DR
The paper addresses unclear and weakly justified metric selection in classification evaluation, especially around “macro” metrics. It analyzes common metrics from bias and prevalence concepts through formal properties, finding distinct behaviors and arguing for more informed, transparent selection.
Problem
Metric choices can affect system rankings, yet papers often leave metric expectations and “macro” terminology unclear or weakly justified.
Method
The paper defines five metric properties and analyzes common metrics using classifier bias, class prevalence, and confusion-matrix behavior.
Results
The analysis distinguishes common metrics by prevalence invariance, monotonicity, and chance correction, with complete class-count-independent chance correction achieved only by MCC and Kappa.
Takeaways & Limitations
The framework supports more informed and transparent decisions when selecting classification evaluation metrics.
Takeaways & Limitations
The idealized score approximation does not account for complex phenomena such as domain shifts.
Abstract
from arXiv · showhide
Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many works use so-called 'macro' metrics to rank systems (e.g., 'macro F1') but do not clearly specify what they would expect from such a `macro' metric. This is problematic, since picking a metric can affect research findings, and thus any clarity in the process should be maximized. Starting from the intuitive concepts of bias and prevalence, we perform an analysis of common evaluation metrics. The analysis helps us understand the metrics' underlying properties, and how they align with expectations as found expressed in papers. Then we reflect on the practical situation in the field, and survey evaluation practice in recent shared tasks. We find that metric selection is often not supported with convincing arguments, an issue that can make a system ranking seem arbitrary. Our work aims at providing overview and guidance for more informed and transparent metric selection, fostering meaningful evaluation.
1 Introduction
The introduction argues that classification metrics compress confusion matrices while metric choices and “macro” terminology often remain insufficiently justified. The paper aims to clarify metric properties and support more transparent selection.
- Classification evaluation compares predictions with reference labels through a confusion matrix and an aggregate metric.
- Reducing a confusion matrix to one number necessarily loses information, so no single metric can be perfect.
- Metric selection can change which system ranks better, yet papers often justify choices only with weak arguments about imbalance or equal class weight.
- Blurry uses of “imbalanced data” and “macro” leave unclear what expectations metrics address and can lead readers and authors to misconceive results.
- The paper provides a reference for understanding metric alignment with stated expectations and constructing arguments for or against particular metrics.
- Its analysis covers common metrics, metric variants, discussion, and metric selection in shared tasks.
2 Preliminaries
The preliminaries formalize classifiers, confusion matrices, metrics, and basic class-wise quantities. A metric maps the confusion matrix to a score used to rank classifiers.
- A classifier maps inputs to classes, while a confusion matrix records each prediction–label combination.
- Diagonal confusion-matrix entries represent correct predictions, whereas off-diagonal entries represent specific errors.
- A metric maps a confusion matrix to a scalar score, with higher scores indicating preferable classifiers.
- The framework defines five basic quantities for analyzing evaluation metrics.
- Class precision concerns the probability of observing a correct class given a specific prediction.
- Class recall approximates the probability of a correct prediction given an input from a particular class.
3 Defining metric properties
The paper defines five properties for distinguishing metrics: monotonicity, class sensitivity, class decomposability, prevalence invariance, and chance correction. These properties formalize expectations about correctness, class treatment, and baselines.
- The framework evaluates metrics using monotonicity, class sensitivity, class decomposability, prevalence invariance, and chance correction.
- Monotonicity: Monotonicity requires scores not to decrease when correct mass increases and not to increase when error mass increases.
- Class sensitivity: A macro metric must be sensitive to classes, whereas a micro metric should depend only on whether predictions are correct.
- Class decomposability: Class decomposability represents a macroaverage as an unweighted mean of class-specific scores.
- Prevalence invariance: An unweighted class mean does not ensure equal treatment because each class-specific score can still depend on other classes’ prevalence.
- Prevalence invariance: Prevalence calibration changes class prevalence while retaining within-class error proportions, enabling strict testing of prevalence invariance.
- Chance correction: Chance correction requires metrics to provide known baseline behavior for random classifiers, with stricter forms controlling dependence on class count.
4 Metric property analysis
The paper analyzes common classification metrics through five properties, showing that their assumptions about prevalence, chance, class weighting, and error structure differ substantially. These differences make metric choice consequential for interpreting classifier comparisons.
- Scope: The analysis covers Accuracy, Macro Recall, Macro Precision, two Macro F1 variants, Weighted F1, Kappa, and MCC.The metrics are evaluated using the paper’s five defined properties.
- Accuracy: Accuracy has only monotonicity and is equivalent to micro Precision, micro Recall, and micro F1 in multi-class evaluation.Accuracy is tied to the class prevalences in the evaluated data set.
- Macro Recall: Macro Recall satisfies all five properties, including prevalence invariance, and is strictly chance corrected with ω(n) = 1/n.It can also be interpreted as prevalence-calibrated Accuracy.
- Macro Precision: Macro Precision lacks prevalence invariance, so score differences can become difficult to interpret when class prevalences or classifier biases change.Prevalence calibration produces a version detached from the class distribution of a specific data set.
- Macro F1: Macro F1 combines macro and micro features but is not prevalence-invariant and does not strictly correct for chance.It favors similar prediction and class-prevalence distributions while also rewarding correctness across classes.
- F1 variants: The two metrics called Macro F1 can diverge by up to 0.5, while Weighted F1 is prevalence-weighted and reduces to Macro F1 after prevalence calibration.Weighted F1 is non-monotonic and has lower interpretability than Accuracy according to the discussion.
- Kappa and MCC: MCC and Kappa provide complete chance correction but are non-monotonic, non-decomposable by class, and not prevalence-invariant.Their stronger dependence on classifier bias can favor uneven biases and reduce interpretability.
5 Metric variants
The paper proposes metric variants by changing how class-wise scores are aggregated and by calibrating prevalence. These variants can emphasize consistent performance across classes or move metrics toward a more macro-oriented interpretation.
- Generalized means: Using generalized means, geometric Macro Recall estimates the probability that randomly selected items from all classes are correctly predicted.This makes it useful when correct performance in every class matters.
- Generalized means: Harmonic Macro Recall emphasizes low class-wise recall and can be used when performance across all classes must remain strong.It is defined through the harmonic mean of the class-wise recall values.
- Prevalence calibration: Prevalence calibration enforces prevalence invariance by standardizing the confusion matrix so that every class has the same prevalence.Under calibration, Kappa and Accuracy reduce to Macro Recall, while Weighted F1 becomes Macro F1.
- Prevalence calibration: Prevalence calibration is appropriate when the goal is to push a metric toward more macro features.It treats classes more equally by detaching scores from the original class distribution.
6 Discussion
The metric analysis distinguishes several notions of balance and shows that metric behavior depends on prevalence calibration, chance correction, and how correctness is distributed across classes.
- Metric properties: Macro Recall has all five analyzed properties, including strict class prevalence invariance, while prevalence calibration gives this property to all metrics.Class prevalence invariance means treating all classes equally in the strict sense.
- Metric properties: Kappa, MCC, and weighted F1 lack monotonicity, so under some circumstances increasing errors can increase their scores.All metrics except Accuracy and weighted F1 show some chance-baseline correction; complete class-count-independent correction is achieved only by MCC and Kappa.
- Metric balance: Macro Recall acts as a prevalence-calibrated version of Accuracy, whereas macro F1 combines high recall across classes with empirical dataset correctness.Macro F1 does not strictly correct for chance in the same way as Macro Recall.
- Metric comparison: MCC and Kappa are similar, while Accuracy and weighted F1 are both strongly affected by class prevalence and lack a clear reason to prefer weighted F1.The discussion also proposes averaging a selected set of metrics when no single metric is clearly justified.
- Value of class-wise Recall: Projected macro-metric scores based on class-wise recall are idealized and do not account for domain shifts.New recall values cannot be projected from old precision scores because precision does not transfer in the same way.
7 Reflecting on SemEval shared tasks
The SemEval analysis shows that different metrics can produce different system rankings, while a survey of shared-task papers finds that metric definitions and justifications are often missing or weak.
- 7.1 Example shared task study: Different metrics can yield different rankings: in the sentiment task, the official winners were selected with macro Recall, but system E had the highest Accuracy.System E also had the most even class recalls, with R1 = 69.8, R2 = 64.0, and R3 = 66.8, compared with system B’s maximum recall gap of 36.4.
- 7.1 Example shared task study: Only four metric pairs completely agreed in ranking all 37 teams, while class-2 recall disagreed with every other metric at ρ ≤14.Macro Recall agrees with calibrated Kappa and calibrated Accuracy because they are equivalent after calibration.
- 7.1 Example shared task study: The first-listed shared-task system had almost twice the citations of the second-listed winner, despite the latter performing better under most metrics.The authors present citation counts as a very coarse proxy for popularity and note that a case could also be made for system E.
- 7.2 Examining metric argumentation: Among 42 recent classification shared-task overview papers, 23.8% provided a metric formula and only 10.9% supplied a sensible metric argument.A further 14.3% used a weak argument, while 73.8% gave no argument or relied on a trope such as “As is standard, ...”.
- 7.2 Examining metric argumentation: ‘Macro F1’ was the most frequent metric, but inconsistent naming and missing formulas sometimes made it unclear which macro-F1 variant had been used.The surveyed ambiguity included macro-average F1, mean F1, macro F1, a distinct macF1′, and balanced Accuracy.
- 7.2 Examining metric argumentation: The broader situation beyond SemEval remains only cursory, although similar unclarities appear across research domains.The paper explicitly limits its characterization beyond SemEval rather than presenting a precise broader survey.
8 Recommendations
The authors recommend transparent, context-sensitive metric selection rather than assuming one universally best metric. They advise justifying formulas, reporting complementary measures, and allowing multiple winners when rankings are metric-dependent.
- State the evaluation metric clearly, preferably with a formula, to avoid ambiguity between similarly named measures.
- Build a case for metric choice using expected application class distributions and desirable metric properties.Greater uncertainty may warrant considering more macro-metric features and consulting the metric analyses.
- Present complementary metrics such as Accuracy and macro Recall to distinguish empirical data-set correctness from robustness to class-distribution shifts.For few classes, class-wise recall can indicate generalizability; for many classes, report variance when the metric is decomposable over classes.
- When no strong case supports one metric, report multiple well-motivated metrics and admit multiple best systems.If one winner is required, averaging across the selected metrics is suggested as a heuristic.
9 Background and related work
The paper situates its metric analysis within broader surveys and prior two-class work while maintaining an agnostic stance toward universally superior metrics. It also notes that specialized label structures and evaluation goals may require methods beyond standard classification metrics.
- Existing surveys and book chapters introduce classification evaluation, while deeper metric analyses have focused mainly on the two-class setting.
- The K-metric can be understood as two-class macro Recall, similarly to Powers’s Informedness.
- The paper adopts an agnostic stance because no perfect metric exists and different cases may call for different choices.Its analysis balances intuitive interpretation with formal metric analysis and desiderata expressed in prior work.
- Hierarchical, ordinal, ambiguous, or unknown class labels can require evaluation methods beyond ordinary classification metrics.The paper also situates precision–recall and receiver-operating-characteristic curves among alternative evaluation approaches.
10 Conclusion
The paper develops a bias- and prevalence-based framework for analyzing common classification metrics and uses it to guide more informed metric selection. Its conclusion emphasizes transparent decisions without claiming a universally best metric.
- The framework starts from classifier bias and class prevalence to analyze common metrics and clarify expectations behind macro evaluation.It is intended to support analysis of existing and new metrics.
- Accuracy has only monotonicity among the analyzed properties, while macro Recall satisfies all five properties, including prevalence invariance.In multiclass evaluation, Accuracy equals micro Precision, micro Recall, and micro F1.
- Weighted F1 can increase even when a classifier makes an error, and MCC can increase when more errors are added in some configurations.
G.1 Monotonicity
The analysis examines whether Kappa and MCC behave monotonically as confusion-matrix entries change. It identifies configurations where both metrics approach zero and an example where MCC increases after adding errors.
- The analysis uses non-normalized confusion matrices, with m_ij denoting the number of data examples and r representing a summed quantity.
- The setup defines z_ij as prevalence(i) + bias(j) before analyzing metric behavior under changes to confusion-matrix entries.
- There are confusion-matrix configurations where Kappa or MCC approach zero under the analyzed conditions.
- MCC increases when more errors are added in the example comparing Table 7 with Table 8.Table 7 reports MCC = 0.0 and Table 8 reports MCC = 0.07.