Analysis and Comparison of Classification Metrics

• Application-relevant evaluation metrics are essential because error costs, class priors, and deployment conditions can differ across use cases.Source: §1 · pp. 1–2
• Expected cost generalizes standard and balanced error rates while offering more intuitive, flexible, and interpretable evaluation of hard decisions.Source: §1 · p. 2
• Proper scoring rules provide a principled family for evaluating posterior probabilities and expose calibration quality.Source: §1 · p. 2
• The paper compares metrics theoretically and empirically, with code available for reproducing its plots and tables.Source: §1 · p. 2

2.1 Expected cost

• Expected cost assigns application-defined costs to each class–decision pair and averages them over evaluation samples, using class-conditional decision rates and priors.Source: §2.1 · pp. 3–4

2.1.1 Normalized Expected Cost

• Normalized expected cost divides EC by the best constant-decision baseline, so values above 1 indicate performance worse than a naive system.Source: §2.1.1 · pp. 4–5

2.1.2 The Special Case of Binary Classification with Square Cost Matrix

• In binary classification with a square cost matrix, NEC depends on one free parameter combining priors and costs.Source: §2.1.2 · p. 5

2.2 Error Rate and Accuracy

• Standard error rate is a 0-1-cost special case of EC, while balanced error rate is another EC special case that weights each class equally through inverse-frequency costs.Source: §2.2 · pp. 5–6
• Unlike error rate and balanced error rate, general EC can assign distinct costs to error types according to their application-specific severity.Source: §2.2 · p. 6

2.3 Net Benefit

• Net benefit is directly related to normalized EC, but lacks NEC’s reference interpretation that 1.0 matches the best naive system.Source: §2.3 · p. 6

2.4 F-beta score

• F-beta is related to EC with class-dependent costs, but can reward labeling more samples as the class of interest even when those detections are not correct.Source: §2.4 · pp. 6–7
• F-beta values should not be compared across datasets with different class priors because changing priors changes the effective weighting of errors.Source: §2.4 · p. 7

2.5 Matthews correlation coefficient

• MCC is related to NECb, becoming negative when NECb exceeds 1, while differing from 1 − NECb by a system-dependent factor.The factor depends on the balance of system decisions, not only on dataset priors.Source: §2.5 · p. 8
• For systems with the same NECb below 1, MCC favors more imbalanced decisions, even when the dataset itself is balanced.The paper characterizes this as undesirable because decision frequency is not inherently a quality criterion.Source: §2.5 · pp. 8–9

2.7.1 EC vs MCC

• Figure 2 compares MCC and NECb across balanced and imbalanced datasets, while Tables 1 and 2 provide selected metric values for those systems.The plots vary confusion-matrix error counts across possible systems; lower NEC is better, whereas higher MCC and Fβ are better.Source: §2.7 · p. 9, §2.7.1 · p. 10, Figure 2 · p. 10

2.7.2 EC vs Fβ

• NECβ2=1 and Fβ=1 correlate for good systems but diverge as systems degrade, so metric choice can change which system appears best.For P1 = 0.1, a system with N21 = 10 and N12 = 90 has NECβ2=1 = 1 but Fβ=1 = 0.62, despite the best naive system having Fβ=1 = 0.18.Source: §2.7.2 · pp. 11–12
• NECβ2=2 correlates better with Fβ=1 for good systems while directly rewarding a lower H2 error rate rather than a high H2 decision rate.The example system has NECβ2=2 = 0.55, significantly better than the best naive system, and the paper recommends choosing NEC parameters directly for the application.Source: §2.7.2 · p. 12

2.8 Discussion on Metrics for Hard Decisions

• EC generalizes standard and balanced error rates by allowing application-specific costs and priors, decisions different from classes, and direct multi-class use.Its normalized form has an intuitive reference: values above 1 indicate performance worse than a naive constant-decision system.Source: §2.8 · p. 12
• The paper recommends EC or NEC as the preferred metric for evaluating hard decisions because its error contributions and priors are directly interpretable and adjustable.It also supports optimal decision-making and calibration analysis.Source: §2.8 · p. 12

3 Bayes Decision Theory

• Bayes decisions minimize expected cost by choosing, for each input, the decision with the smallest posterior-weighted cost.The expected cost is taken over the joint distribution of inputs and true classes, while the decision rule minimizes the inner cost sum for each input.Source: §3 · p. 13

3.1 Calibration

• Calibration means that no transformation of a classifier’s posterior outputs can produce lower expected cost on the evaluation data.The paper notes that empirical calibration assessment depends on the calibration transformation and selected cost function.Source: §3.1 · pp. 13–14

3.2 Special cases

• For a 0-1 cost matrix, expected cost becomes one minus accuracy and Bayes decisions reduce to the standard argmax rule.With unequal costs or different decision and class sets, the corresponding cost-sensitive Bayes rule is required instead.Source: §3.2 · pp. 14–15
• In binary classification, Bayes decisions compare a posterior with a cost-dependent threshold, whereas optimizing Fβ or MCC requires threshold sweeping.The fixed threshold is optimal when scores are calibrated for the selected cost; otherwise, it must be tuned or the posteriors calibrated.Source: §3.2 · p. 15

3.3 Posterior Dependence on Class Priors

• Posterior quality depends on matching the priors used to produce scores with the target test priors; mismatch can make Bayes decisions suboptimal.The paper recommends likelihood-based prior adjustment or calibration using data with the relevant priors when direct likelihoods are unavailable.Source: §3.3 · pp. 15–16

4 Metrics for System Scores

• Score metrics assess classifier outputs independently of the subsequent decision-making stage.The distinction matters because systems commonly produce scores that are later converted into hard decisions.Source: §4 · p. 16

4.1 Expected Value of Proper Scoring Rules

• Expected values of proper scoring rules evaluate posterior scores by applying Bayes decisions for a chosen cost function and averaging the resulting single-sample costs.The construction is proper because its expected value is minimized when the score vector matches the reference class distribution.Source: §4.1 · p. 17

4.1.1 Bayes EC

• Bayes EC evaluates score quality for one operating point defined by the selected costs and priors, including argmax accuracy as a special case.Because it is non-strict, different posterior vectors can receive the same Bayes EC even when they differ under other costs or priors.Source: §4.1.1 · pp. 17–18

4.1.2 Cross-entropy

• Cross-entropy is a strict PSR that heavily penalizes confident incorrect predictions, making it an effective default for posterior probabilities.Its normalized form exceeds 1.0 when the system is worse than the best naive prior-only system.Source: §4.1.2 · pp. 18–19

4.1.3 Brier Score

• The Brier score is a strict PSR based on squared distance between predicted posteriors and the true-class indicator vector.Unlike cross-entropy, Brier score is bounded by 1, while cross-entropy assigns infinite penalty as the true-class probability approaches zero.Source: §4.1.3 · p. 19

4.2 Calibration Loss using the Expected Value of PSRs

• Calibration loss is the difference between the EPSR of raw scores and the EPSR after a calibration transformation.The procedure evaluates raw scores, calibrates them, reevaluates them with the same EPSR, and subtracts the calibrated estimate from the raw value.Source: §4.2 · p. 20
• The decomposition separates discrimination performance from calibration loss, guiding whether development should target score calibration or the classifier itself.Large calibration loss suggests calibration can improve performance, whereas low calibration loss with large residual EPSR points to classifier improvement.Source: §4.2 · p. 20
• Calibration loss depends on the calibration approach, because a better or worse calibrator changes the estimated improvement and can even produce a negative value.The paper makes the assumptions, posterior model, and calibration dataset explicit and flexible.Source: §4.2 · pp. 20–21

4.2.1 Calibration Approaches

• Calibration models should be trained separately from evaluation data, since train-on-test calibration can overfit and exaggerate the raw-to-calibrated gap.The paper particularly discourages this practice with many classes or small evaluation sets.Source: §4.2.1 · p. 21
• The binary square-cost case permits threshold calibration instead of posterior calibration, but this shortcut does not generalize to other cases.The threshold must be selected on held-out data to avoid optimistic results.Source: §4.2.1 · p. 22

4.3.1 Calibration Loss using Brier loss versus ECE

• ECE is conceptually similar to calibration loss because both average functions of raw and calibrated scores over evaluation samples.Source: §4.3.1 · p. 23

4.3.2 Discussion on ECE’s drawbacks

• Calibration loss indicates how much the raw proper-scoring-rule performance could improve through calibration, whereas ECE does not directly quantify that gain.A large CalLoss suggests adding a calibration stage; low CalLoss with high raw EPSR points instead to limited discrimination.Source: §4.3.2 · p. 23
• ECE estimates calibration through histogram binning, whose test-data fitting and bin choice can cause overfitting, instability, or misleadingly low values.A single bin can produce ECE equal to 0, while fixed bin counts may under- or overestimate calibration loss across datasets.Source: §4.3.2 · pp. 23–24
• Multiclass ECEmc evaluates only maximum confidence and can miss severe miscalibration in the remaining posterior values or under non-0-1 decision costs.The paper therefore favors evaluating the full posterior vector with proper-scoring-rule-based metrics.Source: §4.3.2 · p. 24
• ECE also insufficiently penalizes extreme errors because a zero posterior for the true class contributes at most 1 to its average.Source: §4.3.2 · p. 24
• The paper recommends estimating calibration transformations on held-out data and evaluating them with proper scoring rules, such as cross-entropy.This preserves decomposition of overall performance and makes calibration assessment more interpretable and actionable.Source: §4.3.2 · p. 24

4.4 Metrics for Assessing Interpretability of Scores

• A classifier that outputs class priors for every sample can have zero calibration error while providing no information about individual inputs.Source: §4.4 · p. 25
• Proper scoring rules evaluate how close posteriors are to the data distribution, so a lower EPSR can indicate more interpretable scores even when calibration error is higher.Source: §4.4 · p. 25
• The paper treats calibration-only metrics as development diagnostics for deciding whether to add calibration, while EPSRs should evaluate systems when good posteriors are required.Source: §4.4 · p. 25

4.5 Area Under the Curve and Equal Error Rate

• AUC summarizes ROC or precision–recall curves, while EER is the ROC operating point where the two error rates are equal.Source: §4.5 · p. 25
• Under perfect calibration, fEC with the 0-1 cost matrix is upper-bounded by min(EER, P1, P2), whereas without calibration fEC can grow unbounded while EER cannot.Source: §4.5 · p. 25
• ROC-, PR-, AUC-, and EER-based metrics are invariant to monotonic score transformations, so they ignore calibration and implicitly assume optimal threshold selection.Source: §4.5 · p. 25
• AUC and EER do not suffice for evaluating Bayes decisions, interpretable posterior probabilities, or multiclass and non-class decision sets.The paper recommends proper scoring rules for those settings.Source: §4.5 · p. 25

4.6 Specificity at Fixed Sensitivity

• SP@SE recalculates a threshold on each dataset or system, although deployment requires a predefined threshold applied blindly to new data.Source: §4.6 · p. 26
• Because each recalculated SP@SE threshold corresponds to a different expected cost, the metric changes its implicit error-cost assumptions across systems or datasets.Source: §4.6 · p. 26
• SP@SE forces every system to operate at the minimum acceptable sensitivity, preventing better systems from selecting a more favorable error trade-off.Source: §4.6 · p. 26
• A normalized expected cost can encode the target sensitivity using a baseline system, then support threshold selection and comparison across datasets.The procedure optimizes NEC over thresholds for each α, chooses α achieving the target sensitivity, and uses that NEC thereafter.Source: §4.6 · p. 26
• The NEC selected from a baseline may fail to achieve the target sensitivity on another system or dataset, requiring recalibration of α or declaring that system inadequate.Source: §4.6 · p. 27
• The selected sensitivity target implicitly specifies an error-cost ratio, which can be inspected for reasonableness before adoption.For the development set, the ratio is c12/c21 = αP2/P1.Source: §4.6 · p. 27

• One-vs-other evaluation computes a binary metric for each class and averages the results.Each binary problem distinguishes one class from all other classes.Source: §4.7 · p. 27
• This approach permits zero or multiple positive detections, unlike the original classification task, which requires a class decision.Consequently, the averaged metric may not correlate with deployed classification performance.Source: §4.7 · p. 27
• Averaged binary metrics are not recommended for classification, but are appropriate when the actual task permits multiple binary detections.Source: §4.7 · p. 27

4.8.1 Threshold Selection for Binary Classification

• 0.366 versus 0.604: for miscalibrated LR-mc1 scores, the best NECβ2=2 outperforms the Bayes threshold, revealing poor calibration.The Bayes threshold is suboptimal because well-calibrated scores would align it with the optimal threshold.Source: §4.8.1 · p. 29, Figure 3 · p. 29
• For calibrated LR-cal scores, Bayes and best NEC thresholds nearly coincide, producing almost identical NEC values.The best NECβ2=2 and Fβ=1 thresholds remain very similar.Source: §4.8.1 · p. 29, Figure 3 · p. 29

4.8.2 EC with an Abstain Decision for Binary Classification

• As abstention cost α decreases below 0.5, abstentions increase because the system abstains when neither posterior exceeds 1 −α.For α above 0.5, abstention never occurs.Source: §4.8.2 · p. 30
• Calibrated scores yield lower EC and NEC than miscalibrated scores, while adding abstention reduces EC whenever abstaining costs less than choosing a class.The normalized EC need not decrease monotonically because its naive-system normalization changes with α.Source: §4.8.2 · p. 30, Table 3 · p. 31

4.8.3 PSRs and Calibration Analysis

• Scaling log-posteriors without changing ranking leaves 0-1 NEC unchanged, but mismatched-prior posteriors raise total error from 0.027 to 0.111.These errors correspond to 97.3% and 88.9% accuracy, respectively; NEC above 1 identifies performance worse than the naive baseline.Source: §4.8.3 · p. 31, Table 4 · p. 32
• Affine calibration restores perfect calibration for mc1 and mismatched-prior posteriors, whereas temperature scaling succeeds for mc2 but not those broader distortions.Temperature scaling assumes miscalibration is restricted to a log-posterior scaling factor.Source: §4.8.3 · p. 31
• Cross-entropy and Brier calibration loss detect mismatched-prior errors that ECE misses because ECE evaluates only decision confidences rather than the full posterior vector.The mismatched-prior system changes only 10% of decisions, while its posterior vectors are wrong for every sample.Source: §4.8.3 · p. 32, Table 5 · p. 32

• Posterior probabilities provide interpretable outputs and enable Bayes decision theory for optimal categorical decisions.Source: §4.9 · p. 33
• ECE is limited because its absolute value is not directly interpretable, its multiclass form evaluates confidences rather than full posterior vectors, and it relies on fragile histogram binning.The paper retains ECE as a development aid for assessing whether calibration may help.Source: §4.9 · p. 33
• Expected PSRs evaluate posterior quality and calibration, while calibration loss compares actual PSR performance with the best calibrated value.The paper presents calibration loss as a principled replacement for ECE.Source: §4.9 · p. 33
• AUC and EER measure discrimination only and are invariant to invertible score transformations, so they ignore calibration problems.When deployed hard-decision performance matters, calibration-sensitive expected PSRs are preferred.Source: §4.9 · p. 33

• Expected cost unifies hard-decision metrics, while standard and balanced error rates are special cases of EC.The paper also reviews accuracy, F-beta score, and related metrics.Source: §5 · p. 33
• Expected PSRs, including cross-entropy, evaluate posterior probabilities and support calibration assessment for Bayes decisions.Source: §5 · p. 33
• The paper argues that EC is superior to common hard-decision metrics and that PSR-based calibration loss improves on ECE in interpretability and generality.The claims are supported by theoretical and practical arguments.Source: §5 · pp. 33–34
• An open-source repository provides code to compute the metrics and replicate the paper’s plots and tables.Source: §5 · p. 34