Source-linked AI summary
aggmetrics: Aggregating Classification Performance Measures for Machine Learning
Ronaldo Cristiano Prati
TL;DR
Classifier evaluation requires choosing both a measure and how to aggregate it, but fixed averaging interfaces can constrain combinations and yield misleading values in some settings. aggmetrics applies common aggregation operators to binary measures and tracks or infers structural properties to support diagnostics. Its inference check agrees with analytic properties for 21 built-in measures and reveals two hand-annotation errors.
Problem
Fixed averaging arguments attached to a limited measure list can require hand-written combinations, and some measures behave unexpectedly in particular evaluation settings.
Source: §1 · p. 1Method
aggmetrics reduces confusion-matrix counts with micro, macro, exemplar, or fold operators before applying a binary measure, while recording or inferring structural properties used by diagnostics.
Source: §2 · p. 2Results
Inference agreed with analytic properties for all 21 built-in measures, recovered coefficients within 6.1×10−15, and found two hand-annotation errors.
Source: §2 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
aggmetrics is a Python package that extends any binary performance measure expressed in the four confusion-matrix counts to multiclass, multilabel, ordinal, cost-sensitive, soft-label and cross-validation settings, by substituting one of four aggregation operators for what libraries usually expose as an average switch. The package also stores, for every measure, the structural properties that decide which identities apply to it, can infer those properties from a user-supplied formula, and uses them to warn when a measure is used where it does not behave as commonly assumed. Its test suite encodes the results of the companion theory as executable tests. The package is BSD-3 licensed, depends only on NumPy, and is available at https://github.com/rcprati/aggmetrics; documentation at https://aggmetrics.readthedocs.io.
1. Introduction
Classifier evaluation separately chooses a measure and an aggregation scheme, but common libraries restrict averages to fixed measure lists. Measures can also mislead in settings where their behavior is unsuitable; aggmetrics combines aggregation operators with measure-property checks to address these issues.
1. Introduction
- A classifier’s measure and its aggregation over classes, examples, or folds are separate choices, while scikit-learn’s fixed measure list can require hand-written combinations.A function may return a float even when the evaluation setting does not suit the measure.Source: §1 · p. 1
- On a 10-class problem, an always-wrong classifier scores 0.800 on label accuracy and −0.111 on micro-averaged MCC because each error still contributes eight true negatives.Measures depending on true negatives at tp = 0 can have a floor that grows with the number of classes.Source: §1 · p. 1
- On single-label data, micro-averaged precision, recall, and F1 all equal accuracy, while pooled micro balanced accuracy or MCC can fall outside the per-class range.In the latter case, no weighting of per-class values reproduces the pooled micro value.Source: §1 · p. 1
- aggmetrics treats micro, macro, exemplar, and fold-pooled averaging as instances of one operation and records measure properties that determine their behavior.Those properties can be queried and checked.Source: §1 · p. 1
2. Design
aggmetrics reduces confusion-matrix cell counts across classes, examples, or folds before applying a binary measure. It also records or infers measure properties and provides diagnostics and additional evaluation conventions.
2. Design
- Four operators reduce confusion-matrix counts across entries, columns, rows, or cross-validation splits, then substitute the reduced counts into a binary measure.The 21 registered measures and four operators yield 84 combinations through one code path; constructors accept class-index vectors, binary or probability matrices, and ordinal classes.Source: §2 · p. 2
- Each measure records decomposability, skew-invariance, true-negative dependence at tp = 0, and direction, supporting closed-form Bayes thresholds and attainable floors.The example query reports MCC’s attainable floor for 10 classes as −0.111.Source: §2 · p. 2
- User-supplied functions of (tp, fp, fn, tn) can be registered with properties inferred by solving and validating a homogeneous linear system and evaluating the other properties.The inferred measure can then be used with every operator.Source: §2 · p. 2
- Inference agrees with analytic properties for all 21 built-in measures, with recovered coefficients within 6.1×10−15; it also exposed two hand-annotation errors involving tn dependence at tp = 0.The errors were in error rate and false positive rate, which therefore flatter classifiers wrong everywhere.Source: §2 · p. 2
- diagnose(im) returns findings with codes, severity, messages, and triggering values for issues including out-of-range pooled micro values, inflated floors, and micro collapse.The checks also cover undefined entries, degenerate optima, one-vs-rest prior drift, and evaluation suites exceeding m(m −1) degrees of freedom.Source: §2 · p. 2
- Additional options specify four conventions for 0/0 entries, use t-norms for soft ground truth, apply cost matrices, and restrict operators to observed entries.Only the product t-norm preserves marginals; the package also provides Pareto dominance checks and scorers for scikit-learn model-selection tools.Source: §2 · p. 3
3. Example
A 10-class example shows that an always-wrong classifier can receive nonzero label accuracy and a nonzero micro-averaged MCC. The diagnostics identify an inflated floor and the collapse of micro precision, recall, and F1 to accuracy.
3. Example
- 0.800 on label accuracy and −0.111 on micro-averaged MCC are reported for a classifier wrong on every example, despite its accuracy being 0.000.The example uses 500 labels cycling through 10 classes and shifts every prediction to the next class.Source: §3 · p. 3
- 0.4444 is the reported balanced-accuracy floor on this 10-class one-hot problem, while micro-precision, micro-recall, and micro-F1 all equal accuracy.The output illustrates the inflated-floor and micro-collapse diagnostic findings.Source: §3 · p. 3
- The scorer interface applies macro-averaged multiclass MCC in a five-fold cross-validation call.The example uses make_scorer with cross_val_score.Source: §3 · p. 3
4. Comparison with existing software
aggmetrics offers a broader measure–operator grid and diagnostics than fixed-catalogue implementations, while not replacing production-oriented tools such as torchmetrics.
4. Comparison with existing software
- Unlike scikit-learn and torchmetrics’ fixed catalogues with averaging arguments, aggmetrics offers a full measure–operator grid, explicit undefined-value conventions, and diagnostics.It does not replace production metric computation: torchmetrics targets GPU, streaming, and distributed accumulation, which aggmetrics does not.Source: §4 · p. 3
C. Prati
The section compares aggmetrics with PyCM and yardstick, noting differences in input settings, user-defined measures, and aggregation options. The authors have not benchmarked these packages on the discussed cases.
C. Prati
- PyCM offers broad per-class and overall statistics but takes confusion matrices as input, excluding multilabel and soft-label settings.It also does not offer user-defined measures with derived properties.Source: C. Prati · p. 4
- Yardstick supports macro, weighted macro, and micro estimators for multiclass metrics.Source: C. Prati · p. 4
- The authors know of no package that attaches structural properties to measures or warns about floor effects, but have not benchmarked the cases beyond reading documentation.Source: C. Prati · p. 4
5. Project status
The project is BSD 3-Clause licensed, requires only NumPy for its core, and has an extensive automated test suite. Documentation and maintenance materials include a quickstart, public-function docstrings, and a machine-readable guide for AI coding assistants.
5. Project status
- The package is BSD 3-Clause licensed, and its core requires only NumPy; scikit-learn and pandas are optional.It is pure Python, requires Python 3.9 or later, and has no compilation or proprietary dependencies.Source: §5 · p. 4
- 209 tests run in about three seconds, with 100% line and branch coverage enforced by continuous integration.Tests encode companion-paper theoretical results, so changes that break a theorem break a test.Source: §5 · p. 4
- Project materials include a README, runnable quickstart, public-function docstrings, and a machine-readable guide for AI coding assistants.Version 0.1.0 is under review; the package is archived on Zenodo and documentation includes example notebooks.Source: §5 · p. 4
6. Conclusion
aggmetrics makes the aggregation operator explicit for every measure and stores properties that determine which identities apply. It uses those properties to flag misleading evaluations; planned work includes sparse backends, streaming accumulation, and an R implementation.
6. Conclusion
- The package makes the aggregation operator an explicit parameter for every measure.Source: §6 · p. 4
- It stores measure properties that determine which identities hold and uses them to flag misleading evaluations.Source: §6 · p. 4
- Planned work includes sparse backends for large label spaces, a streaming accumulator, and an R implementation.Source: §6 · p. 4