Source-linked AI summary
Macro F1 and Macro F1
Juri Opitz, Sebastian Burst
TL;DR
The paper addresses the use of two different formulas for macro F1 and analyzes how they compare. It defines and mathematically examines the formulas, then evaluates their practical behavior, finding that they can diverge substantially and produce different classifier rankings.
Problem
Two formulas are used to compute macro F1, but their equivalence and evaluation consequences require clarification.
Method
The paper defines the two macro F1 formulas, mathematically analyzes their difference, and examines practical implications through numerical experiments.
Results
The formulas can diverge substantially: the difference can reach 0.5, and they can yield different classifier rankings.
Takeaways & Limitations
Averaged F1 is recommended because it is more robust to error-type distributions; researchers should at least report which formula they use.
Abstract
from arXiv · showhide
The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically, one formula well 'rewards' classifiers which produce a skewed error type distribution. In fact, the difference in outcome of the two computations can be as high as 0.5. The two computations may not only diverge in their scalar result but can also lead to different classifier rankings.
1. Introduction
The paper identifies two formulas used to compute macro F1 and analyzes their definitions and implications. One averages per-class F1 scores, while the other takes the harmonic mean of averaged precision and recall.
- Two formulas for macro F1 are identified: “averaged F1” and “F1 of averages”.
- Precision, recall, and F1-score are defined per class from a classifier’s confusion matrix, with all three set to 0 when the denominator is zero.
- Averaged F1 computes each class’s F1 score and then takes their arithmetic mean.
- F1 of averages computes the harmonic mean of the arithmetic means of precision and recall.
- The paper first mathematically analyzes the two formulas and then examines their practical implications.
2. Mathematical analysis
The mathematical analysis characterizes when the two macro F1 formulas diverge and shows that their difference grows under heterogeneous precision–recall patterns. The largest gap occurs when classes exhibit opposing extreme precision and recall values, while biased error distributions can make one formula overly favorable.
- F1 of averages exceeds averaged F1 exactly when at least one class has unequal precision and recall, equivalently when some classes have precision below recall and others above it.
- The difference is large when many classes have substantially different precision and recall, but changing one class alone need not increase it monotonically.All possible class pairs affect the difference.
- The difference is maximized when some classes approach (precision, recall) = (1, 0) and others approach (0, 1).In this limit, all per-class F1 scores approach 0, while averaged precision and recall approach 0.5.
- A strong bias toward particular error types creates large outcome differences because classes need not share the same bias.The F1 of averages formula rewards such classifiers, although different class-specific error distributions can be desirable in some applications.
3. Numerical experiments
The experiments show that the two macro F1 formulas can produce different classifier rankings and substantially different scores, especially under biased errors or imbalanced data. Across random tasks, score differences remain measurable even when biases are weaker or data are balanced.
- The two metrics correlate imperfectly on 1,000 random data sets, with Pearson’s ρ = 0.72 and Spearman’s ρ = 0.69, allowing different classifier rankings.The tasks use uniformly random dummy classifiers on data sets with varying label distributions.
- The experiments vary classifier performance and label distribution across random classification tasks, using the x-axis for performance and the y-axis for class distribution.The tasks contain 2,000 data points in the reported figure setup.
- Up to 2 percentage points separate the metrics when classifier accuracy is below 1 and the data set is skewed.Both metrics are roughly proportional to classifier accuracy in this weaker-bias setting.
- Differences of up to 0.8 percentage points for n=4 and 1.7 percentage points for n=13 are reported in the balanced-data analysis.The cited passage does not specify the full experimental condition for these values.
4. Discussion and conclusion
The discussion concludes that two formulas for macro F1 often agree when within-class precision and recall are similar but can diverge sharply otherwise. It recommends the arithmetic mean of individual F1 scores and, at minimum, reporting which formula is used.
- Two macro F1 formulas can yield vastly different outcomes, with one producing misleadingly high scores for heavily biased classifiers, especially on imbalanced data.The difference is negligible when precision and recall do not differ much within classes.
- The recommended F1 formula is the arithmetic mean over individual F1 scores because it is more robust to error type distribution.Researchers should at least identify which macro F1 formula they use.
A. Proof Lemma
The proof section defines the summation convention used in the lemma’s formulas: classes with Pi + Ri = 0 are excluded.
- All summations exclude classes where Pi + Ri = 0.
B. Proof Theorem 2.
The theorem proof establishes that equality of the two macro F1 computations follows when class precision and recall align, while certain cross-class discrepancies make their difference positive.
- If no class has Pi ≠ Ri, the proof derives F1 − F1 = 0.
- The proof separately assumes that no class has Pj < Rj in one implication.
- A positive squared cross-class term, (PiRj − PjRi)^2 > 0, implies F1 − F1 > 0.
C. Proof Theorem 3.
The proof constructs configurations that maximize the difference between the two macro F1 computations. It shows that this difference is associated with skewed precision–recall patterns across classes and can approach 0.5.
- Maximum difference: The maximum over the boundary configurations can be approximated by a sequence of suitable confusion matrices.Thus, the extremal abstract precision–recall configurations correspond to increasingly close realizable confusion-matrix examples.
- Proof construction: The maximization procedure swaps precision and recall when their skew matches the configuration’s directional imbalance, increasing ∆.After swapping, the proof sets selected precision or recall values to zero and nonzero variables to one.
- Proof construction: The proof reduces candidate configurations to cases where each class has precision–recall pairs among (0,0), (0,1), and (1,0).It then counts classes with each nonzero pattern using r and s.
- Maximum difference: 1/2 is the maximum difference for an even number of classes, with nearly balanced r and s attaining the corresponding boundary configuration.For odd class counts, the proof uses r and s differing by one.
- Examples: In the demonstrated extreme case, F1 = 0.505 while F1 = 0.0196, although each class’s individual F1 score is 0.0196.A balanced error distribution instead gives F1 = 0.0196 and F1 = 0.0196.
D. Implementation example
The implementation example evaluates the two macro F1 formulas on contrasting binary confusion matrices. A skewed error distribution produces a large discrepancy, whereas a balanced distribution makes the results identical.
- Inputs: The example takes a number of classes and a confusion matrix as input.The supplied skewed example uses [[100, 10000], [0, 100]].
- Skewed errors: 0.504950495049505 is the benevolent macroF1, compared with 1.96078431372549e-2 for the non-benevolent macroF1.For this confusion matrix, both reported delta values are 0.48534265191225007.
- Balanced errors: 0.0 is the delta between the two macro F1 results for the confusion matrix with balanced error type distribution.Both macroF1 benevolent and macroF1 non-benevolent equal 1.96078431372549e-2.
E. Example code
The example code computes both macro F1 variants from a confusion matrix and prints their values and difference. It defines classwise precision, recall, F1, averaging, and the analytic delta calculation.
- Macro F1 formulas: The code computes one macro F1 by averaging classwise F1 scores and another by taking the harmonic mean of average precision and average recall.The shared harmonic-mean function computes 2xy/(x+y).
- Classwise metrics: Classwise precision divides the diagonal entry by its row sum, while recall divides it by its column sum.The implementation obtains diagonal cells with valueAt, row totals with rowSum, and column totals with colSum.
- Difference calculation: The analytic delta function sums pairwise contributions based on precision and recall values for class pairs.Each contribution uses the squared cross-term difference between the paired precision–recall values and a normalizer.
- Program flow: The program converts command-line values into a confusion matrix, computes both macro F1 variants, and prints each score and their delta.It also prints a separately calculated delta for comparison.