Source-linked AI summary
F*: An Interpretable Transformation of the F-measure
David J. Hand, Peter Christen, Nishadi Kirielle
TL;DR
The paper addresses concerns that F-measure combines distinct precision and recall probabilities through a potentially unintuitive harmonic mean. It presents F*, a transformed F-measure with a practical interpretation, while noting scope-related limitations inherited from F's treatment of true negatives.
Problem
F-measure's combination of distinct precision and recall aspects is difficult to interpret, and researchers question whether the harmonic mean is appropriate.
Method
The paper transforms F into F*, which can be calculated from the confusion matrix and interpreted through correctly classified class 1 objects.
Results
F* is a monotonic transformation of F, so comparing F* values yields the same algorithm choices as comparing F values.
Takeaways & Limitations
F* offers a straightforward and familiar intuitive interpretation for researchers who are uneasy about the conventional F-measure.
Takeaways & Limitations
F's omission of true negatives is appropriate in some domains but may be unsuitable where correct classification of both classes matters, such as medical diagnosis.
Abstract
from arXiv · showhide
The F-measure, also known as the F1-score, is widely used to assess the performance of classification algorithms. However, some researchers find it lacking in intuitive interpretation, questioning the appropriateness of combining two aspects of performance as conceptually distinct as precision and recall, and also questioning whether the harmonic mean is the best way to combine them. To ease this concern, we describe a simple transformation of the F-measure, which we call F* (F-star), which has an immediate practical interpretation.
1. Introduction
Classification evaluation compares algorithms using measures derived from a confusion matrix, but F-measure interpretation remains contested because it combines distinct conditional probabilities with a harmonic mean. The paper therefore introduces a transformed F-measure with a straightforward intuitive interpretation.
- Evaluation framework: Classification evaluation uses an independent test set and a two-by-two confusion matrix formed by thresholding scores into two classes.The test set contains scores and true labels; threshold t determines predicted class assignments.
- Evaluation framework: Known test-set size and class proportions reduce the confusion-matrix problem to two degrees of freedom that must be combined into one comparative performance measure.Different choices of degrees of freedom and combination yield measures such as misclassification rate, kappa, the Youden index, Matthews coefficient, and F-measure.
- Existing measures: Performance measures differ in interpretation, ranging from incorrect-classification proportions and chance-adjusted accuracy to AUC's probability comparison between randomly chosen objects.The H-measure instead expresses reduction in expected minimum misclassification loss relative to a random classifier.
- F-measure motivation: F-measure is widely used for classification, especially imbalanced problems, and combines precision and recall, which represent different conditional probabilities.Precision conditions on predicted class 1, whereas recall conditions on true class 1.
- F-measure motivation: Researchers question averaging precision and recall because their arithmetic average has no probability interpretation, while others question the harmonic mean's appropriateness.The harmonic mean is closer to the smaller value and becomes zero when either precision or recall is zero.
- Paper contribution: The paper presents a transformed version of the F-measure intended to provide a straightforward intuitive interpretation.This transformation is introduced specifically in response to concerns about interpreting F-measure.
2. The F-measure and F*
The paper transforms F into F* to give the measure an immediate proportion-based interpretation while preserving the ordering of classifier performance.
- Transformation and interpretation: F′ measures correctly classified class 1 objects per misclassified object, but its ratio is not constrained between 0 and 1.The paper therefore presents F′ as an attractive interpretation with a scale unlike most performance measures.
- Transformation and interpretation: F* is the proportion of relevant classifications that are correct, counting classifications that are truly class 1 or predicted as class 1.This interpretation expresses the correctly classified class 1 objects relative to the union of actual and predicted class 1 objects.
- Transformation and interpretation: F* can be calculated directly from the confusion matrix as TP/(n−TN) and is recognizable as the Jaccard coefficient.The coefficient is used in settings where true negatives may not be relevant, including numerical taxonomy and fraud analytics.
- Examples: For information retrieval, F* is relevant retrieved documents divided by all documents except non-retrieved irrelevant documents; for COVID-19, it divides infected positive tests by the union of positive tests and infections.These examples instantiate the same proportion interpretation in two application domains.
- Relationship to F: F* is a monotonic transformation of F, so comparing F* values yields identical classifier choices and crossing thresholds.The relationship is approximately linear, so F* values remain close to F values; the transformation changes curve shapes but not their ordering.
- Extensions: Weighted versions of F′ and F* follow immediately from the weighted version of F, allowing different importance to be assigned to precision and recall.The paper also gives F* in terms of precision and recall as PR/(P + R −PR).
3. Discussion
The paper emphasizes matching performance measures to the objective because different measures highlight different aspects of classification performance. It presents F* as an interpretable transformation while noting that F-measure’s omission of true negatives can be inappropriate when both classes matter.
- Measure selection: Performance measures should be matched to the classification objective because different measures emphasize different aspects of algorithm performance.A poor choice can contribute to selecting an inappropriate algorithm and consequently to suboptimal decisions and actions.
- Scope and limitation: The F-measure ignores true negatives, which is appropriate in some domains but problematic when correct classification of both classes is important.The paper gives medical diagnosis as an example where this limitation can matter.
- F* interpretation: F* provides a straightforward interpretation of the F-measure through a suitable transformation.The paper frames this interpretation as an alternative way of viewing F that may be useful under some circumstances.