Source-linked AI summary
Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters
Jingxiu Yao, Martin Shepperd
TL;DR
F1 is widely used in software defect prediction despite known bias, raising questions about the reliability of empirical comparisons. The authors systematically compared classifier-pair results reported with F1 and MCC; 23% changed direction under MCC. They conclude that metric choice matters and F1 should be deprecated.
Problem
F1 is widely used despite known bias, and the extent to which it affects confidence in empirical software defect prediction results is uncertain.
Source: §1 · p. 1Method
The authors systematically searched for defect prediction studies reporting both F1 and MCC, then compared classifier-pair preferences under the two metrics.
Source: §5 · p. 7Results
23% of pairwise results changed direction when assessed with MCC rather than F1.
Source: §5 · p. 8Takeaways & Limitations
The authors conclude that metric choice matters and that biased F1 should be deprecated.
Source: §5 · p. 9The tightly defined search may have excluded relevant experiments, although the authors were not aware of any missed papers.
Source: §5.1 · p. 9
Abstract
from arXiv · showhide
Context: There is considerable diversity in the range and design of computational experiments to assess classifiers for software defect prediction. This is particularly so, regarding the choice of classifier performance metrics. Unfortunately some widely used metrics are known to be biased, in particular F1. Objective: We want to understand the extent to which the widespread use of the F1 renders empirical results in software defect prediction unreliable. Method: We searched for defect prediction studies that report both F1 and the Matthews correlation coefficient (MCC). This enabled us to determine the proportion of results that are consistent between both metrics and the proportion that change. Results: Our systematic review identifies 8 studies comprising 4017 pairwise results. Of these results, the direction of the comparison changes in 23% of the cases when the unbiased MCC metric is employed. Conclusion: We find compelling reasons why the choice of classification performance metric matters, specifically the biased and misleading F1 metric should be deprecated.
1 INTRODUCTION
Software defect prediction studies widely compare classifiers, but they lack a standard performance metric. Because F1 is widely used despite known bias, the paper asks whether it makes published comparisons unreliable, especially for meta-analysis.
1 INTRODUCTION
- F1 and its components, precision and recall, are widely used, although statisticians have identified F1 as problematic and MCC as an alternative.The paper focuses on F1 because it is both biased and widely used, particularly with imbalanced data.Source: §1 · p. 1
- Unreliable primary studies can undermine meta-analysis, and prior software defect prediction reviews have aggregated such studies.The paper argues that full reporting could let meta-analysts calculate MCC from confusion matrices.Source: §1 · p. 1
- The paper asks how often classifier comparisons change between F1 and MCC, and whether imbalance or performance differences relate to discordance.It also asks whether past results based on F1 can be trusted and whether researchers should report enough information to calculate MCC.Source: §1 · p. 1
2 RELATED WORK
The paper reviews classification metrics, explains why F1 can mislead in defect prediction, and notes its continued use in research and meta-analyses.
2.1 Classifier performance metrics
- F1 and MCC use different scales, but higher values indicate better performance for both; MCC incorporates all four confusion-matrix counts.AUC instead summarizes a family of threshold-dependent classifiers, so the authors do not analyze it further unless one ROC curve strictly dominates another.Source: §2.1 · p. 2
2.2 A critique of F1 and comparison with MCC
- F1 can conceal different precision–recall trade-offs and is hard to interpret against chance, whereas correlation coefficients indicate direction and performance relative to chance.In defect prediction, correctly identifying defect-free components matters because they may need fewer testing resources; F1 omits that outcome.Source: §2.2 · p. 3
- F1 ignores true negatives, so it can give the same score to chance-level and useful classifications, and can rank below-chance performance as better.In the examples, F1 = 0.5 for both MCC = 0.0 and MCC ≈0.44, while F1 ≈0.52 accompanies MCC ≈ −0.11.Source: §2.2 · p. 3
2.3 Te usage of classification performance metrics in sofware defect prediction
- F1 remained widely used in software defect prediction, and the authors’ review found it in 29 of 31 available papers from 2014–2019.They caution that the search was targeted and cannot support strong inferences about what proportion of all studies this represents.Source: §2.3 · pp. 3–4
- F1 use carries into meta-analyses, while some reviews have discarded studies that reported only F1 results.The authors note that F1’s prevalence in empirical studies may therefore affect evidence synthesis.Source: §2.3 · p. 4
3 SYSTEMATIC REVIEW
The authors used a lightweight systematic review to find cross-validated defect prediction studies reporting both F1 and MCC, then compared classifier-pair preferences under each metric.
3 SYSTEMATIC REVIEW
- The review searched Google Scholar in November 2019 and applied inclusion criteria covering publication date, peer review, language, domain, reporting, cross-validation, and new results.The strategy was lighter than Kitchenham et al.’s full method: it had no formal protocol or quality assessment beyond peer review and cross-validation.Source: §3 · p. 4
- Eight studies met the criteria, providing 4,017 classifier-pair comparisons evaluated with both F1 and MCC.Per-study comparisons ranged from 14 to 1,512, with a median of 282; four of the eight studies also reported AUC.Source: §3 · p. 4
- An additional paper was refereed, in scope, and provided necessary data, but almost all treatment comparisons tied; its 16 treatments across 52 data sets could yield more than 6,000 pairwise comparisons.The paper caused two types of difficulty.Source: §3 · p. 4
4 RESULTS
Across the reviewed results, F1 and MCC often move together, but their comparisons disagree often enough to change classifier preferences. The results also show that discordance is more associated with small F1 differences than with dataset imbalance.
4.1 Summary of the classification accuracy metrics
- F1 and MCC had a positive relationship (Spearman correlation ρ ≈0.82, n = 8034), but substantial scatter and outliers remained; F1 was bimodally distributed.The two outlier clusters came from one study’s extremely imbalanced Log4j data, with over 95% defective components; reported near-perfect F1 scores accompanied MCC values around or below zero.Source: §4.1 · pp. 4–5, Figure 1 · p. 4
4.2 Summary of data set utilisation
- The 112 distinct datasets ranged from 36 to 17,186 cases and from 0.4% to 95.6% defect-prone components; 35 used imbalance-learning or corrective procedures.The median dataset size was 400 cases, and the median imbalance level was about 13.5%.Source: §4.2 · p. 5
4.3 RQ1: Are F1 and MCC concordant?
- 23% (927/4017) of pairwise comparisons are discordant between F1 and MCC, meaning the metrics can reverse which classifier is preferred.The analysis matches F1 and MCC results for each pairwise comparison across eight studies; Fig. 4 presents the proportion of incorrect experimental results.Source: §4.3 · pp. 5, 7, Figure 4 · p. 7
4.4 RQ2: How does data set imbalance impact differences between F1 and MCC?
- Dataset imbalance showed little relationship with discordance; unexpectedly, datasets closer to 50% defect-prone components appeared more likely to have discordant results.The association between post-processed imbalance and the probability of a result changing was small (Spearman’s ρ = 0.24, n = 112).Source: §4.4 · p. 7
4.5 RQ3: How does the magnitude of difference in the classifier performance impact discordancy?
- An odds ratio of 4.36 (confidence interval 3.53–5.39) indicates that comparisons in the lower third of absolute F1 differences were more likely to be discordant than those in the upper third.The middle third was excluded from this comparison; the median absolute F1 difference across classifier comparisons was 0.066.Source: §4.5 · p. 7, Table 9 · p. 7
5 DISCUSSION AND CONCLUSIONS
The review compares F1 and MCC across published software defect prediction results and finds that their comparison directions differ often enough to warrant caution. It recommends avoiding inappropriate F1 use, reporting confusion matrices, and treating validity limits in the review’s search and evidence base seriously.
5 DISCUSSION AND CONCLUSIONS
- 23% of F1-based comparisons changed direction under MCC, meaning the comparison direction was in error in those cases.The analysis used eight studies and 4,017 pairwise comparisons.Source: §5 · pp. 7–8
- The authors recommend stopping inappropriate F1 use and reporting confusion matrices so others can compute alternative metrics and reinterpret results.They also recommend caution with F1-only results and sensitivity analysis when meta-analyses include such studies.Source: §5 · p. 9
- The authors conclude that metric choice matters for software defect prediction and suggest extending examination to grey literature and other problem domains.They state that their analysis does not criticize the eight primary studies, which reported a wide range of metrics.Source: §5 · pp. 8–9
5.1 Treats to validity
- The review treats its direction-based comparison as limited because the sign function captures changes in effect direction, not their magnitude.It also notes that imbalance estimates are affected by stochastic cross-validation sampling and incompletely reported preprocessing.Source: §5.1 · p. 9
- The systematic search was tightly defined and excluded relevant experiments, despite the authors’ belief that they had not missed papers.They note that additional queries, bibliographic databases, or grey literature could have expanded coverage.Source: §5.1 · p. 9
- Only eight experiments were available, limiting the study count, although they contributed over 4,000 pairwise results.The authors state that a larger sample would be preferable, while noting their constraint to available studies.Source: §5.1 · p. 9