PaperlayerView exact revision on arXiv

Source-linked AI summary

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

Diyorbek Musaev

arXiv:2608.16147v1cs.AIcs.LG

TL;DR

Imbalance-handling comparisons are often conducted on a single dataset, limiting evidence for whether their rankings generalize. The study evaluates 45 binary classification tasks spanning imbalance ratios from 1:1.5 to 1:178, with 2,025 model fits across four model families. Across the suite, Random Forest gains more from threshold tuning than any other family, while threshold-tuning benefit peaks at ΔF1 = +0.120 for 1:15–1:40 imbalance.

  • Problem

    Imbalance-handling comparisons are often conducted on a single dataset, limiting evidence for whether their rankings generalize.

  • Method

    The study evaluates 45 binary classification tasks spanning imbalance ratios from 1:1.5 to 1:178, with 2,025 model fits across four model families.

  • Results

    Across the suite, Random Forest gains more from threshold tuning than any other family, while threshold-tuning benefit peaks at ΔF1 = +0.120 for 1:15–1:40 imbalance.

  • Takeaways & Limitations

    Methodologically impeccable single-dataset findings can still be backwards on the point of interest, making external validity essential when evaluating imbalance-handling methods.

  • Takeaways & Limitations

    The suite contains 30 synthetic tasks and only 15 real tasks from three underlying sources, limiting domain diversity and external validity.

Abstract

from arXiv · show

Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.

1. Introduction

This paper shows that imbalance-handling conclusions drawn from a single dataset do not generalize reliably. Across 45 tasks, threshold tuning and SMOTE often help, with effects depending on model family and imbalance severity, while calibration diagnostics fail to predict tuning benefit.

  • Fraud-dataset result: F1 = 0.861 ± 0.021: on the fraud dataset, plain Random Forest at threshold 0.5 gains nothing from tuning (ΔF1 = -0.002), and SMOTE makes performance worse.The dataset contains 284,807 transactions, including 492 fraudulent cases, and is among the field’s most widely used benchmarks.
  • 45-task study: ΔF1 = +0.101: Random Forest benefits most from threshold tuning across 45 tasks, reversing its fraud-dataset result, while three other model families behave similarly across datasets.The study covers imbalance ratios from 1:1.5 to 1:178 and 2,025 model fits under the same protocol.
  • What generalises: Mean ΔF1 = +0.059 for threshold tuning and +0.076 for SMOTE across dataset–model pairs, but gains are strongly model-dependent.SMOTE yields +0.170 F1 for an MLP versus +0.005 for logistic regression; threshold tuning records 138 wins and 31 losses, while SMOTE records 138 wins and 39 losses.
  • Imbalance severity: ΔF1 = +0.120: threshold-tuning benefit peaks at moderate imbalance (1:15–1:40), declining to -0.009 below 1:5 and +0.045 beyond 1:100.This inverted-U pattern makes extreme-imbalance datasets such as credit-card fraud poor venues for studying threshold selection.
  • Calibration diagnostic: r = -0.087: expected calibration error does not predict tuning benefit, while Brier score correlation is only weak (r = +0.137).The proposed validation-set calibration diagnostic therefore does not identify when practitioners should tune thresholds.

2. Related Work

Related work establishes threshold tuning, resampling, and cost-sensitive weighting as standard responses to class imbalance, while multi-dataset studies already show threshold tuning can help. This paper instead addresses external validity: whether a clean result from one dataset supports a general claim, demonstrating that such conclusions can invert.

  • Prior methods: Threshold tuning, resampling, and cost-sensitive weighting are established alternative strategies for addressing class imbalance.Threshold selection traces to cost-sensitive classification and ROC operating-point selection, while SMOTE changes the training distribution.
  • Prior evidence: Multi-dataset studies already report broad threshold-tuning benefits, including substantial gains for Random Forest, so that finding is not this paper’s contribution.GHOST was validated on 138 public drug-discovery datasets, and M-Tune reports comparable findings.
  • Contribution: The paper’s contribution is showing that single-dataset claims about whether an intervention is needed for a model do not necessarily transfer and can invert.The focus is not whether threshold tuning helps in general, but whether conclusions drawn from one dataset generalize.
  • Fraud-dataset literature: Prior credit-card fraud studies report strong results across resampling, ensembles, and deep models, but the literature also contains leakage and validation problems.Reported issues include sampling before train/test splitting, vague reporting, inadequate temporal validation, and metric manipulation.
  • Validity distinction: Unlike prior work focused on internal validity, this paper examines external validity while adopting the recommended leakage controls to show that clean single-dataset findings can still mislead.The two concerns are complementary: internal validity asks whether a result is faithful on its dataset, whereas external validity asks whether it licenses a general claim.

3. Methods

The studies use a leakage-free nested cross-validation protocol that selects thresholds only on inner validation folds and evaluates them on untouched test folds. The design compares four supervised model families and intervention strategies across one fraud dataset and a 45-task suite spanning broad imbalance ratios.

  • Evaluation protocol: Nested stratified cross-validation isolates test performance: preprocessing, fitting, resampling, and threshold selection use only training data and inner validation folds.The fraud study uses five outer folds, while the 45-task suite uses three as a compute-budget concession; thresholds are selected by validation F1 over 0.01–0.99, or fixed at 0.5 for the plain strategy.
  • Metrics and contrasts: The analysis reports precision, recall, F1, ROC-AUC, AUPRC, calibration errors, and Brier score, with calibration statistics computed on validation folds.Threshold effects are evaluated as paired within-fold contrasts on the same test fold, eliminating between-fold and between-model variance.
  • Models and strategies: The comparison includes plain, tuned-threshold, class-balanced, and SMOTE strategies across Logistic Regression, Random Forest, Histogram Gradient Boosting, and MLP models.Class-balanced tuning is omitted for the MLP because its implementation lacks class-weight support; the fraud study additionally includes an autoencoder anomaly detector.
  • Study design: Study A uses the full Kaggle fraud dataset with 284,807 rows, 492 positives, 1:577 imbalance, five outer folds, and 85 fits.Study B contains three folds per task, four model families, and four strategies, totaling 2,025 fits.
  • Study design: The benchmark suite comprises 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178, with 30 synthetic tasks controlling imbalance, class separation, and dimensionality.It also includes 14 real tasks and one rarefied breast-cancer task; three folds, four model families, and four strategies produce 2,025 fits.

4. Study A: The Credit-Card Fraud Dataset

On the credit-card fraud dataset, the simplest Random Forest configuration—without resampling, class weighting, or threshold tuning—performed best. Taken alone, this result suggests that standard imbalance handling is unnecessary and counterproductive for a well-calibrated ensemble, a claim tested in Section 5.

  • Study A: The Credit-Card Fraud Dataset: The best configuration was a plain Random Forest with no resampling, no class weighting, and the default threshold.Table 1 reports five-fold nested cross-validation results for the strongest configuration of each model family.
  • Study A: The Credit-Card Fraud Dataset: Threshold tuning was neutral (ΔF1 = -0.002), while class weighting reduced F1 by 1.7 points and SMOTE reduced F1 by 2.3 points and AUPRC by 1.0 point.Every intervention applied to the Random Forest made it worse relative to the default, unweighted, non-resampled configuration.
  • Study A: The Credit-Card Fraud Dataset: Read alone, the dataset supports the claim that imbalance handling is unnecessary and counterproductive for a well-calibrated ensemble, motivating the test in Section 5.The passage identifies this as the type of conclusion routinely advanced by single-dataset studies.

5. Study B: The Conclusion Reverses

Across 45 binary tasks, Random Forest reverses the fraud-dataset conclusion: threshold tuning is most valuable for it, while three other model families reproduce their earlier behavior. SMOTE is beneficial overall but highly model-dependent, and threshold-tuning benefits follow an inverted-U across imbalance ratios, making the 1:577 fraud dataset unrepresentative.

  • Model-family comparison: Random Forest reverses the fraud-dataset result: threshold tuning is worthless there but most valuable across the suite, by 44 F1 points over the second-placed family.Three of four families replicate their fraud-dataset behavior within 0.02 F1; Figure 1 displays the contrast.
  • Model-family comparison: A default-threshold deployment of Random Forest would forgo an average of 10 F1 points on tasks drawn from the suite.The Study A conclusion applies to Random Forest on the fraud dataset, not to Random Forest generally.
  • Resampling: SMOTE is beneficial on average across the suite, reversing its fraud-dataset result, where it reduced Random Forest F1 by 2.3 points and Logistic Regression F1 by 9.2 points.The suite comparison is paired at the dataset-model level in Table 3.
  • Resampling: SMOTE’s benefit is model-dependent: +0.170 for MLP, +0.092 for Random Forest, +0.036 for Histogram Gradient Boosting, and +0.005 for Logistic Regression.The near-zero Logistic Regression effect is indistinguishable from nothing, so blanket recommendations for or against resampling are unsupported.
  • Imbalance-ratio analysis: Threshold-tuning benefit follows an inverted-U: it is marginally harmful under mild imbalance, peaks at 1:15–1:40, and declines beyond 1:100.At extreme imbalance, too few validation positives may make the F1-maximizing threshold unstable and generalize poorly to the test fold.
  • Imbalance-ratio analysis: At 1:577, the fraud dataset lies in the regime where threshold tuning delivers least and is estimated least reliably, making it unrepresentative for studying threshold selection.This explains why the Study A conclusion does not generalize despite the dataset’s popularity.

6. What Predicts the Benefit of Threshold Tuning?

Calibration error does not provide a usable advance signal for threshold-tuning benefit: global relationships are absent or too weak for a go/no-go decision. Because tuning is inexpensive, the defensible recommendation is to always tune on the validation fold and let it select the threshold.

  • Family-specific relationships: r = +0.259 for Histogram Gradient Boosting, but Logistic Regression, MLP, and Random Forest show no within-family association.The MLP has the largest mean calibration error, 0.086 versus 0.012–0.027 for the other families, yet the smallest mean tuning benefit.
  • Calibration as a predictor: Calibration error is not a usable diagnostic for deciding whether to tune a decision threshold.Expected calibration error is uncorrelated with tuning benefit, while the Brier score explains under 2% of variance and is practically useless for a go/no-go decision.
  • Other recorded predictors: r = +0.137 for Brier score and r = +0.138 for log imbalance ratio, while AUPRC is weakly negative at r = -0.162.Sample size and dimensionality are uninformative, with |r| < 0.09.
  • Practical implication: Always perform threshold tuning and let the validation fold decide, because a single sweep over a fitted model requires no retraining.When tuning does not help, the validation fold selects a threshold near 0.5, so little is lost.

7. Discussion

The discussion shows that a clean single-dataset result can reverse across diverse tasks, making external validity essential for claims about imbalance handling. It recommends routine validation-fold threshold selection, model-specific resampling decisions, and broader multi-dataset evaluation while acknowledging important design limitations.

  • Limitations: The suite is limited because 30 of 45 tasks are synthetic, 15 real tasks come from three sources, and established real-world benchmarks were unavailable.External data downloads were blocked, preventing inclusion of several established imbalanced benchmark collections.
  • Limitations: Maximum imbalance was 1:178 versus 1:577 in the fraud dataset, narrowing evidence for the inverted-U decline and leaving its few-validation-positives explanation inferential.The proposed explanation is unstable threshold estimation from few validation positives, but it was not directly demonstrated.
  • Limitations: Interpretation is further constrained by three folds rather than five, F1-based evaluation, and random rather than temporal validation.Three folds widen suite confidence intervals; F1 may not match deployment costs, and the fraud analysis inherits the absence of temporal validation.
  • For practitioners: Practitioners should routinely select thresholds on validation folds, avoid using calibration diagnostics to decide, and treat resampling as model-specific.Threshold selection is described as cheap and helpful on average; resampling is strongly worth trying for neural networks but marginal for logistic regression.
  • Discussion: F1 = 0.861 ± 0.021 on fraud data, yet across 45 tasks Random Forest benefited most from threshold tuning and SMOTE changed from harmful to beneficial.The fraud result was methodologically clean, but the suite showed it was an artifact of that dataset.
  • Discussion: Methodologically impeccable single-dataset findings can still be backwards on the point of interest, so imbalance-handling claims require evidence across multiple datasets.The discussion distinguishes internal validity, such as leakage-free protocols and honest threshold selection, from neglected external validity.

Reproducibility

The study is reproducible through released code and per-run metrics for both studies. Fixed seeds, short runtimes, consumer hardware, and standard Python dependencies support practical replication.

  • Reproducibility: Released code implements both studies, with 85 nested-cross-validation rows and 2,025 multi-dataset per-run metrics provided.Study B is resumable.
  • Reproducibility: Fixed random seeds are used throughout, enabling deterministic replication of the reported runs.
  • Reproducibility: Study A runs in approximately nine minutes and Study B in approximately eight minutes on a consumer laptop without GPU requirements.
  • Reproducibility: The implementation depends only on scikit-learn, imbalanced-learn, pandas, NumPy, SciPy, and Matplotlib.
Loading 2608.16147v1…