The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

1 Introduction

• The benchmark’s widely reported accuracy reaches 99.01%, while conflicting labels on its encoded feature space yield an empirical irreducible-error estimate of approximately 99.74%.The authors call this a sanity-check estimate, not a strict ceiling, and do not attribute the cited 99.01% result to the audited failure modes.Source: §1 · p. 2

(F1) Pre-split Smote inflates results, but class weighting on the same leaky pipeline does not.

• 13.1 percentage points: pre-split SMOTE inflated churn-class F1 across 10 classifier families and 15 seeds, while class weighting on the same leaky ordering showed no comparable inflation.The asymmetry replicated on Iranian Telecom Churn to within 0.4 pp.Source: (F1) Pre-split Smote inflates results,… · pp. 2–3

(F2) TotalCharges is algebraically redundant and SHAP over-attributes to it.

• TotalCharges had R2 = 0.9991 with tenure and MonthlyCharges, yet TreeSHAP ranked it ninth in mean absolute attribution despite negligible performance change when removed.The authors propose an R2 > 0.95 pre-modelling diagnostic and report replication of the pattern on Iranian Telecom Churn.Source: (F2) TotalCharges is algebraically… · p. 3

(F3) Isotonic regression is the strongest calibration default for class-weighted tree ensembles; temperature scaling fails on bimodal output.

• Isotonic regression was the strongest single-method calibration default, while temperature scaling failed on the bimodal predicted-probability distribution induced by class weighting on tree ensembles.Isotonic was best on four of five models; Platt scaling was a fallback on the worst-calibrated raw models.Source: (F3) Isotonic regression is the… · p. 3

(F4) The cost-optimal decision threshold is approximately 5–10× lower than the F1-optimal threshold.

• The cost-optimal threshold was consistently 0.04–0.07 across five seeds, saving $77,646±$19,698 per 1,000 customers relative to F1-optimal thresholding.The analysis uses a fixed $50 retention-offer cost and a 24-month customer-lifetime-value proxy.Source: (F4) The cost-optimal decision… · p. 3

Contributions and scope.

• The authors recommend a four-part reporting checklist covering pipeline ordering, feature redundancy, calibration, and cost-sensitive thresholds, and release code and materials for reproduction.The paper frames these failures as invisible to accuracy- and F1-centred reporting and potentially misleading for retention-budget sizing or model interpretation.Source: §1 · p. 2, Contributions and scope · p. 4

2.1 High-Accuracy Telco Churn Prediction

• Telco studies commonly use tree ensembles, SMOTE, and held-out accuracy or F1; the most-cited reported result is 99% accuracy using a kernel SVM with Smote-Enn preprocessing.Prior studies also commonly omit details needed to determine whether resampling and scaling preceded the split, and do not report calibration or explicit cost matrices.Source: §2.1 · p. 5

2.2 Data Leakage in Tabular Machine Learning

• Prior leakage work identifies preprocessing before the train–test split as a recurring failure mode, and earlier SMOTE audits report boundary-spanning nearest-neighbour pairs and compare pre-split with pipeline-internal resampling.The paper distinguishes preprocessing placement from imbalance-correction choice by comparing SMOTE and class weighting under the same leaky ordering.Source: §2.2 · pp. 5–6

2.3 Calibration and Cost-Sensitive Decision Support

• Calibration separates ranking ability, measured by ROC-AUC, from agreement between predicted probabilities and empirical frequencies, measured by ECE or Brier score.The literature describes Platt scaling, isotonic regression, and temperature scaling as standard remedies, but the paper identifies a gap in audits for class-weighted tree ensembles on binary churn.Source: §2.3 · p. 6
• Decision Curve Analysis plots net benefit against threshold probability, but is largely absent from churn prediction, where operational evaluation commonly uses lift charts or top-decile capture.The paper brings DCA and an explicit cost matrix to the churn setting; prior profit-aware work often uses the more extensive Expected Maximum Profit framework.Source: §2.1 · p. 5, §2.3 · p. 6

3.1 Datasets

• The study audits the IBM Telco dataset of 7,043 records and uses Iranian Telecom Churn, with 3,150 records, as a within-domain replication benchmark.IBM Telco has 21 columns and a 5,174-negative to 1,869-positive class distribution; Iranian Telecom has 13 numeric features and a 15.71% churn rate.Source: §3.1 · p. 7, Replication benchmark · p. 7

The whitespace TotalCharges quirk.

• The loader treats whitespace TotalCharges entries as $0.0 for tenure-zero customers, retaining 11 brand-new customers who would otherwise be dropped after coercion to missing values.All 11 entries have tenure = 0 and Churn = “No”.Source: The whitespace TotalCharges quirk · p. 7

Duplicate-row audit.

• The diagnostic audit defines exact duplicates, feature-only duplicates, and ambiguous tuples, and finds 22, 40, and 18 cases, respectively.The 18 conflicting-label feature configurations imply an empirical irreducible-error estimate of approximately 99.74% accuracy at n = 7,043, treated as a sanity check rather than a strict bound.Source: Duplicate-row audit · p. 8

Missingness and identifier audit.

• The dataset has no other missing values, and customerID is dropped because it is unique for every row and carries no predictive signal.Source: Missingness and identifier audit · p. 8

Tenure-redundancy diagnostic (F2).

• For rows with tenure > 0, the residual measures the difference between TotalCharges and tenure multiplied by MonthlyCharges; the fit yields R2 = 0.9991.The mean residual is $0.15, and maximum absolute residual is $373.25, which the authors say likely reflects mid-cycle plan changes.Source: Tenure-redundancy diagnostic (F2) · p. 8

Class-balance diagnostic.

• The Telco negative-to-positive ratio is 1 : 2.77, which the authors characterize as mild imbalance and use to motivate direct comparison of SMOTE with class-weighted training.Source: Class-balance diagnostic · p. 8

3.3 Feature Encoding

• The feature matrix excludes customerID and Churn, and one-hot encoding is applied to the 16 categorical features with drop_first=True.Source: §3.3 · p. 8
• The study does not engineer interactions, bin numeric features, or apply log transforms because its goal is to audit standard pipelines rather than optimize predictive performance.Source: §3.3 · pp. 8–9
• The encoded feature matrix has 30 columns, or 29 when TotalCharges is removed for the P-Drop pipeline.Source: §3.3 · p. 9

3.4 Pipelines

• Four pipelines vary the ordering of split, scaling, and SMOTE while holding the encoded features, stratified 80/20 split, and classifiers constant.P-Clean fits scaling and SMOTE on training data; P-Leak applies SMOTE and scaling before splitting; P-Scale isolates scaler leakage; P-Drop removes TotalCharges.Source: §3.4 · pp. 9–10
• P-Leak versus P-Clean tests ordering effects, P-Clean versus P-Drop tests TotalCharges’ predictive contribution, and P-Scale controls for scaler leakage alone.The text identifies the first contrast with classical leakage and says scaler-only leakage is small.Source: §3.4 · p. 10

Imbalance handling.

• Each pipeline uses either SMOTE oversampling to a 1:1 class ratio or class weighting, including class weights for boosters.Source: Imbalance handling · p. 10
• The 4 × 2 factorial design compares SMOTE and class weighting across pipelines to distinguish SMOTE-specific inflation from generic pipeline-ordering effects.Source: Imbalance handling · p. 10

3.5 Classifier Bank

• Ten classifiers span four families: linear, single tree, tree ensembles, and neural models.The bank includes Logistic Regression, Decision Tree, Random Forest, Extra Trees, Balanced Random Forest, Gradient Boosting, LightGBM, XGBoost, CatBoost, and a two-layer MLP.Source: §3.5 · p. 10
• Most classifiers use library defaults; tree ensembles use 300 trees, except Gradient Boosting’s 100, and the study does not tune hyperparameters.The authors state that tuning could obscure the pipeline-ordering signal under audit.Source: §3.5 · p. 10
• Support vector classifiers, Gaussian naive Bayes, and k-nearest neighbours are excluded from the classifier bank.The accompanying passage projects more than two hours per fit for SVC on the SMOTE-augmented training set, making the 1,200-fit benchmark intractable.Source: §3.5 · pp. 10–11

3.6 Evaluation Protocol

• Five held-out test metrics are reported: accuracy, churn-class F1, recall, precision, and ROC-AUC, with emphasis on minority-class metrics.Source: §3.6 · p. 11

Multi-seed factorial design.

• The main factorial benchmark comprises 1,200 fits: 10 classifiers, 15 seeds, four pipelines, and two imbalance strategies.Seeds control both the train–test split and classifier randomness; mean and standard deviation across seeds are reported.Source: Multi-seed factorial design · p. 11

Calibration evaluation.

• Calibration compares raw probabilities, Platt scaling, isotonic regression, and temperature scaling on the top five classifiers across 15 seeds.The evaluation reports test-set Expected Calibration Error and Brier score; Platt and isotonic use a held-out training calibration split.Source: Calibration evaluation · p. 11

Cost-sensitive evaluation.

• Cost evaluation assigns false negatives a 24-month MonthlyCharges-based CLV cost, flagged customers a $50 offer cost, and true negatives zero cost.It sweeps 99 thresholds from 0.01 to 0.99 across five seeds; Decision Curve Analysis uses the seed-42 P-Clean+Cw GBM model.Source: Cost-sensitive evaluation · p. 11

SHAP analysis.

• TreeSHAP mean absolute attributions are computed on the seed-42 P-Clean+Cw GBM test set and checked against eight expected risk factors.Source: SHAP analysis · p. 11

Cross-benchmark replication protocol.

• Iranian Telecom replication runs 900 fits across three pipelines, two imbalance strategies, 15 seeds, and 10 classifiers; P-Drop is omitted.The protocol instead checks numeric columns for redundancy because the benchmark has no a priori derived feature analogous to TotalCharges.Source: Cross-benchmark replication protocol · pp. 11–12

3.7 Reproducibility

• Experiments run in a hosted notebook environment with a single NVIDIA T4 GPU.Source: §3.7 · p. 12
• Seeds are fixed for splits, resampling, and classifiers; measured CatBoost and XGBoost GPU-reduction variation is below 10^-3 in accuracy.The paper describes other reported numbers as deterministic under these settings.Source: §3.7 · p. 12
• Code, intermediate CSV tables, and figures are released under an OSI-approved permissive license.Source: §3.7 · p. 12

Pipeline-aggregated effect.

• 0.8035 accuracy and 0.7055 churn-class F1 under P-Leak exceed P-Clean’s 0.7669 accuracy and 0.5744 F1 by 3.66 and 13.11 pp, respectively.The pipeline-aggregated comparison covers 10 classifiers, 15 seeds, and both imbalance strategies.Source: Pipeline-aggregated effect · p. 12, Table 2 · p. 13

Significance of the per-classifier inflation.

• SMOTE-specific churn-class F1 inflation ranged from +15.13±1.36 pp (LogReg) to +30.23±2.04 pp (MLP), with p < 10^-4 for all ten classifiers.The paired-by-seed Wilcoxon tests used 15 seeds per classifier and were restricted to the SMOTE strategy.Source: Significance of the per-classifier… · p. 14, Table 4 · p. 14

The SMOTE-versus-class-weight isolation.

• 0.8404 accuracy and 0.8393 F1 under leaky P-Leak with SMOTE contrast with 0.7666 accuracy and 0.5716 F1 with class weighting, statistically indistinguishable from P-Clean.Moving SMOTE inside the training split or replacing it with class weighting removes the inflation.Source: The SMOTE-versus-class-weight isolation · pp. 14–15, Table 5 · p. 14

4.2 Feature-Redundancy Finding (F2)

• R2 = 0.9991 for TotalCharges against tenure × MonthlyCharges; dropping it changes accuracy by 0.18 pp and F1 by 0.53 pp, within seed-to-seed variation.The authors recommend testing features with R2 > 0.95 and call this a feature-redundancy diagnostic.Source: §4.2 · p. 15

4.3 Calibration Audit (F3)

• Isotonic regression was best on four of five models, while temperature scaling worsened ECE for BalancedRF (0.1092 → 0.1127) and CatBoost (0.1129 → 0.1160).The authors attribute temperature scaling’s failure on class-weighted tree ensembles to bimodal leaf-vote distributions that single-parameter logit rescaling cannot correct.Source: §4.3 · p. 15, Table 6 · p. 15

4.4 Dollar-Cost Threshold Analysis (F4)

• 0.04–0.07 was the cost-optimal threshold across five seeds, saving $77,646 ± $19,698 per 1,000 customers (95% CI [$56,675, $107,634]).The threshold analysis uses the canonical P-Clean+Cw+GBM+Platt model; Table 7 reports the seed-42 test-set comparison.Source: §4.4 · p. 16, Table 7 · p. 16

4.5 SHAP Trust-Anchor and Over-Attribution

• 8-of-8 expected churn risk factors appeared in the top ten, but TotalCharges ranked ninth despite contributing essentially nothing predictive beyond tenure and MonthlyCharges.The authors caution that SHAP can split attribution among algebraically related features, so a high rank alone does not establish predictive value.Source: §4.5 · p. 16, Table 8 · p. 16

4.6 Cross-Benchmark Replication on Iranian Telecom Churn

• F1 inflation replicated on Iranian Telecom (+12.69 pp) and Bank (+30.5 pp), whereas F2 redundancy generalised within telecom but not to banking.The Bank benchmark’s maximum feature R2 was 0.094, below the proposed R2 > 0.95 diagnostic threshold.Source: §4.6 · p. 17, §4.7 · p. 17

5.1 Why Smote Leaks but Class Weighting Does Not

• 36.6±2.5% of synthetic-train points have a test-fold parent, while class weighting creates no synthetic samples and cannot geometrically contaminate training.SMOTE’s linear interpolation can combine real points that later fall on opposite sides of the split.Source: §5.1 · p. 17

5.2 Calibration on Class-Weighted Tree Ensembles

• Temperature scaling fails on class-weighted tree ensembles because one scalar cannot correct bimodal outputs; isotonic regression absorbs heterogeneous miscalibration shapes.The recommendation of temperature scaling for multi-class softmax outputs should not be transferred to binary class-weighted tree ensembles without testing.Source: §5.2 · pp. 17–18

5.3 Feature Redundancy and the Limits of SHAP

• Feature redundancy leaves P-Clean and P-Drop with indistinguishable test accuracy, yet a nonpredictive redundant feature appears in every ranking’s top ten; the pre-model audit fits TotalCharges from other numeric features and checks R2.Source: §5.3 · p. 18

5.4 Limitations

• F3 and F4 were not replicated on Iranian or Bank churn, and the study did not tune hyperparameters.The cost analysis assumes a fixed $50 retention offer and 24-month CLV proxy; its unconstrained threshold flags 75.1 ± 5.9% of customers. SHAP analysis is limited to TreeSHAP on the canonical GBM model.Source: §5.4 · p. 18

6 Conclusion: A Reporting Checklist for Trustworthy Churn DSS

• The audit identifies four failures: SMOTE-specific leakage, redundant-feature SHAP distortion, calibration failures, and cost-insensitive thresholds; F1 generalises across Iranian and Bank benchmarks, while F2 generalises within telecom.The conclusion reports +13.1 pp F1 for F1 and R2 = 0.999 for F2.Source: §6 · p. 18

6.1 The Four-Component Reporting Checklist

• The checklist calls for pipeline-order disclosure, a feature-redundancy diagnostic, a calibration audit, and cost-sensitive threshold reporting.It specifies an R2 > 0.95 redundancy flag and recommends isotonic regression for binary class-weighted tree ensembles.Source: §6.1 · p. 19

6.2 Closing

• The paper presents its contribution as an audit, checklist, and cross-benchmark validation, not as a better churn classifier.Code, intermediate tables, and figures for reproducing the analyses are released under an OSI-approved permissive license.Source: §6.2 · p. 19

• The research received no external funding, and the author declared no competing financial or personal interests.Source: Declarations · p. 20
• The IBM Telco and Iranian Telecom datasets are publicly available, and the analysis code is released under an MIT License.Source: Declarations · p. 20
• Ethics approval, consent to participate, consent for publication, and materials availability are marked not applicable; the author contribution is sole-author.Source: Declarations · p. 20