Source-linked AI summary
Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models
Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz
TL;DR
The paper asks whether near-0.89 AUROC in survey-based cardiovascular screening reflects learning or target leakage, and whether model class changes that answer. It benchmarks ten classifiers across leakage tiers and audits deployment-relevant properties with frozen temporal validation. Removing two post-diagnostic features reduced every model by 0.049–0.051 AUROC, while the glass-box model remained non-inferior to alternatives and was much faster.
Problem
The paper addresses limited evidence on how much reported survey-screening discrimination survives controlled removal of target-leaking predictors and whether learner class changes the result.
Method
Ten classifiers were evaluated across five decreasing-leakage feature tiers for prevalent myocardial infarction, with audits of discrimination, calibration, fairness, conformal coverage, explanation faithfulness, inference cost, and frozen 2023 transport.
Results
0.049–0.051 AUROC was lost across every model after removing two post-diagnostic features, while the explainable boosting machine was non-inferior to every comparator within δ = 0.005 and scored far faster than foundation models.
Takeaways & Limitations
Reported headroom was a property of the feature set rather than the learner, and transparent modeling supported auditable fairness and uncertainty evaluation without measurable discrimination loss.
Takeaways & Limitations
The outcome was self-reported prevalent, surviving, diagnosis-aware myocardial infarction in one national surveillance system and therefore does not represent incident risk or undiagnosed and fatal events.
Abstract
from arXiv · showhide
Cardiovascular screening models trained on national health surveys routinely report areas under the receiver operating characteristic curve (AUROC) near 0.89. We asked whether that accuracy reflects learning or target leakage, whether tabular foundation models change the answer, and whether the properties deployment requires survive joint examination. We benchmarked ten classifiers spanning linear, tree-ensemble, neural, glass-box, and tabular foundation classes for prevalent myocardial infarction in 442,067 respondents of the 2022 Behavioral Risk Factor Surveillance System across five feature tiers of decreasing leakage risk. Each was audited for discrimination, calibration, fairness at an explicit screening threshold, conformal coverage, explanation faithfulness, and inference cost, then applied -- models and thresholds frozen -- to 430,755 respondents of 2023. Removing two post-diagnostic features cost every model 0.049-0.051 AUROC, collapsing the field into a 0.0045-wide band. The glass-box explainable boosting machine was non-inferior to every alternative within a pre-specified 0.005 margin while scoring the cohort roughly 104 times faster than the strongest foundation model. One threshold detected 75.4% of women's infarctions against 89.0% of men's; editing the model's shape functions reduced the gap to 0.010. Marginal conformal prediction gave 0.86 coverage to men and 0.82 to adults over 60; Mondrian calibration repaired every stratum. Frozen models transported within 0.002 AUROC. Reported headroom in this literature is a property of the feature set, not the learner. Transparency cost nothing measurable and made fairness repair and uncertainty conditioning directly auditable. Evaluation practice, not model capacity, is the binding constraint.
BACKGROUND AND SIGNIFICANCE
Survey-based cardiovascular screening commonly reports AUROC near 0.89, but the literature has not measured how much survives controlled removal of target-leaking predictors or whether results hold across learners and deployment-relevant audits.
- AUROC near 0.89 is repeatedly reported for BRFSS-derived cardiovascular screening models, motivating scrutiny of that benchmark.
- Target leakage occurs when predictors are consequences of the outcome rather than antecedents, inflating apparent performance without providing screening ability.
- The central gap is a controlled dose–response measurement of accuracy after leakage-prone predictors are removed, across different learners.
- Tabular foundation models invite national-survey screening evaluation, but their behavior under leakage stress, subgroup audit, conformal calibration, temporal shift, and population-scale inference cost remains open.
- Existing evaluations rarely examine fairness at an explicit operating threshold, group-conditioned uncertainty, and frozen temporal validation together.
OBJECTIVE
The study asks whether reported cardiovascular-screening discrimination reflects target leakage rather than model capacity and whether transparent models retain deployment-relevant trustworthiness after leakage-prone predictors are removed.
- The study measures how much reported discrimination is attributable to target leakage rather than model capacity across model classes.
- It tests whether a transparent glass-box model is non-inferior to boosted ensembles and large-context tabular foundation models after leakage-prone predictors are removed, including inference cost.
- It evaluates whether calibration, subgroup equity at an explicit threshold, group-conditional uncertainty, and year-to-year transportability hold simultaneously.
MATERIALS AND METHODS
The study trained ten classifier classes on nested BRFSS feature tiers with decreasing leakage risk, audited discrimination and deployment properties under a prespecified protocol, and froze 2022 models and thresholds for 2023 validation.
- The analysis used 442,067 eligible 2022 BRFSS respondents for development and 430,755 2023 respondents for temporal external validation, retaining item missingness natively.
- Leakage-tiered feature design: Thirty-nine predictors were assigned before fitting to nested tiers T0, T1, T1-portable, and T2, with T1-ns as a sibling ablation.
- Leakage-tiered feature design: T0 retained two direct post-diagnostic markers, whereas primary screening tier T1 removed them; T1-portable used cross-year questionnaire items and T2 retained self-reportable non-clinician-mediated items.
- Ten models covered linear, tree-ensemble, glass-box additive, neural, and zero-shot tabular foundation classes, using five-seed stratified splits and validation-only tuning.
- Evaluation: The audit measured AUROC, precision–recall area, calibration, decision-curve utility, thresholded sex TPR gaps, conformal coverage, explanation faithfulness, and inference cost.
- Evaluation: All 2022 T1-portable models and validation-selected thresholds were applied unchanged to 2023 without refitting or re-thresholding, with additional survey-weighted reevaluation.
RESULTS
Removing two post-diagnostic features reduced AUROC similarly for every model, while the glass-box model matched alternatives within the prespecified margin and required dramatically less inference time; calibration differences remained after discrimination converged.
- Two post-diagnostic features account for the reported headroom: 0.049–0.051 AUROC was lost for every model when two post-diagnostic markers were removed, collapsing T1 performance into a 0.0045-wide band of 0.8395–0.8440.
- The glass-box model is statistically equivalent to every alternative: At T1, the explainable boosting machine reached 0.8430 ± 0.0019 versus CatBoost 0.8439, XGBoost 0.8430, the multilayer perceptron 0.8432, and TabICL 0.8440, remaining non-inferior to every comparator within δ = 0.005.
- The glass-box model is statistically equivalent to every alternative: 0.1 s of glass-box scoring contrasted with 1153.8 s for TabICL and 642.5 s for TabPFN v2 on the identical 88,413-respondent workload.
- Discrimination parity, calibration divergence: Class-weighted classical models had expected calibration error 0.234–0.291, foundation models 0.010, and isotonic recalibration reduced every model to error ≤0.005 without changing discrimination.
A single threshold under-detects women; glass-box repair closes the gap
At the fixed screening threshold, the glass-box model detected fewer prior infarctions in women than men, while targeted repair reduced the disparity without materially changing specificity. Conformal results likewise showed that marginal guarantees could conceal weaker coverage for higher-risk groups.
- A single threshold under-detects women; glass-box repair closes the gap: 75.4% of women’s prior infarctions versus 89.0% of men’s were detected, producing a TPR gap of 0.128 ± 0.008.XGBoost was indistinguishable, with a gap of 0.127 ± 0.008.
- A single threshold under-detects women; glass-box repair closes the gap: 0.0104 ± 0.0099 was the TPR gap after glass-box shape repair with intercept equalization, statistically matching per-group thresholds and reweighing at essentially identical specificity.The paired repair difference versus thresholds was −0.0013, with a 95% bootstrap CI of −0.0049 to 0.0022.
- A single threshold under-detects women; glass-box repair closes the gap: 0.043 remained after deleting sex terms outright, while the fitted interaction structure exposed sex-linked proxy pathways through age, general health, and smoking status.The figure encodes main-effect importance as node size and pairwise-interaction importance as chord width.
- A single threshold under-detects women; glass-box repair closes the gap: 0.141 → 0.007 in false-positive-rate gap accompanied TPR-gap closure, but the positive-predictive-value gap widened from 0.038 → 0.070.Baseline age-band disparities were larger and costly to close, whereas race and ethnicity gaps fell from 0.243 to 0.082 at nearly retained specificity.
- A single threshold under-detects women; glass-box repair closes the gap: 0.818–0.819 coverage for adults aged 60+ was lower than 0.992–0.993 for adults aged 18–39 under marginal conformal prediction.Mondrian calibration restored every reported stratum to 0.90, with deferral and empty prediction sets representing its costs.
Post-hoc explanations are not glass-box explanations
Post-hoc explainers only gradually recovered the glass-box model’s exact importances, while frozen models transported to 2023 with minimal AUROC loss. Sensitivity analyses also showed that common preprocessing could reduce performance substantially without changing the broader model ranking.
- Post-hoc explanations are not glass-box explanations: TreeSHAP reached Kendall τ = 0.884 against exact importances, while LIME plateaued near τ ≈0.626 and misidentified roughly half of the ten strongest drivers.KernelSHAP applied to the glass-box model itself improved from τ = 0.724 at 25 samples to 0.870 at 500.
- Post-hoc explanations are not glass-box explanations: 0.0009–0.0016 AUROC was lost when 2022 models and thresholds were applied frozen to all 430,755 respondents in 2023.2023 sensitivities remained 0.855–0.862 against the 0.85 target, with calibration slopes of 0.91–1.19.
DISCUSSION
Across model classes, reported cardiovascular-screening accuracy was driven by post-diagnostic leakage rather than model sophistication. At reduced leakage, transparent models matched alternatives while enabling auditable fairness and uncertainty repair, but the study’s cross-sectional survey outcome and transportability remain bounded.
- Principal findings: 0.049-0.051 AUROC was lost by every model after removing two post-diagnostic markers, showing that the celebrated headroom was leakage rather than learning.The decline generalized across ten models and five leakage tiers.
- Principal findings: The glass-box model was non-inferior to every alternative within the pre-specified margin and scored the cohort four orders of magnitude faster than TabICL.TabICL matched the best boosted trees without tuning, while foundation-model advantages shrank under survey weighting.
- Fairness: At the fixed screening threshold, women’s true-positive rate lagged men’s, while shape repair with intercept equalization provided a reversible, auditable fairness edit.The threshold disparity was essentially identical for the glass-box model and XGBoost, indicating a data-plus-threshold property.
- Uncertainty: Marginal conformal prediction met average coverage but under-covered men and adults over 60, whereas Mondrian calibration restored every stratum.The group-coverage pattern replicated across the glass-box model, XGBoost, and TabICL.
- Explanation faithfulness: The faithfulness analysis cautioned that standard post-hoc explanations can be computationally demanding or unstable even when explaining the glass-box model itself.KernelSHAP required hundreds of samples and LIME never converged.
- Limitations: The task is bounded to prevalent, surviving, diagnosis-aware myocardial infarction in one cross-sectional survey system, with transportability to clinical registries untested.The outcome is self-reported, fatal or undiagnosed events are invisible, and conformal validity can fail under drift.
CONCLUSION
The study concludes that two post-diagnostic features, rather than model sophistication, explain celebrated performance on reused cardiovascular survey data. After their removal, a transparent glass-box model matched stronger alternatives while supporting auditable deployment controls.
- Conclusion: Two post-diagnostic features—not model sophistication—explain the celebrated performance on heavily reused public cardiovascular survey data.
- Conclusion: After leakage reduction, the transparent, editable glass-box model matched tuned gradient boosting and full-context tabular foundation models while supporting explicit operating points and auditable fairness repair.
Data availability
The study uses publicly available 2022 and 2023 BRFSS landline and cellular telephone files, while withholding raw-data redistribution. Cohort construction logs and per-cell results accompany the code release.
- Data availability: The 2022 and 2023 BRFSS Landline and Cellular Telephone files are publicly available without restriction and distributed by the CDC.
- Data availability: Raw data are not redistributed, but derived cohort-construction logs and every reported per-cell result accompany the code release.
Funding
The study was supported by Abu Dhabi University’s Office of Research and Sponsored Programs.
- Funding: The study was supported by Abu Dhabi University’s Office of Research and Sponsored Programs under grant 19300893.
Supplementary Information
The supplementary materials document cohort construction, feature tiers, audit procedures, fairness repairs, conformal methods, and reproducible implementation details. They specify how models were evaluated, edited, and validated across seeds and survey years.
- Supplementary figures: The supplementary figures track cohort construction, geography and demography, mitigation performance, and replication of learned risk shapes across survey years.The risk-shape replication figure compares 2022 contributions with an independent 2023 refit on the year-portable tier, excluding sleep duration from 2023 because it was removed from the core questionnaire.
- Evaluation protocol: Each tier uses five seeds with stratified 60/20/20 splits, validation-only tuning, and a single-read test partition that informs no modeling choice.Configurations and the full protocol were frozen before execution, while seed variation reflects split and fitting variability rather than retuning.
- Audit framework: Calibration, discrimination, clinical utility, conformal coverage, and explanation faithfulness are evaluated alongside pre-specified AUROC non-inferiority testing.The non-inferiority margin is δ = 0.005 AUROC, with paired DeLong tests, Holm correction, equivalence testing, calibration metrics, decision curves, and conformal audits.
- Fairness and explanation audits: Fairness audits fix a validation threshold at sensitivity ≥0.85 and compare four mitigation families using sex-specific true-positive-rate gaps.Shape repair edits sex main and interaction terms, while intercept equalization adds validation-selected per-group offsets; the edits are reversible and machine-readable.