PaperlayerView exact revision on arXiv

Source-linked AI summary

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz

arXiv:2609.11838v1cs.CL

TL;DR

Survey-based cardiovascular screening literature reports AUROC near 0.89, yet controlled evidence on how much survives removal of leakage-prone features and whether this varies by learner is limited. This study benchmarks ten classifier classes across leakage-risk tiers and joint deployment audits, finding that removing two post-diagnostic markers lowers every model by 0.049–0.051 AUROC and compresses T1 results into a 0.0045-wide band.

  • Problem

    Prior survey-based cardiovascular screening literature lacks controlled, stepwise evidence on how much reported accuracy survives removal of leakage-prone predictors and whether that depends on the learner.

    Source: BACKGROUND AND SIGNIFICANCE · p. 2

  • Method

    The study benchmarks ten classifier classes across prespecified tiers of decreasing leakage risk and jointly evaluates discrimination, calibration, threshold fairness, conformal coverage, explanation faithfulness, inference cost, and temporal transport.

    Source: OBJECTIVE · p. 3, Leakage-tiered feature design · p. 4

  • Results

    0.049–0.051 AUROC: removing two direct post-diagnostic markers reduced every model's test discrimination and left T1 results in a 0.0045-wide band.

    Source: Two post-diagnostic features account… · p. 5

  • Takeaways & Limitations

    The explainable boosting machine was non-inferior to every alternative at each tier, supporting transparency without a measurable discrimination penalty in this benchmark.

    Source: The glass-box model is statistically… · p. 8

    The task concerns prevalent, self-reported, ever-diagnosed myocardial infarction in a cross-sectional survey, not incident risk, and fatal or undiagnosed events are structurally invisible.

    Source: Limitations · p. 13

Abstract

from arXiv · show

Cardiovascular screening models trained on national health surveys routinely report areas under the receiver operating characteristic curve (AUROC) near 0.89. We asked whether that accuracy reflects learning or target leakage, whether tabular foundation models change the answer, and whether the properties deployment requires survive joint examination. We benchmarked ten classifiers spanning linear, tree-ensemble, neural, glass-box, and tabular foundation classes for prevalent myocardial infarction in 442,067 respondents of the 2022 Behavioral Risk Factor Surveillance System across five feature tiers of decreasing leakage risk. Each was audited for discrimination, calibration, fairness at an explicit screening threshold, conformal coverage, explanation faithfulness, and inference cost, then applied -- models and thresholds frozen -- to 430,755 respondents of 2023. Removing two post-diagnostic features cost every model 0.049-0.051 AUROC, collapsing the field into a 0.0045-wide band. The glass-box explainable boosting machine was non-inferior to every alternative within a pre-specified 0.005 margin while scoring the cohort roughly 104 times faster than the strongest foundation model. One threshold detected 75.4% of women's infarctions against 89.0% of men's; editing the model's shape functions reduced the gap to 0.010. Marginal conformal prediction gave 0.86 coverage to men and 0.82 to adults over 60; Mondrian calibration repaired every stratum. Frozen models transported within 0.002 AUROC. Reported headroom in this literature is a property of the feature set, not the learner. Transparency cost nothing measurable and made fairness repair and uncertainty conditioning directly auditable. Evaluation practice, not model capacity, is the binding constraint.

BACKGROUND AND SIGNIFICANCE

BRFSS cardiovascular screening reports repeatedly cluster near AUROC 0.89, but leakage may inflate them, and the literature lacks controlled tests of how performance changes when leakage-prone features are removed. No prior study has jointly tested discrimination, fairness, uncertainty, and temporal transport under deployment-relevant conditions.

BACKGROUND AND SIGNIFICANCE

  • Reported BRFSS discrimination clusters near AUROC 0.89, while the survey’s national scope and low cost make it attractive for screening triage.The literature also includes widely redistributed derivatives and public-facing tools.Source: BACKGROUND AND SIGNIFICANCE · pp. 1–2
  • Leakage can inflate performance without screening ability; diagnosis codes recorded with the outcome demonstrate this failure, while controlled dose–response tests remain lacking.Prior cardiovascular survey corrections point in the same direction.Source: BACKGROUND AND SIGNIFICANCE · p. 2
  • Tabular foundation models and editable glass-box models motivate comparison, but their behavior under leakage stress, subgroup audit, conformal calibration, temporal shift, and population-scale inference cost remains unresolved [5] [8] [9].Glass-box editing had not been quantified as a fairness intervention against standard mitigation baselines, nor used as ground truth for post-hoc explanation faithfulness.Source: BACKGROUND AND SIGNIFICANCE · p. 2
  • Fairness is usually reported without an explicit operating threshold, conformal guarantees are seldom group-conditioned, and temporal validation may refit models rather than test frozen deployment models.The section identifies the lack of a study applying all these audits to the same cohort and models.Source: BACKGROUND AND SIGNIFICANCE · p. 2

OBJECTIVE

The study asks whether reported cardiovascular-screening discrimination reflects target leakage rather than model capacity, and whether models remain transparent, equitable, calibrated, uncertainty-aware, and transportable after leakage-prone predictors are removed.

OBJECTIVE

  • The study measures how much reported survey-based cardiovascular-screening discrimination is attributable to target leakage rather than model capacity across model classes.Source: OBJECTIVE · p. 3
  • It tests whether a transparent glass-box model is statistically non-inferior to tuned gradient boosting and large-context tabular foundation models after leakage-prone predictors are removed, and compares inference cost.Source: OBJECTIVE · p. 3
  • It examines whether calibration, subgroup equity at an explicit screening threshold, group-conditional uncertainty, and year-to-year transportability hold simultaneously for the resulting models.Source: OBJECTIVE · p. 3

MATERIALS AND METHODS

The study defined prevalent myocardial-infarction prediction across 2022 and 2023 BRFSS cohorts, pre-specified leakage-tiered features and a ten-classifier protocol, and audited trustworthiness before frozen temporal validation.

Data and cohorts

  • The 2022 development cohort included 442,067 respondents and the 2023 validation cohort 430,755, predicting self-reported ever-diagnosed myocardial infarction.Respondents with missing outcomes were excluded, but item missingness was retained natively; complete-case analysis would have discarded 42.9% of 2022 and 98.7% of 2023 respondents.Source: Data and cohorts · p. 3

Leakage-tiered feature design

  • The 39 predictors were assigned to five pre-specified leakage tiers, from full T0 to clinician-free T2, with T1 as the primary screening tier.T0 retained prior angina/coronary disease diagnosis and chest CT; T1 removed both, T1-ns additionally removed prior stroke, and T1-portable restricted T1 to items asked in both cycles for temporal validation.Source: Leakage-tiered feature design · pp. 3–4

Models and protocol

  • Ten classifiers spanning linear, tree, glass-box, neural, and tabular foundation models were evaluated using five-seed stratified 60/20/20 splits and validation-only tuning.The explainable boosting machine was the reference model; TabPFN v2 and TabICL were applied zero-shot, while the test partition was read once per cell and did not inform modeling choices.Source: Models and protocol · p. 4

Evaluation

  • The audit covered discrimination, calibration, clinical utility, threshold-based fairness, conformal uncertainty, explanation faithfulness, inference cost, and frozen temporal validation.Fairness was assessed at a validation-selected threshold with sensitivity ≥0.85; conformal prediction used α = 0.1, and T1-portable models and thresholds were applied unchanged to all of 2023, including evaluation under survey weights.Source: Evaluation · p. 5

RESULTS

Removing two post-diagnostic markers reduced every model’s AUROC by 0.049–0.051, while the glass-box model remained non-inferior to alternatives. Further audits identified threshold disparities and uneven marginal conformal coverage, alongside strong frozen-year transport.

Two post-diagnostic features account for the reported headroom

  • 0.8905–0.8933 test AUROC on T0 fell 0.049–0.051 across all ten models at T1, collapsing them to 0.8395–0.8440 and locating headroom in features, not learner.Removing prior stroke as well (T1-ns) cost a further 0.0075–0.0086 AUROC; restricting to self-report-only predictors (T2) cost a further ∼0.021.Source: Two post-diagnostic features account… · p. 5, Figure 2 · p. 6, Table 2 · p. 7

The glass-box model is statistically equivalent to every alternative

  • At T1, the glass-box model reached 0.8430 ± 0.0019 AUROC and was non-inferior to every comparator within δ = 0.005 (Holm-adjusted p < 0.001).At T1, CatBoost reached 0.8439, XGBoost 0.8430, the multilayer perceptron 0.8432 and full-context TabICL 0.8440; two-sided equivalence held against all nine comparators at T0 and T1-portable and eight of nine at T1. The glass-box scored the 88 413-respondent test partition in 0.1 s, versus 1153.8 s of GPU inference for TabICL and 642.5 s for TabPFN v2.Source: Figure 2 · p. 6, The glass-box model is statistically… · p. 8

Discrimination parity, calibration divergence

  • 0.234–0.291 expected calibration error in class-weighted classical models contrasted with 0.010 in the unweighted foundation models, despite parity in discrimination.Refitting without class weights yielded error 0.0031 for the glass-box model and 0.0019 for CatBoost; isotonic recalibration restored every model to error ≤0.005 without altering discrimination, after which all competitive models showed positive net benefit across the 1–30% threshold range.Source: Discrimination parity, calibration… · p. 8, Figure 3 · p. 8

A single threshold under-detects women; glass-box repair closes the gap

  • 75.4% of women’s prior infarctions versus 89.0% of men’s were detected at the pre-specified threshold, a TPR gap of 0.128 ± 0.008 across seeds.Shape repair with intercept equalization reduced the gap to 0.0104 ± 0.0099 at essentially identical specificity; the methods were statistically matched. Closing the TPR gap reduced the false-positive-rate gap from 0.141 to 0.007 but widened the positive-predictive-value gap from 0.038 to 0.070; sex-linked proxy pathways included age, general health and smoking status.Source: A single threshold under-detects… · pp. 8–9, Figure 4 · p. 9, Table 3 · p. 10

Marginal conformal guarantees hide group-level failure

  • 0.856–0.858 conformal coverage for men and 0.818–0.819 for adults aged 60+ contrasted with 0.938–0.940 for women and 0.992–0.993 for ages 18–39 under marginal calibration.Mondrian calibration restored every stratum to 0.90, with oldest-stratum deferral rising from 0.586 to 0.712; a hybrid retaining marginal calibration for ages 18–39 and Mondrian elsewhere dominated both pure policies under every examined utility weighting.Source: Marginal conformal guarantees hide… · p. 10, Figure 5 · p. 10

Post-hoc explanations are not glass-box explanations

  • 0.884 Kendall τ was TreeSHAP’s plateau against the glass-box model’s exact importances, while LIME plateaued at τ ≈0.626 and top-10 Jaccard 0.45.KernelSHAP applied to the glass-box recovered its own importances gradually, with τ = 0.724 at 25 samples and 0.870 at 500; LIME misidentified roughly half of the ten strongest drivers.Source: Post-hoc explanations are not… · p. 11, Figure 6 · p. 11

Frozen models and thresholds transport to the next survey year

  • 0.0009–0.0016 AUROC was lost by every T1-portable model when frozen models and thresholds were applied to all 430 755 respondents in 2023.Calibration slopes stayed within 0.91–1.19, and frozen 2022 thresholds delivered 2023 sensitivities of 0.855–0.862 against the 0.85 target.Source: Frozen models and thresholds transport… · p. 11, Table 4 · p. 11

Sensitivity analyses and the survey-weighted population

  • 0.843 to 0.809 AUROC followed SMOTE-NC oversampling at T1, while survey design weights raised discrimination by +0.020 to +0.022 for all ten models without changing their ranking statistically.SMOTE-NC inflated training time ∼150-fold; complete-case analysis cost 0.0023 AUROC while discarding 42.9% of respondents. In the highest design-weight quintile, CatBoost was better than TabICL by 0.0029 (95% CI 0.0009–0.0048), and weight-informed training did not improve weighted-population discrimination.Source: Figure 2 · p. 6, Sensitivity analyses and the… · p. 12

DISCUSSION

Across the evaluated models, high accuracy on the full feature set depended on post-diagnostic information, while removing two such markers reduced AUROC by roughly 0.05. The discussion argues that transparent models can match alternatives, but screening performance also depends on operating thresholds, subgroup conditioning, and the limits of the survey setting.

Principal findings

  • Every model reproduced the ∼0.89 AUROC regime on the full feature set and lost roughly 0.05 AUROC when two post-diagnostic markers were removed.The authors interpret the near-uniform decline as a leakage tax rather than a difference in learning capacity; residual care-related signal may remain after removing direct markers.Source: Principal findings · p. 12
  • At reduced leakage, the glass-box model was non-inferior to every alternative within a pre-specified margin and scored four orders of magnitude faster than TabICL.TabICL matched the best boosted trees without tuning; both foundation models were natively calibrated, while class-weighted classical models required isotonic repair. Foundation-model advantages shrank under survey weighting, and the subsampled-context variant was weak.Source: Principal findings · p. 12
  • At the selected threshold, one in four true cases among women was missed, compared with one in nine among men, for both the glass-box model and XGBoost.Shape repair with intercept equalization matched conventional mitigation at matched specificity and exposed group-specific effective thresholds as reversible model edits.Source: Principal findings · p. 12
  • Marginal conformal prediction gave men 86% coverage and adults over sixty 82%, while KernelSHAP needed hundreds of samples and LIME never converged even for the glass-box model.Group-conditional repair could also produce empty prediction sets in very-low-risk strata, a cost hidden by marginal accounting.Source: Principal findings · pp. 12–13

Limitations

  • The findings concern self-reported, ever-diagnosed prevalent infarction in a cross-sectional U.S. survey, and transportability beyond the same surveillance system and country remains untested.The study also notes residual care-related leakage, point-estimate-only survey weighting, measured-attribute-only fairness audits, and that conformal validity can fail under drift.Source: Limitations · p. 13

CONCLUSION

Removing two post-diagnostic features leaves a transparent glass-box model competitive with gradient boosting and tabular foundation models, while supporting deployment-relevant auditing and uncertainty handling. The conclusion identifies evaluation practice, rather than model capacity, as the field’s binding constraint.

CONCLUSION

  • Removing two post-diagnostic features lets a transparent, editable glass-box model meet tuned gradient boosting and full-context tabular foundation models head-on.The conclusion attributes the celebrated performance on heavily reused public cardiovascular survey data to those features, not model sophistication.Source: CONCLUSION · p. 13
  • The sub-second glass-box model supports explicit operating points, inspectable fairness repair, group-conditional uncertainty with deferral, exact explanations, and thresholds that transport across survey years.Its auditable fairness repair matches conventional mitigation while remaining inspectable.Source: CONCLUSION · p. 13
  • Evaluation practice, not model capacity, is the binding constraint on the field.Source: CONCLUSION · p. 13

Data availability

The study used publicly available 2022 and 2023 BRFSS files; raw data were not redistributed, while derived cohort logs and per-cell results accompany the code release.

Data availability

  • The 2022 and 2023 BRFSS Landline and Cellular Telephone files are publicly available without restriction.They are distributed by the Centers for Disease Control and Prevention.Source: Data availability · p. 13
  • No raw data are redistributed with the study.Source: Data availability · p. 13
  • Derived cohort construction logs and every per-cell result accompany the code release.Source: Data availability · p. 13

Supplementary Information · Supplementary Figures · Supplementary Tables

The supplement documents cohort construction, geography and demographics, feature-tier definitions, and extended fairness and conformal audits. It also shows matched gap closure for selected sex-mitigation methods and replication of glass-box risk shapes across survey years, with sleep duration unavailable in the 2023 core questionnaire.

Supplementary Information

  • The appendix contains the supplementary figures, tables, methods, and note referenced in the main text and matches the journal-filed Supplementary Information.Source: Supplementary Information · p. 17

Supplementary Figures

  • Supplementary Fig. 1 depicts record flow for the 2022 development and 2023 temporal-validation cycles, with complete-case counts shown for reference while primary analyses retain native missingness.Source: Figure 1 · p. 17
  • The 2022 geography and demographic figure maps respondents and crude self-reported MI prevalence by state and shows a male prevalence excess emerging in midlife and persisting beyond age 80.Territories GU, PR, and VI, with n = 9,258, are not mapped; prevalence is unweighted and age–sex intervals are 95% Wilson intervals.Source: Figure 2 · p. 18
  • Per-group thresholds, reweighing, and shape repair with intercept equalisation achieved statistically matched sex-gap closure at matched specificity, while partial and unawareness repairs stalled.The projection omits false-positive-rate and positive-predictive-value gaps; numerical values are reported in Table 3.Source: Figure 3 · p. 19
  • Glass-box risk shapes were shown for 2022 and an independent 2023 refit on the year-portable tier; sleep duration appears only for 2022 because it left the 2023 core questionnaire.Source: Figure 4 · p. 20

Supplementary Tables

  • Split conformal used α = 0.1 with a 0.90 per-group target; its carve-out hybrid used marginal calibration for ages 18–39 and Mondrian calibration elsewhere.Marginal calibration has zero empty sets and obeys average set size = 1 + deferral, an identity that breaks under Mondrian calibration in the lowest-risk stratum; respondents missing age were excluded from age strata.Source: Table 3 · p. 22, Supplementary Tables · p. 22
  • Age-conditioned thresholds closed raw fairness gaps only at heavy specificity and positive-predictive-value cost.The sensitivity-target threshold variant coincides with the TPR-equalisation variant; shape-repair arms targeting Sex terms are not defined for other attributes.Source: Supplementary Tables · p. 23, Table 4 · p. 23

Supplementary Methods · Splits, tuning and the pre-specification regime

The protocol separated tuning, fitting, and evaluation: configurations were selected on a validation split and frozen across seeds, while test partitions were reserved for reporting. The full analysis plan was fixed before full-scale execution.

Splits, tuning and the pre-specification regime

Statistical procedures … Supplementary Note

The supplementary material specifies the inferential procedures, reversible fairness and explanation audits, computing setup, and accompanying transparency materials.

Statistical procedures

  • The primary tests used the pre-specified δ = 0.005 AUROC margin, paired DeLong variances on the seed-0 test partition, and Holm correction.Comparisons were one-sided non-inferiority tests and two-one-sided equivalence tests; seed-level comparisons across five paired replications were descriptive because five replications lacked meaningful testing power.Source: Statistical procedures · p. 24
  • A DeLong permutation audit yielded a 6.0% type-I rate at α = 0.05 over 400 null draws, within Monte-Carlo error.Uncertainty on between-arm fairness-gap differences used paired seed-level contrasts and a 1000-draw stratified (sex×outcome) bootstrap of the seed-0 test partition.Source: Statistical procedures · p. 24

Fairness and explanation details

  • Fairness audits defined sex using BRFSS SEXVAR and assessed age bands and race/ethnicity in threshold-based arms.Shape-repair edits zeroed or attenuated the sex main effect and sex-involving interactions (γ = 0.5); intercept equalisation added validation-selected per-group offsets, reported alongside the arm.Source: Fairness and explanation details · p. 24
  • LIME was applied to the glass-box model with integer-encoded categoricals to distinguish estimator error from cross-model disagreement.Source: Fairness and explanation details · p. 24

Software and hardware

  • Experiments ran on a Ryzen 9 7950X workstation with 128 GB RAM and an RTX 4090; foundation-model inference used the GPU and other models used the CPU.Foundation-model timings used library defaults without batching tuning, quantisation, or context caching, representing out-of-the-box cost rather than an optimised lower bound.Source: Software and hardware · p. 24
  • Every result row recorded stage, tier, model, seed, and split, while all tables and figures were generated from one append-only results file.Versions were pinned in the code release and frozen at run time, avoiding manual transcription of reported numbers.Source: Software and hardware · p. 25

Supplementary Note

  • The submission includes a completed TRIPOD+AI checklist and openly available analysis pipeline, configurations, per-cell results, and model edit logs.Source: Supplementary Note · p. 25

Related papers

Loading 2609.11838v1…