PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing for the decision not the prediction: an exploration of Smooth Net Benefit as a training objective

Koen M. F. Gorgels, Lasai Barreñada, Maarten van Smeden, Ben Van Calster, Ewout W. Steyerberg, Wouter A. C. van Amsterdam

arXiv:2609.12752v1cs.LG

TL;DR

Clinical prediction models are often trained to optimize global likelihood even when downstream decisions depend on specific risk thresholds. This paper introduces sigmoid-smoothed Net Benefit training and finds no consistent substantial gains over NLL, with modest gains concentrated in logistic regression.

  • Problem

    Training commonly optimizes global predictive performance, while clinically relevant decision thresholds may be under-prioritized despite their importance to downstream decisions.

    Source: §1 · p. 3

  • Method

    The paper introduces Smooth Net Benefit, a sigmoid-smoothed approximation of Net Benefit, and evaluates it as a training objective across logistic regression, GAMs, and XGBoost.

    Source: §1 · p. 3, §2.1 · p. 4

  • Results

    Across the evaluated settings, Smooth Net Benefit did not consistently or substantially improve Net Benefit; gains were clearest for logistic regression, while GAMs and XGBoost showed little or no benefit.

    Source: §4 · p. 15, Conclusion · p. 16

  • Takeaways & Limitations

    The findings do not support Smooth Net Benefit as a general replacement for likelihood-based training, and model class was more strongly associated with decision performance in the settings considered.

    Source: §4 · p. 15, Conclusion · p. 16

    Benchmark evaluation focused on thresholds centered around prevalence and its inverse, which may limit generalizability to substantially different or narrowly defined decision thresholds.

    Source: §4 · p. 16

Abstract

from arXiv · show

Objective Prediction models are commonly trained using objectives such as Bernoulli negative log-likelihood (NLL), although downstream clinical decisions may depend on specific risk thresholds. We introduce Smooth Net Benefit ($\sigma$NB), a differentiable approximation of Net Benefit designed to align model training with threshold-specific clinical utility. Materials and Methods We evaluated $\sigma$NB as a training objective for logistic regression, generalized additive models (GAMs), and XGBoost with three Hessian implementations. Experiments used the Framingham cardiovascular risk dataset and 44 TabZilla datasets comprising 72 dataset-threshold combinations. Results $\sigma$NB training did not consistently improve Net Benefit in Framingham. Across the TabZilla benchmark, mean standardized Net Benefit for logistic regression increased from 0.5669 with NLL to 0.5765 with $\sigma$NB (mean difference 0.0096, 95% CI -0.0001 to 0.0193). For GAMs, mean standardized Net Benefit decreased from 0.5921 to 0.5625 (mean difference -0.0296, 95% CI -0.0721 to 0.0129). For XGBoost, NLL achieved 0.6745 compared with 0.6723--0.6735 across $\sigma$NB implementations. In logistic regression, $\sigma$NB gains were positively associated with the performance advantage of XGBoost over NLL-trained logistic regression. Discussion The effect of $\sigma$NB was context dependent, with modest gains concentrated in logistic regression and little benefit for more flexible model classes. This suggests that decision-focused optimization may be most useful when limited model flexibility leaves greater scope for improvement. Conclusion Our results do not support $\sigma$NB as a general replacement for NLL training, but support further investigation of decision-focused objectives in settings where conventional likelihood-based training may not adequately capture decision-relevant structure.

1 Background and significance

Clinical decisions use risk thresholds that encode trade-offs between beneficial and harmful decisions, while common training objectives optimize predictive performance across the probability range. The paper motivates training that gives greater emphasis to clinically relevant threshold regions and evaluates a smooth Net Benefit objective across several model classes and datasets.

1 Background and significance

  • At a decision threshold, Net Benefit weighs true-positive decisions against false-positive decisions according to the threshold odds.Thresholds reflect the relative value assigned to beneficial and harmful decisions.Source: §1 · p. 3
  • NLL optimizes global predictive performance, which may leave clinically relevant threshold regions under-prioritized during training.The paper notes related work on selectively improving model behavior near plausible decision boundaries.Source: §1 · p. 3
  • The study introduces Smooth Net Benefit, a differentiable approximation of Net Benefit, and evaluates it for logistic regression, GAMs, and XGBoost across Framingham and tabular benchmark datasets.The approach replaces the hard decision threshold with a smooth approximation.Source: §1 · pp. 3–4

2.1 Smooth Net Benefit

Net Benefit measures threshold-specific decision utility using true- and false-positive decisions. Smooth Net Benefit replaces the hard threshold with sigmoid smoothing, making the objective amenable to gradient-based optimization.

2.1 Smooth Net Benefit

  • Net Benefit is the proportion of true-positive decisions minus the proportion of false-positive decisions weighted by the threshold odds.Models predicting at or below the threshold for everyone have zero Net Benefit by definition.Source: §2.1 · p. 4
  • Because Net Benefit is piecewise constant in model parameters with zero gradients across most of the parameter space, it is difficult to optimize directly.It is discontinuous when predictions cross the decision threshold.Source: §2.1 · p. 4
  • Smooth Net Benefit uses sigmoid smoothing to approximate Net Benefit, with inverse-temperature k controlling steepness and larger k producing a closer approximation.The smoothed objective has non-zero gradients everywhere, enabling gradient-based optimization.Source: §2.1 · p. 4
  • For multiple decision thresholds, the objective can average Smooth Net Benefit across thresholds or use a weighted average; the study uses unweighted averages.Source: §2.1 · p. 5

2.2 Motivating 1D example: piece-wise linear function

The example shows how a single-line logistic regression can miss a piecewise-linear relationship in logit space. Optimizing around the relevant decision threshold can instead target the corresponding line segment.

2.2 Motivating 1D example: piece-wise linear function

  • For uniform X from -2 to 2, the increase in NB was 0.036 for NLL and 0.063 for optimal NB.The example uses inflection points (-2, -1.667), (0, -1), and (2, 3).Source: §2.2 · p. 5
  • Standard logistic regression fits a single straight line, interpolating between the two segments of the piecewise-linear logit relationship.The true function is outside logistic regression's model class.Source: §2.2 · p. 5, Figure 2 · p. 6
  • Depending on the decision threshold, fitting either the first or second line segment can achieve better Net Benefit.Direct optimization around the relevant threshold targets the corresponding segment.Source: §2.2 · p. 5, Figure 2 · p. 6

2.3 Training procedure

The training procedure warm-starts Smooth Net Benefit optimization from NLL, gradually sharpens its approximation, and selects checkpoints using Net Benefit. Implementations cover logistic regression, GAMs, and XGBoost, with local calibration methods as comparators.

2.3 Training procedure

  • Inverse temperature k increases from 1 to 4 to 10, sharpening the objective after optimization begins with a smoother approximation.NLL warm-start initialization addresses non-convexity and regions with near-zero gradients; checkpoints are selected by NB, with final evaluation on held-out folds.Source: §2.3 · pp. 6–7, Figure 3 · p. 7

2.3.1 Implementation details

  • For logistic regression and GAMs, the objective changes while model structure and cross-validated regularization remain fixed.Training-set NB selects checkpoints during Smooth Net Benefit continuation.Source: §2.3.1 · p. 7
  • XGBoost uses a custom Smooth Net Benefit objective with absw, fixed025, and true Hessian variants.Continuation starts from NLL-trained models; continuation settings and checkpoint length are selected on inner validation folds before refitting on outer training data.Source: §2.3.1 · p. 8

2.3.2 Post-hoc calibration comparators

  • Local Platt scaling and local temperature scaling serve as post-hoc decision-focused calibration comparators.Both are fitted to NLL-model predictions around the target threshold and evaluated over the same threshold band as Smooth Net Benefit.Source: §2.3.2 · p. 8

2.4 Experimental evaluation

The evaluation uses Framingham for an illustrative 15-year CHD prediction task and TabZilla for a broader binary-classification benchmark. It evaluates Net Benefit around selected decision thresholds using held-out data.

2.4.1 Framingham

  • The Framingham analysis includes 3,781 participants, of whom 630 (16.7%) experienced CHD within 15 years.Participants with prevalent CHD or insufficient follow-up to determine the 15-year outcome were excluded.Source: §2.4.1 · p. 8
  • Framingham models use one-hot categorical covariates and threshold ranges of ±0.025 around estimated 15-year cardiovascular risk levels of 5%, 10%, and 20%.Five rotating 80/20 train-test splits support matched comparisons using raw Net Benefit on held-out folds.Source: §2.4.1 · pp. 8–9

2.4.2 Benchmark: TabZilla

  • TabZilla analysis is restricted to binary datasets with at least 1,000 observations and fewer features than observations.The suite includes real-world and synthetic datasets with varied sample sizes and outcome prevalences.Source: §2.4.2 · p. 9
  • TabZilla uses five-fold stratified cross-validation, with model selection and tuning confined to each outer training set.Thresholds are set at observed prevalence and, when prevalence is outside 40% to 60%, also at inverse prevalence; Net Benefit is averaged over a ±0.025 band.Source: §2.4.2 · p. 9
  • Standardized Net Benefit, defined as Net Benefit divided by outcome prevalence, is the primary TabZilla performance measure.Preprocessing is fitted on outer training data; continuous predictors are standardized for logistic regression and XGBoost and represented with spline basis functions for GAMs.Source: §2.4.2 · p. 9

2.5 Statistical analysis

Statistical comparisons use paired analyses suited to the matched splits or dataset-threshold combinations. The flexibility analysis adjusts for dataset size and feature count and uses non-overlapping fold subsets to reduce shared-comparator correlation.

2.5 Statistical analysis

  • Framingham method comparisons use paired Net Benefit differences across five matched held-out splits, with confidence intervals from split-level differences using the t distribution.All tests are two-sided, and 95% confidence intervals excluding zero are treated as evidence of a difference.Source: §2.5 · pp. 9–10
  • TabZilla comparisons use paired standardized Net Benefit differences across dataset-threshold combinations, with paired t-based confidence intervals and two-sided p-values.Each dataset-threshold combination is averaged over five held-out folds; fold-level confidence intervals characterize the direction and consistency of effects.Source: §2.5 · p. 10
  • Flexibility gain and Smooth Net Benefit gain are analyzed with adjusted regression and correlations, using non-overlapping fold subsets to reduce shared-comparator correlation.Adjustments include log-transformed sample size and number of features; fold allocation is reversed in a complementary analysis.Source: §2.5 · p. 10

3 Results

Across TabZilla, the effect of σNB varied by model class, and its logistic-regression gains were positively associated with XGBoost’s performance advantage. Framingham and small-sample analyses did not show consistent improvements.

3.1 Illustrative dataset: Framingham

  • σNB did not consistently improve Framingham Net Benefit; only small 5% threshold gains after local Platt or temperature scaling in logistic regression and GAMs had paired CIs excluding zero.No other comparison showed clear evidence of improvement.Source: §3.1 · p. 10, Table 1 · p. 11

Overall performance

  • Logistic regression mean standardized Net Benefit rose from 0.5669 with NLL to 0.5765 with σNB, while GAM fell from 0.5921 to 0.5625; σNB did not improve XGBoost over NLL.The logistic-regression mean difference was 0.0096 (95% CI −0.0001 to 0.0193); the GAM mean difference was −0.0296 (95% CI −0.0721 to 0.0129). The restricted analysis retained 59 of 72 dataset–threshold combinations from 44 datasets after floor and ceiling exclusions.Source: Overall performance · pp. 11–12, Table 2 · p. 11, Figure 4 · p. 13
  • Post-hoc local calibration did not consistently improve standardized Net Benefit; logistic-regression mean differences were 0.0049 for temperature scaling and 0.0021 for Platt scaling.Both methods slightly reduced mean performance for GAMs and produced larger reductions for XGBoost.Source: Overall performance · p. 13
  • After adjustment for feature count and dataset size, flexibility gain was positively associated with σNB gain in both analyses.Partial Pearson correlations were 0.490 and 0.494; partial Spearman correlations were 0.420 and 0.520. Both quantities were estimated from non-overlapping fold subsets to avoid shared sampling noise.Source: Flexibility and σNB gains · pp. 13–14, Figure 5 · p. 14

3.2.1 Analysis on small datasets

  • In the small-sample analysis, mean standardized Net Benefit was 0.5542 for NLL-trained versus 0.5505 for σNB-trained logistic regression; XGBoost remained higher at 0.6006.The logistic-regression mean difference was −0.0037 (95% CI −0.0125 to 0.0050).Source: §3.2.1 · p. 15

4 Discussion

Across the evaluated settings, σNB did not consistently or substantially improve Net Benefit over NLL, with modest gains concentrated in logistic regression. The discussion relates those gains to threshold-region misspecification while noting calibration, optimization, sample-size, and evaluation-scope constraints.

4 Discussion

  • σNB gains in logistic regression were positively associated with XGBoost’s performance advantage over NLL-trained logistic regression, while Framingham showed neither a substantial XGBoost advantage nor consistent σNB improvement.The authors suggest the gains may depend on model misspecification near the decision threshold and how limited model capacity is allocated there.Source: §4 · p. 15
  • Local temperature and Platt scaling produced only small logistic-regression changes, no GAM improvement, and reduced XGBoost Net Benefit, so local recalibration alone did not explain σNB findings.The pattern is consistent with limited benefit from additional decision-focused optimization when NLL already captures the relevant decision structure.Source: §4 · p. 15
  • σNB aligns more directly with threshold-specific utility but creates a non-convex objective with limited gradients away from the threshold, requiring warm starts and temperature annealing.NLL is strictly proper and yields convex logistic-regression optimization, providing stable probability estimation.Source: §4 · p. 16
  • The modest logistic-regression advantage disappeared with reduced training-set sizes, where sampling variability and optimization instability may outweigh closer objective alignment.Source: §4 · p. 16
  • The benchmark focused on thresholds around prevalence and its inverse, and no dedicated large-scale medical-dataset benchmark was included, potentially limiting generalizability to other threshold settings.The Framingham analyses did not suggest substantial sensitivity to alternative threshold choices.Source: §4 · p. 16
  • Logistic regression and GAMs retained NLL-selected regularization during σNB continuation, so those experiments measured incremental objective change rather than independently tuned σNB models.The authors note objective-specific tuning could help, but the reported pattern suggests it alone is unlikely to explain the main findings.Source: §4 · p. 16

Conclusion

  • σNB training did not consistently or substantially improve Net Benefit over NLL; the clearest gains were in logistic regression, while GAMs and XGBoost showed little or no benefit.The authors do not support σNB as a general replacement for likelihood-based training.Source: §4 · p. 15, Conclusion · p. 16
  • Future work should identify settings where decision-focused objectives add meaningful advantages beyond those obtainable through increased model flexibility.Source: Conclusion · p. 17

Data Availability

The paper reports public access to its code and analyzed datasets, disclosures on funding, competing interests, and AI use, and supplementary procedures for model development, calibration, and preprocessing.

Data Availability

  • The Smooth Net Benefit and analysis-reproduction code, TabZilla datasets, and Framingham data are publicly available through the stated repositories or sources.Source: Data Availability · p. 19

Funding

Competing interests

Artificial intelligence use

  • ChatGPT assisted with language editing and text and code refinement; the authors reviewed and approved scientific decisions, analyses, interpretations, and final content.Source: Artificial intelligence use · p. 19

Supplementary Methods

S2. Local calibration comparators

  • Local calibration subsets began within ±10 percentage points of each threshold and expanded to at least 50 events and non-events; temperature or Platt transformations were fit there and applied to test predictions.Source: S2. Local calibration comparators · p. 21

S3. TabZilla preprocessing

  • Categorical variables were one-hot encoded after retaining the 10 most frequent training-set levels; remaining numeric variables were median-imputed, with continuous predictors standardized for logistic regression and XGBoost or spline-transformed for GAMs.Source: S3. TabZilla preprocessing · pp. 21–22
Loading 2609.12752v1…