PaperlayerView exact revision on arXiv

Source-linked AI summary

The Loss or the Architecture? What Matters in Survival Analysis with Competing Risks

Julie Alberge, Gaël Varoquaux, Judith Abécassis

arXiv:2610.23322v1stat.MLcs.LG

TL;DR

Competing-risks deep-learning studies often bundle architecture and loss, leaving their individual effects difficult to assess. This paper varies both across tabular and pathology-image datasets, finding that the cumulative-incidence loss leads across modalities while the best backbone depends on the modality.

  • Problem

    Competing-risks methods often pair a new architecture with a new loss and are evaluated on few datasets and metrics, confounding their effects.

    Source: §1 · p. 1

  • Method

    The paper independently compares three losses across four backbones on competing-risks and single-event tabular data and pathology images.

    Source: §1 · p. 2

  • Results

    The cumulative-incidence loss leads across modalities, contributes about twice as much as a good backbone in competing-risks settings, and the winning backbone varies by modality.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    RealMLP or TabM trained with the cumulative-incidence loss beats published neural competing-risks methods on Brier score, while proper scoring rules distinguish models where C-index does not.

    Source: §1 · p. 2, §7 · pp. 9–10

    The calibration criteria disagree and neither separates the losses usefully, so the authors draw no calibration conclusion.

    Source: §6 · p. 9

Abstract

from arXiv · show

Often, \emph{eg} in healthcare and predictive maintenance, what matters is which of several competing causes will strike first, and with what probability: survival analysis with competing risks. Corresponding deep learning methods are developed as bundles: a new architecture together with a new training loss, tested on one to five datasets against a handful of baselines. Thus, a state-of-the-art result cannot be attributed to the architecture, the loss, or the evaluation protocol. The coupling is not accidental: with one incidence curve per cause, a network's head is tied to the losses expressible on it. We vary the two axes independently, three losses, two of them strictly proper, on four backbones, across two modalities that tease the axes apart: 20 tabular and 3 pathology-image datasets. One loss wins in both: modelling cumulative incidence directly under censoring weighting. It brings twice as much as benefits a good choice of backbone on competing-risks data, and no more than it on single-event data: the loss is decisive for competing risks. The backbone matters too, and must match the modality: tabular networks lead on tables, pretrained vision encoders on images. The two choices compound rather than substitute. Proper scoring rules reveal these dynamics; the concordance index (C-index), which is not proper, does not separate models: neither axis moves it, not even tuning on the loss built to improve it. This is an avenue of progress: composed, the two give a model nobody has tried, RealMLP or TabM trained with the cumulative-incidence loss, which takes the best Brier score in our benchmark.

1 INTRODUCTION: COMPETING RISKS, AND WHERE THE GAINS COME FROM

Competing-risks studies often couple a new architecture with a new loss, leaving their effects difficult to attribute. This benchmark varies both axes and finds that the cumulative-incidence loss leads across modalities while the strongest backbone depends on the data type.

1 INTRODUCTION: COMPETING RISKS, AND WHERE THE GAINS COME FROM

  • Earlier methods commonly introduced architecture and loss together, while evaluations often used only one to five datasets and one to three metrics.Concordance alone was often used despite not measuring differences in predicted probabilities.Source: §1 · pp. 1–2, Table 1 · p. 2
  • Pairing a recent tabular architecture with the cumulative-incidence loss beats every published neural competing-risks method on Brier score and meets or exceeds SurvivalBoost.The pairing combines components not previously tested together.Source: §1 · p. 2
  • The cumulative-incidence loss brings twice the benefit of a good backbone on competing-risks data, while the two contribute equally on single-event data.The loss ordering persists across modalities, but tabular networks lead on tables and pretrained vision encoders on images.Source: §1 · p. 2
  • The benchmark crosses three losses with four backbones under equal search budgets and compares nine fixed-loss methods across 20 tabular and three pathology-image datasets.The reported metrics span probabilistic accuracy, discrimination, and calibration.Source: §1 · p. 2
  • The comparison excludes time-varying covariates and recurrent events, and parametric-CIF losses remain fixed-loss baselines rather than matrix cells.Those losses cannot be added to an arbitrary backbone.Source: §1 · p. 3

2 SETTING: COMPETING RISKS, LOSSES, AND HOW TO SCORE THEM

The section defines competing-risks prediction around cause-specific cumulative incidence under censoring, then describes losses and metrics for comparing predicted distributions, rankings, and calibration.

2.1 COMPETING RISKS: ONE INCIDENCE CURVE PER CAUSE

  • For cause k, the cumulative incidence function is the probability that its event occurs by time t before any competing cause, conditional on covariates x.The observed time is the minimum of the event time and censoring time, and the observed event indicator is zero for censoring.Source: §2.1 · p. 3

2.2 THREE LOSSES THAT ANY BACKBONE CAN CARRY, TWO OF THEM PROPER

  • HazardLikelihood is a proper competing-risks likelihood, while HLRanking adds a risk-ordering penalty that makes the composite loss improper.Tuning on IBS selects weights that downplay the ranking penalty and pull HLRanking toward the likelihood.Source: §2.2 · p. 3
  • CRNLL directly scores cumulative incidence and reweights observations by the probability of remaining uncensored, making it strictly proper when censoring weights are correctly specified.Inverse Probability of Censoring Weighting up-weights subjects still observed to represent those lost to follow-up.Source: §2.2 · p. 4

2.3 METRICS: PROBABILISTIC ACCURACY, DISCRIMINATION, CALIBRATION

  • Evaluation spans probabilistic accuracy, discrimination, and calibration; the Brier score is proper under independent censoring and correctly estimated censoring weights, unlike the risk-magnitude-blind C-index.The Brier score is minimized if and only if the predicted distribution matches the true distribution under those assumptions.Source: §2.3 · p. 4

3 RELATED WORK: MANY METHODS, FEW CONTROLLED COMPARISONS

Prior competing-risks work spans statistical and machine-learning methods, but evaluations have been limited and neural methods often couple architecture with loss. The paper’s adaptation keeps the backbone intact so these choices can be compared separately.

3 RELATED WORK: MANY METHODS, FEW CONTROLLED COMPARISONS

  • Competing-risks benchmarks have used few datasets and often have not varied training loss, unlike single-event benchmarks with broader comparisons.SurvTRACE, for example, was benchmarked on one real dataset.Source: §3 · p. 4
  • The paper adapts each backbone with minimal changes so the architecture and loss can be varied independently.The adaptation adds cause-specific heads while leaving the upstream architecture unchanged.Source: Figure 1 · p. 5
  • Deep competing-risks methods commonly introduce a network together with its own likelihood-type loss, leaving loss comparisons across backbones untested.The literature ranges across neural networks and transformers as well as classical estimators, support vector machines, Gaussian processes, and trees.Source: §3 · pp. 4–5

4 BUILDING BLOCKS: ONE ADAPTATION THAT FITS EVERY BACKBONE

The competing-risks adaptation adds cause-specific outputs while preserving each backbone. The study tests this design across tabular and image models, separating loss effects from output parameterization and modality.

4 BUILDING BLOCKS: ONE ADAPTATION THAT FITS EVERY BACKBONE

  • Each cause receives a prediction head; direct-CIF losses also use a survival head, and removing that extra head shifts neither IBS nor calibration systematically.That ablation supports attributing reported differences to the loss rather than the output parameterization.Source: §4 · p. 5
  • Training uses a tuned grid of 20 to 60 observed event-time points, excluding censored times, with linear interpolation at inference.Grid resolution is a tuned hyperparameter.Source: §4 · p. 5
  • The tabular comparison trains every loss on an MLP, RealMLP, TabM, and the SurvTRACE transformer.RealMLP and TabM are identified as strong non-foundation tabular models.Source: §4 · p. 5
  • For images, the same adaptation uses frozen pretrained AlexNet, MobileNetV2, and ResNet50 feature extractors, with the tabular MLP carried over as a comparison.Only the prediction heads are optimized for the vision models.Source: §4 · pp. 5–6

5 EXPERIMENTS: 23 DATASETS, EQUAL TUNING BUDGET IN EVERY CELL

The benchmark covers 20 tabular survival datasets and three pathology-image datasets, comparing losses, backbones, and fixed-loss baseline methods. Each loss–backbone cell receives the same tuning budget, with separate searches for IBS and C-index.

5.1 DATASETS: 10 COMPETING-RISKS, 10 SINGLE-EVENT, 3 IMAGE

  • The benchmark includes 10 competing-risks and 10 single-event tabular datasets, plus three pathology-image datasets built from 635 images.The tabular datasets each contain more than 1,000 samples; the image datasets define cancer and other-cause death as competing events.Source: §5.1 · p. 6

5.2 TABULAR BASELINES: NINE METHODS WITH THEIR OWN LOSSES

  • Nine baseline methods retain their original fixed losses, including Aalen–Johansen, Fine and Gray, Random Survival Forests, DeepHit, and Deep Survival Machines.The four shared backbones are also trained under all three benchmark losses, forming the controlled loss × architecture matrix.Source: §5.2 · pp. 6–7

5.3 EVALUATION STRATEGY: TWO HYPERPARAMETER SEARCHES PER CELL

  • Each loss–backbone cell receives 150-iteration hyperparameter searches separately tuned for IBS and C-index, with each selection evaluated on five train/test splits.Splits use 70% for training and validation and 30% for testing, capped at 50,000 samples; IBS is reported as gain over Aalen–Johansen.Source: §5.3 · p. 7

6 RESULTS: THE LOSS AND THE BACKBONE EACH MATTER

Across tabular and image data, CRNLL ranks first within each backbone, while the best backbone depends on modality. On competing-risks tabular data, loss choice has about twice the effect of backbone choice; C-index and calibration provide weaker or conflicting signals.

Tabular data: the loss is decisive with competing risks

  • 0.053 versus 0.027 IBS-gain units: loss effect is about twice the backbone effect on competing-risks datasets; on single-event data, the corresponding effects are 0.072 versus 0.086.On competing-risks data, CRNLL also more than doubles DeepHit’s median IBS gain on its own MLP, from 0.12 to 0.26.Source: §6 · p. 7

Trends across the two modalities

  • CRNLL ranks first within every backbone on both modalities, while RealMLP and TabM lead on tables and pretrained encoders lead on images.Figure 2 panels use different backbones, baselines, and vertical axes, so they should not be compared cell by cell.Source: §6 · p. 7, Figure 2 · p. 8

Pathology images: the backbone decides, the loss still matters

  • Every pretrained image encoder beats the MLP under every loss, while CRNLL remains the best loss within each backbone, including the frozen encoders.The MLP is also less stable across image datasets.Source: §6 · pp. 8–9

C-index

  • Neither loss nor architecture visibly changes C-index; even HLRanking gains only marginally at the 25th percentile and not at the 50th or 75th.Selecting HLRanking on C-index improves median C-index by +0.0018 but degrades IBS by +0.0117.Source: §6 · p. 9

Calibration gives little signal

  • The calibration criteria do not separate losses usefully and disagree, with CRNLL best on AJ-calibration but trailing hazard-based losses on D-calibration.The authors report both measures for completeness and draw no calibration conclusion.Source: §6 · p. 9

7 CONCLUSION: BACKBONE FOR THE MODALITY, LOSS FOR THE CENSORING

The conclusion is that loss and backbone choices should be matched to censoring and modality, respectively. The benchmark supports pairing tabular backbones with the cumulative-incidence loss, while noting limits in dataset composition and sample size.

Limitations

  • Half of the 20 tabular datasets are single-event, and the 50,000-sample split cap leaves the largest datasets outside the large-data regime.The loss ordering is the same across single-event and competing-risks datasets, but effect sizes differ.Source: §7 · p. 9

Combining loss and backbone for the win

  • RealMLP or TabM paired with the cumulative-incidence loss beats every published neural competing-risks model on Brier score.The cumulative-incidence loss leads on all 20 tabular datasets and also leads on pathology images.Source: §7 · p. 9

Only strictly proper scoring rules cleanly separated the models

  • Only the Brier score and log-likelihood consistently order models; C-index and AUC do not, and calibration criteria disagree.The authors recommend using strictly proper scoring rules for comparison and keeping other metrics as diagnostics.Source: §7 · pp. 9–10

REPRODUCIBILITY STATEMENT

The authors state that they will release the benchmark codebase on GitHub after publication.

REPRODUCIBILITY STATEMENT

  • The planned release includes loss implementations, backbone architectures, search spaces, evaluation logic, and dataset preprocessing and splitting.The statement specifies release after publication.Source: REPRODUCIBILITY STATEMENT · p. 11

Appendices

The appendices document dataset preparation and ICU cohort construction, alongside the target definitions used for the ICU competing-risks study.

A.1 ACCESS AND PREPROCESSING

  • The dataset appendix gives preparation instructions for 20 tabular datasets, including cohort-specific event coding and covariate handling.For example, the competing-events METABRIC loader codes relapse as event 1 and qualifying death as event 2.Source: §A.1 · pp. 16–18
  • Dataset-specific loaders prepare covariates and targets, while several cohorts require separate source files or construction steps to reproduce their endpoints or features.The SEER digestive-cancer, loan, TCGA, and ENS preparations have additional source-extraction or construction requirements; shared preparation includes vectorization, mean imputation, and type conversion.Source: §A.1 · pp. 16–19

A.2.1 COHORT SELECTION AND INCLUSION CRITERIA

  • The ICU study selects adults with an ICU stay and randomly samples one hospital admission per patient.The cohort is constructed from MIMIC-IV; the appendix says the same methodology was applied to the ICU datasets.Source: §A.2.1 · p. 19, §A.2 · p. 19

A.2.2 FEATURE ENGINEERING: AGGREGATED CLINICAL LABS

  • For each selected stay, the cohort represents first-day physiology with mean values for the 50 most frequent laboratory tests.The labs are aggregated from observations between admission and one day later.Source: §A.2.2 · p. 20

A.2.3 TARGET DEFINITION: COMPETING RISKS FRAMEWORK

  • The target is time from discharge to the first post-discharge event, with rehospitalization and death as competing outcomes and last hospital contact as censoring.Durations are measured in relative days because MIMIC-IV dates are shifted for de-identification.Source: §A.2.3 · p. 20

A.3 STATISTICS SUMMARIZING THE DATASETS

The appendix summarizes the prepared tabular datasets and explains how their characteristics are counted.

A.3 STATISTICS SUMMARIZING THE DATASETS

  • Table A.1 reports characteristics of the 20 tabular datasets after dataset-specific preparation.For TableVectorizer loaders, feature counts are measured after vectorization.Source: §A.3 · p. 20
  • The table’s CR column indicates whether a dataset has competing events.Source: §A.3 · p. 20
  • The benchmark code is stated to become available on GitHub after publication.Source: §A.3 · p. 20

B THEORY: DEFINITIONS OF THE EVALUATION METRICS

The appendix defines probabilistic, discrimination, and calibration metrics for evaluating predicted cumulative incidence functions, and records the benchmark’s libraries and search setup. It also reports how tuning changes the ranking weight selected for HLRanking.

B THEORY: DEFINITIONS OF THE EVALUATION METRICS

  • Evaluation uses held-out test data, a shared censoring-survival estimate and time grid, and predicted CIFs for each cause.The overall survival prediction is defined as one minus the sum of cause-specific CIFs; time-dependent scores are also reported at event-time quartiles.Source: B THEORY: DEFINITIONS OF THE… · p. 20

B.1 METRICS CONTROLLING PROBABILISTIC PREDICTION METRICS: IBS AND INLL

  • IBS and INLL assess probabilistic predictions using censoring weights, with INLL replacing squared error by Bernoulli log-loss.Both are strictly proper under the stated assumption that the censoring-survival estimate is correctly specified; lower values are better.Source: §B.1 · pp. 20–21

B.2 DISCRIMINATION METRICS: C-INDEX AND TIME-DEPENDENT AUC

  • The competing-risks C-index and time-dependent AUC assess discrimination by ranking predicted incidences, and are insensitive to calibration.The C-index compares weighted pairs; the AUC compares cases experiencing cause k by time t with controls still event-free after t.Source: §B.2 · pp. 21–22

B.3 CALIBRATION METRICS

  • AJ-calibration targets marginal calibration, whereas CR-D-calibration targets conditional calibration; both are lower-is-better measures.Both are computed on a held-out calibration split, with α = 2, and the main text reports gain over the marginal Aalen–Johansen estimator.Source: §B.3 · pp. 22–23

C.1 COMPETING RISKS LIBRARIES

  • The appendix lists open-source libraries used to evaluate or download datasets, including tools for metrics and dataset access.Source: §C.1 · p. 24

C.2 COMPUTE AND HYPER PARAMETER SEARCH

  • HLRanking’s selected α depends on the tuning criterion: the median is 0.59 with IBS tuning and 0.29 with C-index tuning.For C-index tuning, the ranking term dominates; on RealMLP the median α is approximately 0.003. The search uses 150 iterations per cell, and α and σ are searched only for HLRanking.Source: Table C.2 · p. 24, §C.2 · p. 26

C.3 TRAINING-TIME IPCW SENSITIVITY: KAPLAN-MEIER VS. COX VS. STRATIFIED

The training-time censoring-weighting sensitivity analysis supports marginal Kaplan–Meier as the default: stratified Kaplan–Meier generally reproduces it, while Cox is unreliable on these benchmark datasets. The accompanying effect-size analysis finds that loss and backbone importance differ between competing-risks and single-event data.

C.3 TRAINING-TIME IPCW SENSITIVITY: KAPLAN-MEIER VS. COX VS. STRATIFIED

  • Cox produced usable fits on only 35/94 cells, and on the 11 cells with large relative IBS differences it was 139.4% worse on average than Kaplan–Meier.Cox was worse in all 11 cells; stratified Kaplan–Meier stayed within a couple of percent of Kaplan–Meier on that subset.Source: §C.3 · pp. 26–27
  • Cox was also slower, taking a median 1.75× and up to 6.7× Kaplan–Meier’s fitting time where both completed.Censoring rates in unstable Cox cells ranged from 35% to 98%, with no consistent relationship between censoring rate and stability.Source: §C.3 · p. 27
  • The authors support marginal Kaplan–Meier as the default because stratified Kaplan–Meier nearly reproduces it, while Cox instability does not show that reliable conditional censoring models would change conclusions.Cox completed on only a dataset-dependent subset, whereas the stratified estimator reproduced Kaplan–Meier almost everywhere it was estimable.Source: §C.3 · pp. 26–27

D.1 COMPUTING EFFECT SIZES: HOW MUCH THE LOSS MATTERS, AND WHERE

  • The loss effect is about twice the backbone effect on competing-risks data, but slightly smaller on single-event data.The effects are paired IBS-gain spreads; pooling the two dataset families averages opposite regimes and hides the difference.Source: §D.1 · p. 27, Table D.1 · p. 28
  • Across datasets, the loss spread exceeds the backbone spread on most competing-risks datasets and on fewer single-event datasets.The analysis uses available groups with complete members for each spread; it covers 229 of 240 matrix cells.Source: §D.1 · p. 27, Table D.2 · p. 28

D.2 COMPARING THE DIFFERENT MODEL-SELECTION METRICS

Across model-selection metrics and dataset strata, strictly proper scoring rules distinguish losses more clearly than ranking metrics do. The appendix also examines censoring and tuning choices, finding that the CRNLL advantage persists in competing-risks settings and does not depend on its extra survival head.

D.2 COMPARING THE DIFFERENT MODEL-SELECTION METRICS

  • The Brier score and integrated negative log-likelihood both rank CRNLL ahead of hazard-based losses on every backbone, whereas C-index and integrated AUC largely fail to separate models.The C-index boxes overlap across losses, backbones, and baselines; the integrated AUC is also muddied despite being proper but not strictly proper.Source: §D.2 · p. 28, Figure D.2 · p. 29, Figure D.3 · p. 30

D.3 DATASET STRATIFICATION: MULTIPLE EVENTS OR NOT, AMOUNT OF CENSORING

  • On competing-risks datasets, CRNLL consistently achieves the largest IBS gains across datasets and architectures, and the advantage widens under heavy censoring.The heavy-censoring figure reports CRNLL ahead of both hazard-based losses on every backbone; one SurvTRACE–HazardLikelihood cell falls below the Aalen–Johansen reference.Source: §D.3 · p. 32, Figure D.7 · p. 33, Figure D.8 · p. 33

D.5 HLRANKING: TUNING ON ITS OWN TARGET METRIC

  • Tuning HLRanking on C-index buys almost no C-index improvement while costing IBS, according to the table’s comparison of the two selection criteria.The table pairs IBS-selected and C-index-selected hyperparameters by dataset and backbone; the supplied evidence does not provide the cell values.Source: Table D.5 · p. 34, §D.5 · p. 34

D.4 C-INDEXES AT DIFFERENT TIME POINTS

  • At the 25th C-index quantile HLRanking has a marginal advantage, but no clear trend emerges at the 50th or 75th quantiles.Evaluating C-index at specific time points still does not clarify relative benefits of architectures or losses.Source: §D.4 · p. 34, Figure D.9 · p. 35

D.6 CRNLL SURVIVAL-HEAD ABLATION

  • Removing CRNLL’s dedicated survival head leaves IBS and both calibration criteria essentially unchanged, so the advantage is attributed to the loss rather than output parameterization.The paired variants overlap across all four backbones, with neither variant systematically ahead.Source: §D.6 · p. 34, Figure D.10 · p. 36, Figure D.11 · p. 37

E BACKGROUND THEORY: LINK BETWEEN THE HAZARD FUNCTION AND THE CIF FUNCTION

  • The background derivation explains the link among the event-specific hazard, survival function, cumulative incidence function, and its density.The stated purpose is to derive the relationships among λk, S, Fk, and fk.Source: E BACKGROUND THEORY: LINK BETWEEN THE… · p. 36
Loading 2610.23322v1…