Source-linked AI summary
Prediction meets causal inference: the role of treatment in clinical prediction models
Nan van Geloven, Sonja Swanson, Chava Ramspek, Kim Luijken, Merel van Diepen, Tim Morris, Rolf Groenwold, Hans van Houwelingen, Hein Putter, Saskia le Cessie
TL;DR
When treatment can begin after baseline, clinical prediction models may target unclear risks, complicating interpretation and use in treatment decisions. The authors adapt the clinical-trial estimand framework to prediction, defining predictimands and discussing estimators and their assumptions. The passages report 43% of patients dying within 10 years under ignored treatment and a 5% chance of being alive without transplant in 10 years with unchanged transplant availability and allocation (1 minus composite).
Problem
When treatment can begin after baseline, clinical prediction models may target unclear risks, complicating interpretation and use in treatment decisions.
Source: §1 · pp. 2–3Method
Results
The passages report 43% of patients dying within 10 years under ignored treatment and a 5% chance of being alive without transplant in 10 years with unchanged transplant availability and allocation (1 minus composite).
Source: §5 · p. 20Takeaways & Limitations
Treatment handling should be prespecified according to the intended interpretation and future use of predictions, and multiple predictimands may be estimated when multiple questions matter.
Source: §6 · pp. 21, 23Validation of hypothetical-strategy predictions may require assumptions and estimation techniques similar to those used to develop them, and warrants further research.
Source: §6 · p. 23
Abstract
from arXiv · showhide
In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a `predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.
1 Introduction
Clinical prediction models can guide treatment decisions, but their intended risk is often unclear when patients may start treatment after baseline. The paper proposes adapting the clinical-trial estimand framework to define these prediction questions and guide their estimation.
1 Introduction
- Treatment after baseline complicates prediction because future treatment cannot be used as a baseline predictor, yet how it is handled affects how predictions can be used.Ad hoc choices include ignoring treatment, censoring at treatment start or switch, and excluding treated patients; these choices can target a risk different from the one intended.Source: §1 · p. 2
- Untreated cohorts are rare and may be unrepresentative, while mixed treated and untreated data are common in observational studies and trials.Source: §1 · p. 2
- Censoring at transplantation when estimating liver-transplant waiting-list survival has been shown to overestimate observed waiting-list mortality.Source: §1 · p. 3
- The authors adapt the EMA’s five-strategy trial estimand framework to prediction, where each strategy corresponds to a different question about expected outcomes given baseline characteristics.They propose a predictimand framework to make the target quantity explicit.Source: §1 · p. 3
- The paper defines prediction estimands, discusses their assumptions and common estimators, and illustrates the impact of predictimand choice in patients with end-stage kidney disease.Source: §1 · p. 4
2 Notation
The paper considers event-free patients without treatment at baseline, some of whom start treatment during follow-up. Event follow-up may stop or continue after treatment initiation.
2 Notation
- The setup tracks baseline covariates X(0), event time T, treatment-start time V, and time-varying treatment status A(t).Treatment is assumed to continue once started; T and V may be censored by study end or loss to follow-up, assumed non-informative.Source: §2 · p. 4
- Treatment decisions may depend only on baseline prognostic covariates or also on prognostic markers that evolve over time.Source: §2 · p. 4
- Figure 1 contrasts settings where event follow-up stops after treatment initiation with settings where it continues.Source: §2 · p. 4, Figure 1 · p. 5
3 Predictimands
The paper defines four prediction strategies for handling treatment initiated after baseline, each targeting a different risk question. Their interpretation and usefulness depend on treatment policy and whether post-treatment follow-up is available.
3 Predictimands
- The paper presents four strategies analogous to trial estimands, discusses their usefulness and generalizability, and does not map the principal-stratum strategy to prediction.The authors also link the estimands to assumptions and common estimators.Source: §3 · pp. 5–6
3.1 “Ignore treatment” strategy
- The “ignore treatment” strategy uses event time regardless of treatment initiation, targeting risk under the treatment practice in the development data.It requires follow-up after treatment starts and generalizes only when treatment assignment policy is similar across predictor-defined subgroups; with effective treatment, it gives lower risk than untreated risk.Source: §3.1 · p. 6
3.2 “Composite” strategy
- The “composite” strategy counts treatment initiation together with the event, which can be useful when treatment proxies an imminent event or needing treatment captures poor outcome.It does not require post-treatment follow-up, but predictions require a similar treatment assignment policy in new patients.Source: §3.2 · p. 8
3.3 “While untreated” strategy
- The “while untreated” strategy targets events occurring before treatment starts, treating treatment initiation as a competing event.It is useful when treatment is limited, such as by a waiting list, and requires a similar treatment assignment policy for calibration in new patients.Source: §3.3 · pp. 8–9
3.4 “Hypothetical” strategy
- The “hypothetical” strategy targets untreated risk in a world where treatment is eliminated, using the counterfactual event time under no treatment.It can inform treatment decisions and generalizes to the question of what would happen if a new patient were never treated, but estimation is challenging and the strategy relies on strong assumptions.Source: §3.4 · pp. 9–10
4 Estimators and their assumptions
This section links each predictimand to the assumptions and design elements needed for estimation and to common estimators.
4 Estimators and their assumptions
- Estimating each predictimand requires attention to its key assumptions and design elements.Source: §4 · p. 10
- The authors connect the estimands to common estimators.Source: §4 · p. 10
- The overview is not exhaustive.Source: §4 · p. 10
4.1 “Ignore treatment” strategy
The ignore-treatment strategy relates the event to baseline covariates using standard time-to-event regression without censoring at treatment initiation. It requires non-informative censoring for loss to follow-up or end of study and continued follow-up after treatment starts.
4.1 “Ignore treatment” strategy
- Standard time-to-event regression can relate the event to baseline covariates when treatment initiation is ignored.Examples include Cox regression with a Breslow or Efron baseline-hazard estimator and flexible parametric survival models.Source: §4.1 · p. 10
- The strategy requires non-informative censoring for loss to follow-up or end of study; some methods additionally assume proportional hazards for covariates.Source: §4.1 · pp. 10–11
- Because follow-up is not censored when treatment starts, this strategy requires continued follow-up after treatment initiation.Source: §4.1 · p. 11
4.2 “Composite” strategy
The composite strategy counts whichever occurs first—the event of interest or treatment initiation—as an event. Follow-up after treatment initiation is not needed, but assumptions apply to the composite outcome.
4.2 “Composite” strategy
- The event of interest or treatment initiation counts as an event, whichever occurs first.Source: §4.2 · p. 11
- For a Cox-like model, proportionality of covariate effects must hold for the composite outcome, which may be less likely than for single outcomes.Source: §4.2 · p. 11
- Censoring from loss to follow-up or end of study must be non-informative for the composite outcome; continued follow-up after treatment starts is unnecessary.Source: §4.2 · p. 11
4.3 “While untreated” strategy
The while-untreated strategy estimates cumulative incidence using competing-risks methods. Follow-up after treatment initiation is not needed.
4.3 “While untreated” strategy
- Cumulative incidence for the while-untreated strategy can be estimated with competing-risks methods.Source: §4.3 · p. 11
- Without covariates or other censoring, cumulative incidence is the number of patients with the event divided by the number at baseline.Source: §4.3 · p. 11
- With covariates and censoring such as loss to follow-up or end of study, estimation can use various methods; post-treatment follow-up is unnecessary.The passage refers to methods in.Source: §4.3 · p. 11
4.4 “Hypothetical” strategy
The hypothetical strategy targets risk under no treatment, using either censoring at treatment initiation or post-treatment data to model treatment. Its estimation depends on treatment-assignment assumptions and whether time-varying prognostic factors influence treatment.
4.4 “Hypothetical” strategy
- The hypothetical estimand requires exchangeability, consistency, and positivity; baseline-only treatment decisions are often implausible in healthcare.Exchangeability requires measured and appropriately adjusted factors that influence treatment and prognosis; positivity may limit the usable prediction horizon.Source: §4.4 · pp. 11–12
- Other proposed estimators for hypothetical untreated risk include the g-formula, copulas, and multiple imputation, each with its own assumptions.Source: §4.4 · p. 13
4.4.1 Baseline covariates, censoring
- Censoring at treatment start requires treatment initiation to be independent of changing prognostic markers given baseline covariates, a strong, often implausible, untestable assumption.For this censoring approach, positivity is needed only among untreated individuals.Source: §4.4.1 · p. 13, §4.4 · p. 13
4.4.2 Baseline covariates, modelling
- With post-treatment follow-up, a time-dependent treatment covariate can be included and set to zero to predict untreated risk, but the treatment-effect form and interactions need checking.This approach relies on the same assumption as baseline-covariate censoring.Source: §4.4.2 · pp. 13–14
4.4.3 Time varying covariates, censoring
- When time-varying covariates influence treatment, inverse probability of censoring weighting can address informative censoring under exchangeability, consistency, and positivity.Weights are inversely proportional to the conditional probability of not yet being treated.Source: §4.4.3 · p. 14
4.4.4 Time varying covariates, modelling
- When time-varying covariates affect treatment, a marginal structural model with inverse probability of treatment weighting can estimate hypothetical risk by setting treatment to zero.Including time-varying covariates directly is not useful for baseline prediction because they will not be known for a new patient; correct treatment-effect specification is also required.Source: §4.4.4 · pp. 14–15
5 Data application
The NECOSAD application compares four mortality-risk estimands for patients with end-stage renal disease, showing how predicted risk varies with the treatment question and estimation strategy.
5 Data application
- The analysis included 2,045 patients followed for a median 5.1 years; 749 received a transplant and 1,470 died, including 248 after transplantation.Age and baseline dialysis type were the prediction variables; transplantation was the treatment and death the event of interest.Source: §5 · p. 15
- The four strategies target total mortality regardless of treatment, death or transplantation, death before transplantation, or death if transplantation were not performed.The hypothetical strategy used four estimation approaches, including censoring at transplantation, modelling transplantation, and inverse-probability weighting.Source: §5 · pp. 16–17
- Hypothetical mortality curves exceeded ignore-treatment curves, indicating lower mortality under the observed transplantation policy than if nobody received a transplant.The contrast was more apparent at age 50, when more patients were transplanted; peritoneal-dialysis curves were similar to hemodialysis curves and are not shown.Source: §5 · p. 18, Figure 2 · p. 19
- Without kidney transplantation, 59–62% of patients your age were estimated to die within 10 years; with recent-years transplant availability/allocation, about 43% died within 10 years, ignoring treatment; with unchanged availability/allocation, about 34% faced death before transplant in 10 years.With unchanged transplant availability and allocation, the estimated chance of being alive and without transplant at 10 years was 5%.Source: §5 · p. 20
- The two-predictor model was illustrative, not intended for clinical use, and uncertainty intervals were omitted.The example illustrates that different strategies for handling treatment started after baseline answer different risk questions.Source: §5 · p. 20
6 Discussion
The paper argues that treatment handling defines which risk a clinical prediction model targets and should be chosen explicitly for its intended use. Its framework also applies to other modifiable post-baseline events, while prediction and validation depend on assumptions that vary by strategy.
Discussion
- Treatment choices should be prespecified according to the intended risk, and researchers may estimate multiple predictimands when several questions matter.Different strategies can yield very different predictions, and an unplanned choice may enter through the analysis method.Source: §6 · p. 21
- The framework can address modifiable post-baseline events beyond treatment, such as dialysis recovery; the analyses censored recovery, implying a hypothetical strategy for that event.Source: §6 · p. 21
- Five of 16 reviewed on-dialysis mortality models used Cox models censoring at transplantation, implicitly targeting hypothetical risk; none explained the interpretation or non-informative censoring assumption.Three included post-transplant deaths, three excluded transplant recipients, and five did not describe their handling of transplantation.Source: §6 · p. 21
- The hypothetical strategy predicts outcomes as if nobody received treatment and requires untestable assumptions; the other three give real-world predictions but rely on similar treatment policies in development and use.The prediction model's prognostic factors need not be causal, but the treatment policies must be clearly defined and similar for predictions to remain valid.Source: §6 · p. 22
- A prediction estimand also needs a defined target population or setting, outcome and time horizon, and prediction time point.Source: §6 · p. 22
- Each relevant treatment needs an explicit predictimand choice and suitable estimation assumptions; differences between strategies may be less pronounced when few patients are treated or treatment effects are small.Multiple treatments may require a mixture of approaches, with some handled as care as usual and others targeted by a different strategy.Source: §6 · pp. 22–23
- With repeated treatment switching, ignore-treatment and hypothetical definitions can remain unchanged, while composite and while-untreated strategies may count or cover additional treatment episodes.The first treatment episode could define the composite event, while while-untreated risk could extend across untreated episodes.Source: §6 · p. 23
- Model validation was not addressed; hypothetical-strategy validation may require assumptions and estimation techniques similar to those used to develop predictions.Validation methods for this strategy warrant further research.Source: §6 · p. 23
7 Funding
Tim Morris received Medical Research Council support.
7 Funding
- Tim Morris was supported by Medical Research Council grants MC UU 12023/21 and MC UU 12023/29.Source: §7 · p. 24