Causal multi-modal AI for personalized chemosensitivity prediction

1 Introduction

• Chemotherapy benefit varies widely, while treatment toxicity makes accurate individualized benefit estimates important for early-breast-cancer decisions.Source: §1 · p. 2
• Recurrence scores were developed prognostically, but patients with identical genomic risk can have markedly different chemotherapy benefits.Prognosis and treatment benefit are distinct; prognostic de-escalation also cannot identify high-risk patients who will not benefit from chemotherapy.Source: §1 · p. 2
• Existing breast-cancer AI pathology models predict outcomes or mimic prognostic gene-expression assays rather than directly estimating chemotherapy benefit.Source: §1 · p. 2
• Causal AI estimates treatment-specific recurrence probabilities, with their difference serving as a personalized chemotherapy-benefit estimate.The approach balances representations across treatment groups and can combine routine clinical variables with pathology morphology.Source: §1 · p. 2
• CTX integrates clinical characteristics with foundation-model pathology representations and causal adjustment; the study evaluates it across five axes.The assembled multi-national dataset included 11,135 patients spanning ten countries.Source: §1 · p. 2

2.1 Multi-national development and evaluation datasets

• The model was developed on 9,141 patients from twelve cohorts and evaluated on 1,994 HR+/HER2− patients from five held-out cohorts, with inverse propensity weighting reducing absolute average SMD from 0.376 to 0.125.Source: §2.1 · p. 4

2.2 CTX: a causal multi-modal AI model for estimating personalized chemotherapy benefit

• CTX fuses pathology-slide and clinical representations, then uses CFRNet to predict recurrence under both CET and ET and estimate individualized absolute chemotherapy benefit.CTX-τ categories used a 2% benefit threshold for reporting, while the continuous score allows other thresholds to be applied.Source: §2.2 · p. 4

2.4 Evaluation of CTX for predicting chemotherapy benefit

• CTX-τ had the strongest interaction across biomarkers (p = 2.09 ×10−6) and exceeded Oncotype DX in the same-patient comparison (p = 9.68 ×10−4 vs p = 0.0247).CET was associated with lower recurrence in the high-benefit group (HR = 0.42, 95% CI = [0.20, 0.88], p = 0.022), but not significantly in the low-benefit group (HR = 1.40, 95% CI = [0.77, 2.57], p = 0.270). Benefit scaled continuously with CTX-τ (HR per standard deviation = 0.67, 95% CI = [0.57, 0.79], p = 2.09 ×10−6), and pathology added predictive information beyond clinical features (likelihood ratio test p = 9.79 ×10−3).Source: Figure 2 · p. 5, §2.4 · p. 6

2.3 Evaluation of CTX for predicting prognosis

• In 1,994 held-out patients, CTX-prognostic achieved 5-year AUC 0.711 (95% CI = [0.663, 0.756]) and 10-year AUC = 0.765 (95% CI = [0.708, 0.791]), with near-perfect 5-year calibration.It remained an independent predictor at 5 years (adjusted HR = 2.02, 95% CI = [1.44, 2.83], p = 4.65×10−5) and 10 years (adjusted HR = 2.28, 95% CI = [1.73, 3.00], p = 5.74×10−9).Source: §2.3 · pp. 5–6, Figure 2 · p. 5

2.5 Personally tailored therapeutic decision support

• CTX-guided allocation matched the observed 93.7% five-year recurrence-free rate while treating 10.5% rather than 40.7% of patients, a relative reduction in chemotherapy use of 74.2%.At the observed treatment rate, the strategy reduced recurrence from 6.3% to 5.4% (relative reduction 14.3%). Population-average benefit was 0.30%, while clinicopathological subgroup averages varied and individual examples illustrated divergent predicted benefits; these case studies illustrate interpretation, not formal predictive accuracy.Source: §2.5 · pp. 7–8, Figure 3 · p. 7, Figure 4 · p. 8

2.6 Mechanistic insights into the determinants of chemosensitivity

• CTX-τ high-benefit tumors showed invasive carcinoma with high cellularity and mitotic figures, alongside proliferative, cell-cycle, DNA-damage-response, and immune-activation signatures.Low-benefit tumors showed stroma-rich morphology and estrogen-response, EMT, and angiogenesis signatures. The molecular analyses were exploratory and hypothesis-generating in TCGA (n = 348), a development-set cohort.Source: §2.6 · pp. 9–10, Figure 5 · p. 9

2.7 Zero-shot transfer of CTX prognosis and chemosensitivity predictions across cancer types

• CTX-risk was associated with worse overall survival in 11 of 16 cancer types (69%) after zero-shot application, and transfer was significant in 6 of 7 cancer types treated with comparable cytotoxic regimens.Transfer was strongest in endometrial cancer and other epithelial adenocarcinomas, and weakest in central nervous system tumors; the model was frozen and applied without fine-tuning or re-calibration.Source: Figure 6 · p. 10, §2.7 · p. 11
• CTX-τ correctly ordered chemotherapy benefit in 8 of 14 cancer types (57%), including 5 of 7 (71%) treated with anthracycline-, platinum-, and/or taxane-based regimens.In those five cancer types—head and neck, colon, bladder, and lung squamous and adenocarcinoma—the high-benefit subgroup gained more survival time from chemotherapy than the low-benefit subgroup.Source: §2.7 · p. 11
• Transfer failed predominantly in cancers treated with mechanistically unrelated regimens, including alkylator therapy or targeted and immunotherapies.Failures were predominantly in glioblastoma, glioma, and melanoma treated with alkylators, and renal and hepatocellular carcinoma treated with targeted therapies or immunotherapies.Source: §2.7 · p. 11
• The frozen breast-trained model recovered prognostic ordering in most cancer types, suggesting CTX learned generalizable treatment-specific patterns of tumor aggressiveness and chemosensitivity [46].Source: §2.7 · p. 11

Approach

• CTX combines a pathology foundation model with counterfactual regression to predict personalized chemosensitivity in HR+/HER2− early breast cancer.It was developed on 9,141 patients and evaluated on an independent held-out set of 1,994 patients.Source: §3 · pp. 11–12

Results

• CTX-τ met the formal criterion for a predictive biomarker, and its treatment-by-biomarker interaction was strongest among evaluated predictors, exceeding Oncotype DX and standard clinicopathological variables.CTX-prognostic also showed strong discrimination, near-perfect calibration, and performance across 5- and 10-year horizons.Source: §3 · p. 12

Biological correlates

• CTX-τ high-benefit tumors showed proliferation and DNA-damage-response programs, homologous-recombination deficiency, and a primed immune microenvironment.The authors describe these biological links as mechanistically consistent with chemotherapy susceptibility, but still a hypothesis requiring validation.Source: §3 · p. 12

Clinical implications

• Biomarker-guided allocation could distinguish recurrence risk from chemotherapy benefit and reduce overtreatment while directing treatment toward patients predicted to benefit.The discussion reports decision analyses indicating reduced overtreatment relative to historical prescribing patterns.Source: §3 · p. 12

Limitations

• Because all study cohorts were observational for chemotherapy, residual confounding cannot be excluded; CTX also models chemotherapy as a binary CET-versus-ET contrast.The reported mechanistic associations are correlative and require prospective and experimental validation.Source: §3 · p. 12

Generalizability

• Breast-trained CTX transferred prognostic and chemosensitivity predictions to most other cancer types without per-cancer fitting.The authors call for further cross-cancer validation and suggest that shared morphological determinants may underlie the transfer.Source: §3 · p. 12

4.1 Data

• The study allocated 17 early-breast-cancer cohorts from 10 countries to development (12 cohorts, n = 9,141) or evaluation (5 cohorts, n = 1,994), with evaluation restricted to the intended HR+/HER2− population.The endpoint was recurrence-free interval from diagnosis to first invasive recurrence, with deaths treated as censoring events.Source: §4.1 · p. 13

4.2 Pathology foundation model; 4.3 Multi-modal architecture

• Falcon encoded H&E slide patches, while gated-attention aggregation and a tabular ResNet fused pathology and clinical representations for treatment-specific prediction.Falcon was pretrained on over two billion H&E patches; slides were divided into non-overlapping 224 × 224 pixel patches and encoded as 1,536-dimensional embeddings.Source: §4.3 · p. 13, §4.2 · p. 13

4.4 Discrete-time survival modeling

• CTX modeled censored recurrence-free interval with discrete-time hazards and accumulated interval survival probabilities, using risk-set binary cross-entropy to handle right-censoring.The follow-up intervals were based on event-duration quantiles and rescaled to include a boundary at five years.Source: §4.4 · p. 13

4.5 Causal modeling

• Two CFRNet heads predicted survival under CET and ET, while maximum mean discrepancy encouraged alignment of treatment-group representations.CTX-τ was the difference in five-year recurrence-free probability, with values of at least 2 percentage points classified as high benefit.Source: §4.5 · pp. 13–14

4.6 Training methodology

• Training combined observed-treatment survival loss, a representation-discrepancy penalty, and elastic-net regularization, followed by cohort-held-out validation, ensembling, and beta calibration.Three development cohorts were held out in turn for hyperparameter selection, early stopping, and calibration.Source: §4.6 · p. 14

4.7 Adjustment for confounding and propensity score estimation

• Inverse propensity weighting adjusted evaluation contrasts for nonrandom chemotherapy assignment, reducing mean absolute standardized mean difference from 0.376 to 0.125.Weights were inverse probabilities of the treatment received, clipped at 10; maximum standardized mean difference fell from 1.036 to 0.281.Source: §4.7 · p. 14

4.8 Evaluation

• Evaluation tested CTX-τ treatment interactions, benchmarked biomarkers, assessed prognostic performance, and estimated recurrence-free rates under allocation strategies.Molecular analyses were exploratory and used TCGA-BRCA, a development cohort.Source: §4.8 · pp. 14–15

4.9 Statistical analysis; 4.10 Data availability; 4.11 Code availability

• Analyses were retrospective, no prespecified analysis plan was registered, cohort data access is controlled, and the full model codebase is proprietary.Code for main figures and aggregate results may be available upon reasonable request.Source: §4.9 · p. 15, §4.10 · p. 15, §4.11 · p. 15

• Several authors hold Ataraxis AI equity and are inventors on a US patent application related to methodological aspects of the work; the remaining authors declare no competing interests.Source: Competing interests · p. 16

Extended Data Fig. 2

• Falcon achieved the highest mean performance across 24 slide-level histopathology tasks, measured by C-index for survival and AUC for mutation and biomarker tasks.The benchmark also includes per-task slide-level results and a separate suite of 24 patch-level tasks.Source: Figure 2 · p. 27

Extended Data Figs. 1 and 3

• The prognostic evaluation displays five- and 10-year recurrence-free interval ROC curves and adjusted hazard ratios, while the propensity figure shows score distributions and covariate balance before and after IPW adjustment.The propensity scores are clipped so no patient has an inverse propensity weight greater than 10.Source: Figure 1 · p. 26, Figure 3 · p. 28

Extended Data Fig. 4

• The chemotherapy-benefit analyses compare treatment-by-biomarker interactions for CTX-τ and Oncotype DX and test whether pathology adds predictive information beyond clinical features.The Oncotype DX comparison includes 983 patients; the modality ablation uses a 2% CTX-τ threshold and an IPW-adjusted Cox model.Source: Figure 4 · p. 29

Extended Data Fig. 5

• Patches in low-benefit clusters are dominated by fibrotic collagen, adipose tissue, normal ducts, and lymphocytes, whereas high-benefit clusters predominantly contain highly cellular invasive carcinoma nests.Five randomly sampled H&E patches are shown per cluster, ordered by increasing cluster-level CTX-τ coefficient.Source: Figure 5 · p. 30

Extended Data Fig. 6

• The cohort flow diagrams show 9,141 development patients retained from 23,912 and 1,994 evaluation patients retained from 2,877 after specified eligibility and slide-quality filtering.Development cohorts were not restricted by receptor subtype or treatment; the evaluation set was restricted to the intended HR+/HER2− deployment population.Source: Figure 6 · p. 31

A Supplementary Methods

• The methods first describe the data used in the study.This is the first of four parts listed for the section.Source: A Supplementary Methods · p. 35
• The methods then define notation and the two quantities output by CTX.These definitions precede the detailed review of modeling choices.Source: A Supplementary Methods · p. 35
• The remaining parts review modeling choices and describe how CTX is evaluated.The modeling review follows notation, and the evaluation description is last.Source: A Supplementary Methods · p. 35

A.1 Datasets

• 11,135 patients from 17 independent cohorts were assembled, with 9,141 patients in development and 1,994 in the held-out evaluation set.The evaluation set was restricted to HR+/HER2− patients receiving ET or CET.Source: §A.1 · p. 35
• Evaluation included female patients with early-stage disease, known chemotherapy status and RFI outcomes, no prior breast cancer, and at least one pathology slide.The Oncotype DX recurrence score was available for 983 evaluation-set patients.Source: §A.1 · p. 35
• Oncotype DX categories use the stated clinical cutoffs: low RS 0–10, intermediate RS 11–25, and high RS ≥26.Supplemental Table 1 reports score-category distributions across evaluation datasets.Source: §A.1 · p. 35, Table 1 · p. 35

A.2 Notation and CTX outputs

• For each patient, the observed data include clinicopathological and morphological features, treatment assignment, follow-up time, and recurrence-event status.CET is coded as A = 1 and ET as A = 0; the primary outcome is five-year recurrence-free interval.Source: §A.2 · p. 35
• CTX estimates five-year recurrence-free probability under each treatment, and the difference between CET and ET probabilities is personalized chemotherapy benefit.The CTX-τ score is the estimated CET probability minus the estimated ET probability.Source: §A.2 · p. 35
• The factual prediction corresponds to the treatment received and is used to assess prognostic discrimination and calibration; CTX-risk is recurrence risk under ET.The factual prediction is the model's estimate for the observed treatment assignment.Source: §A.2 · pp. 35–36

A.3 Model development

• Four methodological challenges inherent to estimating personalized chemotherapy benefit from observational data motivated CTX’s development components.The section describes those components in detail.Source: §A.3 · p. 36
• Supplemental Table 2 summarizes each methodological challenge alongside the component of the approach that addresses it.The table caption identifies its columns as “Challenge” and “Our solution.”Source: §A.3 · p. 36, Table 2 · p. 36
• The model-development section explains the components used to address those challenges.Source: §A.3 · p. 36

A.3.1 Multi-modal model architecture

• Clinical variables are encoded with a tabular ResNet, while image patches are embedded and aggregated into a slide-level representation with gated attention.Clinical inputs include tumor and nodal stage, age, AI genomic risk, receptor status, and ductal or lobular histology.Source: Figure 1 · p. 37, §A.3.1 · p. 37
• The multi-modal branch predicts recurrence-free probability under chemoendocrine therapy and endocrine therapy alone; their difference is the estimated chemotherapy benefit.Clinical-only and pathology-only treatment heads provide auxiliary predictions used only during training.Source: §A.3.1 · pp. 36–37, Figure 1 · p. 37
• Gated attention fuses the clinical and pathology representations into a multi-modal representation for treatment-specific prediction.Source: §A.3.1 · pp. 36–37, Figure 1 · p. 37
• Treatment-specific heads output discrete-time hazards that are accumulated into recurrence-free probabilities, including a five-year prediction at the bin ending at five years.The model derives its five-year chemotherapy-benefit score from the difference between the two treatment-specific probabilities.Source: §A.3.1 · pp. 37–38

A.3.2 Training objective

• The objective addresses right-censoring and treatment-group distribution differences with separate loss terms combined for gradient-based optimization.Source: §A.3.2 · p. 38
• The factual-treatment negative log-likelihood sums interval losses only through the observed follow-up bin, so intervals after a patient’s last follow-up contribute nothing.For censored patients, the censoring time is treated as occurring at that bin’s right boundary.Source: §A.3.2 · p. 38
• The maximum mean discrepancy penalty measures the distance between multi-modal representation distributions for chemoendocrine-therapy and endocrine-therapy patients.Minimizing it reduces embedding-space covariate shift and encourages representations supporting prediction for either treatment arm.Source: §A.3.2 · p. 38
• The overall loss combines weighted per-modality outcome losses, the balancing penalty, and elastic-net regularization; the balancing weight is annealed during training.The MMD weight starts at zero for 20% of epochs, ramps over the next 60%, then remains at its target for the final 20%.Source: §A.3.2 · pp. 38–39

A.3.3 Multi-source cross-validation and model training

• CTX used three-fold leave-one-cohort-out validation, with held-out cohorts for validation, hyperparameter selection, early stopping, and calibration; the five evaluation cohorts were excluded from model development.The ten best validation configurations were ensembled, and beta calibration was fitted to factual predictions before averaging calibrated predictions across folds.Source: §A.3.3 · p. 39

A.3.4 AI genomic risk as an auxiliary clinical input

• The auxiliary AI genomic risk input was trained to predict a 21-gene recurrence-score formulation, using pan-cancer TCGA pretraining followed by breast-cancer fine-tuning.The held-out NYU validation set contained 472 slides from 301 patients with recorded scores.Source: §A.3.4 · pp. 39–40

A.4.1 Causal assumptions; A.4.2 Propensity score estimation and inverse propensity weighting

• The appendix interprets τ(x) causally as five-year recurrence-free probability difference under chemoendocrine versus endocrine-only assignment, relying on consistency, ignorability, treatment positivity, non-informative censoring, and censoring positivity.Source: §A.4.1 · p. 40

A.4.3 Evaluation of CTX as a predictor of prognosis and chemosensitivity

• The held-out evaluation assessed CTX prognostic discrimination, calibration, and recurrence association at five and ten years, and tested whether predicted benefit modified chemotherapy’s association with recurrence-free interval.Benchmarking compared the treatment-interaction framework with clinical features and Oncotype DX, including a direct comparison in the 983 patients with recorded Oncotype DX scores.Source: §A.4.3 · pp. 41–42

A.4.4 Therapeutic decision support analysis

• The therapeutic decision-support analysis ranked patients by CTX-τ and estimated five-year recurrence-free rates when chemotherapy was assigned to the highest-benefit fraction.It derived the smallest treated proportion matching observed practice’s recurrence-free rate and the recurrence reduction at the observed treated proportion.Source: §A.4.4 · p. 42

A.4.5 Pathology foundation model explainability

• Pathology explainability analysis used one evaluation cohort to cluster image-patch embeddings and relate cluster prevalence to CTX-τ, with blinded pathologist review of representative patches.The analysis was conducted in UNIRAD (n = 362).Source: §A.4.5 · p. 42

A.4.6 Genomic and transcriptomic correlates

• Exploratory molecular analyses related CTX-τ to differential gene expression, pathway enrichment, and homologous-recombination-deficiency scores in 348 HR+/HER2− TCGA-BRCA samples.TCGA-BRCA contributed to model development, which the authors identify as a reason these analyses are exploratory.Source: §A.4.6 · pp. 42–43

A.4.7 Pan-cancer generalizability analysis

• Pan-cancer transfer evaluation applied CTX to TCGA cohorts for prognostic analysis across 16 cancer types and predictive analysis across 14 cancer types.The predictive analysis used cancer-type-specific IPW and compared chemotherapy-versus-endocrine restricted mean survival time differences between high- and low-benefit groups.Source: §A.4.7 · p. 43