PaperlayerView exact revision on arXiv

Source-linked AI summary

CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets

Robert N. Spaans, Catherine Chia, Tongjie Wang, Adam Kowalewski, Parandzem Khachatryan, Domingos Oliveira, Khrystyna Faryna, Jean-Paul A. van Basten, Geert Litjens, Nadieh Khalili

arXiv:2608.21497v1eess.IVcs.CV

TL;DR

Most oncology prognostic benchmarks address a single modality, leaving standardized evaluation of paired imaging and structured data open despite the integrated nature of clinical risk assessment. CHIMERA provides a controlled benchmark pairing prostate MRI, pathology slides, and clinical variables; when pathology-derived variables were withheld, multimodal models retained C-indices of 0.6762–0.6881, while unimodal clinical models fell to near-random performance after permutation.

  • Problem

    Most oncology prognostic benchmarks address single modalities, leaving standardized evaluation of paired raw imaging and structured clinical data an open challenge despite the integrated nature of clinical risk assessment.

    Source: §1 · p. 3

  • Method

    CHIMERA established a curated, controlled benchmark pairing patient characteristics and clinician-derived variables with mpMRI and prostatectomy whole-slide images from 267 patients at two Dutch centers.

    Source: §2.1.1 · p. 4, §5 · p. 29

  • Results

    C ≈0.50 (q < 0.05) after pathology-variable permutation in unimodal clinical models, while multimodal models without pathology-derived variables retained C-indices 0.6762–0.6881 without significant decline.

    Source: §4 · p. 25

  • Takeaways & Limitations

    Multimodal approaches were less dependent on expert feature engineering when clinician-derived variables were withheld, a relevant property when complete pathological annotation is unavailable.

    Source: §5 · p. 29

    Requiring complete mpMRI, pathology, and clinical data limited the eligible cohort and did not reflect the incomplete modality availability common in real-world clinical data.

    Source: §4 · p. 27

Abstract

from arXiv · show

Biochemical recurrence (BCR), defined as any detectable prostate-specific antigen level after prostatectomy with confirmatory elevation, is widely used as a surrogate endpoint and typically assessed using clinical and pathological variables. Currently, no standardized benchmark exists for multimodal prognostic modeling in urological cancers, partly because curating heterogeneous multimodal data remains challenging. We developed the CHIMERA Challenge, a multimodal benchmark integrating preoperative mpMRI, post-prostatectomy histopathology, patient characteristics, and clinician-derived variables from 267 patients across two institutions. The dataset comprises 801 MRI sequences, 13 clinical variables per case, and 942 WSIs. Training (n=95), validation (n=23), and test (n=149) splits were established and hosted on the Grand Challenge platform. Baseline clinical and pathological characteristics did not differ significantly across splits. Models were evaluated on predicting time to BCR using the C-index. Post-challenge analyses tested how each model type performed when clinician-derived variables were withheld or randomized. Unimodal clinical models achieved the highest test C-index of 0.7402 but proved sensitive to the integrity of these variables, with performance collapsing toward chance (C approximately 0.50) when they were randomized. Multimodal models retained near-baseline performance when these variables were withheld (delta C at most 0.04), indicating their ability to recover prognostic signal directly from imaging data. CHIMERA is the first public, standardized multimodal benchmark for prostate cancer prognosis. Although models using only patient characteristics and clinician-derived variables yielded the highest leaderboard performance, multimodal models demonstrated greater robustness in clinically realistic scenarios where complete expert annotation is not guaranteed.

1. Introduction

CHIMERA addresses the lack of standardized prognostic benchmarks integrating paired imaging and structured clinical data by releasing an open multimodal prostate cancer benchmark and testing modality masking.

1. Introduction

  • Most oncology benchmarks evaluate single modalities, leaving paired raw imaging and structured-data prognostic modeling under-addressed despite recent multimodal architectures.Examples include PANDA for Gleason grading, PI-CAI for mpMRI detection, and LEOPARD for recurrence prediction from pathology images.Source: §1 · p. 3
  • Biochemical recurrence occurs in 27% to 53% of patients after prostatectomy, making risk assessment relevant to individualized follow-up and salvage-treatment decisions.CAPRA-S already combines PSA, grade, surgical margin status, capsular penetration, seminal vesicle invasion, and lymph node invasion in tabular form.Source: §1 · p. 3
  • Paired imaging, patient characteristics, and clinician-derived variables could test whether models learn prognostic information from radiology and pathology and match, complement, or improve on clinical risk models.Source: §1 · pp. 3–4
  • CHIMERA releases a curated, openly accessible multimodal benchmark for predicting time to biochemical recurrence after radical prostatectomy and uses modality masking to examine data-source contributions and redundancy.The cohort pairs radiological and pathological imaging with structured clinical and clinician-derived variables.Source: §1 · p. 4

2. Material and methods

CHIMERA assembled a fully paired prostatectomy cohort with MRI, pathology, patient, and clinician-derived data for a patient-level time-to-BCR prediction challenge. The methods describe cohort selection, data preparation and splits, challenge evaluation, and post-hoc analyses of variable and modality contributions.

2.1.1. Inclusion criteria

  • Of 819 initially considered patients, 552 were excluded for incomplete or uncertainly linked modalities, leaving 267 fully paired cases from two Dutch centers.The cohort included patients undergoing prostatectomy between 2012 and 2021 with preoperative mpMRI, digitized H&E slides, and permission to use clinical data.Source: §2.1.1 · p. 4, §2.1.2 · p. 6

2.1.2. Data split

  • The final splits contained 95 training, 23 validation, and 149 test cases, with allocation shaped by overlap with public and hidden PI-CAI sets to prevent data leakage.The proportions were determined by the PI-CAI overlap constraint rather than chosen in advance.Source: §2.1.2 · p. 6

2.1.3. Pathology images

  • Pathologists selected slides corresponding to each case’s highest reported ISUP grade from 1,993 initial WSIs, yielding 942 WSIs digitized at 0.25 µm/pixel.Submissions were limited to 10 WSIs per patient, with slides grouped and downsampled to 0.5 µm/pixel for evaluation.Source: §2.1.3 · pp. 6–7

2.1.4. Radiology images

  • Preoperative mpMRI comprised axial T2-weighted, high b-value diffusion-weighted, and apparent diffusion coefficient sequences, totaling 801 sequences across 267 patients.Scans were acquired at RUMC using 1.5T or 3.0T scanners under standards consistent with PI-CAI.Source: §2.1.4 · pp. 7–8

2.1.5. Patient characteristics & clinician-derived variables

  • The dataset paired three patient characteristics and 10 clinician-derived pathology variables with time-to-event labels; recurrence PSA was masked at inference, and validation/test outcome labels were withheld.BCR required a confirmed rising PSA after prostatectomy; patients without recurrence by five-year follow-up or lost to follow-up were censored.Source: §2.1.5 · p. 8

2.1.6. Additional data

  • Alongside the challenge data, organizers provided tissue-segmentation masks and pre-extracted 1024-dimensional UNI v1 patch embeddings, while allowing participants to choose their own feature encoders.WSIs were divided into 224×224-pixel patches for embedding extraction; MRI prostate gland masks were provided using the PI-CAI model.Source: §2.1.6 · p. 9, §2.2 · p. 10

2.2. Challenge Design and Organization

  • The challenge evaluated fully automatic patient-level predictions of time to BCR, ranking submissions by test-set C-index across three phases on Grand Challenge.Participants could use any subset of the available modalities; validation allowed up to five submissions, while testing allowed one final submission.Source: §2.2 · pp. 9–10

2.3. Analysis

  • Post-hoc analyses randomized clinical variables at test time, removed clinician-derived pathology variables when retraining multimodal models, and tested all modality combinations in one multimodal model.Randomization used valid synthetic values, individually, in related groups, or across the full clinician-derived subset; the single-model ablation was selected based on team availability.Source: §2.3 · pp. 11–12

3. Results · 3.1. Challenge Leaderboard and Overview of Evaluated Algorithms

Across six final submissions, the clinical-only TIA-Pegasus model led test performance with a C-index of 0.7402, while multimodal models also appeared on the leaderboard. Reported performance varied across teams and between validation and test sets.

3.1. Challenge Leaderboard and Overview of Evaluated Algorithms

  • The challenge drew 336 participants from 43 countries; six teams submitted final models, including clinical-only, clinical-plus-WSI, and full multimodal approaches.Two teams submitted models using all available data for each case.Source: §3.1 · p. 13

3.1.1. TIA-Pegasus

  • On test, TIA-Pegasus’s clinical-only model achieved the top C-index, 0.7402; the top three leaderboard models used only patient characteristics and clinician-derived variables.The team’s clinical-and-WSI model achieved C-index 0.8770 internally but 0.6446 on validation, whereas its clinical-only model achieved 0.8182 on validation.Source: Table 2 · p. 13, §3.1 · p. 13, §3.1.1 · p. 15

3.1.2. WL

  • The standard MLP with 11 time bins was selected for the final test phase after better internal-validation performance, achieving a C-index of 0.7294; on CHIMERA validation, it scored 0.7521 versus the ensemble at 0.7423.Source: §3.1.2 · pp. 15–16

3.1.3. SMILE

  • SMILE achieved C-index 0.7355 on validation and 0.7280 on test.The team used a three-layer MLP with four discrete time-bin outputs and trained it with three-fold cross-validation.Source: §3.1.3 · p. 16

3.1.4. IU CompPath

  • IU CompPath achieved C-index 0.7107 on validation and 0.7197 on test, the only submitted model with a higher test than validation score.Its multimodal model fused clinical, MRI, and pathology representations using a Transformer encoder and a Cox proportional hazards head.Source: §3.1.4 · pp. 16–17

3.1.5. VCMI

  • VCMI achieved C-index 0.7521 on validation and 0.7153 on test, after reporting a training C-index of 0.97.Its final model used gated late fusion and DeepHit survival modeling with modified Weibull time bins.Source: §3.1.5 · pp. 18–19

3.1.6. OHSU-Cedar

  • OHSU-Cedar achieved C-index 0.7686 on validation and 0.6885 on test using clinical features combined with six-category FTU pathology histograms.The pathology features summarized the relative proportions of six tissue classes across 2,000 sampled tiles per WSI.Source: §3.1.6 · p. 19

3.2. Unimodal Feature Importance Analysis

Randomization analysis showed that unimodal models relied strongly on pathology-derived variables, while the importance of individual features varied across models. Several other clinical and pathological variables had little or inconsistent independent impact.

3.2. Unimodal Feature Importance Analysis

  • Positive lymph node status was the strongest individual feature for WL and SMILE, whereas primary Gleason had the largest individual impact for TIA-Pegasus.For WL, positive lymph node randomization caused ∆C = 0.1292 (q = 0.012) and primary Gleason randomization caused ∆C = 0.0889 (q = 0.025). For SMILE, positive lymph nodes caused ∆C = 0.1017 (q = 0.028); secondary Gleason caused ∆C = 0.0397 (q = 0.047), and all Gleason components combined caused ∆C = 0.0630 (q = 0.046). For TIA-Pegasus, primary Gleason caused ∆C = 0.0642 (q = 0.010), while positive lymph node randomization had ∆C = 0.0000.Source: §3.2 · p. 20
  • Capsular penetration, tertiary Gleason, pT stage, pre-operative PSA, and lymphovascular invasion generally had small, inconsistent effects with confidence intervals overlapping zero and non-significant q-values.Their independent contributions appeared limited in the presence of other predictors.Source: §3.2 · p. 20
  • Randomizing all pathology variables reduced performance to C ≈ 0.50–0.56, with ∆C = 0.1790 for WL, ∆C = 0.2112 for SMILE, and ∆C = 0.2519 for TIA-Pegasus.These results indicate that structured pathological variables predominantly drove predictive performance in the unimodal clinical models.Source: §3.2 · p. 20, Table 3 · p. 21

3.3. Multimodal performance using no pathology-derived variables

Multimodal models largely preserved predictive performance after pathology-derived structured variables were removed, with no statistically significant reductions reported. The observed declines ranged from 0.0004 to 0.0391 in ∆C-index.

3.3. Multimodal performance using no pathology-derived variables

  • IU-CompPath declined from C0 = 0.7197 to 0.6869 (∆C = 0.0328), while VCMI declined from C0 = 0.7153 to 0.6762 (∆C = 0.0391).IU-CompPath excluded patient characteristics as well and relied solely on imaging data.Source: §3.3 · p. 21, Table 4 · p. 22
  • OHSU-Cedar decreased from C0 = 0.6885 to 0.6881 (∆C = 0.0004), showing minimal sensitivity to pathology-derived variable removal.Source: §3.3 · p. 21, Table 4 · p. 22
  • No statistically significant performance reductions occurred across the three models (all p > 0.05) after pathology-derived structured variables were removed.The analysis compared baseline C0 with post-removal Cr performance.Source: Table 4 · p. 22, §3.3 · p. 22

3.4. Analyzing Modality Importance in the VCMI Model

The VCMI ablation found that Clin_full was the strongest single modality and that modality contributions depended on the combination: WSI benefited from added clinical information, while MRI’s contribution was inconsistent. Clin_filtered alone or with MRI performed near chance, but improved performance when paired with WSI.

3.4. Analyzing Modality Importance in the VCMI Model

  • Clin_full alone was closest among single modalities to the full-model baseline (C0 = 0.7153; Cr = 0.6831; ∆C = 0.0322, q = 0.4402).Its difference from the full multimodal reference was not statistically significant.Source: §3.4 · pp. 22–23, Table 5 · p. 23
  • WSI alone reached Cr = 0.6460 (∆C = 0.0693, q = 0.0355), while MRI alone was lowest at Cr = 0.4595 (∆C = 0.2558, q = 0.0040).Both were below the full multimodal reference; WSI’s decrease was significant.Source: Table 5 · p. 23, §3.4 · p. 23
  • WSI+Clin_full reached Cr = 0.7076 (∆C = 0.0077, q = 0.5435), and WSI+MRI reached Cr = 0.6902 (∆C = 0.0251, q = 0.4254).Both combinations improved over WSI alone, with WSI+Clin_full closest to the reference model.Source: Table 5 · p. 23, §3.4 · p. 23
  • Adding MRI to Clin_full did not improve performance (Cr = 0.6687 versus Cr = 0.6831 for Clin_full alone), so MRI’s benefit depended on the modality combination.MRI improved performance with WSI but added no benefit to the full set of patient characteristics and clinician-derived variables.Source: Table 5 · p. 23, §3.4 · p. 23
  • Clin_filtered alone (Cr = 0.5062; ∆C = 0.2091, q = 0.0073) and MRI+Clin_filtered (Cr = 0.5010; ∆C = 0.2143, q = 0.0073) performed near chance.Clin_filtered denotes the clinical dataset after clinician-derived variables were removed.Source: §3.4 · pp. 22–23, Table 5 · p. 23
  • Adding Clin_filtered to WSI improved performance (Cr = 0.6981 versus Cr = 0.6460), but adding it to WSI+MRI did not (Cr = 0.6762 versus Cr = 0.6902).The passage attributes the WSI combination’s improvement to complementary information from age and PSA, and describes Clin_filtered’s contribution as configuration-dependent.Source: Table 5 · p. 23, §3.4 · pp. 23–24

4. Discussion

CHIMERA provides a fixed-protocol setting for comparing prognostic models across matched clinical and imaging data. Although clinician-derived variables led the leaderboard, post-hoc analyses showed that multimodal models retained prognostic signal when pathology-derived variables were withheld.

Discussion

  • Clinical models fell to C ≈0.50 after pathology-variable permutation, while multimodal models retrained without those variables retained C-indices of 0.6762–0.6881 without significant decline.The retained performance remained above chance but below the full unimodal leaderboard result, suggesting that MRI and WSIs can partially compensate for missing manually curated pathology variables.Source: §4 · p. 25
  • The leaderboard gap partly reflects clinician-derived variables’ compressed expert pathology signal, whereas multimodal models must learn equivalent representations from raw imaging with only 95 training cases.Producing the structured variables requires expert pathology review and manual extraction, making them a labor-intensive, high-quality feature set rather than a purely non-imaging input.Source: §4 · pp. 24–25
  • The top-ranked combined model’s internal validation C-index of 0.8770 fell to 0.6446 on the validation leaderboard, while its clinical-only submission generalized better.Structured pathology remains a strong baseline where available, but larger cohorts could make manual extraction infeasible; pathology-only AI models have performed competitively in larger benchmarks.Source: §4 · p. 25
  • CHIMERA’s top test C-index of 0.7402 was below CAPRA-S’s 0.77 and the Stephenson MSKCC nomogram’s 0.79–0.81, though cohort and validation differences limit direct comparison.The passage also notes Hu et al.’s single-center internal C-index of 0.860, which is not directly comparable with CHIMERA’s held-out challenge test result.Source: §4 · p. 26
  • In VCMI ablations, MRI alone was near chance but improved WSI performance, while MRI added no benefit to Clin_full; Clin_filtered was weak alone but additive with WSI.These results came from one architecture trained on 95 cases; modality effects may reflect its gated fusion and capacity, so the MRI pattern is context- and architecture-dependent.Source: §4 · pp. 26–27
  • Requiring complete paired modalities limited cohort size and generalizability; expansion across institutions, missing-modality methods, and unimodal pretraining followed by CHIMERA fusion could address these constraints.The challenge did not clearly demonstrate that multimodal models outperform clinician-derived pathology variables; larger paired cohorts would also support more robust modality ablations.Source: §4 · pp. 27–28
  • The C-index evaluates risk ordering, not calibration or clinically meaningful event-time estimates; future editions could add calibration, time-dependent AUC, and Brier score.The high censoring rate (≈70%) also reduces observed events and informative comparisons for training and evaluation.Source: §4 · p. 28
  • CHIMERA-Agent extends the benchmark toward agentic multimodal reasoning under realistic conditions, including missing modalities, while CHIMERA training data will remain publicly available.Source: §4 · p. 29

5. Conclusion

CHIMERA establishes a curated multimodal benchmark for prostate cancer prognostic modeling. Its results show that multimodal models did not consistently outperform unimodal clinical models, but retained more predictive performance when clinician-derived variables were withheld.

5. Conclusion

  • CHIMERA introduces the first urological multimodal benchmark for prostate cancer prognosis, enabling reproducible comparison of methods across clinical data, mpMRI, and prostatectomy whole-slide images.Six teams submitted methods spanning modality combinations and fusion strategies.Source: §5 · p. 29
  • When clinician-derived variables were withheld, unimodal models lost nearly all predictive performance, while multimodal models retained substantially more.The challenge results also showed that pathological variables most strongly drove predictive performance and that multimodal models did not consistently outperform unimodal clinical models on the held-out test set.Source: §5 · p. 29
  • Future CHIMERA iterations should expand sample size and center diversity, add metrics beyond the concordance index, and handle missing data more faithfully to clinical reality.The benchmark provides a foundation for fairer, structured evaluation as multimodal prognostic modeling develops.Source: §5 · p. 29

CRediT authorship contribution statement

The authors’ contributions span conceptualization, data curation, analysis, methodology, software, supervision, and writing. Robert N. Spaans is credited with the broadest listed range of roles.

CRediT authorship contribution statement

  • Spaans contributed to conceptualization, data curation, analysis, investigation, methodology, administration, software, validation, visualization, and writing.He is credited with both original-draft writing and review and editing.Source: CRediT authorship contribution statement · p. 30
  • Chia contributed to conceptualization and project administration; Wang to review and editing; and Kowalewski, Khachatryan, and Oliveira to data curation and review and editing.Faryna contributed to conceptualization and software; van Basten to resources and supervision; and Litjens to conceptualization, data curation, resources, software, and supervision.Source: CRediT authorship contribution statement · p. 30
  • Khalili contributed to conceptualization, funding acquisition, methodology, project administration, resources, software, and supervision; nine named collaborators are credited with methodology.Those collaborators are Noorul Wahab, Ethar Alzaid, Jiaqi Lv, Adam Shephard, Shan E Ahmed Raza, Yeonwoo Seo, Emily Xie, Jun Ma, and Bo Wang.Source: CRediT authorship contribution statement · p. 30
  • Bakas contributed to methodology, software, and review and editing; Chen and Huang contributed to methodology and review and editing.The passage also begins a list of further contributors, but their roles are not included in the supplied text.Source: CRediT authorship contribution statement · p. 30

6. Supplementary Material · B. Supplementary Figures

The supplementary figures show team performance under variable randomization and feature-removal analyses, including cohort- and center-specific comparisons.

B. Supplementary Figures

  • Supplementary Figure S1 plots unimodal-team performance with randomized variables across cohorts.Source: Figure S1 · p. 37
  • Supplementary Figure S2 plots multimodal-team performance after removing pathology-derived variables, split by data-origin center.Source: Figure S2 · p. 37
  • Supplementary Figure S3 presents VCMI ablation results for different modality and clinical-feature configurations across cohorts.Source: Figure S3 · p. 38
Loading 2608.21497v1…