PaperlayerView exact revision on arXiv

Source-linked AI summary

CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets

Robert N. Spaans, Catherine Chia, Tongjie Wang, Adam Kowalewski, Parandzem Khachatryan, Domingos Oliveira, Khrystyna Faryna, Jean-Paul A. van Basten, Geert Litjens, Nadieh Khalili

arXiv:2608.21497v1eess.IVcs.CV

TL;DR

Multimodal prognostic benchmarks integrating imaging and structured clinical data remain limited in prostate cancer. CHIMERA establishes such a benchmark and shows that multimodal models were more robust than unimodal clinical models when clinician-derived variables were withheld, despite lower leaderboard performance.

  • Problem

    Most prognostic benchmarks use a single modality, leaving limited standardized evaluation of models integrating paired imaging and structured clinical data.

  • Method

    CHIMERA benchmarked patient-level time-to-biochemical-recurrence prediction using paired mpMRI, histopathology whole-slide images, patient characteristics, and clinician-derived variables from 267 prostatectomy patients.

  • Results

    Multimodal models did not consistently outperform unimodal clinical models, but retained C-indices of 0.6762–0.6881 without pathology-derived variables while unimodal models fell near chance (C ≈0.50).

  • Takeaways & Limitations

    Multimodal models may reduce dependence on expert feature engineering when complete structured pathology annotations are unavailable.

  • Takeaways & Limitations

    The fully paired multimodal design limited the eligible cohort size and did not reflect incomplete modality availability common in real-world clinical data.

Abstract

from arXiv · show

Biochemical recurrence (BCR), defined as any detectable prostate-specific antigen level after prostatectomy with confirmatory elevation, is widely used as a surrogate endpoint and typically assessed using clinical and pathological variables. Currently, no standardized benchmark exists for multimodal prognostic modeling in urological cancers, partly because curating heterogeneous multimodal data remains challenging. We developed the CHIMERA Challenge, a multimodal benchmark integrating preoperative mpMRI, post-prostatectomy histopathology, patient characteristics, and clinician-derived variables from 267 patients across two institutions. The dataset comprises 801 MRI sequences, 13 clinical variables per case, and 942 WSIs. Training (n=95), validation (n=23), and test (n=149) splits were established and hosted on the Grand Challenge platform. Baseline clinical and pathological characteristics did not differ significantly across splits. Models were evaluated on predicting time to BCR using the C-index. Post-challenge analyses tested how each model type performed when clinician-derived variables were withheld or randomized. Unimodal clinical models achieved the highest test C-index of 0.7402 but proved sensitive to the integrity of these variables, with performance collapsing toward chance (C approximately 0.50) when they were randomized. Multimodal models retained near-baseline performance when these variables were withheld (delta C at most 0.04), indicating their ability to recover prognostic signal directly from imaging data. CHIMERA is the first public, standardized multimodal benchmark for prostate cancer prognosis. Although models using only patient characteristics and clinician-derived variables yielded the highest leaderboard performance, multimodal models demonstrated greater robustness in clinically realistic scenarios where complete expert annotation is not guaranteed.

1. Introduction

Existing prostate cancer benchmarks and risk models have established the value of standardized evaluation and multimodal prognostic information, but multimodal benchmarking with paired raw data remains a gap. CHIMERA addresses this gap through a curated, openly accessible prostatectomy cohort combining radiological, pathological, clinical, and clinician-derived data.

  • Standardized oncology benchmarks have demonstrated the value of fixed datasets and evaluation protocols, including PANDA, PI-CAI, and LEOPARD.
  • CAPRA-S combines PSA, Gleason grade, surgical margin status, capsular penetration, seminal vesicle invasion, and lymph node invasion to predict BCR after RP.
  • Multimodal benchmarks with paired raw imaging, patient characteristics, and clinician-derived variables are needed to evaluate whether AI learns prognostic information directly from radiology and pathology data.
  • CHIMERA released a curated paired multimodal prostatectomy cohort containing radiological and pathological imaging alongside structured clinical and clinician-derived variables.
  • CHIMERA is presented as a standardized, openly accessible benchmark for prognostic multimodal AI development in prostate cancer.

2. Material and methods

CHIMERA Task 1 used a fully paired multimodal cohort of 267 radical-prostatectomy patients, combining preoperative mpMRI, pathology whole-slide images, patient characteristics, and clinician-derived variables. The challenge targeted patient-level time-to-biochemical-recurrence prediction, with a dedicated multimodal modality-ablation analysis conducted across MRI, WSI, and clinical-variable combinations.

  • Cohort: 267 patients treated at two Dutch centers between 2012 and 2021 formed the CHIMERA challenge cohort.Eligibility required radical prostatectomy, preoperative mpMRI, digitized H&E-stained prostate WSIs, and consent for clinical-data use.
  • Cohort: Each included case required patient characteristics, clinician-derived variables, digitized pathology images, and preoperative multiparametric MRI.Cases were excluded when one or more required modalities were incomplete or unavailable.
  • Data modalities: 801 MRI sequences across 267 patients comprised axial T2W, high b-value diffusion-weighted, and apparent diffusion coefficient imaging.Scans were acquired at RUMC using 1.5T or 3.0T MRI scanners, with acquisition standards consistent with PI-CAI.
  • Clinical data and outcome: The released dataset contained 13 variables per patient—3 patient characteristics and 10 clinician-derived pathology variables—paired with time-to-event ground truth.The outcome was time to BCR or last available follow-up in months; BCR-related labels and PSA levels were masked during validation and testing.
  • Challenge task and ablation: Task 1 predicted patient-level time to BCR after radical prostatectomy for postoperative prognostic risk assessment.Algorithms produced a single patient-level prediction, and one multimodal model was retrained across all combinations of MRI, WSI, Clin_full, and Clin_filtered.

3. Results · 3.1. Challenge Leaderboard and Overview of Evaluated Algorithms

The CHIMERA challenge attracted 336 participants, with six teams submitting final algorithms for evaluation. Clinical-only models led the leaderboard, while teams also explored WSI-based and fully multimodal approaches.

  • 3.1. Challenge Leaderboard and Overview of Evaluated Algorithms: 336 participants from 43 countries took part, while 21 teams registered and six submitted final algorithms for testing.Participants could submit algorithm versions during development and debugging before the final test phase.
  • 3.1. Challenge Leaderboard and Overview of Evaluated Algorithms: The top three leaderboard models used only patient characteristics and clinician-derived variables; other teams incorporated WSIs or all available modalities.One team combined clinical variables with WSIs, and two submitted full multimodal models.
  • 3.1.1. TIA-Pegasus: 0.7402 was the TIA-Pegasus clinical-only model’s test C-index, following 0.8182 on validation.Its clinical and WSI model reached 0.8770 internally but dropped to 0.6446 on the CHIMERA validation set.
  • 3.1.2. WL: 0.7294 was WL’s final test C-index after selecting a standard MLP with 11 time bins.On validation, the standard MLP achieved 0.7521 versus 0.7423 for the ensemble.
  • 3.1.3. SMILE: 0.7355 and 0.7280 were SMILE’s validation and test C-indexes, respectively, for its imputed and cross-validated clinical-only model.The model used negative log-likelihood training with patient stratification by BCR status and discrete survival-time risk bins.
  • 3.1.4. IU CompPath: 0.7107 and 0.7197 were IU CompPath’s validation and test C-indexes for PROFUSEme, a multimodal model using clinical, MRI, and WSI embeddings.Its internal nested cross-validation C-index was 0.8610, and it was the only model with higher test than validation performance.
  • 3.1.5. VCMI: 0.7521 and 0.7153 were VCMI’s validation and final test C-indexes for its gated late-fusion multimodal survival model.The model integrated clinical, MRI, and WSI branches and achieved a training C-index of 0.97.
  • 3.1.6. OHSU-Cedar: OHSU-Cedar combined a 13-dimensional clinical vector with a 6-dimensional pathology histogram into a 19-dimensional normalized representation.Its pathology branch classified 2000 randomly sampled WSI tiles into six biologically meaningful tissue categories before aggregation.

3.2. Unimodal Feature Importance Analysis

Randomization analyses showed that unimodal clinical models relied primarily on structured pathological variables, with positive lymph node status and Gleason grading most consistently affecting performance. Removing pathology information reduced discrimination toward near-random levels, while feature reliance differed across models.

  • Cross-model feature importance: Positive lymph node status and Gleason grading were the most consistently impactful variables across unimodal models.Capsular penetration, tertiary Gleason, pT stage, pre-operative PSA, and lymphovascular invasion generally had small or inconsistent effects.
  • WL: 0.1292 was the largest WL performance drop from randomizing positive lymph nodes, followed by 0.0889 for primary Gleason.WL baseline performance was C0 = 0.7294; randomizing all pathology parameters produced ∆C = 0.1790 (q = 0.033).
  • SMILE: 0.1017 was the largest individual SMILE performance drop, caused by randomizing positive lymph nodes.For SMILE, C0 = 0.7280; secondary Gleason produced ∆C = 0.0397, all Gleason components combined produced ∆C = 0.0630, and all pathology variables produced ∆C = 0.2112 (q = 0.028).
  • TIA-Pegasus: 0.2519 was the largest TIA-Pegasus performance drop, produced by randomizing all pathology parameters.For TIA-Pegasus, C0 = 0.7402 and primary Gleason produced ∆C = 0.0642 (q = 0.010); randomizing positive lymph nodes had no impact (∆C = 0.0000).
  • Pathology ablation: C ≈0.50–0.56 was the near-random performance range after simultaneous randomization of all pathology-derived parameters.This result indicates that unimodal clinical models’ predictive performance was predominantly driven by structured pathological variables.

3.3. Multimodal performance using no pathology-derived variables · 3.4. Analyzing Modality Importance in the VCMI Model

Removing pathology-derived structured variables did not significantly reduce multimodal performance, with OHSU-Cedar showing minimal sensitivity. In VCMI ablations, Clin_full was strongest among single modalities, while MRI’s contribution depended on the accompanying modalities and clinical features.

  • 3.3. Multimodal performance using no pathology-derived variables: 3.3: ∆C = 0.0004 for OHSU-Cedar, whose performance decreased from C0 = 0.6885 to 0.6881 after pathology-derived variables were removed.The IU-CompPath and VCMI models showed larger declines of ∆C = 0.0328 and ∆C = 0.0391, respectively.
  • 3.3. Multimodal performance using no pathology-derived variables: 3.3: No statistically significant performance reductions occurred in any multimodal model after pathology-derived variables were removed (all p > 0.05).Predictive performance was largely preserved under the variable-removal condition.
  • 3.4. Analyzing Modality Importance in the VCMI Model: 3.4: Clin_full achieved the highest single-modality performance and was not significantly different from the full multimodal reference model (Cr = 0.6831; ∆C = 0.0322, q = 0.4402).The full WSI, MRI, and Clin_full configuration provided the baseline at C0 = 0.7153.
  • 3.4. Analyzing Modality Importance in the VCMI Model: 3.4: MRI alone performed lowest among configurations (Cr = 0.4595; ∆C = 0.2558, q = 0.0040), while WSI alone also significantly underperformed the reference (Cr = 0.6460; ∆C = 0.0693, q = 0.0355).Both imaging-only configurations performed worse than Clin_full.
  • 3.4. Analyzing Modality Importance in the VCMI Model: 3.4: WSI combined with Clin_full approached the reference most closely (Cr = 0.7076; ∆C = 0.0077, q = 0.5435), whereas adding MRI to Clin_full did not improve performance.WSI combined with MRI also improved over WSI alone, reaching Cr = 0.6902; ∆C = 0.0251, q = 0.4254.
  • 3.4. Analyzing Modality Importance in the VCMI Model: 3.4: Clin_filtered alone achieved near-random discrimination (Cr = 0.5062; ∆C = 0.2091, q = 0.0073), and MRI combined with Clin_filtered performed similarly (Cr = 0.5010; ∆C = 0.2143, q = 0.0073).Clin_filtered denotes data after clinician-derived variables were removed.
  • 3.4. Analyzing Modality Importance in the VCMI Model: 3.4: Adding Clin_filtered to WSI improved performance from Cr = 0.6460 to 0.6981, but adding it to WSI and MRI reduced performance from Cr = 0.6902 to 0.6762.The contribution of age and PSA was therefore configuration-dependent, while Clin_full remained the most informative structured-data configuration.

4. Discussion

CHIMERA showed that clinician-derived variables outperformed multimodal submissions, while multimodal models retained prognostic performance when pathology-derived variables were withheld. The discussion attributes this pattern to asymmetric information content and highlights dataset size, modality-ablation, and evaluation limitations.

  • Interpretation of challenge results: Clinician-derived unimodal models outperformed multimodal models, reflecting compressed expert signal embedded in variables that are multimodal in origin.The curated non-imaging modality combined manually extracted pathology-report variables with patient characteristics.
  • Post-hoc robustness analysis: C ≈0.50 was reached after pathology-derived variables were permuted in unimodal clinical models, whereas multimodal models without those variables retained C-indices of 0.6762–0.6881.The permutation result indicates that unimodal predictive capacity was driven primarily by structured pathology-derived variables; multimodal performance showed no statistically significant decline relative to the full models.
  • Generalization: 0.8770 fell to 0.6446 for the top-ranked team’s combined model from internal validation to the validation leaderboard, consistent with overfitting.The submitted model using only patient characteristics and clinician-derived variables generalized better.
  • Modality ablation: Clin_full alone achieved near-reference performance, while modality contributions depended on information already available and showed partial redundancy.The ablation used the VCMI model across combinations of WSI, MRI, Clin_full, and Clin_filtered; interpretation was limited because all ablations used one architecture trained on 95 cases.
  • Limitations: The primary limitation was the small, fully paired cohort required to provide complete mpMRI, pathology, and clinical data for standardized comparison.This inclusion criterion reduced the eligible patient pool and may not reflect incomplete modality availability in practice; expansion to additional institutions could increase sample size and diversity.
  • Future directions: CHIMERA-Agent extends the benchmark toward agentic multimodal reasoning under realistic clinical conditions, including missing modalities.The CHIMERA training data will remain publicly available for the foreseeable future.

5. Conclusion

CHIMERA introduces the first urological multimodal benchmark for prostate cancer prognostic modeling, integrating clinical data, multiparametric MRI, and digitized H&E whole-slide prostatectomy images. Results indicate that pathological variables drove performance most strongly, while multimodal models were less dependent on clinician-derived variables than unimodal models.

  • Benchmark contribution: CHIMERA is the first urological multimodal benchmark for prostate cancer prognostic modeling, integrating clinical data, multiparametric MRI, and digitized H&E whole-slide prostatectomy images.The dataset was curated from experienced urological centers in the Netherlands.
  • Challenge results: Pathological variables drove predictive performance most strongly, and multimodal models did not consistently outperform unimodal clinical models on the held-out test set.When clinician-derived variables were withheld, unimodal models lost nearly all predictive performance, whereas multimodal models retained substantially more.
  • Future directions: Future CHIMERA iterations should expand sample size, add centers to capture population heterogeneity, evaluate metrics beyond the concordance index, and address missing data.The benchmark is intended to support fair and structured evaluation as multimodal prognostic modeling develops.

CRediT authorship contribution statement

The CRediT statement assigns Robert N. Spaans broad leadership across conceptualization, data curation, analysis, methodology, software, validation, visualization, project administration, investigation, and writing. Other contributors are credited with project, data, methodological, software, and writing responsibilities.

  • Contributions: Robert N. Spaans is credited with conceptualization, data curation, formal analysis, investigation, methodology, project administration, software, validation, visualization, and writing.His writing contributions include the original draft, review, and editing.
  • Contributions: Catherine Chia contributed to conceptualization and project administration, along with writing review and editing.
  • Contributions: Adam Kowalewski and Parandzem Khachatryan are credited with data curation and writing review and editing.
  • Contributions: Spyridon Bakas contributed methodology, software, and writing review and editing, while Xulin Chen and Junzhou Huang contributed methodology and writing review and editing.

CHIMERA Consortium

The CHIMERA Consortium comprises contributors from multiple institutions and countries, including the University of Warwick, Korea University, the University of California San Diego, and Canadian organizations.

  • The consortium includes Noorul Wahab, Ethar Alzaid, Jiaqi Lv, Adam Shephard, and additional contributors.
  • Additional consortium members include Felipe C. M. R. Coutinho, Pedro Vitor Lima, Leonardo M. Ferreira, Inês P. Machado, and others.
  • Affiliations span the Tissue Image Analytics Centre at the University of Warwick, Korea University, the University of California San Diego, and Canadian organizations.

6. Supplementary Material · A. CHIMERA Challenge Organizers

The supplementary material identifies the CHIMERA Challenge organizers and their institutional affiliations across pathology, urology, dermatology, and research organizations in the Netherlands.

  • A. CHIMERA Challenge Organizers: Robert N. Spaans, Catherine Chia, Tongjie Wang, and Farbod Khoraminia are listed among the CHIMERA Challenge organizers.
  • A. CHIMERA Challenge Organizers: Khrystyna Faryna, Maryam Mohammadlou, Tahlita Zuiverloond, Jean-Paul A. van Basten, Sita Vermeulen, Geert Litjens, and Nadieh Khalilia are also named as organizers.
  • A. CHIMERA Challenge Organizers: The organizers include researchers affiliated with the Department of Pathology at Radboud University Medical Center in Nijmegen.
  • A. CHIMERA Challenge Organizers: The affiliations include the Department of Urology at Canisius Wilhelmina Hospital in Nijmegen.
  • A. CHIMERA Challenge Organizers: The Oncode Institute in Utrecht is represented among the listed institutional affiliations.
  • A. CHIMERA Challenge Organizers: Additional affiliations include pathology, urology, and dermatology departments at Erasmus University Medical Center in Rotterdam.

B. Supplementary Figures

The supplementary figures examine model performance under altered input conditions. They cover randomized variables in unimodal teams, pathology-variable removal in multimodal teams, and VCMI ablations across modality and clinical-feature configurations.

  • B. Supplementary Figures: Supplementary Figure S1 presents unimodal-team performance with randomized variables, stratified by cohort.The results are shown as a forest plot for each unimodal team and cohort.
  • B. Supplementary Figures: Supplementary Figure S2 presents multimodal-team performance after removing pathology-derived variables, split by data-origin center.The analysis is displayed as a forest plot separated according to the originating center of the data.
  • B. Supplementary Figures: Supplementary Figure S3 shows VCMI ablation results under different modality and clinical-feature configurations, stratified by cohort.The ablation analysis compares the VCMI model across input configurations for each cohort.
Loading 2608.21497v1…