Source-linked AI summary
Real-World Multi-Modal and Longitudinal Lung Cancer Dataset
Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago, Catarina Barata
TL;DR
Multimodal medical learning is limited by incomplete real-world data and difficult modality integration. This paper releases a curated longitudinal lung cancer dataset and evaluates fusion methods for survival prediction. Across realistic missingness, the resource supports multimodal benchmarking, with longitudinal hazard prediction achieving a C-index of 85.59 ± 4.03%.
Problem
Real-world multimodal learning lacks well-curated datasets and must integrate heterogeneous, frequently incomplete medical data.
Method
The paper constructs a 1,365-patient dataset combining clinical, imaging, transcriptomic, follow-up, and treatment data, then benchmarks survival and hazard prediction.
Results
85.59 ± 4.03% C-index was achieved for longitudinal hazard survival analysis.
Takeaways & Limitations
The dataset enables evaluation of multimodal fusion and missing-modality methods under clinically realistic heterogeneous missingness.
Takeaways & Limitations
The authors identify a need for nested cross-validation to reduce pretraining-data leakage and support fairer external comparisons.
Abstract
from arXiv · showhide
Multi-modal learning has demonstrated strong potential in medical applications by integrating heterogeneous data sources such as medical imaging, clinical records, and genomics to improve predictive performance and support clinical decision-making. However, advances in this area are often constrained by two key challenges: the limited availability of well-curated, ready-to-use datasets that accurately reflect real-world conditions, where medical data are frequently collected inconsistently and are often incomplete; and the inherent difficulty of integrating heterogeneous data modalities. In this work, we introduce a newly curated multi-center, multi-modal, and longitudinal dataset designed to support the evaluation of a wide range of learning pipelines under realistic conditions. The dataset comprises a total of 1,365 lung cancer patients and has three imaging modalities (whole-slide images, CT scans, and PET scans), structured clinical data, transcriptomic, and longitudinal follow-up and treatment information. For each imaging modality the dataset contains more than one instance. Moreover, the dataset exhibits substantial and non-uniform missingness across modalities, making it well-suited for studying robust multi-modal fusion strategies. We further provide both uni-modal and multi-modal benchmarks on the task of 12-month overall survival prediction, disease-specific survival, as well as longitudinal benchmark of hazard prediction under severe missing data. Our results show that, despite high levels of missingness, integrating complementary modalities consistently improves predictive performance over uni-modal approaches, highlighting the value of multi-modal fusion in realistic clinical settings. The dataset and benchmark code are available at https://github.com/ritacmendes/MMIST-LUNG.
1 Introduction
The paper addresses the difficulty of integrating incomplete heterogeneous medical data and the lack of curated, directly usable multimodal datasets. It introduces a curated lung cancer resource and benchmarks for multimodal survival and hazard prediction.
- Uni-modal systems capture only partial patient information, motivating models that integrate heterogeneous modalities.
- Existing resources such as TCGA, TCIA, and CPTAC are distributed across platforms and lack consistent curation and integration.
- The dataset contains 1,365 lung cancer patients with clinical, imaging, genomic, and longitudinal follow-up data under realistic modality missingness.
- Each imaging modality includes multiple observations, including repeated imaging studies and follow-up time points.
- Benchmarks cover uni-modal and multimodal prediction of 12-month overall survival, disease-specific survival, and longitudinal hazard.
- The benchmark suite includes fusion approaches that handle missing or incomplete modalities, including Transformer attention masking.
2 Lung Dataset
The curated cohort combines lung cancer patients from multiple repositories with clinical, imaging, genomic, and longitudinal treatment and follow-up data. Its modality availability and follow-up are heterogeneous, providing realistic conditions for survival analysis.
- The cohort contains 1,365 patients from TCGA and CPTAC, spanning LUAD and LSCC with clinical, transcriptomic, imaging, and longitudinal data.
- 1,059 patients have survival or follow-up time information, enabling survival analysis with censoring and individual follow-up durations.
- Clinical preprocessing excluded variables with more than 70% missing values, leaving 15 selected clinical features with substantial residual incompleteness.
- The dataset includes 5,427 WSI slides, 482 CT scans, and 144 PET volumes, with modality availability varying substantially across patients.
- Transcriptomic data were collected from cBioPortal, with one patient-level vector selected preferentially from primary tumor samples.
- Longitudinal records include tumor-status follow-up and treatment information for chemotherapy, radiation, immunotherapy, and surgery.
3 Benchmarks
The benchmarks evaluate survival prediction with clinical and multimodal inputs under missing data, comparing fusion strategies and classifiers. They also use temporal modeling for longitudinal hazard prediction.
- The benchmark covers 12-month overall survival, disease-specific survival, and discrete hazard prediction across successive time intervals.
- Clinical data are evaluated independently before multimodal experiments compare fusion strategies, classifiers, and missing-data effects.
- The pipeline extracts WSI features with TITAN and CT/PET embeddings with MeD-3D before fusion and classification.
- Early fusion uses Transformer self-attention or mean/max pooling, while late fusion averages predictions from independently processed available modalities.
- MLP and XGBoost serve as survival predictors, with class weighting and threshold tuning addressing severe imbalance.
- Longitudinal fusion uses learnable time encodings so self-attention can distinguish observations acquired at different time points.
4 Benchmark Results
The benchmarks cover disease-specific survival, 12-month survival, dynamic-horizon prediction, and longitudinal hazard prediction. Results indicate that multimodal approaches can improve robustness to missing clinical information, while longitudinal treatment information substantially improves dynamic prediction.
- The benchmark evaluates disease-specific survival, 12-month survival, dynamic-horizon survival, and discrete-time hazard prediction.
- Disease-Specific Survival Prediction: MLP improves multimodal performance across fusion strategies, whereas XGBoost benefits particularly from late fusion with modality-specialized models.
- Disease-Specific Survival Prediction: Removing AJCC tumor stage generally decreases predictive performance, but multimodal models show greater robustness than clinical-only models.
- Survival Prediction at 12 months: Multimodal models do not consistently outperform the strongest unimodal baseline for 12-month survival, but they mitigate degradation when AJCC stage is unavailable.
- Dynamic Horizon Leveraging Longitudinal Multi-Modal Data: 81.73 ± 0.37% BAcc was achieved when longitudinal treatment information was incorporated, substantially improving performance over diagnosis-only prediction.
- Longitudinal Hazard Survival Analysis: The hazard model achieved a concordance index of 85.59 ± 4.03% and updated survival estimates after new treatment and follow-up information became available.
5 Conclusion
The paper contributes a curated, real-world multimodal lung cancer dataset and benchmarks designed to evaluate fusion, missing-data robustness, and longitudinal clinical modeling. It also identifies nested cross-validation as future work to reduce pretraining-data leakage and enable fairer external comparisons.
- The dataset contains over 1,300 lung cancer patients with multiple imaging instances, clinical variables, transcriptomic profiles, and longitudinal follow-up and treatment information.
- Heterogeneous, clinically realistic missing-modality patterns make the dataset a challenging benchmark for multimodal fusion and missing-data methods.
- The dataset supports binary survival prediction, missing-modality handling, inter-modal interaction analysis, and longitudinal survival-risk modeling.
- Nested cross-validation is proposed as future work to mitigate pretraining-data leakage and support fairer external comparisons.