Source-linked AI summary
Integrated multimodal artificial intelligence framework for healthcare applications
Luis R. Soenksen, Yu Ma, Cynthia Zeng, Leonard D. J. Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M. Wiberg, Michael L. Li, Ignacio Fuentes, Dimitris Bertsimas
TL;DR
Healthcare AI systems need to integrate heterogeneous clinical information, yet unified and scalable multimodal pipelines remain challenging. This paper proposes and evaluates the HAIM framework using standardized multimodal processing and modeling stages. Across healthcare tasks, multimodal systems consistently outperformed comparable single-source approaches, while Shapley analysis showed that modality importance varied by task.
Problem
Unified and scalable pipelines for healthcare AI systems that combine multiple data sources and modalities remain challenging to develop.
Method
The HAIM framework uses standardized preprocessing, feature extraction, and modeling stages for tabular, time-series, text, and image inputs.
Results
Multimodal HAIM systems consistently improved predictive analytics over single-modality approaches across healthcare applications, with average improvements of 9–28% across evaluated tasks.
Takeaways & Limitations
Shapley analyses showed heterogeneous modality and data-source importance across tasks, supporting multimodal inputs and holistic patient-data collection for healthcare AI.
Takeaways & Limitations
The systematic evaluation was based on the MIMIC-IV dataset, which was used to exemplify the pipeline and provide empirical evidence in a canonical ICU scenario.
Abstract
from arXiv · showhide
Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (N=34,537 samples) containing 7,279 unique hospitalizations and 6,485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6-33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48-hour mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.
INTRODUCTION
AI/ML systems may support clinical practice and healthcare operations, but effective systems often require knowledge from multiple data sources and modalities. Because unified, scalable multimodal pipelines remain challenging, the HAIM framework is proposed as an adaptable solution.
- AI/ML systems are anticipated to become fundamental tools in next-generation clinical practice and healthcare operations.
- Clinical and operational healthcare predictions can require knowledge from multiple data sources and input modalities.
- Multimodality has improved AI/ML performance over single-modality approaches, but unified and scalable multimodal pipelines remain challenging to develop.
- The HAIM framework is introduced as a modular machine-learning pipeline adaptable to tabular data, images, time-series, and text.
Demonstration of HAIM framework on multimodal clinical dataset
The HAIM framework processes multimodal clinical data through standardized stages to generate predictive models. Across its evaluation, increasing modalities and data sources was associated with improved and more consistent predictive performance.
- HAIM supports scalable patient-centric preprocessing and standardized feature extraction for rapid prototyping, testing, and deployment of predictive models.
- The framework uses independent processing streams for tabular, time-series, text, and image data before combining their representations.
- 14,324 individual model AUROCs were evaluated across chest diagnosis, length-of-stay, and mortality prediction tasks.
- 9–28% average percent improvement was observed for multimodal HAIM predictive systems across all evaluated tasks.
Analysis of source and multimodality contributions on model performances
The HAIM framework evaluates how progressively adding modalities and data sources affects predictive performance and quantifies their task-specific contributions. Multimodal systems show consistent performance gains, while the most informative modality varies by healthcare task.
- Shapley analysis found vision most useful for chest pathology diagnosis, whereas historical time-series records were most relevant for length-of-stay and 48 h mortality prediction.The distributions of modality and source contributions differed across predictive tasks.
- Increasing the number of modalities and sources was associated with higher average AUROC across evaluated tasks.AUROC standard deviations also appeared to decrease as more modalities and sources were used.
- 14,324 model experiments systematically tested combinations of modalities and data sources, providing evidence for positive monotonic performance trends alongside local non-monotonic and diminishing-return effects.
- HAIM uses modular feature extractors to create unified patient representations, allowing selected extractors to be updated without retraining the others.
- Aggregated Shapley values quantify heterogeneous modality and source importance across healthcare tasks, and the framework is provided as open-source code.
Dataset
The study uses MIMIC-IV and related multimodal data to construct patient-specific records for intensive-care and emergency-care research. HAIM-MIMIC-MM combines these records into a database supporting subsequent multimodal modeling.
- MIMIC-IV contains de-identified records for 383,220 patients admitted to the ICU or ED at BIDMC between 2008 and 2019.
- HAIM-EHR files contain current and previous admissions, transfers, demographics, laboratory measurements, orders, cultures, medications, and prescriptions.Files include available patient data collected during the admission and prior information before discharge or death.
Patient data processing and multimodal feature extraction
The framework converts heterogeneous patient data into standardized modality-specific embeddings and concatenates them into fixed-dimensional multimodal representations. It uses modality-specific processing and pretrained feature extractors to support consistent downstream modeling.
- Each patient file is processed into fixed-dimensional embeddings for tabular, time-series, text, and vision inputs.
- HAIM distinguishes four broad modalities—tabular, time-series, text, and images—from individual data sources within those modalities.
- Clinical BERT generates 768-dimensional vectors for each unstructured text type.
- Densenet121-res224-chex processes time-stamped chest X-rays, while aggregated multi-image embeddings incorporate all available X-rays per patient.
- Pretrained modality-specific feature extractors can be exchanged for updated models without requiring other extractors to be retrained.
- Single-modality embeddings are flattened, normalized, and concatenated into one 4,845-dimensional multimodal fusion embedding for downstream tasks.
Modeling
The study trains models for chest pathology diagnosis, length-of-stay prediction, and 48 h mortality prediction using multimodal fusion embeddings. Evaluation uses repeated patient-stratified train–test splits.
- Models address chest pathology diagnosis, length-of-stay prediction, and 48 h mortality prediction.
- Available embeddings were randomly split into 80% training and 20% testing sets across five patient-stratified splits.
Tasks of interest
The task targets early prediction of 10 common thorax-level pathologies in diagnostic imaging, supporting earlier intervention rather than delayed treatment.
- 10 thorax-level pathologies were selected for prediction, including fractures, pneumonia, pneumothorax, edema, and cardiomegaly.The passage also lists lung lesions, enlarged cardio mediastinum, consolidation, lung opacities, atelectasis, and cardiomegaly.
- Early detection in CT and other diagnostic imaging modalities can enable clinicians to focus on early intervention.
- The clinical motivation is to avoid delayed treatment for advanced stages of relevant pathologies.
Chest pathology diagnosis prediction.
The study evaluates multimodal predictive systems across clinical and operational tasks using selected machine-learning architectures, repeated randomized testing, and extensive model training.
- Clinical and operational tasks: Accurate length-of-stay forecasts support patient satisfaction, hospital resource allocation, and treatment planning.Predicting next 48 h discharges helps clinicians prioritize patients ready for discharge and accelerates discharge preparation.
- Clinical and operational tasks: 48-hour mortality prediction supports rapid ICU evaluations and can provide advanced warnings of possible deterioration.The passage notes that existing severity estimates such as APACHE scores do not incorporate characteristics beyond acute physiology.
- Model selection: XGBoost was selected for follow-up experiments after outperforming seven explored machine-learning architectures and supporting fast large-scale computation.The explored architectures included logistic regression, trees, random forest, multilayer perceptron, LightGBM, and TabNet.
- Model selection: 10,230 models were trained for chest pathology diagnosis prediction, with each XGBoost model repeated across five data splits to compute average metrics and standard deviations.
- Evaluation: 14,324 models were trained and tested to compare HAIM multimodal systems with single-modality systems across the defined tasks.Performance was summarized using testing-set AUROC averaged over five randomized train-test iterations.
- Evaluation: Aggregated fivefold test averages and standard deviations were grouped by the number of data sources and modalities in Fig. 2.Computational experiments used parallelization on MIT’s Supercloud with 30GB RAM and one NVIDIA Tesla V100 GPU per instance.
Reporting summary
Additional information on the research design is provided in the Nature Research Reporting Summary linked to the article.
- Further information on research design is available in the linked Nature Research Reporting Summary.
- The Nature Research Reporting Summary is linked to the article for additional design information.
- Readers can consult the linked reporting summary for further research-design information.
CODE AVAILABILITY
The authors provide code and supplementary materials, report the study’s submission dates, and identify the source datasets used to generate HAIM-MIMIC-MM.
- Code availability: The dataset-preparation, model-generation, and evaluation code is available in the authors’ GitHub repository and supplementary materials.
- Publication record: The manuscript was received on 26 February 2022 and accepted on 31 August 2022.
- Data sources: HAIM-MIMIC-MM was generated using MIMIC-IV v1.0 and MIMIC Chest X-ray v2.0.The work states that it contains no additional information involving human participants obtained by the authors.
ADDITIONAL INFORMATION
The article provides supplementary-material access, correspondence information, reprint details, publisher information, and licensing terms.
- Supplementary material: Supplementary material is available through the online version.The supplementary information is linked via the article’s DOI.
- Correspondence: Correspondence and material requests should be addressed to Dimitris Bertsimas.
- Reprints and permissions: Reprints and permissions information is available through Nature’s reprints page.
- Publisher’s note: Springer Nature states neutrality regarding jurisdictional claims in maps and institutional affiliations.
- Open access: The article is licensed under Creative Commons Attribution 4.0 International, subject to attribution, license-linking, and change-disclosure requirements.Third-party material is covered unless otherwise indicated.