Source-linked AI summary
TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction
Minjun Kim, Jong Hak Moon
TL;DR
Inpatient risk prediction must model heterogeneous clinical evidence that is irregularly observed over time, while prediction tasks use different observation windows. TRACE organizes recorded multimodal events into daily global–local trajectories, and achieves the highest AUROC in all nine dataset–task settings.
Problem
Inpatient records are heterogeneous, incomplete, and irregularly sampled, while clinical risk tasks require different observation windows and depend on when evidence occurs.
Source: §1 · pp. 1–2Method
TRACE organizes recorded multimodal events within task-specific windows as day-level local tokens alongside day-invariant global context, encoded with temporal global–local attention.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
TRACE achieved the highest AUROC in all nine dataset–task settings across MIMIC-IV, eICU, and MIMIC-III for mortality, early Long LOS, and 30-day readmission prediction.
Source: §7 · p. 11Takeaways & Limitations
The results indicate that temporal organization complements multimodal fusion and missing-modality handling, while phenotype-guided contrastive learning provides additional representation-level benefits.
Source: §7 · p. 11Hospital-day resolution may miss rapidly evolving intra-day physiology or treatment response, and finer temporal resolutions may be more appropriate for acute outcomes.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Accurate inpatient risk prediction requires modeling heterogeneous clinical evidence that accumulates irregularly over time. This challenge spans in-hospital mortality, prolonged length of stay, and readmission, each with task-specific observation windows. Existing multimodal approaches combine structured health record data, chest x-rays, and clinical text, but primarily focus on modality fusion or missing-modality compensation, with limited attention to temporal trajectories over an admission. We propose TRACE (TRAjectory modeling of multimodal Clinical Events), a daily-scale framework organizing clinical events into day-level trajectories, separating day-invariant global context from day-specific local events drawn from EHR data, chest x-rays, radiology reports, and discharge summaries. The trajectory is encoded using temporal attention, allowing unavailable modalities to emit no tokens. A phenotype-guided contrastive objective further organizes admissions under heterogeneous disease distributions. Across cohorts from MIMIC-IV, eICU, and MIMIC-III comprising nearly 180,000 total stays, we evaluate TRACE on mortality, length of stay, and readmission prediction. TRACE achieves the highest AUROC across all nine dataset-task settings and the highest ACC in eight, outperforming baselines. Ablation studies validate our attention masks and contrastive learning, while trajectory-level attribution localizes risk-associated hospital days and modalities.
1 Introduction
Inpatient risk prediction must account for heterogeneous, irregularly recorded evidence whose meaning changes over time and whose observation window depends on the task. TRACE organizes recorded multimodal events into daily global–local trajectories and evaluates them across three datasets and three prediction tasks.
1 Introduction
- Clinical records are incomplete and irregular, and risk tasks require different observation windows, from the first 48 hours for Long LOS to completed-admission data for mortality and readmission.Prediction also depends on when events occur and how patient status changes over time.Source: §1 · p. 1
- TRACE separates day-invariant global context from day-specific local events, organizing recorded EHR, imaging, radiology-report, and clinical-documentation inputs within each task’s observation window.Unavailable modalities emit no tokens rather than requiring a complete multimodal grid.Source: §1 · p. 2
- Temporal global–local attention keeps global context accessible while ordering local events by hospital day and supporting same-day multimodal fusion.This combines stable context with evolving daily events without reconstructing unobserved modalities.Source: §1 · p. 2
- TRACE defines contrastive positive pairs using both outcome status and diagnosis-derived phenotype to organize admission representations.The approach addresses heterogeneous disease trajectories among patients reaching the same endpoint.Source: §1 · p. 2
- TRACE achieved the highest AUROC across all nine dataset–task settings for in-hospital mortality, Long LOS, and 30-day readmission prediction.The evaluation used MIMIC-IV, eICU, and MIMIC-III.Source: §1 · p. 3
- Trajectory-level attribution localizes hospital days and modality-specific inputs associated with changes in predicted risk.The evaluation also includes ablations of modalities, temporal attention, and phenotype-guided contrastive learning.Source: §1 · pp. 2–3
2 Related Work
Prior work combines multimodal clinical data, models admission relationships, or addresses missing modalities, but gives limited attention to the admission trajectory itself. TRACE instead organizes recorded events by hospital day and uses outcome and phenotype labels to structure representations.
2.1 Multimodal Fusion and Relational Clinical Risk Prediction
- Multimodal fusion, ensemble and collaborative predictors, and graph-based models combine clinical modalities or admission relationships, while temporal information is often aggregated at admission level or modeled within individual modalities.Examples include MultimodalPred, Chen et al., and MM-STGNN.Source: §2.1 · p. 3
2.2 Missing-Modality-Aware Clinical Representation Learning
- Missing-modality methods reconstruct, compensate for, or exploit absent inputs, whereas TRACE leaves unavailable modalities tokenless and aligns recorded events with hospital days.TRACE’s contrastive positive pairs use outcome labels and diagnosis-derived phenotypes rather than cross-modal alignment or missing-modality recovery.Source: §2.2 · p. 3
3 Methods
TRACE represents each admission as a task-window-specific daily trajectory of global context and local multimodal events, then encodes it with temporal global–local attention. It combines admission-level prediction with phenotype-guided contrastive learning.
3.1 Problem Formulation
- TRACE defines mortality, Long LOS, and readmission as admission-level tasks with task-specific observation windows.The task record combines day-invariant global context with daily multimodal events recorded within the relevant window.Source: §3.1 · p. 3
3.2 Multimodal Trajectory Representation
- Unavailable modalities contribute no local token, while recorded events retain modality identity and hospital-day position in the ordered trajectory.Structured EHR events from a day are concatenated into one descriptor; multiple images or reports on that day remain separate tokens.Source: §3.2 · p. 4
3.3 Temporal Global–Local Trajectory Encoding
- TGL attention lets local tokens attend to global context and same-day or earlier events, while same-day tokens interact bidirectionally for multimodal fusion.Global context tokens remain accessible across the trajectory, and the task and phenotype tokens summarize context for their respective objectives.Source: §3.3 · pp. 4–5
3.4.1 Admission-Level Risk Prediction
- The admission-level prediction head maps the task-specific admission representation to an outcome probability and is trained with binary cross-entropy.The prediction head uses task-specific parameters.Source: §3.4.1 · p. 5
3.4.2 Phenotype-Guided Contrastive Objective
- Phenotype-guided contrastive learning treats admissions as positives only when they share both the task outcome and diagnosis-derived phenotype.The phenotype is assigned from diagnosis codes mapped to chapter-level clinical super-groups; the primary diagnosis supplies the single major phenotype when multiple groups are present.Source: §3.4.2 · p. 5
3.4.3 Joint Training Objective
- The joint training objective combines admission-level risk prediction with the phenotype-guided contrastive objective, weighted by λ.The temperature τ and loss weight λ are selected independently for each dataset–task setting.Source: §3.4.3 · p. 6
4 Experiments
The experiments use MIMIC-IV, eICU, and MIMIC-III and evaluate mortality, early Long LOS, and 30-day readmission with task-specific observation windows. Cohort inputs reflect each dataset’s available modalities.
4.1 Datasets
- The study used MIMIC-IV, eICU, and MIMIC-III, splitting patients into training, validation, and test sets at a 75:10:15 ratio.Admissions shorter than two days were excluded; MIMIC analyses use hospital admissions, while eICU analysis uses ICU stays.Source: §4.1 · p. 6
- Mortality and 30-day readmission use data through the final hospital day, whereas Long LOS prediction uses only the first 48 hours.Long LOS means a hospital stay of at least seven days; readmission means a subsequent admission within 30 days after discharge.Source: §4.1 · p. 6
- MIMIC cohorts include structured EHR events, radiology reports, discharge summaries, and MIMIC-IV chest x-rays; eICU uses structured EHR inputs.For Long LOS, local inputs are restricted to the first two hospital days, excluding later events and terminal discharge summaries.Source: §4.1 · p. 6
4.2 Implementation Details
TRACE uses pretrained modality-specific encoders, shared-space projections, and a masked Transformer trajectory encoder; pretrained vision and language encoders remain frozen during training.
4.2 Implementation Details
- TRACE encodes each available modality with a pretrained modality-specific encoder, projects representations into a shared space, and applies the masked Transformer with TGL attention.Only projection layers, the trajectory encoder, and task-specific prediction and contrastive heads are updated.Source: §4.2 · p. 6
- Structured EHR values are converted to descriptors that preserve recorded precision and units, while radiology reports and discharge summaries use the clinical language encoder.Text inputs use a maximum sequence length of 512 tokens, and their 768-dimensional [CLS] representations are projected into TRACE’s shared hidden dimension.Source: §4.2 · p. 6
- Training runs for up to 30 epochs with AdamW and warmup–cosine scheduling, with early stopping based on validation performance.Across settings, selected contrastive loss weight λ averaged 0.42 and temperature τ averaged 0.09.Source: §4.2 · p. 6
4.3 Comparison with State-of-the-Art Methods
Across three datasets and three tasks, TRACE achieved the highest AUROC in all nine settings and the highest ACC in eight. Gains in AUPRC and macro-F1 were less uniform.
4.3 Comparison with State-of-the-Art Methods
- TRACE achieved the highest AUROC in all nine dataset–task settings and the highest ACC in eight.Table 1 compares TRACE with fusion- and relation-centered and missing-modality-aware baselines across MIMIC-IV, eICU, and MIMIC-III.Source: §4.3 · pp. 6, 8, Table 1 · p. 7
- TRACE’s mortality AUROCs were 0.9804 on MIMIC-IV, 0.9490 on eICU, and 0.9770 on MIMIC-III, exceeding second-best results by 0.0015, 0.0009, and 0.0018, respectively.CRL-MMNAR achieved higher AUPRC and mF1 in several mortality settings.Source: §4.3 · p. 8
- For Long LOS, TRACE’s AUROCs were 0.7675 on MIMIC-IV, 0.7971 on eICU, and 0.7864 on MIMIC-III, exceeding the second-best method by 0.0125, 0.0360, and 0.0113, respectively.TRACE also achieved the highest mF1 on MIMIC-IV and highest AUPRC on eICU and MIMIC-III.Source: §4.3 · p. 8
- For readmission, TRACE’s AUROCs were 0.8371 on MIMIC-IV, 0.7584 on eICU, and 0.8306 on MIMIC-III, exceeding the second-best method by 0.0245, 0.0694, and 0.0064, respectively.TRACE achieved the highest AUPRC on MIMIC-III; AUPRC and mF1 varied across cohorts.Source: §4.3 · p. 8
4.4 Ablation Studies
The ablations test modality combinations, the TGL attention mask, and phenotype-guided contrastive learning. Complete TRACE configurations led AUROC across both datasets, while the contrastive objective improved mean phenotype-stratified AUROC in all nine dataset–task settings.
4.4.1 Contribution of Input Modalities
- Structured EHR events were the strongest unimodal modality for mortality and readmission, while the strongest Long LOS modality differed by dataset.Multimodal combinations generally improved AUROC, especially when EHR events were combined with imaging or radiology reports; discharge summaries were unavailable for first-48-hour Long LOS.Source: §4.4.1 · p. 8, Table 2 · p. 9
- 0.9804/0.7675/0.8371 on MIMIC-IV and 0.9770/0.7864/0.8306 on MIMIC-III were the complete TRACE AUROCs for mortality, Long LOS, and readmission, respectively.These task-specific complete configurations achieved the highest AUROC for all tasks on both datasets.Source: §4.4.1 · p. 8, Table 2 · p. 9
4.4.2 TGL Attention-Mask Ablation
- 0.9804, 0.7675, and 0.8371 were TGL’s AUROCs for mortality, Long LOS, and readmission, exceeding both All-Causal and Full-Attention on MIMIC-IV.TGL also led AUPRC for mortality and Long LOS, but not every threshold-dependent metric.Source: §4.4.2 · p. 8, Table 3 · p. 9
4.4.3 Phenotype-Guided Contrastive Objective
- Adding Lpheno improved mean phenotype-stratified AUROC in all nine dataset–task settings, while mean AUPRC improved in seven and declined slightly in two.The largest AUROC gains were +0.0274 for eICU mortality and +0.0164 for eICU Long LOS; effects varied by phenotype.Source: §4.4.3 · p. 9
4.5 Trajectory-Level Interpretability
TRACE’s trajectory-level attribution links admission predictions to day- and modality-specific events. The cases illustrate that risk estimates can change with new findings, with different tasks responding differently to the same events.
4.5 Trajectory-Level Interpretability
- Attribution multiplies each input embedding by the gradient of task probability, then aggregates scores by hospital day and modality.The authors characterize this as qualitative interpretation, not a quantitative test of clinical causality.Source: §4.5 · p. 9
- In the respiratory case, Long LOS stayed high from admission, while mortality and readmission changed with new labs, medications, and report findings.The Long LOS prediction was 0.96 at admission and 0.93 through day 2; mortality and readmission rose on day 2, then fell on day 3.Source: §4.5 · pp. 9–10
- A day-16 pleural effusion raised readmission from 0.22 to 0.84 but mortality only from 0.03 to 0.11.Later, both estimates rose on day 17 despite a negative report, alongside new electrolyte/pCO2 abnormalities and added medications.Source: §4.5 · p. 10
- In the circulatory case, mortality and readmission fell by day 27 after improving opacities and new diuretics, then peaked again on day 36.The day-40 DVT and anticoagulants did not raise risk; the report modality had 3–4% attribution versus over 75% for EHR.Source: §4.5 · p. 10
5 Discussion
The discussion frames temporal organization as complementary to multimodal fusion and missing-modality handling. It highlights task- and time-dependent modality contributions and a less uniform additional benefit from phenotype-guided contrastive learning.
5 Discussion
- Admission-level context contributed strongly to early Long LOS estimates, whereas mortality and readmission estimates continued to change as new day-specific findings were recorded.The discussion treats temporal organization as a complementary modeling dimension beyond multimodal fusion and missing-modality handling.Source: §5 · p. 10
- TGL achieved the highest AUROC across mortality, Long LOS, and readmission, with gains more consistent for AUROC and AUPRC than for mF1.The discussion attributes this pattern to the global context remaining accessible while local events stay temporally ordered.Source: §5 · p. 10
- Structured EHR data were particularly informative for mortality and readmission, while chest x-rays provided stronger unimodal Long LOS discrimination on MIMIC-IV.Modality influence shifted during admissions, supporting event-driven input organization rather than fixed modality importance.Source: §5 · pp. 10–11
- Phenotype-guided contrastive learning improved mean phenotype-stratified AUROC across all nine dataset–task settings, although AUPRC and individual phenotype effects varied.The discussion presents it as a phenotype-aware representation regularizer.Source: §5 · p. 11
6 Limitations and Future Work
The authors identify day-level temporal resolution and retrospective public-dataset evaluation as limitations that bound the study’s scope.
6 Limitations and Future Work
- Hospital-day resolution may miss rapidly evolving intra-day physiology or treatment response, and evaluation was limited to public retrospective datasets with varying modality configurations.The authors call for finer temporal resolutions, prospective and external validation, and assessment in deployment settings.Source: §6 · p. 11
7 Conclusion
TRACE organizes heterogeneous inpatient data into temporally structured global–local trajectories. Across three datasets, it achieved the highest AUROC in all nine dataset–task settings; the conclusion identifies temporal organization and phenotype-guided contrastive learning as complementary benefits.
7 Conclusion
- TRACE represents heterogeneous inpatient data as a temporally organized global–local trajectory.Source: §7 · p. 11
- TRACE achieved the highest AUROC in all nine dataset–task settings across MIMIC-IV, eICU, and MIMIC-III.Source: §7 · p. 11
- The results identify temporal organization as complementary to multimodal fusion and missing-modality handling, with phenotype-guided contrastive learning providing additional representation-level benefits.Source: §7 · p. 11