Source-linked AI summary
Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning
Andrea Agostini, Simon Böhi, Moritz Vandenhirtz, Samuel Ruiperez-Campillo, Max Krähenmann, Silke Mühlstedt, Irene Cannistraci, Ece Özkan Elsen, Julia E. Vogt, Thomas M. Sutter
TL;DR
Medical foundation models often pretrain separately by modality, despite clinical assessment combining complementary modalities and medical studies having internal structure. LAMAE uses shared latent attention in self-supervised pretraining to learn patient-level representations across ECG, Echo, and CXR. It outperforms modality-specific and strong baseline approaches on hospital-stay tasks, while also improving single-modality representations.
Problem
Medical foundation models generally learn modality-specific representations, although clinical assessment integrates complementary modalities and medical observations have within- and cross-modality structure.
Source: §1 · pp. 1–2Method
LAMAE is a structure-aware masked autoencoder that uses shared latent attention over a study–view–entity hierarchy to learn patient-level representations during self-supervised pretraining.
Source: §3 · pp. 3–4, §5 · p. 9Results
LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines on hospital-stay tasks, and its latent-attention module improves single-modality representations.
Source: §1 · p. 2Takeaways & Limitations
Multimodal pretraining outperforms unimodal pretraining on hospital-stay tasks, while matching or exceeding unimodal models on most unimodal tasks.
Source: §4 · p. 6The entity-masking formulation assumes a constant entity count across views within each modality and equal token counts for entities within a view.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Cardiovascular diagnosis and treatment rest on integrating complementary modalities, such as electrocardiogram, echocardiography, and chest X-rays, each capturing distinct but complementary aspects of cardiac pathophysiology. Yet most medical foundation models remain modality-specific, combining modalities only for finetuning or post-training. This discards the cross-modal evidence clinicians naturally integrate and ignores the structure within each modality. We introduce Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining. Instead of fusing modalities post hoc, LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study-view-entity hierarchy, enabling aggregation of variable observations and handling of missing modalities. Pretrained on over 500'000 MIMIC-IV hospital stays, LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal hospital-stay tasks, such as in-hospital mortality, ICD-10 and DRG coding, and length of stay, while remaining competitive on unimodal tasks. Modeling this structure also pays off within a single modality: even without cross-modal information, the latent-attention module improves representations over modality-specific pretraining.
1 INTRODUCTION
Clinical assessment integrates complementary cardiac modalities, but existing pretraining is largely modality-specific and often overlooks structure within and across studies. LAMAE learns patient-level representations through shared latent attention, with reported gains on hospital-stay and unimodal tasks.
1 INTRODUCTION
- Clinicians integrate ECG, echocardiography, and chest radiographs because each contributes complementary evidence to cardiovascular diagnosis and prognosis.Cardiovascular diseases account for more than 20 million deaths annually, and early diagnosis and risk stratification are described as critical to improving outcomes.Source: §1 · p. 1
- Existing approaches largely pretrain ECG, Echo, or CXR separately, while multimodal methods commonly use contrastive alignment or late fusion rather than directly modeling hierarchical interactions.Medical observations also have internal structure, such as ECG leads and complementary imaging views.Source: §1 · pp. 1–2
- LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines on hospital-stay tasks, while latent attention also improves single-modality representations.Multimodal pretraining also outperforms unimodal pretraining on hospital-stay tasks without sacrificing unimodal performance.Source: §1 · p. 2
- The proposed LAMAE uses shared latent attention over a study–view–entity hierarchy to exchange information during pretraining, with entity masking that drops complete leads, frames, or views.It is designed to accommodate heterogeneous modalities, variable observations, and missing data.Source: §1 · p. 2
2 RELATED WORK
Prior medical representation-learning work includes modality-specific self-supervision, multimodal alignment and reconstruction, and methods for modeling within-study structure. LAMAE combines modality-specific encoders with shared latent attention to model structure within and across modalities.
Self-supervised pretraining in medicine
- Medical self-supervised encoders have been developed separately for CXR, Echo, and ECG, leaving their reconstruction signals siloed within each modality.Contrastive methods also treat images as independent instances.Source: §2 · p. 2
Multimodal masked autoencoders
- LAMAE’s framework masks patches and complete entities, encodes visible tokens with modality-specific encoders, applies shared latent attention, and reconstructs masked inputs.Complete entities include ECG leads, CXR views, and Echo frames.Source: Figure 1 · p. 3
Multimodal medical models
- Vision–language and cross-modal contrastive approaches align modalities, but text describes findings rather than physiological relationships between recordings, and contrastive alignment requires paired samples.The passage notes paired samples are scarce for many modality combinations in routine care.Source: §2 · p. 3
Multimodal masked autoencoders
- MultiMAE, 4M, and M3AE reconstruct masked tokens across modalities, while LAMAE retains entity-level modality-specific encoders and fuses through shared latent attention to use any observed modality subset.Turgut et al. (2025) combine masked data modeling with contrastive alignment but require paired studies of two modalities.Source: §2 · p. 3
Structured studies
- Prior structured-study methods use multiview contrast, hierarchical fusion, latent-space MAEs, or lead alignment, whereas LAMAE jointly models studies, views, and entities across modalities.The passage characterizes many prior approaches as relying on pairwise objectives.Source: §2 · p. 3
3 METHOD
LAMAE organizes multimodal observations in a study–view–entity hierarchy, encodes them with modality-specific modules, and uses shared latent attention to integrate available information. Entity-level masking supports reconstruction across observations, while the resulting embeddings represent the hospital stay.
3 METHOD
- Each modality is represented as a study containing views, and each view contains base entities such as Echo frames, ECG leads, or CXR images.The ECG example has one examination comprising 12 lead entities.Source: §3 · p. 4, Figure 2 · p. 4
3.1 ENCODERS AND DECODERS
- Modality-specific transformer encoders and decoders process the data, with decoders reconstructing each modality’s base entities separately.The paper refers to modality-specific adaptations for details.Source: §3.1 · p. 4
3.2 LATENT ATTENTION FOR MULTIMODAL INTEGRATION
- Shared latent attention jointly attends over embeddings from observed entities, views, and modalities, transforming entity-specific representations to include available multimodal information.The module operates on concatenated latent embeddings from observed modalities.Source: §3.2 · p. 5
3.3 ENTITY MASKING
- Dropping complete entities such as leads, frames, or views makes latent attention connect information across entities during reconstruction.Dropped entities are fully masked, and the reconstruction target includes their tokens as well as masked tokens of visible entities; the procedure targets a modality-level mask ratio in expectation under an equal-token assumption within each view.Source: §3.3 · p. 5
3.4 LAMAE
- The model trains and performs inference on any subset of modalities without architectural changes or retraining, because reconstruction uses observed modalities and latent attention aggregates their variable-size token set.Each post-attention entity-level CLS embedding can carry information from all observed modalities.Source: §3.4 · p. 6
4 EXPERIMENTS & RESULTS
The experiments assess LAMAE on hospital-stay and unimodal tasks. Multimodal pretraining leads on hospital-stay comparisons, while within-modality results show benefits from modeling structure.
4.1 DATA
- The dataset contains 658,847 hospital stays, including 227,835 CXR studies, 7,243 Echo studies, and 800,035 12-lead ECG measurements.The CXR studies include 377,110 images, and the Echo studies include 525,328 videos.Source: §4.1 · pp. 6–7
4.2 MULTIMODAL EVALUATION
- Across six hospital-stay tasks, LAMAEMM beats Independent MAEMM on all six, both strong baselines on five, and every unimodal LAMAE on the complete test set.The evaluation reports AUROC or macro AUROC; each test stay has at least one ECG, Echo, or CXR measurement.Source: §4 · p. 6, Table 1 · p. 7, Table 2 · p. 8
4.3 UNIMODAL EVALUATION
- On CXR, LAMAECXR raises macro AUROC from 82.58 to 83.29 for findings and from 85.03 to 85.56 for ICD-10; LAMAEMM reaches 83.42 and 85.88.LAMAEMM was evaluated with CXR input only.Source: §4.3 · p. 8
- For Echo, both LAMAE variants raise LVEF R2 from 0.273 to 0.414 and ICD-10 AUROC from 57.49 to 59.21 and 59.10, within seed variance of each other.Source: §4.3 · p. 9
- On ECG, both LAMAE variants improve PR, QT, QTc and ICD-10 AUROC from 70.08 to 76.87 and 79.13; QRS is within seed variance and P-wave R2 is ≤0.Source: §4.3 · p. 9
- Overall, LAMAE improves unimodal representation learning, extending its benefits beyond multimodal integration by leveraging inherent medical-data structure.Source: §4.3 · p. 9
5 CONCLUSION
LAMAE learns patient-level representations from ECG, Echo, and CXR through latent-space information exchange over a study–view–entity hierarchy. It outperforms several pretraining baselines on hospital-stay tasks and improves unimodal representations, with experiments limited to one institution and Echo available for few stays.
5 CONCLUSION
- LAMAE outperforms an otherwise identical model without cross-modal interaction, single-modality models, and contrastive and vision–language baselines on hospital-stay tasks.Pretrained on more than 500,000 MIMIC-IV hospital stays.Source: §5 · p. 9
- A shared latent-attention module exchanges information across ECG, Echo, and CXR during self-supervised pretraining rather than only at prediction time.The module operates over a study–view–entity hierarchy and handles heterogeneous data, variable observation counts, and missing modalities within one reconstruction objective.Source: §5 · p. 9
- The latent-attention module improves over independent MAEs on CXR, Echo, and ECG tasks, while multimodal-pretrained models match or exceed modality-specific counterparts in unimodal finetuning.The authors report that cross-modal pretraining does not come at the expense of unimodal performance.Source: §5 · p. 9
- Experiments use data from one institution, Echo is available for only a small fraction of stays, and no model recovers some fine-grained targets such as P-wave duration.The authors identify cross-institutional validation and adding modalities such as laboratory time series and clinical notes as next steps.Source: §5 · p. 9
A IMPLEMENTATION DETAILS
The implementation represents each stay through a study–view–entity hierarchy, with modality-specific encoders and decoders joined by latent attention. Entity-level masking supports reconstruction, and downstream prediction pools entity embeddings after the attention module.
A IMPLEMENTATION DETAILS
- CXR views are radiographs, ECG entities are the 12 leads of a sampled 10 s recording at 500 Hz, and Echo entities are eight frames sampled from a video.Up to three CXR views are used; Echo frames are spaced across a random window of up to 32 consecutive frames. Missing modalities and views are padded and ignored in attention and loss.Source: A IMPLEMENTATION DETAILS · p. 15, Table 4 · p. 15
- Each modality has its own ViT-B encoder and lighter decoder, while a single-layer latent-attention transformer processes concatenated patch and CLS tokens from observed modalities.The encoders are initialized from DINOv3; learned embeddings encode position, entity, view, and modality, and the latent-attention module carries a multimodal CLS token not used downstream.Source: A IMPLEMENTATION DETAILS · p. 15
- LAMAE masks tokens within views and drops complete entities, then minimizes masked-patch mean squared error averaged by modality and summed over observed modalities.Visible entities are masked more lightly so the overall masking ratio remains at the modality-specific target.Source: A IMPLEMENTATION DETAILS · p. 15
- For finetuning, decoders are discarded, masking is disabled, and each stay is represented by the average of entity-level CLS embeddings after latent attention.A LayerNorm and linear head follow; LAMAEMM unimodal evaluation uses only the respective modality.Source: A IMPLEMENTATION DETAILS · p. 15
- CXR training uses random resized crops, flips, and brightness/contrast jitter; Echo uses crops and brightness jitter without flips, while ECG uses z-score normalization and STAR time warping.Images are normalized with dataset statistics.Source: A IMPLEMENTATION DETAILS · p. 16
B TRAINING DETAILS
Training uses AdamW with global-batch-scaled learning rates, gradient clipping, and FP32 precision. Pretraining and finetuning use different learning-rate schedules, and reported finetuning results average three seeds.
B TRAINING DETAILS
- AdamW training uses weight decay 0.05, gradient clipping at 1.0, FP32 precision, and a learning rate scaled by global batch size.Pretraining uses linear warm-up, a constant learning rate, and a short final cooldown; finetuning uses linear warm-up followed by cosine decay.Source: B TRAINING DETAILS · p. 16
- LAMAE pretraining with all available modalities requires approximately 4,198 GPU hours, while finetuning one downstream task and seed requires approximately 4.33 GPU hours.Source: B TRAINING DETAILS · p. 16
- The hospital-stay comparison uses a 3,000-epoch CXR baseline, while all models in unimodal evaluation use 1,000-epoch checkpoints for fairness.Finetuning settings start from the final pretraining checkpoint, use warm-up followed by cosine decay, and are repeated with three seeds.Source: Table 5 · p. 16, Table 6 · p. 16
C DATA
The data are split by patient into training, validation, and test sets, and the same split is used for pretraining and downstream finetuning. The cohort contains heterogeneous modality subsets, while hospital-stay evaluation uses 50,000 labeled stays across six tasks.
C DATA
- The patient-level 80/10/10 split assigns 138,250 patients and 527,805 stays to training, 17,263 patients and 65,296 stays to validation, and 17,321 patients and 65,746 stays to testing.The same split is used for pretraining and all downstream finetuning, so no test patient is seen during training; 608,170 of 1,267,017 MIMIC stays without any of the three modalities are excluded.Source: C DATA · p. 16
- Hospital-stay finetuning uses 50,000 labeled stays for six tasks: mortality, ICD-10 Chapter IX, DRG code, DRG severity, DRG mortality risk, and length of stay.Experiments start from the final pretraining checkpoint and are repeated over three seeds.Source: Table 7 · p. 17
- Hospital stays may contain any subset of CXR, ECG, and Echo rather than a complete multimodal examination.The availability plot’s horizontal bars show stays containing each modality, and its vertical bars show counts for modality combinations.Source: C DATA · p. 17, Figure 4 · p. 17
- ICD-10 Chapter IX label distributions are reported by modality subset, including prevalence and positive-stay counts for retained diagnosis codes or code groups.The plots provide context on cohort composition and class imbalance alongside per-label classification results.Source: C DATA · p. 17
D EXPERIMENTS
The experiments evaluate hospital-stay and unimodal prediction tasks, define coverage-filled metrics for modality-limited models, and provide per-label results for selected classification and regression tasks.
D EXPERIMENTS
- ProbMED's MIMIC pretraining split may overlap downstream evaluation samples, potentially favoring its comparison; MedSigLip is finetuned with frequency-image ECG input to access all three modalities.ProbMED uses a contrastive objective on combinations of CXR, ECG, Echo, and corresponding text samples.Source: D EXPERIMENTS · pp. 17–18
- ICD-10 Chapter IX label distributions by modality subset show variation in sample availability and class imbalance.The figure reports prevalence and positive hospital-stay counts for retained diagnosis codes or code groups.Source: Figure 5 · p. 18
- Hospital-stay evaluation covers in-hospital mortality, ICD-10 Chapter IX diagnoses, DRG code and severity tasks, and three-class length of stay, using AUROC or macro one-vs-rest AUROC.Mortality uses AUROC; the other hospital-stay tasks use macro-averaged one-vs-rest AUROC.Source: D EXPERIMENTS · pp. 18–19
- Unimodal evaluation includes 14-label CXR classification, LVEF regression from Echo videos, and six ECG-interval regressions; ECG records with unreliable or out-of-range targets are excluded.An ECG record is retained only when all six interval targets are valid.Source: D EXPERIMENTS · p. 19
- Coverage-filled metrics score every labelled evaluation sample, substituting a training-set prior for samples with no requested modality and thereby penalizing limited coverage.Fallbacks are training prevalence for binary targets, the empirical class distribution for multiclass targets, and the training-label mean for regression; metrics converge to covered-only metrics as coverage approaches 100%.Source: D EXPERIMENTS · p. 19
- Additional figures report per-code ICD-10 Chapter IX AUROC, per-label CXR AUROC, and per-interval ECG regression R2 across the named models.Figure 6 maps ICD-10 codes from the root toward fine-grained leaves; Figure 7 shows means and standard deviations over three seeds, and Figure 8 reports interval R2 over three seeds.Source: D EXPERIMENTS · p. 19, Figure 6 · p. 20, Figure 7 · p. 21