Source-linked AI summary
MR-JEPA: A General Purpose Video Foundation Model for Cardiac MRI
Athira J. Jacob, Puneet Sharma, Dorin Comaniciu, Daniel Rueckert
TL;DR
Most CMR deep learning methods discard sequential context by processing individual 2D slices, motivating a unified model for temporal cine and spatial multi-sequence data. MR-JEPA extends LeJEPA to 3D spatiotemporal CMR with tubelet embeddings, masking, and 2D CMR initialization, then evaluates a frozen encoder across six tasks. It outperforms compared methods on all five regression tasks and remains competitive for disease classification, while its evaluation is limited by single-vendor pretraining and restricted external validation.
Problem
Most CMR deep learning methods process individual 2D slices, disregarding sequential context, while adapting video models is difficult because of limited medical video, heterogeneous sequences, and multi-view integration needs.
Method
MR-JEPA extends LeJEPA to 3D spatiotemporal CMR using tubelet embeddings, spatiotemporal masking, and initialization from a 2D CMR foundation model, with gated attention over frozen-encoder views.
Results
MR-JEPA achieves the best performance on all five regression tasks and remains competitive for disease classification, including LV EF MAE 4.79% and GLS MAE 1.87.
Takeaways & Limitations
The results support a unified encoder for multi-view utilization of diverse CMR sequences in cardiac quantification and diagnosis.
Takeaways & Limitations
Pretraining uses single-vendor data; most downstream tasks come from one pretraining center, and RV EF and strain references derive from a validated deep learning model rather than manual annotations.
Abstract
from arXiv · showhide
Cardiac magnetic resonance imaging (CMR) produces rich sequential data such as temporal cine videos and spatial LGE/mapping stacks, yet most deep learning approaches process individual 2D slices, discarding this context. We present MR-JEPA, a self-supervised video foundation model for CMR that extends LeJEPA to 3D spatiotemporal inputs through tubelet tokenization, spatiotemporal masking augmentation, and initialization from a 2D CMR foundation model. Unlike prior CMR video models limited to cine data, MR-JEPA is pretrained on multi-sequence data (cine, LGE, mapping) from 10,505 patients across two centers without annotations. We evaluate the frozen encoder on six downstream tasks using a unified multi-view gated attention architecture: LV ejection fraction, RV ejection fraction, three myocardial strains (GLS, GCS, GRS), and four-class disease detection. MR-JEPA outperforms other compared methods on all five regression tasks, including both a domain-specific CMR model pretrained on more data with text supervision and a natural-video foundation model, achieving an LV EF MAE of 4.79% (r =0.764) and a GLS MAE of 1.87 (r=0.805), with 21-27% MAE reductions over baselines on strain tasks. For disease detection, MR-JEPA achieved a macro AUG of 0.868, remaining competitive with the domain-specific baseline despite using a fully self-supervised pretraining objective. These results demonstrate the potential of a unified video encoder for robust, multi-view utilization of diverse CMR sequences in clinical cardiac quantification and diagnosis.
1 Introduction
CMR contains rich temporal, spatial, and tissue-characterization information, but most deep learning methods analyze individual 2D slices. MR-JEPA addresses this gap with a self-supervised video foundation model designed for heterogeneous multi-sequence CMR.
- CMR combines cine, LGE, and parametric mapping sequences containing temporal and spatial information relevant to cardiac function, mechanics, and disease characterization.
- Most CMR deep learning methods process individual 2D slices, disregarding sequential context in cine and multi-slice acquisitions.
- MR-JEPA is a fully self-supervised CMR video foundation model pretrained on multi-sequence data without annotations.
- MR-JEPA extends LeJEPA with tubelet embeddings, spatiotemporal masking, and initialization from a pretrained 2D CMR foundation model.
- MR-JEPA outperforms natural-video and prior CMR-specific foundation models on all regression tasks while remaining competitive for disease classification.
2 Related Work
Self-supervised visual representation learning has expanded from images to video, while cardiac MRI foundation models have explored 2D, masked-autoencoder, and multimodal approaches. Existing CMR video methods remain limited by cine-focused inputs or dependence on paired clinical reports.
- Image-based self-supervised methods learn invariant representations from augmented views using contrastive or self-distillation objectives.
- Video self-supervised learning includes masked autoencoders that reconstruct masked patches and JEPA methods that predict latent representations.
- Prior cardiac MRI foundation models include a 2D DINO-based model and CineMA, which uses masked-autoencoder pretraining on cine images.
- Shad et al. proposed multimodal foundation-model pretraining on cine MRI and clinical reports, but such approaches require paired report-image data.
3 Method
MR-JEPA extends LeJEPA into a unified spatiotemporal encoder for diverse CMR sequences. A frozen encoder produces view embeddings that gated attention aggregates for cardiac function regression and disease classification.
- LeJEPA trains a single shared encoder to learn invariant representations across multiple augmented views without teacher-student components.
- MR-JEPA adapts LeJEPA to CMR video using 3D tubelet embeddings, spatiotemporal masking, and initialization from a 2D CMR foundation model.
- The Vision Transformer jointly encodes spatial and temporal tubelets, while its [CLS] output is projected for the pretraining loss.
- The frozen encoder is evaluated on five cardiac-function regression tasks covering LV EF, RV EF, GLS, GCS, and GRS.
- A sixth task classifies patients as normal, DCM, HCM, or IHD using cine views and an LGE view.
- Gated attention aggregates projected [CLS] embeddings from multiple views before a task-specific linear head produces regression or classification outputs.
4 Data and Experiments
MR-JEPA is pretrained and evaluated within a frozen-encoder, multi-view framework using diverse CMR data and matched baselines. Experiments cover cardiac function, disease classification, and analyses of key pretraining choices.
- Pretraining uses 160,172 CMR clips from 10,505 patients across two clinical centers, acquired on 1.5T and 3T Siemens scanners.
- RV EF and strain tasks use Center 1 data with reference measurements generated by a previously validated CMR analysis pipeline.
- Disease labels are normal, IHD, HCM, or DCM, with inputs comprising three cine videos and one whole-heart LGE stack.
- Auxiliary analyses assess cine-view classification and LGE scar detection to evaluate representation quality across temporal and spatial sequences.
- Pretraining uses a 27.0M-parameter ViT-S/16 for 300 epochs with eight input frames, multi-scale crops, and two-stage resolution training.
- All methods use frozen encoders with the same gated-attention downstream architecture and identical downstream data and splits for fair comparison.
- MR-JEPA is compared with a text-supervised CMR model pretrained on 293K OMR videos and V-JEPA2, a natural-video JEPA model.
5 Results
MR-JEPA achieved the strongest performance across the five cardiac function regression tasks, while remaining competitive for disease classification. Ablations identified initialization, spatiotemporal masking, and shorter clips as important design choices.
- Cardiac Function Regression: MR-JEPA achieved the best MAE and Pearson correlation on all five cardiac function regression tasks.
- LV Ejection Fraction: 4.79% LV EF MAE and r=0.764 outperformed Shad et al. and V-JEPA2 on the Kaggle dataset.MR-JEPA also showed minimal systematic bias of 0.22%.
- RV Ejection Fraction: 8.67% RV EF MAE and r=0.506 were the best reported results, improving over Shad et al. by 0.68 percentage points.All methods had wider limits of agreement than for LV EF.
- Myocardial Strain: Strain MAEs were 1.87 for GLS, 2.39 for GCS, and 5.33 for GRS, representing 15-27% reductions versus the strongest baseline.Correlations ranged from 0.726 to 0.805, with absolute systematic bias below 0.97.
- Cardiac Disease Detection: MR-JEPA reached macro AUC 0.868 for disease classification, below Shad et al.'s 0.882 but above V-JEPA2's 0.741.It achieved the highest normal-class one-vs-rest AUC at 0.889 and remained within 0.01-0.03 AUC of Shad et al. across disease classes.
- Model Analysis: Removing 2D-CMR initialization or spatiotemporal masking reduced scar-detection accuracy from 0.850 to chance level, while 16-frame clips lowered view-classification accuracy from 0.932 to 0.841.The 8-frame configuration also required fewer optimization epochs, 103 versus 190.
6 Conclusions
MR-JEPA extends LeJEPA to 3D spatiotemporal cardiac MRI and learns representations from multi-sequence data without annotations. It outperformed compared methods on five regression tasks and remained competitive for disease detection, while the study retains important data and evaluation scope limitations.
- Conclusions: MR-JEPA extends LeJEPA to 3D spatiotemporal inputs and uses spatiotemporal masking with 2D-CMR foundation-model initialization.It was pretrained on 160K multi-sequence clips from 10,505 patients without annotations.
- Conclusions: MR-JEPA achieved state-of-the-art performance on LV EF, RV EF, GLS, GCS, and GRS regression tasks against domain-specific and natural-video baselines.
- Conclusions: MR-JEPA remained competitive on disease detection despite less pretraining data and a simpler unsupervised objective.
- Limitations: The study is limited by single-vendor pretraining, center-limited downstream evaluation for most tasks, model-derived RV EF and strain ground truth, and cine-dominated data.The authors identify more mapping data, mapping-specific tasks, and comparisons with task-specific state-of-the-art models as future work.