Source-linked AI summary
CMRVision: A Foundation Model for Cardiac MR Image Analysis
Athira J. Jacob, Puneet Sharma, Daniel Rueckert
TL;DR
CMR analysis must handle complementary information across diverse sequences, views, contrasts, and anatomies, while the effectiveness of transferred natural-image representations and multi-sequence CMR pretraining remains limited or unclear. The paper introduces CMRVision, a DINOv3-style CMR foundation model pretrained on over 36 million multi-center images and evaluates it on segmentation, classification, and zero-shot transfer. CMRVision consistently improves downstream performance, including 9 of 11 best-or-tied segmentation targets, 0.692 zero-shot LGE long-axis Dice, and 0.906 cine-view accuracy.
Problem
CMR analysis spans diverse sequences, contrasts, views, and anatomies, while the effectiveness of natural-image transfer and multi-sequence CMR foundation models remains unclear or limited.
Method
CMRVision is a CMR foundation model trained with DINOv3-style self-supervised learning on diverse multi-sequence data using global, patch-level, regularization, and Gram-anchoring objectives.
Results
CMRVision achieved the best or tied Dice in all 11 segmentation targets, a 0.692 average Dice on unseen LGE long-axis views, and 0.906 average cine-view classification accuracy.
Takeaways & Limitations
CMR-specific pretraining, smaller patch sizes, and patch-level objectives support transferable cardiac representations across multiple sequences and views.
Takeaways & Limitations
The study focuses on DINO-family objectives, evaluates only CMR, does not exhaustively compare fine-tuning strategies beyond LoRA, and includes relatively small task-specific subsets.
Abstract
from arXiv · showhide
Cardiac magnetic resonance (CMR) imaging provides complementary information on cardiac anatomy, function, and tissue characterization across multiple sequences and views. In this work, we investigate foundation model pretraining for 2D CMR and introduce CMRVision, a CMR-specific foundation model trained using DINOv3-style self-supervised learning on a multi-center, multi-sequence cohort of 36 million CMR images. We systematically evaluate architectural and training design choices for domain-specific pretraining. CMRVision is evaluated on two downstream tasks: multi-task segmentation across cine, late gadolinium enhancement (LGE), and mapping sequences, and cine view classification. Our experiments show that CMR-specific pretraining, smaller patch sizes, and patch-level objectives consistently improve downstream performance. Across a multi-task segmentation benchmark, CMRVision achieved the strongest overall performance, outperforming prior natural-image (NI), medical-image, supervised, and CMR foundation model baselines. Improvements were modest but consistent across structures and sequences, with Dice scores ranging from 0.940-0.967 for LV and 0.855-0.905 for myocardium, and reaching 0.929 for RV, 0.920 for LA, and 0.931 for RA. The largest gains were observed for myocardium segmentation in LGE and mapping images. In a zero-shot segmentation task on unseen LGE long-axis views, the model achieved an average Dice score of 0.692, demonstrating cross-view generalization. For cine view classification, CMRVision achieved the highest average accuracy (0.906), compared to prior methods reported in the literature. These results highlight the potential of CMRVision to support robust and generalizable cardiac MRI analysis across multiple sequences and views.
1 Introduction
CMR spans diverse sequences, contrasts, views, and anatomies, creating a need for models that generalize across this variation. CMRVision addresses limited multi-sequence CMR foundation-model research with domain-specific pretraining and reports consistent downstream gains.
- CMR combines cine, late gadolinium enhancement, and mapping sequences that provide complementary clinical information across anatomical views.
- Automated CMR analysis requires models that generalize across diverse contrasts, views, and anatomies.
- Directly transferring natural-image representations to CMR remains uncertain because image appearance, acquisition physics, scanner variability, and subtle anatomy differ substantially.
- Systematic foundation-model investigations spanning multiple CMR sequences remain limited despite recent cine-focused CMR studies.
- CMRVision uses diverse multi-sequence CMR data and DINOv3-style self-supervised learning, achieving the best Dice in 9 of 11 segmentation targets and tying the remaining two.
- CMRVision achieved a Dice score of 0.692 on unseen LGE long-axis segmentation and an average cine-view classification accuracy of 0.906.
2 Related Works
Prior work spans contrastive, generative, predictive, natural-image, medical, and CMR-specific foundation models. Existing CMR analysis studies are often task-specific, motivating broader multi-sequence representation learning.
- Pretraining: Self-supervised pretraining includes contrastive learning, generative reconstruction, and JEPA embedding prediction.
- Pretraining: DINOv3 combines self-distillation with patch-level integration and Gram statistics for task-agnostic image encoding.
- Medical Foundation Models: Medical foundation models include domain-specific systems for chest X-rays and CT, promptable models such as MedSAM, and several CMR-specific models.
- Medical Foundation Models: CMR foundation-model studies have targeted multi-sequence, cine-image, and cine-video applications, while natural-image DINOv3 transfer has also been examined in medical tasks.
- Cardiac Image Analysis: CMR cardiac-image studies typically use small, task-specific datasets and report sequence- and structure-dependent segmentation performance.
- Cardiac Image Analysis: Reported segmentation ranges include 0.90-0.95 Dice for LV in SAX cine, 0.84-0.88 for myocardium in LGE, and 0.85-0.86 for T1/T2 mapping.
3 Methods
CMRVision pretraining combines DINOv3-style global and patch-level objectives with CMR-adapted architectural, augmentation, and training choices. The evaluation covers unified segmentation across multiple sequences and cine view classification, including zero-shot LGE long-axis transfer.
- Pretraining: CMRVision pretrains a ViT-S backbone with DINO, iBOT, KoLeo, and Gram-anchoring objectives.These objectives enforce global consistency, local patch alignment, embedding uniformity, and second-order patch-correlation alignment.
- Pretraining: The method preserves spatial structure with axial rotary positional embeddings across crops.
- CMR Adaptation: CMR adaptation uses 8x8 patches, early Gram anchoring, two-stage resolution training, CMR-specific augmentations, and a minimum crop scale of 0.1.
- Downstream Evaluation: The downstream evaluation includes unified segment-all segmentation across cine SAX, cine LAX, LGE SAX, and mapping SAX without view information.
- Downstream Evaluation: Zero-shot segmentation applies the best model directly to unseen LGE LAX images, testing cross-sequence and cross-view generalization.
- Representation Analysis: Figure 2 visualizes foreground-masked feature spaces for natural-image and CMR-pretrained encoders across cine and mapping examples.
4 Data & Experiments
Pretraining uses over 36 million images from 27,000 studies, while downstream experiments combine multi-center segmentation and classification benchmarks. Evaluation compares pretraining strategies and foundation-model or supervised baselines, with explicit attention to class imbalance and data overlap.
- Data: Pretraining uses over 36 million images from 27,000 studies across multiple scanners, sequences, views, and patient populations.The data come from three sources and include both 1.5 T and 3 T scanners.
- Segmentation Data: The segmentation benchmark combines four datasets totaling 1,873 patients and 11,022 images, with patient-level splits.
- Segmentation Data: Zero-shot evaluation uses 22 patients and 57 LGE LAX images from a fifth dataset.
- Evaluation Scope: Some Center 1 and Center 2 patients may overlap with pretraining, although downstream labels were not used during pretraining.
- Reporting: Segmentation results are reported as test-set Dice in pretraining-strategy and general/CMR-foundation-model comparison blocks.
- Evaluation Scope: The unified benchmark includes view-dependent structures with fewer samples, including 150 RV images and 346 atrial images versus 7,478 LV and myocardium images.
- Classification Data: Classification uses 2,036 images from 147 patients across SAX, long-axis, aorta, and other views, independently of pretraining.
- Baselines: Comparisons include NI, CMR-pretrained, LoRA-adapted, supervised, SAM-based, and scratch-trained encoder or segmentation baselines.
5 Results
CMRVision consistently outperformed or matched competing approaches across segmentation, while gains for cine view classification were comparatively small. Zero-shot evaluation showed transfer to unseen LGE long-axis views, with 2CH remaining more difficult than 4CH.
- Pretraining Strategy Comparison: CMR-specific pretraining improved average Dice by 1.7 percentage points over natural-image-pretrained DINOv3 (ViT-S/16).Reducing patch size from 16 to 8 produced further gains across most structures and sequences.
- Pretraining Strategy Comparison: Performance differences among CMR-pretrained models were comparatively small for view classification.The result suggests that global anatomical cues were already captured by existing self-supervised representations, unlike dense segmentation features.
- CMRVision achieved the best segmentation performance in 9 of 11 tasks and tied for the remaining two.
- Comparison with Baselines: CMRVision achieved an average accuracy of 0.906 for cine view classification, comparable to the CMR-pretrained DINOv2 baseline at 0.908.Its stronger performance on the less frequent “Others” category drove the superior overall result.
- Zero-shot evaluation: Zero-shot segmentation on unseen LGE long-axis myocardium views reached an average Dice score of 0.692.Scores were 0.612 on 2CH and 0.794 on 4CH; 2CH remained difficult when chambers were incompletely captured or anatomically variable.
6 Conclusions
The study introduces CMRVision, a DINOv3-style cardiac MRI foundation model whose domain-specific pretraining supports transfer across segmentation, classification, and zero-shot tasks. The authors identify 2D-only modeling, limited scope of objectives and fine-tuning comparisons, possible patient overlap, and small task-specific subsets as limitations.
- CMRVision uses DINOv3-style self-supervised pretraining to learn transferable representations for cardiac MRI.
- Patch-level objectives and smaller patch sizes improved performance, highlighting their value for capturing fine-grained cardiac features.
- The evaluation is restricted to CMR and DINO-family objectives, without an exhaustive comparison of fine-tuning strategies beyond LoRA.
- CMRVision operates on 2D images and does not explicitly model volumetric or temporal information.The authors identify extension to 3D and spatiotemporal inputs as future work.
- Some task-specific subsets are relatively small, and patient-level overlap between pretraining and downstream data from Centers 1 and 2 may exist.No downstream labels were used during pretraining.