Source-linked AI summary
DINO-Med: A Unified Patch-Based Adaptation Framework for Multi-Modal Medical Image Analysis Applied to Liver Fibrosis Staging
Boya Wang, Ruizhe Li, Chao Chen, Xin Chen
TL;DR
The paper tackles the domain gap between natural-image foundation models and multimodal medical imaging by introducing a unified patch-based DINOv3 adaptation framework for liver fibrosis staging. It combines standardized multimodal preprocessing, frozen patch representations, and subject-level aggregation, and reports the strongest baseline comparison with peak accuracies of 78.4% for S1 and 75.8% for S4.
Problem
Effective adaptation of natural-image foundation models to multimodal medical imaging remains underexplored despite the need for robust medical image representations.
Method
The framework combines training-free registration, automated localization, mask-guided patch extraction, frozen DINOv3 multimodal descriptors, and hierarchical patch-to-subject aggregation.
Results
78.4% accuracy for S1 and 75.8% for S4 were achieved, while DINOv3 consistently outperformed radiomics, ResNet, and SAM-Med2D baselines.
Takeaways & Limitations
Frozen vision foundation models provide a powerful, scalable diagnostic baseline for multimodal liver fibrosis staging without domain-specific fine-tuning.
Takeaways & Limitations
The DINOv3 encoder requires 3-channel RGB input, making the mapping of multimodal MRI sequences into pseudo-RGB a critical design choice.
Abstract
from arXiv · showhide
Adapting natural-image foundation models like DINOv3 to multi-modal medical imaging is challenging due to the significant domain gap between natural color images and multi-channel medical scans. We present a unified, patch-based framework that processes raw multimodal imaging through training-free registration, automated localization, and mask-filtered patch extraction. This architecture culminates in a hierarchical strategy that aggregates patch-level insights into subject-level diagnostics. Using liver fibrosis staging as a case study, we evaluate four patch-level feature representations: handcrafted Radiomics features, learned ResNet features, pre-trained foundation model SAM-Med2D features, and frozen DINOv3 features. To ensure a controlled comparison, all models utilize the same lightweight MLP head and are evaluated across both rigid and deformable registration settings. Our training protocol focuses on mild fibrosis (S1) and cirrhosis (S4) classes only, enabling a single classifier to address both substantial fibrosis detection and cirrhosis staging. Evaluated via 10 random train (90%)/ test (10%) splits on 360 subjects from the CARE 2025 Liver Track 4 cohort, our DINOv3-based framework significantly outperforms all baselines, achieving the best classification accuracy of 78.4% for S1 and 75.8% for S4.
1 Introduction
The paper addresses the persistent domain gap between natural-image foundation models and multimodal medical imaging by proposing a unified DINOv3 adaptation framework and evaluating it on liver fibrosis staging.
- The domain gap in intensity distributions, spatial resolution, and dimensionality limits direct use of natural-image foundation models on multimodal medical images.
- The proposed framework adapts DINOv3 for multimodal medical imaging and investigates its use for subject-level classification.
- Liver fibrosis staging provides the clinical testbed because biopsy is invasive and prone to sampling errors, while multiparametric MRI offers a non-invasive assessment option.
- The pipeline processes raw unaligned multimodal images through registration, localization, mask-guided patch extraction, and hierarchical patch-to-subject aggregation.
- Frozen DINOv3 consistently outperforms radiomics, ResNet, and SAM-Med2D across rigid and deformable registration settings.
2 Related Works
Related work spans radiomics, deep learning, and visual foundation models for medical imaging, while effective adaptation of natural-image models to multimodal medical data remains underexplored.
- Deep learning and radiomics have played central roles in medical image analysis by improving diagnostic accuracy and interpretability.
- Visual foundation models provide transferable representations through large-scale pretraining, reducing dependence on task-specific annotation.
- Medical imaging foundation-model research includes natural-image backbones such as DINO and medical-domain models such as SAM-Med2D and SAM-Med3D.
- Effective adaptation of natural-image foundation models to multimodal medical imaging remains underexplored.
- Radiomics represents tissue using handcrafted shape, intensity, and texture features, but relies heavily on precise segmentation and manual intervention.
3 Methods
The method standardizes multimodal MRI, extracts frozen DINOv3 patch representations, classifies valid ROI patches, and aggregates their predictions into subject-level fibrosis scores.
- 3.1 Overview: The framework coregisters raw 3D volumes, propagates a reference mask, crops a mask-defined ROI, and resizes axial slices to standardized K × 224 × 224 inputs.
- 3.3 Feature Extraction and Patch-level Classification: Frozen DINOv3 independently encodes each modality, after which valid patch features are concatenated across modalities and classified by a lightweight MLP.
- 3.3 Feature Extraction and Patch-level Classification: The mask filter partitions each slice into 196 nonoverlapping 16×16 patches and retains patches whose mask-to-patch area ratio exceeds 0.7.
- 3.4 Patch-to-Subject Aggregation: The binary patch classifier uses only S1 and S4 classes, reflecting the hypothesis that fibrosis is spatially heterogeneous, especially in intermediate stages.
- 3.4 Patch-to-Subject Aggregation: Patch predictions are aggregated by computing the proportion of Stage 4-like patches across all valid liver ROI patches.
- 3.4 Patch-to-Subject Aggregation: Subject-level scores are mapped to fibrosis likelihoods using two piecewise linear functions with thresholds τ12 and τ34 optimized on full-range validation data.
4 Experiments and Results
Experiments compare four patch-level representations under rigid and deformable registration using nested evaluation on 360 subjects. DINOv3 consistently outperforms the radiomics, ResNet, and SAM-Med2D baselines, with the strongest gains in cirrhosis classification.
- Experimental settings: The cohort comprised multi-parametric MRI from the CARE 2025 Liver Track 4 dataset, with preprocessing addressing unaligned volumes and limited liver-mask annotations.Sequences included T1, T2, DWI, and dynamic contrast phases; GED4 was available for every subject.
- Experimental settings: The evaluation used 10 repeated 90%/10% splits, nested four-fold validation, and subject-level AUC and accuracy for S1 and S4 classification.The protocol trained 40 models across the repeated outer and inner splits, with thresholds calibrated on validation folds.
- Results: DINOv3 consistently outperformed all baselines across rigid and deformable registration, with rigid-registration Mean AUC 0.843 and Mean ACC 0.766.Under rigid registration, SAM-Med2D, ResNet, and Radiomics reached Mean AUC/ACC values of 0.780/0.728, 0.753/0.708, and 0.582/0.644, respectively.
- Results: DINOv3 reached 0.784 accuracy for S1 with deformable registration, while its S4 accuracy was approximately 76% in both registration settings.The S4 gap was around 4% over SAM-Med2D, 8% over ResNet, and 20% over Radiomics; the S1 statistical advantage was less significant.
- Results: Registration effects varied by method: Radiomics favored rigid alignment, ResNet favored deformable alignment, and DINOv3 and SAM-Med2D were comparatively registration-insensitive.The authors associate this pattern with handcrafted-feature sensitivity, learned alignment benefits, and pretrained-model immunity to local geometric variation.
- Results: DINOv3 achieved a threshold margin of 0.48 in both registration settings, exceeding SAM-Med2D, ResNet, and Radiomics margins.Radiomics margins fell below 0.05, while ResNet showed a narrow gap and SAM-Med2D remained less discriminative than DINOv3.
5 Conclusions
The study presents a unified patch-based DINOv3 framework for liver fibrosis staging and reports strong performance without domain-specific fine-tuning. It identifies foundation-model robustness across registration settings while outlining extensions to intermediate stages and other organs.
- Conclusion: The framework combines training-free registration with hierarchical patch aggregation and outperformed radiomics, ResNet, and SAM-Med2D, reaching 78.4% S1 and 75.8% S4 accuracy.The conclusion reports these as peak accuracies for the DINOv3-based liver fibrosis staging pipeline.
- Conclusion: DINOv3 and SAM-Med2D produced comparable results across rigid and deformable registration, indicating greater immunity to local geometric variations.The authors describe frozen vision foundation models as a scalable diagnostic baseline without domain-specific fine-tuning.
- Future work: Future work targets lightweight adapters, additional multi-parametric MRI organs, cross-sequence fusion, and classification of intermediate S2 and S3 fibrosis stages.The planned extensions include LoRA, cardiac and prostate imaging, early- and late-fusion strategies, and expert-demanding intermediate-stage classification.