Source-linked AI summary
USFM: A Universal Ultrasound Foundation Model Generalized to Tasks and Organs towards Label Efficient Image Analysis
Jing Jiao, Jin Zhou, Xiaokang Li, Menghua Xia, Yi Huang, Lihong Huang, Na Wang, Xiaofan Zhang, Shichong Zhou, Yuanyuan Wang, Yi Guo
TL;DR
US image analysis lacks generality across organs and tasks, while US foundation-model development faces data, image-quality, and feature challenges. USFM uses a large, organ-balanced database and spatial-frequency masked pre-training; the paper reports performance across common tasks and organs, including with only 20% annotated data.
Problem
US analysis methods have inadequate generality across organs and tasks, and US foundation models face database insufficiency, low image quality, and ineffective features.
Source: §1 · pp. 1–2Method
Results
USFM shows strong performance across common US tasks and diverse organs, with performance ahead of existing methods reported even with only 20% annotated data.
Source: §6 · p. 14, §7 · p. 15Takeaways & Limitations
The authors conclude USFM can serve as a plug-and-play module to support rapid expansion of automatic US analysis across tasks and organs.
Source: §7 · p. 15The study primarily focuses on 2D US images; directly establishing a 3D US foundation model remains a future direction.
Source: §6 · p. 15
Abstract
from arXiv · showhide
Inadequate generality across different organs and tasks constrains the application of ultrasound (US) image analysis methods in smart healthcare. Building a universal US foundation model holds the potential to address these issues. Nevertheless, the development of such foundational models encounters intrinsic challenges in US analysis, i.e., insufficient databases, low quality, and ineffective features. In this paper, we present a universal US foundation model, named USFM, generalized to diverse tasks and organs towards label efficient US image analysis. First, a large-scale Multi-organ, Multi-center, and Multi-device US database was built, comprehensively containing over two million US images. Organ-balanced sampling was employed for unbiased learning. Then, USFM is self-supervised pre-trained on the sufficient US database. To extract the effective features from low-quality US images, we proposed a spatial-frequency dual masked image modeling method. A productive spatial noise addition-recovery approach was designed to learn meaningful US information robustly, while a novel frequency band-stop masking learning approach was also employed to extract complex, implicit grayscale distribution and textural variations. Extensive experiments were conducted on the various tasks of segmentation, classification, and image enhancement from diverse organs and diseases. Comparisons with representative US image analysis models illustrate the universality and effectiveness of USFM. The label efficiency experiments suggest the USFM obtains robust performance with only 20% annotation, laying the groundwork for the rapid development of US models in clinical practices.
1. Introduction
US foundation models face data imbalance, noisy images, and difficulty extracting generalizable features, while existing US analysis often depends on organ- or disease-specific annotated datasets. USFM combines a large multi-organ database, balanced sampling, and spatial-frequency masked pre-training to target these challenges.
1. Introduction
- US image analysis often relies on fully annotated organ- or disease-specific datasets, motivating a label-efficient model adaptable across organs and tasks.US images also differ from natural images in imaging principles, motivating a US-specific foundation model and evaluation across downstream tasks.Source: §1 · pp. 1–2
- US foundation models must address database scarcity and organ imbalance, low image quality, and complex features that include spatial and frequency information.US images can have low resolution, contrast, and signal-to-noise ratio, while useful features may be implicit.Source: §1 · p. 2
- USFM combines spatial and frequency masking with organ-balanced sampling on a database of over two million multi-organ US images.The 3M-US database spans multiple centers and devices; the spatial-frequency method is used for self-supervised pre-training.Source: §1 · p. 2
- USFM is reported to generalize across organs and tasks, and label-efficiency experiments show practical performance with minimal annotated data.Ablations also support the effectiveness of organ-balanced sampling and spatial-frequency dual masking.Source: §1 · pp. 2–3
2. Related work
Visual foundation models use large-scale pre-training to support multiple tasks, while medical imaging models are often tailored to their modality. The related work positions a US-specific foundation model as a response to these modality differences and the complexity of medical-image features.
2.1. Visual foundation model
- Visual foundation models use annotated task-specific or self-supervised task-agnostic pre-training to support broad visual tasks; examples include SAM [Kirillov et al., 2023] and MAE.The text notes that extensive annotation can limit task-specific model development, while task-agnostic methods learn from unannotated data.Source: §2.1 · p. 3
2.2. Foundational models in medical imaging
- Medical foundation models are commonly modality-specific because imaging principles and implicit mapping relationships vary; CT, retinal, and endoscopy models are cited as examples.The cited models include MIS-FM (Wang et al., 2023a), RETFound, and Endo-FM (Wang et al., 2023b).Source: §2.2 · p. 3
3. Materials and methods
USFM is pretrained self-supervised on the 3M-US database using organ-balanced sampling and spatial-frequency dual masking, then adapted to downstream tasks as either a trainable backbone or a frozen feature extractor.
3.1. The 3M-US database
- 3M-US contains 2,187,915 ultrasound images across 12 organs and uses weighted organ-balanced resampling to address organ imbalance.Minority-organ images are resampled more frequently.Source: §3.1 · pp. 3–4
3.2. Spatial-Frequency dual masked image modeling
- USFM uses self-supervised masked image modeling with complementary spatial mean masking and frequency band-stop masking.Organ-balanced images are masked, encoded, and reconstructed during pretraining.Source: §3 · p. 3, §3.2 · p. 4, §3.2.3 · p. 6
3.2.1. Spatial mean masked learning
- Spatial mean masking replaces randomly selected patches with the image-wide mean, simulating ultrasound noise while preserving grayscale continuity.Reconstruction targets spatial grayscale and structural information, including boundaries, positions, and regional activations.Source: §3.2.1 · pp. 4–5
3.2.2. Frequency band-stop masked learning
- Random band-stop filters mask frequencies from low to high for reconstruction, targeting structural and textural information in complementary spectrum regions.The final frequency mask combines several filters; low frequencies represent morphology and deformation, while high frequencies capture texture.Source: §3.2.2 · p. 5, Figure 4 · p. 5
3.2.3. Optimization of self-supervised pre-training
- Pretraining combines pixel-level L1 spatial reconstruction with focal frequency loss, which upweights the currently most-lost components (α = 1).The frequency-domain loss is intended to balance reconstruction across relevant frequency components.Source: §3.2.3 · p. 6
3.3. USFM adaptation in downstream tasks
- For downstream tasks, USFM can be fine-tuned as a backbone with task heads or frozen as a feature extractor within task-specific networks.Backbone optimization uses smaller weights in shallower layers and larger weights in deeper layers.Source: §3.3 · p. 6, Figure 5 · p. 6
4. Experiments
The experiments evaluate USFM pre-training and its downstream use across organs and tasks, including segmentation, classification, enhancement, label efficiency, and ablations.
4.1.1. Visualization of USFM pre-training
- Pre-training visualizations use masked-image reconstructions and UMAP projections of images from 12 organs to inspect recovery and feature distributions.The UMAP analysis randomly selects 100 images from each organ; the reconstruction examples span the 3M-US database.Source: §4.1.1 · pp. 6–7
4.1.2. Comparison on diverse downstream tasks and organs
- Downstream evaluation covers segmentation, classification, and image enhancement across multiple organ datasets.Segmentation includes single-object Neck, Thyroid, and Breast tasks and a multi-object Fetus task; classification covers Breast, Abdomen, and Fetus, while enhancement uses Thyroid, Carotid, Liver, and Kidney images.Source: Table 1 · p. 7, §4.1.2 · p. 7
- Label efficiency is tested at 20%, 40%, 60%, and 80% of training data, with identical sampling indices across methods and evaluation on the full test set.The experiment covers segmentation and classification tasks, which the paper identifies as having high annotation requirements.Source: §4.1.3 · p. 8
4.1.4. Ablation study
- Ablations compare USFM with versions lacking organ-balanced sampling or frequency masking to assess each component's role in pre-training.USFM-woS omits organ-balanced sampling, while USFM-woF omits frequency masking.Source: §4.1.4 · p. 8
4.2. Implementation details
- Pre-training uses 224 × 224 images, 0.4 spatial masking probability, seven frequency band-pass filters, and a preserved 10 × 10 central spectrum area.The spatial and frequency reconstruction losses are balanced at 0.4; downstream images are resized to 256x256 and models are trained for 200 epochs.Source: §4.2 · p. 8
5. Results
USFM performs strongly across the reported organ-specific tasks, and the experiments also examine its pre-training features, annotation efficiency, and component ablations. Its reported results include gains over comparison methods and performance with only 20% of training labels.
5.1. Effectiveness of USFM pre-training
- Reconstructions recover masked structures in both domains, while UMAP shows organ clustering and non-overlapping feature representations within each organ.The authors report that USFM recovers features across majority and minority organs and that fetal organs cluster near one another.Source: §5.1 · p. 8, Figure 6 · p. 9, Figure 7 · p. 9
5.2.1. Segmentation on various organs
- USFM improved DSC over UperNet, with the same network structure but no USFM weight, from 76.6% to 80.6% on Neck, 71.1% to 85.9% on Thyroid, 77.9% to 85.8% on Breast, and 66.0% to 84.3% on Fetus.Source: §5.2.1 · p. 9
5.2.2. Classification of diseases and organs
- USFM outperforms the comparison methods across the reported classification metrics for Breast tumors, Abdomen structures, and Fetus planes.The reported ACC values are 87.7%, 93.%, and 93.6%, respectively; the Abdomen value is printed as 93.% in the passage.Source: §5.2.2 · p. 10, Table 3 · p. 11
5.2.3. US image enhancement
- SSIM (%) for USFM versus CycleGAN was 74.3% versus 72.0% on Thyroid, 75.1% versus 73.9% on Carotid, 76.6% versus 75.8% on Liver, and 75.5% versus 75.3% on Kidney.USFM also had higher LNCC than CycleGAN on all four organs.Source: §5.2.3 · pp. 11–12, Table 4 · p. 12
5.3. Label efficiency
- USFM led label efficiency on segmentation and classification; at 20% labels, segmentation DSCs were 72.1% on Neck, 76.0% on Thyroid, 72.4% on Breast, and 74.2% on Fetus.The paper reports sufficient performance at this label ratio and says USFM consistently achieved the highest label efficiency among the compared methods.Source: §5.3 · p. 12, Figure 10 · p. 13
5.4. Ablation analysis
- Removing organ-balanced sampling or frequency masking reduces performance; USFM performs best across organs and tasks in the ablation comparison.The reported drops include -3.2% and -1.4% for USFM-woS segmentation on Breast and Thyroid, and -3.9% classification on Breast tumors for USFM-woF.Source: §5.4 · pp. 13–14, Figure 11 · p. 14
6. Discussion
USFM is reported to generalize across organs and tasks, including with limited annotations, while its discussion attributes this performance to diverse pretraining data and spatial-frequency learning. The authors note that the study primarily focuses on 2D images and that larger architectures remain unexplored because of computational limits.
6. Discussion
- USFM surpasses representative methods across eight organs and three common tasks, and with 20% annotation it leads full-data methods on specified segmentation tasks.The reported comparisons include UperNet on Neck and Thyroid segmentation and Segformer on Thyroid and Breast segmentation.Source: §6 · p. 14
- The 3M-US database addresses limited training-data diversity with over two million images across organs, centers, and devices; USFM outperforms UperNet without 3M-US pretraining across all downstream tasks and organs.The discussion says natural-image SimMIM underperforms USFM, especially on classification and enhancement tasks requiring high-level US features.Source: §6 · p. 14
- Organ-balanced sampling helps USFM learn features across all 12 organs; removing it biases features toward Breast and degrades performance across tasks and organs.The degradation is particularly marked in classification and enhancement, and the organ-universal model also outperforms the Breast-biased variant on breast tumor segmentation and classification.Source: §6 · pp. 14–15
- Spatial mean masking trains USFM to recover images from noise-added inputs, and the authors report robust feature extraction on noisy images and strong label efficiency.They report higher mean and lower standard deviation on all metrics compared with other methods.Source: §6 · p. 15
- Frequency band-stop masking captures frequency information that spatial-only USFM-woF misses, with reported gains across tasks and organs, especially classification and enhancement.The discussion links the spatial-frequency combination to superior performance and label efficiency.Source: §6 · p. 15
- The study primarily focuses on 2D US, while larger architectures remain unexplored because of computational limits; detection, denoising, and generation are named as potential future tasks.The authors say a 3D foundation model could better use relationships between images.Source: §6 · p. 15
7. Conclusion
The conclusion presents USFM as a plug-and-play US foundation model with organ versatility, task adaptability, and label efficiency. Experiments report performance across segmentation, classification, and enhancement, with label-efficiency results supporting its potential to accelerate US model development.
7. Conclusion
- USFM is presented as a plug-and-play foundation model with high organ versatility, task adaptability, and label efficiency across segmentation, classification, and enhancement.The authors state that its label-efficiency results confirm its potential to accelerate US model development and expand automated US analysis applications.Source: §7 · p. 15