Source-linked AI summary
UltraPIPS: Improving model perception in B-mode ultrasound with foundation models
Tal Grutman, Tali Ilovitsh
TL;DR
The paper asks whether LPIPS backbones pretrained on natural or general medical images are suitable for B-mode ultrasound, whose speckle and acoustic statistics are distinctive. It compares backbone groups across classification, segmentation, and reconstruction, finding that ultrasound-exposed models had the strongest supervised-task correlations and balanced reconstruction quality with realistic texture, while releasing UltraPIPS.
Problem
LPIPS backbone choice is unsettled in medical imaging, and B-mode ultrasound has distinctive speckle and acoustic-specific statistics that may require different perceptual features.
Method
The study compares natural-image, general-medical, and ultrasound foundation-model backbones by evaluating LPIPS relationships with classification, segmentation, and image reconstruction.
Results
Ultrasound-exposed backbones achieved the highest correlations with supervised downstream performance and balanced reconstruction quality with realistic texture without distorting anatomic details.
Takeaways & Limitations
Backbone selection is an important implementation choice for ultrasound LPIPS, and exposure to ultrasound images may improve model suitability for this role.
Takeaways & Limitations
Further evaluation is limited by the lack of realistic augmentations outside echocardiography.
Abstract
from arXiv · showhide
In medical imaging, it is common to use learned perceptual image patch similarity (LPIPS) to compare images semantically in feature space. Although backbones pretrained on natural images are widely used for LPIPS computation, B-mode ultrasound images possess distinct speckle patterns and acoustic-specific image statistics that are fundamentally different from natural images and even from other images in radiology. Consequently, we propose that domain-specific models are needed to measure perceptual similarity in ultrasound data, a finding which is not necessarily the case for other imaging modalities. We compare LPIPS metrics across downstream tasks like classification, segmentation and reconstruction using natural image, medical generalist and ultrasound backbone models and show that selection of LPIPS backbone is a non-trivial design choice. In particular, the ultrasound backbone models were more correlated with downstream performance of supervised models than classical and natural image models, and optimization of the LPIPS loss with an ultrasound backbone achieved a strong balance between reconstruction quality and realism. Our code is available at https://github.com/talg2324/UltraPIPS and introduces the UltraPIPS library, a set of LPIPS metrics based on the open-source foundation models analyzed in this paper.
1 Introduction
LPIPS measures perceptual differences through pretrained encoder features, but its backbone choice remains unsettled in medical imaging. Because B-mode ultrasound has distinctive speckle characteristics and image statistics, this work evaluates ultrasound-specific alternatives across downstream tasks and releases UltraPIPS.
- LPIPS and backbone choice: LPIPS measures distances between image patches in a pretrained encoder’s feature space, incorporating task-grounded context beyond pixel-wise metrics.Its learned features provide a middle ground between human perception and deep-model performance.
- LPIPS and backbone choice: Backbone importance for LPIPS is unresolved in medical imaging, with prior studies reporting marginal, worse, or substantially more relevant in-domain features than ImageNet features.The mixed findings motivate examining domain-specific backbones rather than assuming one pretraining domain is universally preferable.
- Ultrasound-specific motivation: Ultrasound is especially challenging because its speckle characteristics and image statistics differ from natural images and other medical modalities.Pixel-wise MSE can quickly lose speckle patterns that carry diagnostic meaning, whereas deep backbones can accommodate them.
- Study objective: The study compares natural-image, general-medical, and ultrasound foundation-model backbones by relating LPIPS distances to classification, segmentation, and reconstruction.It tests feasible ultrasound perceptual metrics using open-source pretrained models and datasets.
- Study objective: The authors introduce UltraPIPS, an open-source library containing multiple backbones and tests for perceptual ultrasound metrics.The library accompanies the comparative evaluation of LPIPS backbones in B-mode ultrasound.
2 Experiments
The experiments compare LPIPS backbones from cross-domain medical, natural-image, and ultrasound foundation models across supervised perception and INR reconstruction. Results assess both correlation with downstream performance and the trade-off between reconstruction fidelity and ultrasound texture realism.
- Backbone candidates: The candidate backbones span general medical models such as RadImageNet and MedSAM, plus ultrasound models USFM, TUSA, and Ultrasound-CLIP.USFM uses masked auto-encoding, TUSA learns cross-anatomy texture kernels, and Ultrasound-CLIP learns global semantics from image-report pairs.
- Metric construction: LPIPS distances are computed from channel-normalized backbone feature vectors, with ViT and SwinViT variants using final-layer features.The LPIPS loss is the L2 distance between the normalized feature vectors of two images.
- Experimental setup: EchoGains creates realistic echocardiogram degradation through geometric warping, transducer-field masking, and diffusion-based inpainting.The resulting augmentations support evaluation of classifier confidence and segmentation performance under image perturbations.
- Supervised tasks: Ultrasound models achieved the highest correlations with classifier confidence, while MedSAM and USFM were among the least correlated models and trailed ImageNet ViT.BiomedCLIP and UltrasoundCLIP outperformed CLIP, indicating that ultrasound exposure may improve correlation, although performance varied substantially across ultrasound backbones.
- Supervised tasks: Segmentation correlations improved for rotation but failed for depth deformations because the evaluated datasets were within UltraSam’s training distribution.TUSA and Ultrasound-CLIP ranked highest overall, followed by CNN models and BiomedCLIP; MedSAM correlated weakly despite being a segmentation model.
- INR reconstruction: Ultrasound backbones balanced reconstruction quality and realistic texture, whereas L2 gave the best SSIM and HFEN but blurred speckle and generalist LPIPS models hallucinated details.USFM and Ultrasound-CLIP produced highly realistic speckle detail comparable to the SSIM reconstruction baseline.
3 Conclusion
The study finds that ultrasound exposure improves the suitability of LPIPS backbones for B-mode ultrasound, while noting that current evidence remains limited in scope.
- Ultrasound-exposed ViT backbones achieved the highest correlations with supervised-task performance, while CNN backbones were also highly correlated.The conclusion reports this pattern across supervised tasks.
- Ultrasound backbones balanced reconstruction quality and realistic texture, and did not distort anatomic details in qualitative inspection.
- Correlation with model performance is only a proxy for visual perception, so the relationship between human perception and LPIPS backbones remains to be quantified.
- Testing more datasets and downstream models is limited by the lack of realistic augmentations outside echocardiography.
Supplementary Material
The supplementary material documents the realistic ultrasound perturbations and evaluation tables used to relate perceptual distances to classification and segmentation performance.
- Supplementary Figures: EchoGains uses a pretrained diffusion model to induce realistic, controlled perturbations in B-mode ultrasound images.
- Supplementary Figures: Increasing rotation makes UltraSam more error prone while distance metrics capture increasing perturbation.
- Supplementary Tables: Table S1 reports Spearman correlations between EchoPrime view-classifier confidence and candidate distance metrics across increasingly strong augmentations.
- Supplementary Tables: Table S2 reports Spearman correlations between UltraSam dice scores and candidate distance metrics across increasingly strong augmentations.