Source-linked AI summary
Medical Foundation Model Features as Perceptual Loss for Brain MRI Contrast Dose Simulation
Changsheng Fang, Dayang Wang, T. Campbell Arnold, Enhao Gong, Srivathsa Pasumarthi
TL;DR
Natural-image perceptual features may not align well with MRI anatomy and enhancement, motivating a test of medical foundation model features for brain MRI dose simulation. The study benchmarks frozen extractors, selects RadImageNet, and replaces only VGG16 in the existing framework. RadImageNet yields modest endpoint-metric changes but clearer visual suppression of residual enhancement and closer low-dose appearance, while clinical relevance remains unvalidated.
Problem
Natural-image VGG16 and ResNet50 features may provide a weak perceptual proxy for MRI anatomy, contrast enhancement, and acquisition variability.
Method
The study benchmarks five frozen feature extractors across four public medical imaging tasks, then replaces only VGG16 with the selected RadImageNet representation while keeping the Gformer objective fixed.
Results
RadImageNet provides the strongest Stage I representation and produces modest endpoint-metric changes but clearer visual suppression of residual enhancement and low-dose appearance in Stage II.
Takeaways & Limitations
Radiology-pretrained features are supported as a practical perceptual feature space for MRI dose simulation within the tested fixed Gformer setting.
Takeaways & Limitations
The evidence is limited to one VGG16-to-RadImageNet replacement in a fixed Gformer objective, and clinical equivalence requires larger cohorts and reader-centered validation.
Abstract
from arXiv · showhide
Perceptual losses are widely used in medical image synthesis because they encourage agreement in high-level structure beyond voxel-wise intensity similarity. In practice, most perceptual losses are still computed with natural-image backbones such as VGG16 or ResNet50, even when the target domain is magnetic resonance imaging (MRI). This mismatch may weaken supervision for anatomy, contrast enhancement, and acquisition variability. We test whether medical foundation model features provide a more suitable perceptual loss for brain MRI contrast dose simulation. The study has two stages. First, we compare RadImageNet, SegVol, and BrainIAC with ImageNet-pretrained VGG16 and ResNet50 as frozen feature extractors on four public medical imaging benchmarks: thyroid ultrasound, breast ultrasound, anterior cruciate ligament knee MRI, and meniscus knee MRI. RadImageNet achieves the lowest mean rank across the Stage I representation suite and is selected as $φ^\star$. Second, we replace only the VGG16 feature extractor in an existing iterative brain MRI dose simulation framework with $φ^\star$. The generator, reconstruction loss, adversarial loss, auxiliary losses, optimization schedule, and loss weights are kept unchanged. Standard metrics change modestly, with PSNR increasing from 41.63 to 41.74, SSIM from 0.9739 to 0.9754, RMSE decreasing from 0.1384 to 0.1369, and residual-uptake CNR from 0.0085 to 0.0082. The visual results show the main effect: RadImageNet reduces residual enhancement in marked structures, follows a more faithful dose-reduction trajectory, and remains close to the acquired 10% low-dose target. These results support domain-aligned radiology features as a practical perceptual feature space for MRI dose simulation, while leaving clinical equivalence and larger-cohort validation as future work.
1 Introduction
The study asks whether medical foundation model features can better supervise brain MRI contrast dose simulation than natural-image perceptual features. It uses a two-stage design to benchmark candidate representations and then isolate the effect of replacing VGG16 in an existing simulator.
- Contrast dose reduction seeks to preserve diagnostic information while reducing gadolinium exposure and its associated patient, cost, renal, and environmental concerns.
- The Gformer framework simulates arbitrary enhancement levels from pre-contrast, post-contrast, and sparse low-dose observations by learning a gradual reduction toward 10% low-dose images.
- ImageNet-pretrained VGG16 and ResNet50 may be poorly aligned with grayscale MRI, volumetric anatomy, gadolinium uptake, and disease morphology.
- RadImageNet, SegVol, and BrainIAC offer medical representations learned from radiology, volumetric CT, or brain MRI, but their use as frozen perceptual extractors had not been systematically evaluated.
- The study benchmarks candidate extractors on four public medical imaging tasks before inserting the selected representation into the dose simulation framework.
- Only the VGG16 perceptual feature extractor is replaced while the remaining framework components and loss weights stay unchanged, isolating the representation choice.
2 Related Work
Related work frames the study as a narrow test of perceptual feature choice in MRI dose simulation. It connects dose simulation, natural-image perceptual losses, and medical foundation models with different imaging priors.
- Brain MRI dose simulation generates arbitrary enhancement levels from pre-contrast, post-contrast, and sparse low-dose observations.
- The study keeps the prior generator, reconstruction and adversarial losses, and training weights fixed while replacing only the VGG16 feature extractor.
- Perceptual loss compares predicted and reference images in pretrained-network activation space to penalize structural discrepancies underweighted by voxel-wise or structural losses.
- VGG16 and ResNet50 use natural-image features that may be weak proxies for MRI fidelity because brain-MRI information depends on subtle intensity and spatial relationships.
- RadImageNet provides a radiology-pretrained 2D alternative, while SegVol supplies a 3D CT-oriented representation with volumetric anatomical priors.
- BrainIAC is a self-supervised foundation model trained for human brain MRI analysis and is the most domain-specific candidate for this application.
3 Method
The method benchmarks frozen feature extractors across medical imaging tasks, then replaces only the perceptual branch in an iterative dose simulator with the selected medical feature extractor.
- Candidate Feature Extractors: Five pretrained backbones—VGG16, ResNet50, RadImageNet, SegVol, and BrainIAC—are compared as candidate feature extractors.VGG16 and ResNet50 are ImageNet-pretrained, while the remaining candidates are medical models with 2D, volumetric, or brain-MRI-specific pretraining.
- Feature Extraction: Feature layers are chosen from intermediate and high-level blocks to balance local texture, anatomical boundaries, and global structure.For 2D inputs, single-channel images are normalized and replicated as needed; volumetric inputs are resampled to checkpoint-specific resolutions.
- Stage I Representation Benchmark: Stage I evaluates frozen features on thyroid and breast ultrasound classification, ACL tear detection, and meniscus tear detection using linear probing, retrieval consistency, and intensity-perturbation robustness.The selection score averages ranks across benchmark datasets and selected metrics, excluding intensity-perturbation cosine similarity.
- Stage II Dose Simulation: The iterative simulator repeatedly applies generator F to synthesize lower-enhancement images, using intermediate and final reconstruction terms plus adversarial and perceptual losses.The final low-dose prediction receives adversarial and perceptual supervision, while intermediate soft labels provide weak supervision for gradual dose reduction.
- Stage II Dose Simulation: Stage II replaces only the VGG16 perceptual extractor with the selected RadImageNet extractor while keeping the generator, weights, preprocessing, optimizer, and training schedule unchanged.This controlled design isolates the effect of replacing ImageNet perceptual features with foundation-model features; single-channel slices are replicated for RadImageNet input.
4 Experiments
The experiments evaluate frozen feature extractors on four medical imaging tasks, select RadImageNet by Stage I ranking, and substitute it for VGG16 in iterative brain MRI dose simulation. RadImageNet produces modest endpoint changes and more consistent dose-trajectory and image-space agreement with low-dose targets.
- Stage I Evaluation: Stage I evaluates frozen encoders on four public diagnostic tasks and reports linear-probe AUC, retrieval consistency, and feature robustness under intensity perturbations.The selection rank excludes intensity-perturbation robustness from the final choice.
- Stage I Results: RadImageNet attains the lowest mean rank and remains first when intensity-perturbation robustness is excluded, so it is selected as ϕ⋆.The selection score is based on the descriptive cross-dataset mean rank.
- Stage I Results: RadImageNet occupies the outer envelope on probe AUC, accuracy, F1, and pathology-retrieval metrics, while SegVol and BrainIAC lead selected robustness axes.Intensity-perturbation cosine similarity is treated as a robustness diagnostic rather than a selection readout.
- Stage I Results: Only RadImageNet exceeds the matched ImageNet-ResNet50 probe-F1 baseline on all four datasets, indicating a pretraining effect rather than an architecture-capacity difference.The largest gain occurs on meniscus.
- Stage II Evaluation: Stage II replaces only the VGG16 perceptual extractor with RadImageNet while keeping the generator, auxiliary losses, and loss weights unchanged.Table 2 compares the final simulated 10% volume with the acquired low-dose target.
- Stage II Results: Across the reduction schedule, RadImageNet yields consistently higher SSIM and lower MAE against dose-specific reference maps than VGG16.The numerical gaps are small, but their direction remains consistent as nominal remaining contrast decreases.
- Qualitative Analysis: RadImageNet suppresses residual enhancement more completely and preserves low-dose target geometry more closely in qualitative 10% predictions.The 10% prediction approaches the acquired low-dose image rather than retaining stronger post-contrast uptake.
5 Discussion
The study argues that domain-aligned medical features are more suitable perceptual representations for MRI dose simulation, with RadImageNet producing clearer visual improvements despite modest standard-metric changes. The conclusion remains bounded to a single controlled replacement under the original Gformer objective and requires broader clinical validation.
- Metric interpretation: PSNR, SSIM, RMSE, and residual-uptake CNR do not directly measure whether a perceptual feature space captures clinically relevant enhancement patterns.Visual results are therefore central to interpreting the Stage II comparison.
- Visual findings: RadImageNet better suppressed residual uptake in enhancing structures while preserving the expected low-dose appearance of surrounding tissue.
- Controlled feature-space comparison: Replacing VGG16 with RadImageNet yielded modest traditional-metric changes but consistent reductions in residual enhancement across the dose trajectory and image-space comparisons.The controlled design kept the generator, objective, and loss settings unchanged, associating the observed differences with the perceptual feature space.
- Scope of inference: RadImageNet, SegVol, and BrainIAC are not interchangeable; the Stage II conclusion applies specifically to RadImageNet replacing VGG16 under the original Gformer objective.Only RadImageNet was inserted into the simulator after Stage I selection.
- Limitations: The study remains limited by one controlled replacement experiment, potential pretraining-data bias, and the absence of full clinical-equivalence validation.The authors identify larger cohorts, downstream clinical tasks, and reader studies as future validation steps.
6 Conclusion
The study evaluates medical foundation model features as perceptual losses for brain MRI contrast dose simulation. RadImageNet produced modest endpoint-metric changes but clearer visual improvements, while clinical relevance still requires larger cohorts and reader-centered validation.
- RadImageNet provided the strongest frozen representation among eligible medical models across the public benchmark suite.
- Replacing VGG16 with RadImageNet produced modest endpoint-metric changes but clearer suppression of residual enhancement and preservation of low-dose appearance.
- Radiology-pretrained features are supported as a practical perceptual feature space for MRI synthesis within the evaluated setting.
- Clinical relevance of the gains requires larger cohorts and reader-centered validation.
7 Appendix
The appendix reports supplementary Stage I performance, ranking, robustness, embedding, and attribution analyses. These displays clarify how frozen encoders were evaluated and how RadImageNet’s selection was supported without duplicating the main tables and figures.
- Appendix overview: Stage I supplementary quantities include per-dataset accuracy, F1, retrieval scores, per-fold AUC, dataset ranks, perturbation cosine similarity, t-SNE embeddings, and Grad-CAM examples.AUC and the cross-dataset mean rank remain in the main text.
- Numeric breakdowns: Table A1 reports 5-fold mean±std accuracy and F1, pathology retrieval, and perturbation cosine similarity, while AUC appears in Table 1.R@5 is undefined for breast ultrasound because subject IDs are unavailable.
- Numeric breakdowns: Table A2 gives per-fold Stage I linear-probe AUC, with fold-wise mean±std reported in Table 1.Bold values identify the best encoder within each dataset and fold.
- Ranking analysis: Table A3 summarizes per-dataset and cross-dataset descriptive average ranks, with lower values indicating better overall ranking.The ranks aggregate classification, retrieval, and perturbation-similarity measures.
- Ranking analysis: Excluding intensity-perturbation cosine similarity leaves RadImageNet first on all four datasets, with a selection mean rank of 1.46.Cosine similarity is not used to select ϕ⋆.
- Robustness analysis: Table A4 decomposes the robustness axis using frozen-feature cosine similarity under input perturbations, where higher values indicate greater invariance.ViT pipelines use normalization that largely cancels global intensity shifts.
- Qualitative embeddings: Figure A1 visualizes fold-1 frozen-feature embeddings with pathology-label colors across RadImageNet, ImageNet, BrainIAC, and SegVol encoders.The panels cover ACL, breast, meniscus, and thyroid datasets with the stated sample sizes.
- Attribution analysis: Figure A2 shows class-specific Grad-CAM overlays for CNN extractors, excluding BrainIAC and SegVol because they lack comparable convolutional feature maps.