Source-linked AI summary
Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, Akshay Chaudhari
TL;DR
Medical imaging lacks high-quality labeled datasets, and natural-image foundation models do not directly capture its clinical concepts. This paper evaluates Stable Diffusion components and adaptation strategies for synthetic chest-radiograph generation. U-Net fine-tuning produces images that outperform the baseline quantitatively and visually while representing abnormalities, although diversity and clinical prompt realism remain limited.
Problem
Medical imaging has a paucity of high-quality labeled datasets, while pretrained vision-language models are not explicitly trained to represent medical concepts.
Method
The study evaluates Stable Diffusion’s VAE, U-Net, and text encoder for chest-radiograph generation using quantitative metrics and radiologist assessment.
Results
U-Net fine-tuning produced synthetic CXRs that visually and quantitatively outperformed the baseline and correctly represented abnormalities.
Takeaways & Limitations
Stable Diffusion can represent radiology concepts and generate synthetic CXRs with custom pathologies within the evaluated setting.
Takeaways & Limitations
The fine-tuned model lacks image diversity, metrics incompletely assess clinical correctness, and synthetic prompts do not fully match clinical wording.
Abstract
from arXiv · showhide
Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions, frequently obtained through web-crawling approaches. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. Thus, in this study, we seek to research and expand the representational capabilities of large pretrained foundation models to medical concepts, specifically for leveraging the Stable Diffusion model to generate domain specific images found in medical imaging. We explore the sub-components of the Stable Diffusion pipeline (the variational autoencoder, the U-Net and the text-encoder) to fine-tune the model to generate medical images. We benchmark the efficacy of these efforts using quantitative image quality metrics and qualitative radiologist-driven evaluations that accurately represent the clinical content of conditional text prompts. Our best-performing model improves upon the stable diffusion baseline and can be conditioned to insert a realistic-looking abnormality on a synthetic radiology image, while maintaining a 95% accuracy on a classifier trained to detect the abnormality.
1 Introduction
Latent diffusion models offer controllable image generation, but medical imaging lacks high-quality labeled datasets and differs substantially from natural-image training domains. This study evaluates Stable Diffusion components for representing chest-radiograph concepts and generating synthetic images with abnormalities.
- 1 Introduction: Medical imaging has few high-quality labeled datasets, while radiology reports provide text that can support prompt-based synthetic-data generation.Annotations require trained experts to recognize subtle but meaningful image features.
- 1 Introduction: Stable Diffusion components were evaluated for representing medical imaging studies and concepts despite lacking explicit training on those concepts.The study focuses on chest X-rays, a common, inexpensive modality that can reveal and monitor diverse pathologies.
- 1 Introduction: The pretrained VAE reconstructs chest X-rays at arbitrary resolutions, and the frozen CLIP text encoder produces medically useful embeddings with sufficient context.These findings support retaining key visual and medical information without initially replacing the Stable Diffusion text encoder.
- 1 Introduction: Replacing frozen CLIP with a projected in-domain text encoder did not produce better images.The projection-based alternative was compared with the original conditioning approach.
- 1 Introduction: Textual inversion learns complex concepts such as pleural effusion from few examples, while U-Net fine-tuning enables high-fidelity images with inserted pathologies.The study substantiates these findings using image-quality and classification metrics plus thoracic-radiologist evaluation.
2 Materials and Methods
The study uses publicly available chest-radiograph datasets and selected image-report pairs to evaluate Stable Diffusion components and domain-adaptation strategies. Experiments examine the pipeline’s VAE, text encoder, and component fine-tuning under radiology-specific prompts.
- 2 Materials and Methods: Experiments use CheXpert and MIMIC-CXR, sampling 1,000 frontal radiographs from each dataset.The datasets contain 224,316 and 377,110 images, respectively, before study sampling.
- 2 Materials and Methods: Figure 2 depicts the Stable Diffusion architecture operating in the radiology setting to generate synthetic radiology images.The diagram provides the pipeline-level context for the component studies.
- 2 Materials and Methods: Five normal images and five images with isolated pleural effusion were paired with synthetic prompts for Stable Diffusion component fine-tuning.Unusually cropped or colorized images were excluded.
- 2 Materials and Methods: The pipeline encodes prompts with CLIP into a 768-dimensional latent representation, conditions a denoising U-Net, and decodes latent images into pixels with the VAE.The original model was trained on natural-image captions, motivating evaluation in radiology.
- 2 Materials and Methods: Methods assess VAE reconstruction, compare CLIP with in-domain text encoders, and fine-tune Stable Diffusion components for radiology.A one-million-prompt LAION-400M sample supports textual-projection training and experiments.
2.3 Variational Autoencoder
The VAE is tested for preserving radiological information during latent-space reconstruction, while text encoders are evaluated by similarity-based agreement with CheXpert labels. Reconstruction is assessed quantitatively, qualitatively, and through downstream pathology classification.
- 2.3 Variational Autoencoder: Stable Diffusion first compresses chest X-rays into latent representations, which are decoded into reconstructions without VAE fine-tuning.This tests whether medical information survives the pretrained VAE’s perceptual-compression pathway.
- 2.3 Variational Autoencoder: Reconstruction quality is measured with RMSE, PSNR, SSIM, and batch-level FID against original-image distributions.These metrics quantify pixel error, signal quality, structural similarity, and distributional similarity.
- 2.3 Variational Autoencoder: A radiologist with seven years of CXR experience additionally scores reconstruction quality on a 1–5 diagnostic-quality scale.The qualitative assessment complements image metrics with domain-specific review.
- 2.3 Variational Autoencoder: The reconstruction process is also evaluated by measuring DenseNet-121 pathology-classification performance on reconstructed versus original CXRs.AUC, accuracy, and F1 are calculated for selected labels across CheXpert, MIMIC-CXR, and the pretrained model.
- 2.3 Variational Autoencoder: The text-encoder evaluation uses impression sections from radiology reports and compares document-level representations across biomedical and radiology encoders.The representations are extracted from parsed report impressions before similarity computation.
- 2.3 Variational Autoencoder: Text encoders produce 768-dimensional report embeddings, whose latent-space similarities are evaluated with CheXpert@k.The metric retrieves the k most similar reports and measures the proportion sharing the same CheXpert label.
2.5 Textual Projection
Textual projection adapts an in-domain text encoder to Stable Diffusion’s CLIP-conditioned latent space rather than replacing CLIP without alignment. The approach trains a projection on LAION prompts and evaluates prompt styles during generation.
- 2.5 Textual Projection: The method replaces frozen CLIP with an in-domain biomedical or radiology encoder to provide more domain-specific representations for generation.The architectural motivation is to retain medically relevant information from radiology reports for the downstream U-Net.
- 2.5 Textual Projection: A projection is introduced because independently trained text encoders may organize their latent spaces differently, making direct replacement unsafe.The projection translates part of the in-domain representation into the CLIP-compatible space.
- 2.5 Textual Projection: The projection is a 768-to-768 MLP trained with maximum-likelihood loss on LAION-400M text.Its first implementation composes Linear, ReLU, LayerNorm, and Linear layers.
- 2.5 Textual Projection: Generation experiments compare object-oriented prompts such as “a photo of a ...” with style-oriented prompts such as “a photo in the style of a ...”.Lexical variants of both prompt templates are also explored.
2.6 Textual Embeddings Fine-tuning
Textual Inversion fine-tunes Stable Diffusion by learning new token embeddings while freezing the VAE, U-Net, and existing text-encoder layers.
- 2.6 Textual Embeddings Fine-tuning: Textual Inversion introduces a new token representing clinical features and learns only its embedding from paired radiology prompts and images.The token can encode patient-level, procedure-level, or pathology-level concepts.
- 2.6 Textual Embeddings Fine-tuning: Training updates the newly introduced token embeddings using the denoising objective while gradients for existing token embeddings remain zeroed.Generic prompts containing the new tokens are paired with radiology images, and the resulting noise predictions are optimized with MSE loss.
2.7 U-Net Fine-tuning
U-Net fine-tuning adapts Stable Diffusion for radiology by updating the denoising U-Net while keeping the other pipeline components frozen.
- 2.7 U-Net Fine-tuning: U-Net fine-tuning updates only the denoising U-Net, unlike approaches that replace the text encoder or learn new token embeddings.The VAE and text encoder remain frozen during this adaptation.
- 2.7 U-Net Fine-tuning: The experimental generation setup compares prompts for a lung X-ray without findings and one with visible pleural effusion, using 50 samples per setup.Generated images are classified by DenseNet-121 after conditioning on these prompts.
3 Results
Across reconstruction, text representation, and image-generation experiments, Stable Diffusion components retained substantial medical information, while U-Net fine-tuning produced the strongest generation results and domain-specific evaluation exposed limitations of generic metrics.
- 3.2 Variational autoencoder: The pretrained VAE reconstructed MIMIC-CXR images with RMSE 31.8±6.5, PSNR 35.1±1.6, and SSIM 0.93±0.02 at 512x512 resolution.CheXpert reconstruction quality was lower, with RMSE 47.8±7.4, PSNR 31.5±1.4, and SSIM 0.85±0.03.
- 3.2 Variational autoencoder: Visual assessment gave reconstructed MIMIC-CXR images a mean score of 4.51±0.54, preserving diagnostic features while usually scrambling burnt-in text and sometimes blurring devices or rib contours.No reconstruction was rated completely non-diagnostic or potentially diagnostically altered.
- 3.2 Variational autoencoder: Reconstruction slightly improved pleural-effusion classification on MIMIC-CXR but reduced performance for some categories, including lung lesion from AUC 0.766 to 0.728.DenseNet-121 embeddings remained highly similar between original and reconstructed images, with cosine similarity 0.98±0.02 for MIMIC and 0.97±0.02 for CheXpert.
- 3.3 Text Encoders: For radiology text representations, CLS hidden states generally worked best, CXR-BERT-specialized performed strongest overall, and CLIP also retained high clinical-information performance.These results led the authors not to replace CLIP in subsequent textual-inversion and U-Net experiments.
- 3.4 Radiology Image Generation: The baseline generated lung X-rays with FID 0.097 for a simple prompt, while U-Net fine-tuning reduced FID to 0.034; for pleural-effusion prompts, baseline FID was 0.151 and textual inversion and U-Net fine-tuning outperformed it.Textual projection degraded image quality, with FID scores at least doubling for both token-level and document-level projections.
- 3.4 Radiology Image Generation: U-Net fine-tuning with prior achieved the strongest generation outcome, producing the lowest FID scores and more realistic outputs, while FID missed improved abnormality conditioning.The authors therefore call for domain-specific metrics that assess whether generated abnormalities match the conditioning prompt.
- 3.5 Classification of synthetic CXR: U-Net fine-tuning with prior yielded a nearly perfect pleural-effusion classification AUC of 0.98, compared with 0.5 for the Stable Diffusion baseline.Textual inversion reached AUC 0.74, whereas U-Net fine-tuning without prior slightly reduced performance relative to baseline.
4 Discussion
The study evaluates Stable Diffusion components for medical image generation and finds that U-Net fine-tuning produces clinically controllable synthetic CXRs. The work also identifies limitations in clinical evaluation, image diversity, and prompt realism.
- U-Net fine-tuning enables high-fidelity CXR generation with custom pathology insertion, illustrated by synthetic radiology images conditioned on pleural effusion prompts.
- The fine-tuned pipeline generates synthetic CXR images that visually and quantitatively outperform the Stable Diffusion baseline and correctly represent abnormalities.
- The approach extends prior latent-diffusion medical imaging work by using natural-language conditioning to convey a broader spectrum of medical information.
- The findings motivate future studies on learning and combining broader abnormality sets across additional imaging modalities and body parts.
- Clinical correctness is difficult to assess with the employed metrics, while few-shot U-Net fine-tuning limits image diversity and synthetic prompts differ from clinical report wording.
5 Conclusion
The conclusion evaluates Stable Diffusion for text-conditioned synthetic radiological image generation. It reports component-specific findings and concludes that U-Net fine-tuning improves generation quality while preserving abnormality representation.
- The VAE reconstructs CXRs while preserving medical features, and CLIP embeddings retain abnormality content despite limited gains from in-domain alternatives.
- Textual projection degraded image quality and produced out-of-domain samples compared with baseline Stable Diffusion.
- U-Net fine-tuning produces synthetic CXRs that visually and quantitatively outperform the baseline while correctly representing abnormalities.
A Image reconstruction on CheXpert dataset
Table 6 compares DenseNet-121 classification performance on original and VAE-reconstructed CheXpert chest X-rays across reported labels.
- Table 6 reports DenseNet-121 classification performance for original and reconstructed CheXpert CXR images.
- The comparison uses original versus reconstructed images from the CheXpert dataset.
- The table includes an abbreviated label, “Enl Mediastinum,” for Enlarged Mediastinum.
B Text-encoder clusterization capabilities for various chest abnormalities
The study visualizes radiology-report embeddings with t-SNE to examine clustering across chest abnormalities. It compares CLIP, SapBERT-from-PubMedBERT-full, and CXR-BERT-specialized representations using abnormality-specific colors.
- t-SNE projections visualize clusters of radiology reports grouped by different chest abnormalities for each evaluated text encoder.
- The compared embedding spaces are produced by CLIP, SapBERT-from-PubMedBERT-full, and CXR-BERT-specialized encoders.
- Each abnormality is encoded by a distinct color, including Atelectasis, Cardiomegaly, Pleural Effusion, Pneumonia, and Pneumothorax.