Source-linked AI summary
RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, Akshay Chaudhari
TL;DR
Curated annotated medical imaging datasets are scarce, and natural-image models may not generalize to medical images and vocabulary. RoentGen adapts a pretrained latent diffusion model for text-conditioned CXR generation. It produces controllable images and supports downstream classification improvements when real and synthetic data are combined.
Problem
High-quality annotated medical imaging datasets are scarce, and models trained on natural image-text pairs do not tend to generalize well to medical images and their domain-specific language.
Source: §1 · p. 2Method
The study adapts Stable Diffusion for CXR generation using free-form medical text prompts and a domain-adapted model pipeline.
Source: §1 · p. 2, §10 · p. 15Results
RoentGen generates controllable CXR across findings, and joint real-synthetic training improved classifier performance by 5% in the reported setup.
Source: §1 · p. 3, §10 · p. 15Takeaways & Limitations
Fine-tuned Stable Diffusion improved downstream use both as data augmentation and with purely synthetic training data in the reported experiments.
Source: §10 · p. 16Fine-tuning and evaluation used only MIMIC-CXR from a single institution, leaving cross-institutional robustness unmeasured.
Source: §9 · p. 14
Abstract
from arXiv · showhide
Multimodal models trained on large natural image-text pair datasets have exhibited astounding abilities in generating high-quality images. Medical imaging data is fundamentally different to natural images, and the language used to succinctly capture relevant details in medical data uses a different, narrow but semantically rich, domain-specific vocabulary. Not surprisingly, multi-modal models trained on natural image-text pairs do not tend to generalize well to the medical domain. Developing generative imaging models faithfully representing medical concepts while providing compositional diversity could mitigate the existing paucity of high-quality, annotated medical imaging datasets. In this work, we develop a strategy to overcome the large natural-medical distributional shift by adapting a pre-trained latent diffusion model on a corpus of publicly available chest x-rays (CXR) and their corresponding radiology (text) reports. We investigate the model's ability to generate high-fidelity, diverse synthetic CXR conditioned on text prompts. We assess the model outputs quantitatively using image quality metrics, and evaluate image quality and text-image alignment by human domain experts. We present evidence that the resulting model (RoentGen) is able to create visually convincing, diverse synthetic CXR images, and that the output can be controlled to a new extent by using free-form text prompts including radiology-specific language. Fine-tuning this model on a fixed training set and using it as a data augmentation method, we measure a 5% improvement of a classifier trained jointly on synthetic and real images, and a 3% improvement when trained on a larger but purely synthetic training set. Finally, we observe that this fine-tuning distills in-domain knowledge in the text-encoder and can improve its representation capabilities of certain diseases like pneumothorax by 25%.
1. Introduction
Natural-image models face a distribution shift from medical images and their specialized vocabulary, while curated annotated medical datasets are scarce. This work adapts Stable Diffusion for free-form, text-conditioned CXR generation and evaluates its outputs and downstream uses.
1. Introduction
- The work systematically evaluates and expands an out-of-domain pretrained latent diffusion model for language-conditioned medical image generation beyond few- or zero-shot use.Source: §1 · p. 2
- RoentGen generates high-fidelity CXR and can insert, combine, or modify imaging findings through free-form medical-language prompts.Source: §1 · p. 2
- The study presents an evaluation framework using classification, radiology report generation, and image-image and text-image retrieval to assess factual correctness.Source: §1 · p. 3
- Fine-tuning both the U-Net and CLIP text encoder yielded the highest image fidelity and conceptual correctness among the tested adaptation approaches.Source: §1 · p. 3
- Replacing CLIP with a domain-specific text encoder improved the fine-tuned model when the encoder stayed frozen and only the U-Net was trained.Source: §1 · p. 3
- Fine-tuning Stable Diffusion alongside the U-Net can distill in-domain knowledge into the text encoder, improving representations of medical concepts such as rare abnormalities.Source: §1 · p. 3
- Joint training on real and synthetic images improved classifier performance by 5% in the reported setup; synthetic-only training performed comparably to real-data training.RoentGen could be fine-tuned on 1.1–5.5k images and prompts for downstream classification.Source: §1 · p. 3
2. Related Work
Prior synthetic CXR work largely used GANs for targeted or single-modality tasks; earlier latent diffusion studies left gaps in text prompting and downstream evaluation. Related latent diffusion work also examined CT and MRI generation, with reported anatomical inconsistencies in some evaluated scans.
2.1. Generative Models for Chest X-Ray Generation
- GAN-based CXR generation covered disease-specific classification, multi-pathology classification, and other tasks, while prior approaches were largely pathology-specific and imaging-only.Source: §2.1 · p. 3
- Earlier CXR latent diffusion studies explored few-shot or class-conditional generation, but did not report generation metrics; no study assessed downstream benefits or text-prompt conditioning.Source: §2.1 · p. 3
2.2. LDM for other medical imaging modalities
- In CT and MRI work, synthetic-data pretraining improved breast MRI segmentation Dice to 0.95 (+0.04), while radiologists found major anatomical inconsistencies in up to 32% of 50 evaluated scans.Source: §2.2 · p. 3
3. Datasets
The study uses MIMIC-CXR reports and images to construct training and test sets for CXR generation. It focuses on report impressions and controls for view type and label imbalance.
3. Datasets
- Figure 2 illustrates prompt control of finding presence or absence, size, and laterality, including the radiological left-right display convention.Source: Figure 2 · p. 4
- MIMIC-CXR provides 377,110 images and associated reports from 227,827 studies at Beth Israel Deaconess Medical Center.Source: §3 · p. 4
- Impression sections were used as text inputs; impressions shorter than seven characters or longer than CLIP’s 77-token limit were excluded, the latter filtering out 14%.Source: §3 · p. 4
- Capping “No finding” reports was used to reduce class imbalance and support learning positive abnormalities, especially rare ones.Source: §3 · p. 5
- The dataset distinguishes PA-only and all-view training sets, with a PA-view P19 holdout and an official MIMIC test set; the table caption identifies train/test composition details.Source: §3 · pp. 4–5, Table 1 · p. 5
4. Stable Diffusion Fine-Tuning
The section describes Stable Diffusion’s image-generation components, then contrasts earlier few-shot adaptation with training on a larger radiology image-text dataset. Its experiments keep the VAE frozen while varying U-Net and text-encoder training, alongside training settings.
4.1. Model Architecture
- Stable Diffusion combines a VAE that encodes and decodes image representations, a denoising U-Net, and a CLIP text encoder limited to 77 tokens.The safety checker was disabled after showing a high false-positive rate for medical prompts.Source: §4.1 · p. 5
4.2. Few-shot fine-tuning
- Prior few-shot strategies include token-embedding Textual Inversion, projected domain-text conditioning [3], and DreamBooth U-Net tuning, but DreamBooth risks overfitting and low diversity [3].Textual Projection did not produce satisfactory few-shot results [3].Source: §4.2 · pp. 5–6
4.3. Fine-tuning and training from scratch
- Training predicts sampled latent noise from a noisy encoded CXR conditioned on its encoded prompt, using MSE between true and predicted noise.The prompt and image are encoded by the text encoder and VAE, respectively, and the VAE remains frozen in the experiments.Source: §4.3 · p. 6
- The U-Net is trained either from original weights or random initialization, while the text encoder is frozen, jointly trained, or replaced with a domain-specific encoder.Training can vary dataset size and distribution, steps, learning rate, and precision.Source: §4.3 · p. 6
4.4. Training details
- Experiments used SD 1.4 and batch size 256 across 64 A100 GPUs; inference used guidance scale 4 and 75 steps.Training took about 20 minutes for 1k steps, 5 hours for 12.5k steps, and one day for 60k steps.Source: §4.4 · p. 6
5. Fidelity and diversity of generated images
The section evaluates synthetic CXR fidelity and diversity using FID and MS-SSIM, with results varying by training steps, model components, and prompt length. The reported best fidelity was at 60k steps with a learning rate of 5e-5, while the authors caution that the metrics have limitations.
Learning rate, train steps
- At 60k steps and learning rate 5e-5, the best quality results had FIDXRV 3.6 and FIDIncepV3 54.9.The CLIP FID did not align with the domain-specific FIDXRV and manual review, which the authors cite as a limitation of CLIP-based evaluation for this task.Source: Table 2 · p. 7, §5 · p. 7
Fidelity and diversity assessment
- FID and MS-SSIM compare generated images with original impression-section p19 test images, using image-feature representations from InceptionV3, CLIP, and CXR-trained DenseNet-121.Smaller FID indicates greater fidelity, while smaller MS-SSIM indicates greater intra-prompt diversity.Source: §5 · pp. 6–7, Table 2 · p. 7
Components
- After 60k steps, the jointly trained text encoder and U-Net achieved FIDXRV 3.6, versus 4.9 with a randomly initialized U-Net and 9.2 when training only the original U-Net.Training only a randomly initialized U-Net achieved FIDXRV 16.5; the authors conclude that fine-tuning the text encoder with the U-Net improves learning speed and asymptotic performance.Source: §5 · pp. 7–8
Text Encoders
- At 60k steps, frozen SapBERT and RadBERT text encoders with a randomly initialized U-Net achieved FIDXRV 6.0 and 6.7, respectively, versus 16.5 for the random-U-Net-only model.The authors describe this as a 60% improvement in FIDXRV and suggest that domain-specific encoder knowledge can be leveraged for image generation.Source: §5 · p. 8
Diversity
- MS-SSIM can misrepresent diversity: the 1k-step model scored as more diverse than the 60k-step model despite frequent errors and out-of-domain outputs.Manual inspection found richer diversity in the 60k-step model; the multi-view model had the lowest MS-SSIM among fine-tuned models, while the PA-only test set partly invalidated its FID evaluation.Source: §5 · p. 8
- Fine-tuned models tended to show lower diversity as CLIP token length increased, unlike original Stable Diffusion’s consistently high diversity across lengths.The figure summarizes mean MS-SSIM for 5,000 prompts with four generated images each; lower MS-SSIM indicates higher diversity.Source: §5 · p. 8, Figure 3 · p. 8
6. Factual correctness of generated images
The study evaluates factual correctness beyond image-feature similarity using classification, report generation, and image-retrieval tasks, alongside radiologist review. Across retrieval tasks, models trained on multiple views performed best, while expert ratings found realistic appearance but incomplete prompt fulfillment.
6.1. Multi-label classification
- The classification evaluation used DenseNet-121 on real test images and 5,000 generated images per fine-tuned model, with report-derived labels as ground truth.The original Stable Diffusion pipeline had an approximately 0.5 AUROC, while DreamBooth scored a filtered average AUROC of 0.61.Source: §6.1 · p. 9
- AUROC was 0.82 after 1k steps versus 0.81 after 60k at lr = 5e-5, although manual review suggested longer training improved image quality.The authors suggest short training can make abnormalities overly obvious, while longer training yields subtler representations; noisy CheXpert labels and limited classifier performance also complicate score comparisons.Source: §6.1 · p. 10
6.2. Radiology Report Generation
- factENT rose from 13.7 for DreamBooth to 22.9 at 1k steps and 24.1 at 60k; RadGraph rose from 9.0 to 15.0 and 18.9.The multiple-view 60k model was superior on these report-generation metrics; PA-only variants scored around 2–5% below it.Source: §6.2 · p. 12
6.3. Zero-shot Image-Image Retrieval
- Training on multiple views outperformed the second-best PA-only variant by 7–13% on all image-image precision scores.The multiple-view model also achieved the best scores on every image-text retrieval metric and was superior on the report-generation metrics.Source: §6.3 · p. 12, §6.2 · p. 12, §6.4 · p. 13
6.4. Zero-shot Image-Text Retrieval
- For image-text retrieval, SapBERT-based training improved over RadBERT and could outperform CLIP-based models on tasks that directly use text.The authors hypothesize that in-domain text encoders capture fine-grained impression details more effectively.Source: §6.4 · p. 13
6.5. Qualitative evaluation
- Radiologists rated image realism 1.67 ± 0.63 and 1.81 ± 0.46, but prompt alignment averaged 0.41 ± 1.41 and 0.29 ± 1.36.They commonly identified synthetic images through unrealistic device features, and prompts were often only partly reflected; generated samples also varied in appearance per prompt.Source: §6.5 · p. 13
7. Data augmentation
The authors tested synthetic CXRs as training data for a six-finding classifier using small- and larger-dataset generation settings. Mixed real and synthetic training achieved the largest reported improvement over the real-data baseline.
7. Data augmentation
- The classifier was trained on six findings, using synthetic images from models trained for 12.5k or 60k steps on small or larger MIMIC-CXR subsets.The highest-validation-AUROC classifiers were evaluated on the p19 test set.Source: §7 · p. 13
- AUROC improved by 0.11 over the 0.73 baseline when training combined real and synthetic data.Training exclusively on 30k synthetic samples from the larger dataset improved AUROC by 0.07 over baseline; using 1.1k synthetic samples alone reduced AUROC by 0.04.Source: §7 · pp. 13–14
- Adding 1.1k synthetic samples to real data improved AUROC by 0.04, while adding 5.5k yielded a smaller increase.The authors describe the smaller increase from the larger synthetic augmentation as counter-intuitive.Source: §7 · p. 14
8. Distilling in-domain knowledge and potential catastrophic forgetting
Fine-tuning the Stable Diffusion components on CXRs transfers in-domain knowledge but also causes catastrophic forgetting. The text encoder’s in-domain score partially recovered with longer training, with pneumothorax an exception under a stronger learning rate.
8. Distilling in-domain knowledge and potential catastrophic forgetting
- Fine-tuning both the U-Net and text encoder accelerated learning of in-domain concepts but also accelerated forgetting of prior-domain knowledge.The authors attribute forgetting to the absence of a method to control it in their setting.Source: §8 · p. 14
- At lr = 5e-5, the text encoder’s macro CheXpert@10 score dropped 43% after 1k steps and 55% after 12.5k, then recovered to a 25% drop at 60k.For pneumothorax, its knowledge was fully recovered or improved when using lr = 1e-4.Source: §8 · p. 14
9. Limitations
The authors identify limits in image realism, dataset and report scope, small-data training, and retention of pretrained knowledge. These constrain the model’s use and the generality of the reported evaluation.
9. Limitations
- RoentGen outputs are images rather than actual radiographs, have limited grayscale values, and cannot support realistic windowing; they should not replace chest X-ray studies.These limitations constrain their use as substitutes for clinical radiographs.Source: §9 · p. 14
- Fine-tuning and evaluation used only MIMIC-CXR from one institution, leaving cross-institutional robustness unmeasured.The authors suggest multi-institutional training and evaluation could improve generalizability, prompt understanding, and image diversity.Source: §9 · p. 14
- Training used only report impression sections, which may omit image details that findings sections could support for finer-grained conditioning.Source: §9 · p. 15
- The model was prone to overfitting when trained on datasets of a few hundred images.Source: §9 · p. 15
- Fine-tuning both the U-Net and text encoder caused catastrophic forgetting of previously acquired knowledge.The authors identify methods to limit forgetting as a direction for retaining prior knowledge while speeding training.Source: §9 · p. 15
10. Conclusion and future work
RoentGen adapts Stable Diffusion to generate high-fidelity, diverse chest X-rays controlled by free-form natural-language prompts, including medical vocabulary. The conclusion also reports improved downstream classification performance and stronger in-domain text-encoder representations.
10. Conclusion and future work
- RoentGen generates high-fidelity, diverse CXR images and allows free-form prompts to control findings and their appearance.Prompts can combine or modify findings, including side and size, using natural or medical language.Source: §10 · p. 15
- Joint fine-tuning of the pretrained U-Net and text encoder performed best, while a domain-specific encoder improved performance when the U-Net was trained from scratch.The latter result points to potential training speed-up for domain adaptation.Source: §10 · p. 15
- The study's evaluation framework assesses synthetic-image medical correctness through downstream tasks including radiology report generation and image-image or image-text retrieval.The framework compares models in terms of clinical consistency and completeness.Source: §10 · pp. 15–16
- Fine-tuned Stable Diffusion improved downstream training with synthetic data and distilled in-domain knowledge into the text encoder, improving its in-domain representations.The study used fine-tuned models to augment training data or replace real training data with synthetic data.Source: §10 · p. 16
- Future work will extend the approach to other study types and modalities and investigate fine-tuning strategies that limit catastrophic forgetting.The authors specifically mention domain-specific text encoders as one possible strategy.Source: §10 · p. 16
A. Details of the factually-oriented metrics
The section defines three factually oriented metrics for comparing generated and reference reports: factENT, factENTNLI, and F1CheXbert.
A. Details of the factually-oriented metrics
- factENT measures the harmonic mean of precision and recall between named entities extracted from generated and reference reports.A named entity recognizer extracts the two entity sets.Source: A. Details of the factually-oriented… · p. 18
- factENTNLI extends factENT by checking that a generated-report entity's sentence does not contradict its matched reference sentence.The matched sentence has the highest BERTScore, and an NLI model evaluates contradiction.Source: A. Details of the factually-oriented… · p. 18
- F1CheXbert is the F1-score between CheXbert abnormality predictions for generated and reference reports, calculated over five observations.The observations are atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.Source: A. Details of the factually-oriented… · pp. 18–19