Source-linked AI summary
Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models
Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, Tanuja Ganu
TL;DR
Pure retrieval-based report generation does not let users modulate retrievals for different clinical settings and may not produce required report formats. CXR-RePaiR-Gen retrieves radiology text aligned with an input X-ray using CXR-ReDonE embeddings, then prompts an OpenAI LLM to generate an impression. BERTScore 0.2865 (∆+ 25.88%) and Semb score 0.4026 (∆+ 6.31%) improve over the previous state-of-the-art retrieval method CXR-ReDonE.
Problem
Pure retrieval-based report generation does not let users modulate retrievals for different clinical settings and may not produce required report formats.
Source: §1 · p. 2Method
CXR-RePaiR-Gen retrieves radiology text aligned with an input X-ray using CXR-ReDonE embeddings, then prompts an OpenAI LLM to generate an impression.
Source: §1 · p. 3Results
BERTScore 0.2865 (∆+ 25.88%) and Semb score 0.4026 (∆+ 6.31%) improve over the previous state-of-the-art retrieval method CXR-ReDonE.
Source: Generalizable Insights about Machine… · p. 3Takeaways & Limitations
Prompt instructions and few-shot examples allow users to tailor report content and format to downstream clinical needs.
Source: §1 · p. 3Generated clinical entities are limited by retrieval unless the model can use general knowledge; experiments restricted it to retrieved context to alleviate hallucinations.
Source: §5.1 · p. 18
Abstract
from arXiv · showhide
We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing that leverages multimodally aligned embeddings from a contrastively pretrained vision language model for retrieval of relevant candidate radiology text for an input radiology image and a general domain generative model like OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4 for report generation using the relevant radiology text retrieved. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire leveraging the instruction following capabilities of these generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 ({\Delta}+ 25.88%) and Semb score of 0.4026 ({\Delta}+ 6.31%). Our approach can be broadly relevant for different clinical settings as it allows to augment the automated radiology report generation process with content relevant for that setting while also having the ability to inject user intents and requirements in the prompts as part of the report generation process to modulate the content and format of the generated reports as applicable for that clinical setting.
1. Introduction
Pure retrieval-based report generation has limits in handling irrelevant or noisy content, adapting reports to clinical needs, and producing requested formats. CXR-RePaiR-Gen combines aligned image–text retrieval with instructed LLM generation, and the paper reports improved clinical metrics and adaptable outputs.
1. Introduction
- Pure retrieval-based systems can return incoherent, irrelevant, duplicated, or noisy content and cannot modulate outputs or reliably produce requested report formats.The cited examples include mixed-patient sentences, prior references, doctor names, and user-specific details.Source: §1 · p. 2
- General-domain LLMs can follow instructions but lack the up-to-date or domain-specific information needed in medical settings; retrieved context is intended to ground their generations.The introduction motivates RAG as a way to make generations less prone to hallucination.Source: §1 · pp. 2–3
- CXR-RePaiR-Gen retrieves radiology reports or sentences aligned to an input X-ray and uses OpenAI LLMs to generate a consolidated impression.CXR-ReDonE’s contrastively pretrained model embeds both the image and text for similarity-based retrieval.Source: §1 · p. 3, Figure 1 · p. 4
Generalizable Insights about Machine Learning in the Context of Healthcare
- Prompt instructions and few-shot examples let users inject requirements and tailor report content and format to clinical settings.The paper also describes measuring generated reports against retrieved text for hallucinations to inform practical-use decisions.Source: §1 · p. 3, Generalizable Insights about Machine… · p. 3
- RAG is presented as bridging domain-specific healthcare encoders and general-domain generative models while improving clinical metrics.The paper identifies this combination as a generalizable insight for machine learning in healthcare.Source: Generalizable Insights about Machine… · p. 3
- BERTScore 0.2865 (∆+ 25.88%) and Semb 0.4026 (∆+ 6.31%) improve over CXR-ReDonE, while RadGraph F1 remains on par.The paper attributes the RadGraph F1 ceiling to generation being based on records retrieved by CXR-ReDonE.Source: Generalizable Insights about Machine… · pp. 3–4
2. Related Work
Prior radiology report-generation work spans generative models, retrieval methods, and retrieval-augmented generation. This paper builds on CXR-ReDonE’s multimodal retrieval to study LLM generation with retrieved radiology text and user-controlled outputs.
2. Related Work
- R2Gen used a Transformer decoder, while M2 Trans used reward-based reinforcement learning to generate complete, consistent, and clinically accurate reports.These are examples of framing report generation as a generative task.Source: §2 · p. 4
- CXR-RePaiR used retrieval over MIMIC-CXR with a contrastively pretrained vision-language model and introduced Semb alongside BERTScore for semantic similarity evaluation.Its retrieval approach established a benchmark using clinically reliable metrics.Source: §2 · pp. 4–5
- CXR-ReDonE removed prior references through CXR-PRO, retrained CXR-RePaiR with ALBEF, and established a new report-generation benchmark.It also used RadGraph F1 to assess clinical-entity completeness and accuracy.Source: §2 · p. 5
- RAG was introduced as a way to augment LLMs with external knowledge; this work supplies radiology-report text and studies multimodal retrieval, output modulation, and hallucination measurement.The paper uses CXR-ReDonE’s contrastively pretrained vision-language model for image–text retrieval.Source: §2 · p. 5
3. Methods
CXR-RePaiR-Gen retrieves radiology text using contrastively aligned image and report embeddings, then prompts a general-domain LLM to generate impressions. The method supports free-text and structured outputs, with iterative refinement proposed for contexts that exceed an LLM’s token limit.
3. Methods
- CXR-RePaiR-Gen uses ALBEF-aligned image and report embeddings to retrieve relevant text, then prompts a general-domain LLM with retrieved impressions to generate a report.The authors test report-level and sentence-level retrieval and use text-davinci-003, gpt-3.5-turbo, and gpt-4.Source: §3 · pp. 5–6
3.1. Retrieval Corpus
- The retrieval corpus uses CXR-PRO training impressions, considering both whole reports and their component sentences; prior references are omitted from the adapted dataset.The dataset contains 374,139 reports and associated chest radiographs.Source: §3.1 · pp. 6–7
3.2. Baselines
- CXR-ReDonE, using the CXR-PRO dataset as its retrieval corpus, serves as the baseline for evaluating whether LLM augmentation improves clinical metrics.Source: §3.2 · p. 7
3.3. Prompt Design
- Prompts instruct the LLM to generate an impression from retrieved text, using system and user prompts for gpt-3.5-turbo and gpt-4.Source: §3.3 · p. 7
3.4. Structured Report Output
- Few-shot prompts specify structured outputs containing an impression summary and attributes such as pathology, severity, size, or position.Source: §3.4 · p. 7
3.5. Experiments
- Experiments use three OpenAI LLMs with report- and sentence-level retrieval, applying zero-shot prompts for free-text impressions and few-shot prompts for structured reports.Tables 1 and 2 describe the zero-shot and few-shot prompt templates, respectively.Source: §3.5 · p. 7, Table 1 · p. 8, Table 2 · p. 9
4. Results on Real Data
The evaluation uses two radiologist-created benchmarks with different text granularity and compares generations with retrieval-based baselines using semantic and clinical-entity metrics. It also examines structured output and semantic similarity to retrieved context as a check on hallucination.
4.1. Evaluation Dataset
- MS-CXR contains 1,162 image–sentence pairs across eight findings, while CXR-PRO has 2,188 images and longer impressions; both datasets were created with board-certified radiologists.MS-CXR provides precise, concise phrases and bounding boxes; CXR-PRO contains longer impression text.Source: §4.1 · p. 10
4.2. Evaluation Approach
- Evaluation compares generated impressions with ground truth using BERTScore and Semb for semantic similarity and RadGraph F1 for overlap in clinical entities.The study evaluates report-level retrieval and sentence-level retrieval with K=1, 2, and 3 against CXR-ReDonE impressions.Source: §4.2 · p. 10
- Structured JSON impressions are shown qualitatively to assess whether generated output can follow user-specified formats; they are not quantitatively measured.Source: §4.2 · p. 10
4.3. Measuring Hallucinations
- Hallucination is assessed qualitatively and by comparing generated impressions with their top-K sentence context using Semb as a semantic-equivalence measure.Source: §4.3 · p. 10
- CXR-RePaiR-Gen improves BERTScore and Semb over CXR-ReDonE for both report- and sentence-level retrieval across K values, while matching its RadGraph F1 at K=3.The table caption states that exceeding CXR-ReDonE on RadGraph F1 would indicate hallucinated clinical entities absent from the context.Source: Table 3 · p. 11
4.4. Results – CXR-PRO
On CXR-PRO, retrieval-augmented generations improve semantic metrics over CXR-ReDonE while keeping RadGraph F1 at parity at K=3. The approach also improves BERTScore and Semb on MS-CXR phrase-grounding text.
4.4. Results – CXR-PRO
- At K=3 on CXR-PRO sentence retrieval, RAG improves BERTScore by 5.06% and Semb by 2.43% over CXR-ReDonE, with RadGraph F1 at parity.RadGraph F1 is slightly lower at lower K values; the authors state that exceeding the baseline would imply hallucinating clinical entities absent from the context.Source: §4.4 · p. 11
4.5. Results – MS-CXR
- On MS-CXR phrase-grounding text, BERTScore improves by an absolute value of 8.67 and Semb by an absolute value of 3.86.Source: §4.5 · p. 11, Table 4 · p. 12
- The MS-CXR comparison uses text-davinci-003 with sentence-corpus retrieval at K=3 and reports RadGraph F1 at parity with the baseline.Source: Table 4 · p. 12
4.6. Qualitative Analysis
RAG-generated impressions are described as more concise and less noisy than pure retrieval outputs while retaining relevant clinical entities. Examples also show it excluding insignificant details such as follow-up recommendations.
4.6. Qualitative Analysis
- RAG impressions are more concise and less noisy than pure retrieval outputs while retaining important clinical entities.Table 5 compares CXR-RepaiR-Gen outputs with reference reports and CXR-ReDonE at K=3.Source: §4.6 · p. 12, Table 5 · p. 12
- RAG retains the relevant clinical entities in its concise impression summaries.The examples are from CXR-PRO and MS-CXR.Source: §4.6 · p. 12, Table 5 · p. 12
- RAG omits insignificant details, including follow-up recommendations, to produce cleaner impressions.Source: §4.6 · p. 12
4.7. Instruction Driven Output
GPT-4 prompts instructed the system to exclude prior-report references and follow-up actions. The reported evaluation removed all retrieved follow-up recommendations and excluded most prior-report references from the impressions.
4.7. Instruction Driven Output
- GPT-4 was prompted to exclude prior-report references and follow-up actions from generated impressions.Source: §4.7 · p. 14
- All 40 retrieved follow-up recommendations were omitted, and 80 of 87 prior-report references were excluded from impressions.Source: §4.7 · p. 14
- Table 7 presents CXR-PRO examples where instruction-based prompts removed prior-report references and follow-up recommendations.Source: §4.7 · p. 14
4.8. Modulate Report Generation Format
Prompt engineering lets RAG modulate report output format, including generating structured JSON that extracts attributes of interest. The supplied examples also describe reductions in noisy or duplicate details and removal of prior-report references and follow-up recommendations.
4.8. Modulate Report Generation Format
- RAG can modulate report-generation format through prompt engineering, including structured JSON output.Source: §4.8 · p. 14, Table 8 · p. 17
- MS-CXR examples describe RAG outputs as avoiding noisy and duplicate details while retaining important clinical entities.Outputs were generated with top K retrievals set to 3.Source: Table 6 · p. 15
- CXR-PRO examples show GPT-4 RAG generations removing prior-report references and follow-up recommendations based on instruction prompts.Source: Table 7 · p. 16
- Few-shot prompts generate structured outputs extracting pathology, positional information, severity, and size from retrieved context.Source: Table 8 · p. 17
4.9. Hallucinations in Retrieval Augmented Generation
The authors compare generated impressions with retrieved context using Semb scores at K=3. They report high average similarity and say most test impressions exceed 0.70, which they regard as an indication that generations did not hallucinate.
4.9. Hallucinations in Retrieval Augmented Generation
- 0.8466 was the average Semb similarity between generated impressions and retrieved context at K=3.Source: §4.9 · p. 18
- Around 87% of test impressions had a Semb score above 0.70, which the authors consider a good indication that generations did not hallucinate.Source: §4.9 · p. 18
- The authors say even the lowest-Semb records did not hallucinate, and suggest their scores may reflect concise impression summaries.Source: §4.9 · p. 18
5. Discussion
CXR-RepaiR-Gen combines contrastive-model retrieval with OpenAI LLM generation to produce concise, precise impressions and improve BERTScore and Semb without compromising relevant clinical entities. The discussion identifies retrieval-dependent content as a limitation and calls for more detailed contrastive models and improved medical prompt engineering.
5. Discussion
- CXR-RepaiR-Gen uses contrastive-model embeddings to retrieve radiology text and OpenAI LLMs to generate concise, precise impressions, improving BERTScore and Semb without losing relevant clinical entities.The authors say prompting can modulate content and format, reduce noisy text and repetition, and support structured extraction of attributes of interest.Source: §5 · p. 18
5.1. Limitations
- Generated clinical entities are limited by those in retrieved sentences unless the model is allowed to use general knowledge; the experiments restricted generation to retrieval context to alleviate hallucinations.The authors therefore emphasize retrieving all relevant clinical entities.Source: §5.1 · p. 18
5.2. Future Work
- More fine-grained contrastive models and advances in medical prompt engineering are identified as directions for future work.The desired image details include severity, size, position, and pathology.Source: §5.2 · p. 18
- The authors report no hallucinations in the listed examples where Semb scores were lower between retrieved sentences and LLM generations.The passage describes Semb as CheXbert-based cosine similarity between final-hidden-state embeddings and an indicator of hallucination.Source: Table 9 · p. 19
- Table 10 reports RAG generation metrics across temperature settings for text-davinci-003 using sentence-corpus retrieval with K=3.Source: Table 10 · p. 20
6. Appendix
In zero-shot text-davinci-003 impression generation with top K=3 sentence-corpus retrievals, the tested temperature settings had little influence on output scores.
6.1. GPT-3 Hyperparameters
- 0, 0.5, and 1 produced little change in zero-shot text-davinci-003 impression-generation scores with sentence-corpus top K=3 retrievals.The authors suggest that instructions to use the retrieved corpus as context may explain this behavior.Source: §6.1 · p. 20