Source-linked AI summary
R2GenGPT: Radiology Report Generation with Frozen LLMs
Zhanyu Wang, Lingqiao Liu, Lei Wang, Luping Zhou
TL;DR
R2GenGPT addresses the challenge of applying LLMs to radiology report generation caused by the mismatch between visual inputs and textual generation. It aligns visual features with LLM word embeddings through parameter-efficient modules, achieving competitive performance across two benchmark datasets while delta tuning trains only 5M parameters.
Problem
Radiology report generation requires connecting visual information to LLM-based textual generation despite their modality mismatch.
Method
R2GenGPT uses a visual encoder and mapper to align image features with LLM word embeddings through shallow, deep, and delta alignment strategies.
Results
R2GenGPT shows promising and robust performance on IU-Xray and MIMIC-CXR, with delta tuning training 5M parameters.
Takeaways & Limitations
R2GenGPT demonstrates that LLM-based radiology report generation can combine competitive performance with reduced computational complexity.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have consistently showcased remarkable generalization capabilities when applied to various language tasks. Nonetheless, harnessing the full potential of LLMs for Radiology Report Generation (R2Gen) still presents a challenge, stemming from the inherent disparity in modality between LLMs and the R2Gen task. To bridge this gap effectively, we propose R2GenGPT, which is a novel solution that aligns visual features with the word embedding space of LLMs using an efficient visual alignment module. This innovative approach empowers the previously static LLM to seamlessly integrate and process image information, marking a step forward in optimizing R2Gen performance. R2GenGPT offers the following benefits. First, it attains state-of-the-art (SOTA) performance by training only the lightweight visual alignment module while freezing all the parameters of LLM. Second, it exhibits high training efficiency, as it requires the training of an exceptionally minimal number of parameters while achieving rapid convergence. By employing delta tuning, our model only trains 5M parameters (which constitute just 0.07\% of the total parameter count) to achieve performance close to the SOTA levels. Our code is available at https://github.com/wang-zhanyu/R2GenGPT.
1. Introduction
R2GenGPT addresses the modality gap between visual radiology data and LLM-based text generation by aligning image features with the LLM feature space. It explores parameter-efficient alignment strategies and reports robust performance on IU-Xray and MIMIC-CXR.
- Motivation: Radiology report generation is motivated by rising imaging volume, radiologist workload, fatigue, and diagnostic-error risk.R2Gen aims to produce coherent paragraphs describing observations and findings in radiology images.
- Motivation: LLMs may help with long, coherent medical reports because they encode grammar, syntax, and semantic coherence.
- Approach: R2GenGPT bridges visual and textual modalities by mapping visual features into the LLM feature space.
- Approach: Three alignment strategies vary trainable parameters: shallow trains the mapper, deep trains encoder and mapper, and delta trains the mapper plus incremental encoder parameters.
- Results: R2GenGPT achieves promising and robust performance on IU-Xray and MIMIC-CXR compared with multiple state-of-the-art methods.
2. Relate Works
Prior R2Gen research largely uses encoder-decoder architectures adapted from image captioning, while addressing longer report generation and difficulty detecting abnormalities under biased data. Proposed solutions include image-text alignment, memory modules, knowledge graphs, and semantic multi-task learning.
- Challenges: R2Gen differs from image captioning because reports are longer and abnormalities are harder to detect under training-set data bias.
- Image-text alignment: Image-text matching branches use report features to augment image representations and reduce the impact of data bias.
- Long-text generation: Hierarchical LSTMs and memory modules have been used to improve long-text generation by retaining or reusing prior information.
- External knowledge: Knowledge graphs encode disease-organ relationships and are integrated with graph neural networks to improve abnormality feature learning.
- External knowledge: Multi-task methods incorporate semantic information through medical-term or diagnostic-result classification to assist report decoding.
3. Methodology
R2GenGPT combines a visual encoder, visual mapper, and LLM to generate reports from chest X-rays. Its alignment strategies differ in how many visual-encoder parameters are trainable, while report tokens alone receive autoregressive training supervision.
- Overview: The visual encoder extracts chest X-ray features, and the visual mapper projects them into the LLM’s word-embedding space for report generation.
- Feature Alignment: Visual features are converted into a sequence of visual tokens with the same dimensionality as the LLM word embeddings.
- Alignment Strategies: Shallow alignment freezes the Swin Transformer and trains only the linear Visual Mapper.
- Training Objective: The input sequence concatenates visual tokens, prompt tokens, and report tokens, with masks selecting tokens included in autoregressive training.
- Alignment Strategies: Deep alignment jointly fine-tunes all Swin Transformer and Visual Mapper parameters.
- Alignment Strategies: Delta alignment uses LoRA to fine-tune a limited set of incremental parameters in the Swin Transformer alongside the Visual Mapper.
- LLM Component: R2GenGPT uses Llama2-7B as its LLM component for understanding and generating complex language.
- Training Objective: Instruction tuning applies the negative-log-likelihood objective only to report tokens conditioned on visual information and the instruction prompt.
4. Experiments
Experiments evaluate R2GenGPT on IU-Xray and MIMIC-CXR using standard report-generation and clinical-efficacy metrics, comparing alignment variants with prior methods. Results show strong performance, reduced training cost, faster epochs, and qualitative gains from Delta and Deep Alignment.
- Datasets: R2GenGPT is evaluated on IU-Xray and MIMIC-CXR, with IU-Xray using a 7:1:2 train/test/validation partition and MIMIC-CXR using its official partition.IU-Xray contains 3,955 reports and 7,470 images; MIMIC-CXR contains 377,110 images and 227,835 reports.
- Evaluation Metrics: BLEU, ROUGE-L, METEOR, and CIDEr assess report quality, while CheXpert-based precision, recall, and F1 assess clinical efficacy across 14 thoracic categories.Generated reports are annotated with CheXpert and compared against ground-truth annotations.
- Comparison with SOTA: R2GenGPT variants outperform compared methods across nearly all evaluation metrics, with MIMIC-CXR BLEU_4 increasing from 0.124 to 0.134 over METransformer.The 0.134 BLEU_4 score represents an 8.1% increase; CIDEr remains lower than METransformer because that method uses an expert-voting strategy.
- Model Efficiency: R2GenGPT (Deep) trains 90.9 million parameters versus METransformer’s 152 million, while R2GenGPT (Delta) achieves comparable performance with 5 million parameters.These results are reported in the model-efficiency and clinical-efficacy analysis on MIMIC-CXR.
- Model Efficiency: Training epochs take 1.75, 1.83, and 2.75 hours for Shallow, Delta, and Deep Alignment, respectively, versus 3.62 hours for METransformer.The comparisons were conducted on four NVIDIA A100 40G GPUs.
- Clinical and Qualitative Results: Deep and Delta Alignment achieve clinical-efficacy F1 scores of 0.389 and 0.358, surpassing METransformer’s 0.334, while Shallow Alignment misses mild pulmonary edema in a qualitative example.Delta and Deep Alignment correctly identify the abnormality in that example.
5. Conclusions
R2GenGPT applies LLMs to radiology report generation through shallow, delta, and deep visual alignment methods. The framework combines competitive SOTA performance with reduced computational complexity, positioning it as a promising approach for automated radiology reporting.
- 5. Conclusions: R2GenGPT is an LLM-based framework for Radiology Report Generation that explores shallow, delta, and deep visual alignment methods.The conclusion describes the framework as capitalizing on LLM capabilities.
- 5. Conclusions: The framework achieves competitive SOTA performance while reducing computational complexity.The conclusion presents these as the paper’s dual achievement.
- 5. Conclusions: R2GenGPT is positioned as a promising solution to automate and improve radiology reporting.This is the stated practical conclusion of the paper.