Source-linked AI summary
Generating Radiology Reports via Memory-driven Transformer
Zhihong Chen, Yan Song, Tsung-Hui Chang, Xiang Wan
TL;DR
Radiology reports are time-consuming to write and difficult to generate automatically because they are long clinical narratives. The paper introduces a memory-driven Transformer that records generation patterns and incorporates them into the decoder. On IU X-Ray and MIMIC-CXR, the approach achieves state-of-the-art performance across language-generation and clinical-efficacy evaluations, while producing long reports with medical terms and meaningful image-text attention mappings.
Problem
Radiology report generation must produce long clinical narratives, while retrieval-based methods require large databases or explicitly constructed template lists.
Method
A memory-driven Transformer uses relational memory to record prior generation information and memory-driven conditional layer normalization to integrate it into the decoder.
Results
The full model achieves state-of-the-art performance on IU X-Ray and MIMIC-CXR across natural-language-generation and clinical-efficacy metrics.
Takeaways & Limitations
The approach generates long reports with necessary medical terms and produces meaningful image-text attention mappings.
Takeaways & Limitations
Retrieval-based alternatives remain constrained by the need for large databases or explicitly constructed template lists.
Abstract
from arXiv · showhide
Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory-driven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.
1 Introduction
Radiology report generation aims to automate free-text descriptions of clinical radiographs, but reports are long, multi-section narratives that challenge conventional image-captioning methods. The paper addresses this challenge with a memory-driven Transformer and evaluates it on benchmark datasets.
- Radiology report generation seeks to automatically produce free-text descriptions for clinical radiographs.
- Reports typically contain findings describing normal and abnormal observations, followed by an impression summarizing prominent observations.This multi-section structure contributes to the task’s complexity.
- Conventional image-captioning approaches may be insufficient because they are designed for brief visual descriptions rather than long radiology narratives.
- Retrieval-based approaches are limited by the need for large databases or explicitly constructed template lists capturing report patterns.
- The paper proposes a memory-driven Transformer with relational memory and memory-driven conditional layer normalization for radiology report generation.The relational memory records prior generation information, while conditional normalization incorporates it into the Transformer decoder.
2 The Proposed Method
The proposed memory-driven Transformer generates radiology reports by integrating a relational memory into the Transformer decoder through memory-driven conditional layer normalization. The model uses recurrent pattern memory, gated updates, and memory-conditioned decoding to support long, informative reports.
- Model Structure: Radiology images are encoded as source features, processed by a standard Transformer encoder, and decoded into report token sequences.The visual extractor uses pretrained CNNs, while the encoder produces hidden states for the decoder.
- Model Structure: The decoder adds a relational memory and memory-driven conditional layer normalization to each Transformer decoding layer.The relational memory records information from previous generation steps, and MCLN incorporates its output into Transformer normalization.
- Relational Memory: Relational memory transfers states across generation steps, using multi-head attention to model relations among report patterns and previous outputs.The previous memory state serves as the query, while its concatenation with the previous output supplies the key and value.
- Relational Memory: Residual connections and a gated update mechanism balance previous memory states with new decoder outputs during recurrent memory updates.Forget and input gates balance contributions from the prior memory and previous output, respectively.
- Experimental Evaluation: The model is evaluated on IU X-RAY and MIMIC-CXR using natural-language-generation metrics and, for MIMIC-CXR, CheXpert-based clinical-efficacy metrics.Table 1 summarizes dataset statistics, while Table 2 reports baseline and full-model performance averaged over five runs.
- Memory-driven Conditional Layer Normalization: MCLN feeds relational-memory outputs into the scaling and shifting parameters of layer normalization while limiting their influence on other Transformer parameters.At each decoding layer, the memory is expanded and processed to predict changes to normalization parameters.
3 Experiment Settings
Experiments evaluate the proposed model on IU X-RAY and MIMIC-CXR against Transformer, memory, captioning, and medical-domain baselines using language-generation and clinical-efficacy metrics.
- Datasets: Experiments use IU X-RAY and MIMIC-CXR, applying conventional dataset splits after excluding samples without reports.IU X-RAY contains 7,470 images and 3,955 reports; MIMIC-CXR contains 473,057 images and 206,563 reports from 63,478 patients.
- Baselines: The main baselines are vanilla Transformer and BASE+RM, which concatenates relational memory with Transformer outputs before softmax.BASE uses three layers, 8 heads, and 512 hidden units; BASE+RM tests memory as an added component rather than integrating it within Transformer.
- Baselines: Additional comparisons include conventional image-captioning models and medical-domain report-generation models.Compared models include ST, ATT2IN, ADAATT, TOPDOWN, COATT, HRGR, and CMAS-RL.
- Evaluation Metrics: Performance is evaluated with BLEU, METEOR, and ROUGE-L for natural-language generation, plus CheXpert-based precision, recall, and F1 for clinical efficacy.Clinical-efficacy metrics compare generated reports with ground truths across 14 thoracic-disease and support-device categories.
- Evaluation Metrics: Clinical-efficacy metrics apply only to MIMIC-CXR because the CheXpert labeling schema differs from that of IU X-RAY.The evaluation uses CheXpert labels for generated reports and ground truths.
4 Results and Analyses
The memory-driven models improve report generation across language and clinical metrics, while analyses show that memory size, integration strategy, report length, medical terminology, and image-text alignment affect generation quality.
- Benchmark results: BASE+RM and BASE+RM+MCLN outperform vanilla Transformer on NLG metrics across IU X-RAY and MIMIC-CXR, with the full model best across metrics.On MIMIC-CXR, the full model also outperforms baselines on clinical precision, recall, and F1.
- Comparison with previous studies: Transformer-based models outperform sequence-to-sequence and conventional image-captioning baselines, supporting task-specific modeling for long radiology reports.The full model also matches the goal of reinforcement learning with a simpler method and exceeds models using manually extracted templates or extra information.
- Memory size: Three memory slots achieve the best overall performance, whereas four reduce performance by introducing redundant or invalid information.Adding slots increases parameter counts only modestly relative to the total model size.
- Report length: BASE+RM and BASE+RM+MCLN generate more long reports than BASE, and MCLN produces a length distribution closer to the ground truth.The authors attribute this to integrating memory throughout Transformer layers rather than only at the final output.
- Case study: Qualitative examples show that BASE+RM+MCLN covers nearly all necessary medical terms that vanilla Transformer misses.Examples include “enlarged cardiac silhouette,” “atelectasis,” and “small pleural effusion.”
- Attention analysis: BASE+RM+MCLN better aligns image locations with indicated diseases or anatomical parts in image-text attention mappings.The analysis suggests improved interaction between chest X-ray images and generated text.
5 Related Work
Radiology report generation differs from ordinary image captioning because it requires longer outputs and must model recurring report patterns. Prior work uses hierarchical sequence models, templates, or reinforcement learning, while this paper presents memory for Transformer decoding as an alternative.
- Task characteristics: Radiology report generation requires longer outputs and has recurring patterns that distinguish it from standard image captioning.These characteristics motivate specialized solutions rather than direct application of short-captioning methods.
- Previous approaches: Prior approaches include hierarchical LSTMs with co-attention, manually extracted template databases, and reinforcement learning for clinical accuracy.Template-based methods require substantial databases or explicitly constructed template lists.
- Positioning of this work: The proposed approach uses relational memory for Transformer decoding, learning from previous generation processes and patterns for long-text generation.The authors distinguish this decoder-focused use of memory from related memory methods applied primarily to encoding.
6 Conclusion
The memory-driven Transformer achieves state-of-the-art radiology report generation by integrating relational memory into the decoder. Analyses show effects of memory size and support generation of long, medically informative reports with meaningful attention mappings.
- The memory-driven Transformer uses relational memory to record prior generation information and integrates it into Transformer layer normalization.
- The approach achieves state-of-the-art performance on two benchmark radiology report datasets.
- Analyses examine how memory size affects model performance.
- The model generates long reports containing necessary medical terms and meaningful image-text attention mappings.