Source-linked AI summary
Cross-modal Memory Networks for Radiology Report Generation
Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan
TL;DR
Radiology report generation aims to automate free-text descriptions of radiographs, but prior encoder-decoder methods have limited explicit use of image-text mappings. CMN adds a shared memory that records cross-modal alignment and supplies queried responses to the encoder and decoder. The approach achieves state-of-the-art performance on IU X-RAY and MIMIC-CXR and is reported to better align image and text information for report generation.
Problem
Prior radiology report generation studies mainly focus on text generation with encoder-decoder models and few explicitly exploit mappings between images and reports.
Method
CMN uses a shared memory matrix to record image-text alignment, query related memory vectors, and provide responses to the encoder and decoder.
Results
State-of-the-art performance is achieved on the IU X-RAY and MIMIC-CXR benchmark datasets.
Takeaways & Limitations
Further analyses report better alignment of radiology image and text information and more accurate reports in terms of clinical indicators.
Takeaways & Limitations
CheXpert clinical-evaluation metrics apply only to MIMIC-CXR because their labeling schema differs from IU X-RAY.
Abstract
from arXiv · showhide
Medical imaging plays a significant role in clinical practice of medical diagnosis, where the text reports of the images are essential in understanding them and facilitating later treatments. By generating the reports automatically, it is beneficial to help lighten the burden of radiologists and significantly promote clinical automation, which already attracts much attention in applying artificial intelligence to medical domain. Previous studies mainly follow the encoder-decoder paradigm and focus on the aspect of text generation, with few studies considering the importance of cross-modal mappings and explicitly exploit such mappings to facilitate radiology report generation. In this paper, we propose a cross-modal memory networks (CMN) to enhance the encoder-decoder framework for radiology report generation, where a shared memory is designed to record the alignment between images and texts so as to facilitate the interaction and generation across modalities. Experimental results illustrate the effectiveness of our proposed model, where state-of-the-art performance is achieved on two widely used benchmark datasets, i.e., IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is able to better align information from radiology images and texts so as to help generating more accurate reports in terms of clinical indicators.
1 Introduction
Radiology report generation can reduce radiologists’ manual workload, but conventional encoder-decoder studies have limited use of mappings between radiology images and reports. The paper proposes cross-modal memory networks to explicitly facilitate interactions across visual and textual modalities.
- Radiology report generation automatically produces free-text descriptions from radiographs to ease radiologists’ workload while maintaining health-care quality.
- Most existing studies use CNN encoders with recurrent or non-recurrent decoders following the image-captioning paradigm.
- These methods remain constrained in fully exploiting mappings between radiology images and their corresponding reports.
- The proposed cross-modal memory networks use a memory matrix to query and respond to visual and textual features, feeding shared responses into the encoder and decoder.
- Experiments on IU X-RAY and MIMIC-CXR report state-of-the-art performance, while further analyses examine the model’s alignment of image and text information.
2 The Proposed Approach
The proposed approach treats radiology report generation as image-to-text sequence generation and enhances a Transformer encoder-decoder with cross-modal memory responses. CMN stores shared image-text information, queries related memory vectors, and feeds visual and textual responses into generation.
- Problem formulation: Radiology report generation is formulated as sequence-to-sequence image-to-text generation, with visual features as the source sequence and report tokens as the target.The source contains visual features extracted from an image, while the target contains generated report tokens.
- Cross-modal memory networks: CMN uses a shared memory matrix to record cross-modal information connecting images and texts.The matrix contains N memory vectors, each with dimension d, and serves as an intermediate medium for cross-modal mapping.
- Memory querying: Memory querying transforms visual, textual, and memory features into a common representation space before selecting related memory vectors and assigning normalized importance weights.The number of queried vectors is controlled by K, and querying is performed across multiple representation subspaces.
- Memory responding: Memory responding linearly transforms queried memory vectors and combines them using the visual and textual querying weights to produce modality-specific responses.The responding process is applied across threads to obtain responses from different memory representation subspaces.
- Encoder-decoder: The Transformer encoder receives visual memory responses, while the decoder combines encoded visual states with textual memory responses from previously generated tokens.The decoding process repeats until the complete report is generated.
3 Experiment Settings
The experiments evaluate CMN-based report generation on IU X-RAY and MIMIC-CXR using Transformer-based baselines, NLG metrics, and clinical efficacy metrics. The setup follows prior work while applying CheXpert-based clinical evaluation only to MIMIC-CXR.
- Datasets: Experiments use IU X-RAY and MIMIC-CXR, generating only the findings section with established dataset splits.IU X-RAY uses a 70%/10%/20% train/validation/test split, while MIMIC-CXR uses its official split.
- Baselines: BASE is a three-layer, eight-head Transformer with 512 hidden units, while BASE+MEM adds separate memory networks for images and text.BASE+MEM provides a reference for evaluating cross-modal memory.
- Evaluation Metrics: Table 2 reports NLG and clinical efficacy evaluations on the test sets and includes average NLG improvement over BASE.BL-n denotes BLEU through 4-grams, while MTR and RG-L denote METEOR and ROUGE-L.
- Baselines and Evaluation Metrics: The comparison includes conventional image-captioning models and medical-domain systems such as R2GEN, evaluated with NLG and clinical efficacy metrics.NLG metrics are BLEU, METEOR, and ROUGE-L; clinical efficacy uses CheXpert precision, recall, and F1 across 14 categories.
- Implementation: The implementation uses ResNet101 patch features and a three-layer, eight-head Transformer, with two IU X-RAY images and one MIMIC-CXR image per patient.The visual features have 512 dimensions and are extracted from ResNet101 pretrained on ImageNet.
- Evaluation Scope: Clinical efficacy metrics apply only to MIMIC-CXR because CheXpert’s labeling schema differs from that of IU X-RAY.This restricts cross-dataset comparison of clinical efficacy results.
4 Results and Analyses
Across IU X-RAY and MIMIC-CXR, CMN improves radiology report generation over memory-free and separate-memory baselines, while analyses examine memory capacity, querying, and image-text alignment.
- Cross-modal memory effectiveness: BASE+CMN outperforms BASE and BASE+MEM on NLG metrics across IU X-RAY and MIMIC-CXR, with particularly large gains over BASE+MEM.The comparison supports using a shared cross-modal structure rather than separate memory representations.
- Cross-modal memory effectiveness: BASE+CMN also outperforms its baselines on MIMIC-CXR clinical efficacy metrics in precision, recall, and F1.The reported size-effect discussion links these gains to the dataset and its visual-textual mappings.
- Comparison with previous studies: Cross-modal memory outperforms COATT, while using a shared memory matrix to softly align visual and textual features in one representation space.The authors also report better results than more complicated approaches using templates or reinforcement learning, and than R2GEN, which applies memory only during decoding.
- Memory size: Within a relatively small memory, increasing memory vectors improves overall performance up to N ≤1024; beyond a threshold, further enlargement no longer improves it.The analysis attributes the initial benefit to storing more cross-modal information.
- Queried memory vectors: BLEU-4 improves as queried vectors increase through K ≤32, then drops; the reported optimal number of queried vectors is K=32.The proposed explanation is over-updating of essential memory vectors when querying becomes intensive.
- Image-text alignment: Qualitative visualizations compare image regions with report words and phrases, while t-SNE shows lung and hemidiaphragm features querying similar memory vectors.These analyses support memory vectors as an intermediate medium for image-text interaction.
5 Related Work
Related work frames radiology report generation as a medical extension of image captioning and includes encoder-decoder, co-attention, and memory-based approaches. CMN differs by using a soft intermediate layer to align visual and textual features.
- Memory-based approaches: Prior memory-based methods model patternized information or visual prior knowledge, but some apply memory only during encoding.The cited comparison distinguishes CMN’s cross-modal alignment role from encoding-only memory operations.
- Radiology report generation: Radiology report generation extends image captioning by describing radiology images in professional free-text reports.The broader image-captioning task is described as a cross-modal natural-language and computer-vision problem.
- Existing approaches: Most existing radiology report generation studies use CNN encoders with recurrent or non-recurrent decoders following the image-captioning paradigm.Jing et al. is identified as a related exception using multi-task learning and co-attention to link image parts with reports.
- CMN’s distinction: CMN provides a soft intermediate layer for mappings between visual and textual features, offering an effective yet simple alternative that produces more accurate descriptions.Its distinction from earlier studies is the explicit facilitation of visual-textual mappings rather than memory use limited to one modality or stage.
6 Conclusion
The proposed cross-modal memory network records image–text alignment and interaction in a shared memory, achieving state-of-the-art performance on two benchmark datasets. Analyses indicate improved alignment supports more accurate report generation, while enlarging the memory matrix does not significantly increase model size.
- CMN uses a memory matrix to record alignment and interaction between radiology images and texts.Memory querying and responding obtain shared cross-modal information.
- The model achieves state-of-the-art performance on two benchmark datasets.The conclusion identifies this as evidence of the model’s effectiveness.
- Better image–text alignment helps CMN generate more accurate radiology reports, especially according to clinical indicators.
- Enlarging the memory matrix does not significantly affect the model’s overall size.