Source-linked AI summary
Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment
Shuxin Yang, Xian Wu, Shen Ge, S. Kevin Zhou, Li Xiao
TL;DR
Radiology reporting is clinically important but burdensome, and accurate generation requires disease-specific medical knowledge. The paper introduces an automatically learned knowledge base and multi-modal alignment for chest X-ray report generation; it achieves state-of-the-art performance across natural language and clinical efficacy metrics on IU-Xray and MIMIC-CXR, with both modules improving results.
Problem
Radiology reports guide clinical decision making, but writing them is time-consuming, and existing knowledge-enhanced methods require laborious manual knowledge-base construction.
Method
The model learns and stores medical knowledge automatically during training and aligns image features with report embeddings and disease labels.
Results
The model achieves state-of-the-art performance on natural language generation and clinical efficacy metrics across IU-Xray and MIMIC-CXR, while each proposed component improves performance.
Takeaways & Limitations
Automatic knowledge-base learning avoids manual construction, while multi-modal alignment and the learned knowledge base improve both natural-language quality and clinical correctness.
Takeaways & Limitations
Prior knowledge-enhanced methods may require rebuilding their knowledge graph or template database for images of other diseases, limiting migration across datasets.
Abstract
from arXiv · showhide
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical knowledge from textual embedding without manual labour; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, with the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics.
Introduction
Radiology report generation must capture disease-specific image information while reducing the time burden on radiologists. The paper addresses this with an automatically learned knowledge base and multi-modal alignment, achieving strong results on IU-Xray and MIMIC-CXR.
- Radiology reports support clinical decision making but are time-consuming and tedious for radiologists to write.
- Unlike image captioning, radiology report generation must identify abnormal regions and describe disease-specific information accurately.
- Existing knowledge-enhanced methods require manually constructed knowledge graphs or template databases, limiting labor efficiency and dataset migration.
- The learned knowledge base automatically stores medical knowledge during training through a knowledge updating mechanism without manual labor.
- Multi-modal alignment aligns image features with report embeddings and disease labels so visual learning focuses on clinically important abnormalities.
- The model achieves state-of-the-art performance on almost all natural-language-generation and clinical-efficacy metrics across IU-Xray and MIMIC-CXR.
Related Work
Prior work adapts image-captioning architectures, injects manually constructed medical knowledge, or uses memory mechanisms for report generation. The proposed approach differs by learning its knowledge base automatically from complete medical reports during training.
- Early radiology report generation methods adapt CNN-RNN or encoder-decoder image-captioning architectures to describe detected diseases.
- HRGR, KERP, MKG, and PPKED inject manually extracted templates or pre-constructed knowledge graphs into report generation.
- R2Gen uses relational memory built from already generated words, whereas this paper builds memory from the entire medical report.
- Unlike KVMN, the proposed knowledge base is learned from scratch during training without manual labor and supports report generation rather than question-answer classification.
Method
The model learns a knowledge base from reference reports during training and uses it with visual features to generate reports during inference. Multi-modal alignment connects visual, textual, and disease-label representations to guide knowledge storage and report generation.
- Framework: During training, the model uses images, disease labels, and reference reports; during inference, only the image is required.The knowledge base is learned during training and fixed for inference.
- Visual Encoding: A CNN extracts visual features V from the image, and average pooling produces the aggregated feature zimg.V contains K visual features of dimension D, while zimg also has dimension D.
- Text Encoding: A Transformer-based report encoder extracts textual embeddings from reference reports, using the CLS representation to obtain ztxt.The textual embedding T is derived from the final hidden states, and ztxt is transformed to dimension D.
- Knowledge Base Updating: The knowledge base is initialized as a memory matrix, updated with missing knowledge identified from reference-report features, and normalized after integrating incremental knowledge.The memory is updated during training and then fixed for inference.
- Knowledge Retrieval: Multi-head attention retrieves supporting knowledge from the knowledge base using the pooled visual feature zimg.The retrieved support is denoted MS and is passed to the report generator with visual features.
- Report Generation and Alignment: The Transformer generator decodes visual features concatenated with supporting knowledge into a report through autoregressive decoding.The alignment module additionally combines textual-textual and visual-textual objectives within the training framework.
Experiment
The experiments evaluate the proposed report-generation model on IU-Xray and MIMIC-CXR using natural language generation and clinical efficacy metrics. Ablation studies examine alignment methods, knowledge-base updating, and knowledge-base size, while qualitative examples illustrate generated-report differences.
- Evaluation Setup: The model is evaluated on IU-Xray and MIMIC-CXR, using NLG metrics and clinical efficacy metrics derived from disease-label agreement.MIMIC-CXR contains 377,110 images and 227,827 reports; IU-Xray contains 7,470 images and 3,955 reports.
- Quantitative Results: The model achieves state-of-the-art performance on almost all metrics, including 26.1% higher precision, 24.2% higher recall, and 27.5% higher F1 score.These clinical-efficacy improvements are reported relative to previous models.
- Ablation Study: Combining visual-textual and visual-label alignment produces the best performance, while either alignment alone outperforms the model without alignment.The two alignment mechanisms are reported to assist each other in improving report quality.
- Ablation Study: Increasing the number of knowledge-base updating heads from 1 to 8 consistently improves performance across the tested settings.The authors associate this trend with greater expressive ability to learn features from different sub-spaces.
- Ablation Study: Knowledge-base size has dataset-dependent effects: performance peaks at size 30 on IU-Xray but improves continuously with larger sizes on MIMIC-CXR.Compared with a model without a knowledge base, the proposed model’s performance is significantly improved.
- Qualitative Results: Qualitative MIMIC-CXR examples show the proposed model recovering disease descriptions and support-device information that the basic model misses.The examples include pleural effusions, atelectasis, cardiomegaly, and support-device descriptions.
Conclusion
The proposed model combines a learned knowledge base with multi-modal alignment and achieves state-of-the-art performance on IU-Xray and MIMIC-CXR. Ablations support the effectiveness of the alignment component, while larger knowledge bases and more updating heads improve performance.
- The model achieves state-of-the-art performance on both IU-Xray and MIMIC-CXR using natural language generation and clinic efficacy metrics.
- Ablation studies show that multi-modal alignment effectively guides visual-feature learning.
- Performance improves when the knowledge-base updating mechanism uses more heads and a relatively large knowledge base.
- The knowledge-base updating method can automatically build knowledge or template bases across datasets without manual labor.