Source-linked AI summary
KiUT: Knowledge-injected U-Transformer for Radiology Report Generation
Zhongzhen Huang, Xiaofan Zhang, Shaoting Zhang
TL;DR
Handwritten radiology reports are time-consuming and error-prone, and medical report generation must account for visual, contextual, and clinical information. KiUT combines a U-connected encoder-decoder with a symptom graph and decoding-stage knowledge distillation. It achieves state-of-the-art performance on IU-Xray and MIMIC-CXR.
Problem
Handwritten reports are time-consuming and error-prone, while report generation must address dataset biases and combine visual, contextual, and clinical information.
Source: §1 · p. 1Method
KiUT uses a U-connection between encoder and decoder, a Region Relationship Encoder, and a symptom graph whose clinical, visual, and contextual information is distilled during decoding.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4Results
KiUT achieves state-of-the-art performance on the MIMIC-CXR and IU-Xray datasets, and ablation studies support the effectiveness of its proposed parts.
Source: §5 · p. 8Takeaways & Limitations
The reported U-connection improvement extends to both radiology report generation and natural image captioning.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Radiology report generation aims to automatically generate a clinically accurate and coherent paragraph from the X-ray image, which could relieve radiologists from the heavy burden of report writing. Although various image caption methods have shown remarkable performance in the natural image field, generating accurate reports for medical images requires knowledge of multiple modalities, including vision, language, and medical terminology. We propose a Knowledge-injected U-Transformer (KiUT) to learn multi-level visual representation and adaptively distill the information with contextual and clinical knowledge for word prediction. In detail, a U-connection schema between the encoder and decoder is designed to model interactions between different modalities. And a symptom graph and an injected knowledge distiller are developed to assist the report generation. Experimentally, we outperform state-of-the-art methods on two widely used benchmark datasets: IU-Xray and MIMIC-CXR. Further experimental results prove the advantages of our architecture and the complementary benefits of the injected knowledge.
1. Introduction
Handwritten radiology reports are time-consuming and error-prone, while medical report generation must address data biases and combine visual, contextual, and clinical information. KiUT uses a U-connected encoder-decoder and decoding-stage knowledge distillation, and is reported to achieve state-of-the-art performance on two benchmarks.
1. Introduction
- Hand-crafted reports are time-consuming and error-prone, although encoder-decoder architectures inspired by neural machine translation are widely used for report generation.Radiology reports support later treatment, and experienced radiologists are needed to interpret images accurately and write findings.Source: §1 · p. 1
- Radiology report generation must address case-level and region-level biases, capture abnormal regions and cross-modal interactions, and use clinical knowledge without embedding inconsistency or visual distraction.The cited work describes normal cases and normal-region descriptions as dominant in datasets and reports, respectively.Source: §1 · p. 1
- KiUT achieves state-of-the-art performance on IU-Xray and MIMIC-CXR, with ablations supporting the effectiveness of its proposed components.The contributions also report that U-connection improves radiology report generation and natural image captioning.Source: §1 · p. 2
- The model introduces a U-connection to exploit multi-level visual information and a symptom graph whose clinical knowledge is distilled with visual and contextual information during decoding.Its Region Relationship Encoder models intrinsic and extrinsic relationships among image regions to extract abnormal-region features.Source: §1 · p. 2
2. Related Work
Related work spans image-captioning architectures and methods for generating longer radiology reports. KiUT is positioned against approaches that model image regions or incorporate medical knowledge, emphasizing decoder-stage knowledge distillation.
2.1. Image Caption
- Image-captioning methods model salient regions and region relationships, but the cited discussion identifies full use of visual information and encoder-decoder interaction as remaining concerns.Prior work explored complex mesh-like connectivity; KiUT proposes a simpler connection schema.Source: §2.1 · p. 2
2.2. Radiology Report Generation
- Radiology report generation extends image captioning to longer, context-dependent descriptions, with prior work using hierarchical generation, attention, and cross-modal memory.The examples include separate normal and abnormal sentence generation, topic-guided sentence generation, image-region localization, and memory-based cross-modal alignment.Source: §2.2 · p. 2
- Prior report-generation methods use external prototypes, medical graphs, or prior and posterior knowledge; KiUT instead distills multiple modalities during decoding.The related-work discussion distinguishes this from approaches that apply knowledge to visual feature extraction.Source: §2.2 · p. 2
3. Methodology
KiUT generates a report token by token from a radiology image using a Cross-modal U-Transformer, an Injected Knowledge Distiller, and a Region Relationship Encoder.
3. Methodology
- KiUT maps a radiology image to a descriptive report and comprises a Cross-modal U-Transformer, an Injected Knowledge Distiller, and a Region Relationship Encoder.The report is generated recursively as a sequence of word tokens.Source: §3 · p. 3
3.1. Cross-modal U-Transformer
- The U-connection sends each encoder layer’s output to a corresponding decoder layer, allowing the decoder to aggregate visual features across layers before passing its last-layer output to the knowledge distiller.Encoder layer i feeds decoder layer N − i + 1.Source: §3.1 · pp. 3–4
3.2. Injected Knowledge Distiller
- The Injected Knowledge Distiller uses visual, contextual, and clinical signals, including a symptom graph initialized from classifier probabilities and processed with graph attention.The graph reflects symptom correlations, characteristics, and occurrence locations; attention distills useful information from the signals.Source: §3.2 · p. 4
3.3. Region Relationship Encoder
- The Region Relationship Encoder augments self-attention to model both spatial relationships between regions and intrinsic relationships among their features.It incorporates relative geometry for extrinsic relationships and extends the attention keys and values with learnable matrices for intrinsic relationships.Source: §3.3 · p. 5
4. Experiment
Experiments on IU-Xray and MIMIC-CXR compare KiUT with existing methods and examine its implementation, components, and generated reports. KiUT leads across MIMIC-CXR metrics and most IU-Xray metrics, with ablations and examples probing its design.
4.1. Dataset
- IU-Xray contains 7,470 chest X-ray images and 3,955 reports, while MIMIC-CXR contains 377,110 images and 227,835 reports; both use 70%/10%/20% splits.IU-Xray samples without a Findings section are excluded; MIMIC-CXR uses the official splits.Source: §4.1 · p. 5
4.4. Comparison with state-of-the-art
- KiUT outperforms state-of-the-art methods across all MIMIC-CXR metrics and most IU-Xray metrics, though some IU-Xray benchmarks remain below XPRONET [38].The comparison covers NLG and clinical-efficacy metrics on both datasets.Source: Table 1 · p. 6, §4.4 · pp. 6–7
4.2. Implementation Details
- Image regions are extracted with ImageNet-pretrained ResNet-101 and projected to 512 dimensions; IU-Xray uses paired views, while MIMIC-CXR uses one image.A pretrained DenseNet-121 generates probabilities for symptoms in the graph.Source: §4.2 · p. 6
4.3. Metrics
- Evaluation combines BLEU, METEOR, and ROUGE-L with CheXpert-derived precision, recall, and F1 across 14 disease categories.Source: §4.3 · p. 6
4.5. Ablation Studies
- Adding extrinsic and intrinsic region relationships raises base-transformer BLEU-1 from 0.373 to 0.379 and 0.382, respectively; intrinsic relationships yield the larger gain.The authors associate the gain with anatomical similarities among human organs.Source: Table 2 · p. 7, §4.5 · p. 7
- Removing both contextual and clinical features reduces performance from 0.393 to 0.337, while the two features support contextual text and abnormality descriptions, respectively.Source: Table 2 · p. 7, §4.5 · p. 7
- U-connection raises R2GenCMN BLEU-1 from 0.353 to 0.363 and achieves performance comparable to meshed connection with fewer parameters.The authors report that one-to-one encoder-decoder connections also improve on using only the last encoder layer.Source: Table 3 · p. 7, §4.5 · p. 8
4.6. Qualitative Analysis
- Qualitative examples show KiUT describing subtle observations and distinguishing different clinical manifestations of the same symptom.Examples include support devices and slight elevation of the right hemidiaphragm.Source: Figure 6 · p. 8, §4.6 · p. 8
5. Conclusion
KiUT combines multi-level visual information, region relationships, and injected knowledge to generate radiology reports. It achieves state-of-the-art performance on MIMIC-CXR and IU-Xray, while its components are supported by ablation studies.
5. Conclusion
- KiUT achieves state-of-the-art performance on MIMIC-CXR and IU-Xray and outperforms recent research from the external-knowledge view.Ablation studies also support the effectiveness of the proposed parts.Source: §5 · p. 8
- KiUT encodes images using extrinsic and intrinsic relationships among image regions and decodes words through an injected knowledge distiller.The framework focuses on extracting and distilling multi-level information and multiple kinds of injected knowledge.Source: §5 · p. 8
- Its U-connection schema exploits interaction between the encoder and decoder in this cross-modal task.The paper describes this schema as unprecedented among architectures in this scenario.Source: §5 · p. 8
- The authors leave sophisticated knowledge construction and structured report-template filling for future work.Source: §5 · p. 8