PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

Shuxin Yang, Xian Wu, Shen Ge, Shaohua Kevin Zhou, Li Xiao

arXiv:2112.15009v2eess.IVcs.CLcs.CV

TL;DR

Radiology report generation is time-consuming and error-prone, while encoder-decoder methods can exhibit visual-textual bias and lack expert knowledge. The paper combines general and image-specific knowledge with knowledge-enhanced multi-head attention, achieving state-of-the-art results across IU-Xray and MIMIC-CXR evaluations. A major limitation is the labor required to build and rebuild the knowledge graph for new datasets.

  • Problem

    Existing encoder-decoder report generators can favor normal descriptions and cannot precisely incorporate expert knowledge, despite the clinical importance of efficient and accurate reporting.

  • Method

    The framework combines broad input-independent knowledge from a pre-constructed graph with fine-grained image-dependent knowledge retrieved from similar reports using knowledge-enhanced multi-head attention.

  • Results

    The model achieves state-of-the-art performance on various natural-language-generation and clinical-efficacy metrics for IU-Xray and MIMIC-CXR, with consistent improvements from its modules.

  • Takeaways & Limitations

    Both general and specific knowledge benefit radiology report generation, while the framework is intended to support report automation and radiologist assistance.

  • Takeaways & Limitations

    Building the knowledge graph is laborious, making direct transfer to other datasets difficult because the graph must be rebuilt.

Abstract

from arXiv · show

Automatic radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two types of medical knowledge: 1) General knowledge, which is input independent and provides the broad knowledge for report generation; 2) Specific knowledge, which is input dependent and provides the fine-grained knowledge for report generation. To fully utilize both the general and specific knowledge, we also propose a knowledge-enhanced multi-head attention mechanism. By merging the visual features of the radiology image with general knowledge and specific knowledge, the proposed model can improve the quality of generated reports. Experimental results on two publicly available datasets IU-Xray and MIMIC-CXR show that the proposed knowledge enhanced approach outperforms state-of-the-art image captioning based methods. Ablation studies also demonstrate that both general and specific knowledge can help to improve the performance of radiology report generation.

1. Introduction

Automatic radiology report generation addresses time-consuming, error-prone reporting, but encoder-decoder approaches can miss abnormalities and lack expert knowledge. The paper introduces general and specific knowledge with knowledge-enhanced attention and reports strong performance on IU-Xray and MIMIC-CXR.

  • Motivation: Radiology reporting takes at least 10 minutes on average and can miss important findings, motivating automated report generation.The paper describes workload relief for experienced radiologists and reminders about potential abnormalities for inexperienced radiologists.
  • Limitations of existing methods: Encoder-decoder report generators can favor normal descriptions because abnormal regions and their report descriptions occupy small fractions of the image and text.These approaches also lack precise incorporation of expert knowledge.
  • Knowledge gap: Existing knowledge-enhanced methods use limited manually constructed templates or knowledge graphs, whereas RadGraph contains 6 million entities.One cited pre-constructed graph contains only 20 entities, which the paper describes as insufficient for radiology report generation.
  • Results: Experiments on IU-Xray and MIMIC-CXR report state-of-the-art performance on natural-language-generation and clinical-efficacy metrics, with benefits from both knowledge types.The introduction also reports consistent improvements when adding the proposed modules.
  • Proposed approach: The framework adds input-independent general knowledge from a pre-constructed graph and input-dependent specific knowledge retrieved from reports with similar label distributions.This pairing supplies broad and fine-grained knowledge for report generation.
  • Proposed approach: Knowledge-enhanced multi-head attention combines structural knowledge information with visual features.The mechanism is designed to merge the visual representation with both general and specific knowledge.

2. Related Works

Related work mainly uses image-captioning architectures, attention, curriculum learning, or injected medical knowledge. These approaches address reporting and data-bias challenges but can depend on restricted disease categories, normal references, extra extraction work, or limited knowledge resources.

  • Image-based generation: CNN-RNN report generation describes detected diseases from visual features but is restricted to predefined disease categories.
  • Image-based generation: Attention-based methods improve visual-text generation, while normal-image retrieval methods are restricted by the quality of their normal input images.
  • Bias and sequence modeling: Curriculum learning and AlignTransformer methods target limited data, data bias, and long-sequence modeling in radiology report generation.
  • Multi-stage generation: A two-stage generation approach incorporates high-level concepts but requires extra labor to extract high-level context for first-stage guidance.
  • Knowledge-enhanced generation: HRGR uses manually extracted templates, while KERP, MKG, and PPKED combine report generation with pre-constructed knowledge graphs.

3. Method

The method section presents a radiology report-generation framework that integrates general and specific medical knowledge into the modeling pipeline. It organizes the proposed approach into an overview, knowledge integration mechanisms, and report-generation components.

  • Method overview: The proposed method uses both general and specific knowledge for radiology report generation.The section structure covers notation and overview, general-knowledge integration with enhanced attention, specific-knowledge integration, and multimodality semantic attention.

3.1. Overview

The overview defines the image-to-report task and describes a four-module architecture: visual extraction, general-knowledge embedding, specific-knowledge retrieval and embedding, and report generation. General knowledge comes from RadGraph, while specific knowledge is retrieved from visually similar reports.

  • Task formulation: A radiology image Img ∈ R^C×H×W is mapped to a report sequence W = {w_1, ..., w_T} describing normal and abnormal findings.C, H, and W denote image channels, height, and width; each w_t is a vocabulary word index.
  • Knowledge representation: A knowledge graph represents entities as nodes and relationships as edges in a multi-relational semantic network.The paper illustrates concepts such as opacity and suggestive of as connected graph elements.
  • General knowledge: General knowledge uses RadGraph, manually built by board-certified radiologists from 500 MIMIC-CXR reports and represented as source-entity, relation, target-entity triplets.
  • Specific knowledge: Because general knowledge may be too broad for a particular image, the method retrieves visually similar reports and extracts triplets as image-specific knowledge.
  • Architecture: The architecture contains four modules: visual feature extraction, general-knowledge embedding, specific-knowledge retrieval and embedding, and report generation.The report generator fuses general and specific knowledge to generate the radiology report.
  • Visual feature extraction: The visual feature extraction module uses a CNN to extract visual features and predict disease-label distributions from one or multiple x-ray views.Its feature representation is I = CNN(Img), with I ∈ R^k×d.
  • General knowledge embedding: The general-knowledge embedding module obtains entity and relation embeddings with RotatE and attends to visual features using knowledge-enhanced multi-head attention.

3.2. General Knowledge Embedding

The model embeds general radiology knowledge from RadGraph and combines entity, relation, and visual information through knowledge-enhanced multi-head attention. This preserves structural information in the knowledge graph while augmenting visual representations for report generation.

  • General knowledge representation: RadGraph supplies general medical knowledge as triplets representing radiology entities and their relations.The knowledge graph is manually built by board-certified radiologists.
  • General knowledge representation: RotatE produces entity and relation embeddings from the manually built radiology knowledge graph.Entity and relation embeddings represent the graph components used by subsequent attention modules.
  • Knowledge-enhanced attention: Knowledge-enhanced attention incorporates structural relation information into multi-head attention over visual and knowledge features.The mechanism is formulated as a knowledge-enhanced attention function and extended to multi-head attention.
  • Knowledge-enhanced attention: The method aggregates relation embeddings between entity pairs rather than using entity embeddings alone.Multiple relations between two entities are averaged to form the aggregate relation embedding.
  • Knowledge-enhanced attention: The resulting general-knowledge-augmented visual representations include both entity embeddings and structural relation information.Layer normalization is applied to embeddings before fusion.

3.3. Specific Knowledge Retrieval & Embedding

Specific knowledge is retrieved for each input image by finding similar training records and mining radiology facts from their reports. The retrieved triplets are encoded with ClinicalBERT and fused with visual features.

  • Specific knowledge retrieval: The method retrieves image-specific knowledge because general knowledge may be too broad for an individual radiology image.Specific knowledge is obtained through similar-report retrieval followed by knowledge mining.
  • Similar record retrieval: Similar records are selected from a training-set repository using visual disease-label distributions and KL-divergence.The repository contains image-report pairs, and the input distribution is compared with repository distributions.
  • Knowledge mining: Named entities are extracted from the reports of the top-k retrieved records before querying specific knowledge.Stanza is used for named-entity recognition; an example extracts pneumothorax, pleural, and effusion.
  • Knowledge mining: The extracted entities query RadGraph triplets automatically extracted from MIMIC-CXR training reports.This triplet collection contains 6M entities and 4M relations and is larger than the manually built subset.
  • Specific knowledge embedding: All acquired triplets are concatenated into one sentence and encoded with ClinicalBERT to obtain image-specific knowledge representations.The pooled ClinicalBERT output is denoted as E_s and is fused into visual features through multi-modality semantic attention.
  • Specific knowledge embedding: The specific-knowledge attention uses zero structural bias because the knowledge is organized as a fact sequence containing structural information.The resulting representation is passed through the knowledge-enhanced multi-head attention module.

3.4. Radiology Report Generator

The report generator concatenates visual features with general and specific knowledge and uses a Transformer decoder to produce the report autoregressively. Training optimizes the negative conditional log-likelihood of the report given the image and knowledge.

  • Report generation: A standard Transformer decoder generates reports from concatenated visual features, general knowledge, and specific knowledge.The three modalities are combined before decoding.
  • Report generation: The generator produces the medical report sequence autoregressively from a conditional distribution.Each report sequence is generated incrementally as a sequence of words.
  • Training objective: The whole model is optimized with the negative conditional log-likelihood of the report given the image and its general and specific knowledge.The objective trains report generation conditioned on all three information sources.

4. Experiments and Analysis

Experiments on IU-Xray and MIMIC-CXR evaluate the knowledge-enhanced model with dataset-specific, language-generation, clinical-efficacy, qualitative, and ablation analyses. The results consistently support incorporating general knowledge, specific knowledge, and relation-aware attention.

  • Datasets and metrics: The evaluation uses IU-Xray and MIMIC-CXR, with NLG metrics including BLEU-n, CIDEr, METEOR, and ROUGE-L.Clinical efficacy metrics are additionally used on MIMIC-CXR to assess medically relevant terminology.
  • Quantitative results: 0.178 versus 0.173 BLEU-4 on IU-Xray and 0.115 versus 0.107 on MIMIC-CXR were reported against the state-of-the-art model.On IU-Xray, the proposed method leads on all reported metrics except a slightly lower ROUGE-L than M2TR.
  • Quantitative results: 37.5% higher Precision, 27.5% higher Recall, and 34.4% higher F1-Score were reported against R2Gen on MIMIC-CXR clinical-efficacy metrics.These metrics compare critical radiology terminology extracted from generated and reference reports.
  • Ablation study: Removing KEMHA caused a significant performance drop on both datasets, supporting aggregation of relation embeddings between knowledge-graph entities.The ablation contrasts relation-aware knowledge-enhanced attention with regular multi-head attention using entity embeddings only.
  • Ablation study: Removing either general or specific knowledge reduced performance, while an incomplete knowledge graph could introduce bias and cause performance degeneration.General knowledge supplies standard radiology terms, whereas specific knowledge retrieves target-related information from similar records.
  • Ablation study: BLEU-4 values of 0.168, 0.171, and 0.178 for TransE, TransR, and RotatE show that embedding choice affects performance, while all outperform the no-knowledge-graph model.The corresponding CIDEr values are 0.368, 0.371, and 0.382 on IU-Xray.

5. Discussion and Conclusion

The framework combines general and image-dependent specific knowledge with visual features through knowledge-enhanced multi-head attention, improving report generation on two public datasets. The authors note that laborious knowledge-graph construction limits transferability and that findings generation is only a preliminary step toward full report automation.

  • Contributions: The proposed framework combines general knowledge, specific knowledge, and visual features through a knowledge-enhanced multi-head attention mechanism.General knowledge is common across reports, whereas specific knowledge is customized to the current image.
  • Results: State-of-the-art performance is achieved across various metrics on both IU-Xray and MIMIC-CXR.The paper also reports consistent performance improvements from its proposed modules.
  • Limitations: Building the knowledge graph is laborious, making direct transfer to other datasets difficult because the graph must be rebuilt.The authors identify this limitation as also affecting several other knowledge-based report-generation models.
  • Limitations: Generating findings is only a preliminary attempt because typical radiology report generation consists of multiple steps.The authors propose future work on a more generalizable model that automatically learns and stores medical knowledge during training.

Appendix

The appendix compares reference reports with reports generated by the proposed and basic models, using BLEU-4 scores and highlighted findings. Across the displayed cases, the proposed model often has higher BLEU-4 scores, though the comparison is not uniformly favorable.

  • Case comparison: Each appendix case presents a radiologist reference report, the proposed model’s report, and the basic model’s report.BLEU-4 scores are provided for comparison, with positive findings highlighted in blue and false-negative findings in red.
  • Case examples: 0.611 / 0.139 BLEU-4 is reported for Case 2 for the proposed and basic models, respectively.The proposed report reproduces several reference findings, including sternotomy wires, mediastinal clips, and absence of focal consolidation, effusion, or pneumothorax.
  • Case examples: 0.518 / 0.383 BLEU-4 is reported for Case 3 for the proposed and basic models, respectively.The reports concern unchanged findings, left hemidiaphragm elevation, and associated atelectasis.
  • Case examples: 0.505 / 0.400 BLEU-4 is reported for Case 4 for the proposed and basic models, respectively.The reference describes low lung volumes, cardiomegaly, retrocardiac atelectasis, and no pneumonia; the proposed report mentions mild fluid overload.
  • Case examples: 0.473 / 0.0 BLEU-4 is reported for Case 5, while Cases 6 and 7 report 0.0 / 0.0 and 0.0 / 0.088, respectively.These cases include findings such as no pneumonia, emphysema, compression deformity, interstitial markings, and absence of pleural effusion.
Loading 2112.15009v2…