Source-linked AI summary
When Radiology Report Generation Meets Knowledge Graph
Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan Yuille, Daguang Xu
TL;DR
Radiology report generation requires accurate disease mentions and clinically meaningful evaluation beyond N-gram overlap. The paper uses a prior-knowledge chest abnormality graph to support classification and report generation, and reports better language and MIRQI performance on IU-RR.
Problem
Radiology reporting prioritizes accurate disease mentions and associated attributes, which conventional N-gram metrics may not measure reliably.
Method
A prior-knowledge chest abnormality graph learns node-specific and relational features that support classification and report generation through graph convolution and attention.
Results
The model outperforms previous approaches in language and MIRQI metrics on the IU-RR chest-radiograph dataset, with about 2% average AUC improvement and 1.3%-1.8% MIRQI improvement.
Takeaways & Limitations
Graph-based prior knowledge supports report generation that is more accurate from both language and clinical-correctness perspectives.
Takeaways & Limitations
The study uses a defined 20-keyword graph, while more sophisticated disease relationships and graph structures remain future considerations.
Abstract
from arXiv · showhide
Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting quality should focus more on matching the disease keywords and their associated attributes instead of counting the occurrence of N-gram. Based on these concerns, we propose to utilize a pre-constructed graph embedding module (modeled with a graph convolutional neural network) on multiple disease findings to assist the generation of reports in this work. The incorporation of knowledge graph allows for dedicated feature learning for each disease finding and the relationship modeling between them. In addition, we proposed a new evaluation metric for radiology image reporting with the assistance of the same composed graph. Experimental results demonstrate the superior performance of the methods integrated with the proposed graph embedding module on a publicly accessible dataset (IU-RR) of chest radiographs compared with previous approaches using both the conventional evaluation metrics commonly adopted for image captioning and our proposed ones.
Introduction
Radiology report generation aims to reduce radiologists’ workload, but medical reporting requires accurate disease mentions and evaluation beyond N-gram overlap. The proposed graph-based system improves classification and report-generation performance on IU-RR.
- Motivation: Radiology reporting is laborious and error-prone, motivating systems that provide candidate reports for radiologist verification.Learning from free-text reports also supports data-hungry machine-learning approaches.
- Approach: The proposed model embeds a prior-knowledge graph of chest findings to learn dedicated node features and relationships for classification and report generation.CNN features initialize graph nodes, graph convolutions propagate information, and separate classifier and decoder branches use the resulting features.
- Motivation: Natural-image captioning metrics may misrepresent medical-report quality because positive and negated disease mentions are clinically important.BLEU, ROUGE, and CIDEr mainly match N-gram occurrence, weighting individual words equally.
- Results: 2% average AUC improvement was achieved for classification on the publicly accessible IU-RR dataset.The model performed better in most categories than previous approaches.
- Results: 1.3%-1.8% improvement on MIRQI metrics accompanied better or equivalent conventional language-metric performance for report generation.The proposed metric evaluates disease-reporting correctness more directly than conventional captioning metrics.
Related Works
Prior radiology-reporting systems largely adapt CNN-RNN and multi-task captioning methods. This work retains successful practices while adding graph-based interaction among representative finding features.
- Radiology Reporting: Radiology reports contain sections such as comparison, indication, findings, and impression, and this work generates findings and impression together.Findings describe observations, while impressions usually summarize them in one or two sentences.
- Prior Methods: Existing systems use text-image embeddings, multi-task medical tags, co-attention, and hierarchical decoders for classification and report generation.These methods build on CNN-RNN encoder-decoder frameworks adapted from image captioning.
- Proposed Framework: The proposed framework fuses frontal and lateral views and uses a two-level topic-and-sentence decoder, following established practices.Its stated goal is to measure gains from adding a graph module with prior knowledge.
- Graph-Based Difference: Unlike image-specific scene graphs, the method uses one universal chest-finding graph because radiology images have less overall-content variability and lack finding bounding boxes.The setting therefore requires alternative ways to annotate findings and initialize dedicated node features.
Graph Construction with Prior Knowledge
The composed graph represents common chest-X-ray findings as nodes and organizes related abnormalities by body part. Its clinically motivated structure encodes within-organ relationships while leaving room for richer future modeling.
- Graph Organization: Each graph node represents a chest-X-ray finding denoted by a disease keyword, with virtual organ or body-part groups organizing most findings.‘Normal,’ ‘other,’ and ‘foreign object’ are exceptions to organ-based grouping.
- Graph Organization: Findings associated with the same organ or tissue are connected, while a global root node connects to the other graph nodes.The figure distinguishes target-class nodes from dotted organ or tissue groupings.
- Clinical Prior Knowledge: The graph is based on clinical prior knowledge that abnormalities in the same body part may correlate and share features more strongly than abnormalities across organs.The defined graph contains 20 keywords or categories for classification and report generation.
- Scope: The predefined graph is a scope choice rather than a fixed modeling limit, since more sophisticated disease relationships could be annotated in future graph structures.The authors state that the model is not limited to the underlying graph.
Embedding
The embedding framework converts CNN features into graph-node representations, propagates information through graph convolutions, and branches into classification and report generation. Node features are then pooled for multi-label prediction.
- Framework: Frontal and lateral image features from a DenseNet-121 backbone are passed through attention-based graph embedding before classification and report-generation branches.The classification branch is trained first and then remains fixed during decoder training.
- Node Initialization: Each graph node corresponds to a finding category, and CNN-based attention initializes its feature before graph convolution propagates messages across nodes.The global node is the exception to the finding-category correspondence.
- Graph Convolution: Graph convolution generates and aggregates neighbor messages using the normalized adjacency matrix, then updates node features layer by layer.The implementation uses one-dimensional convolutions, batch normalization, ReLU activations, and residual connections.
- Classification Branch: After graph convolution, global average pooling and a sigmoid fully connected layer produce multi-label probabilities for the finding categories.Weighted binary cross entropy addresses positive-negative class imbalance during classifier training.
Report Generation via Graph Embedding
The report-generation decoder uses graph-embedded chest-finding features with attention at topic and word levels. Its hierarchical recurrent structure supports reports whose sentences focus on individual topics and findings.
- Report Generation via Graph Embedding: The report decoder is trained after graph embedding and uses two levels because medical reports usually contain multiple sentences focused on different topics.The implementation experiments with LSTM recurrent units.
- Report Generation via Graph Embedding: Graph-node attention combines the previous topic-LSTM hidden state with embedded finding features to produce a context vector for the next sentence.The attention weights are computed by a two-layer network with softmax activation.
- Report Generation via Graph Embedding: The attention module uses what has been predicted to determine which finding concepts should receive focus for the next sentence.The topic-LSTM hidden state is initialized from global averaged CNN features and updated after each sentence.
- Report Generation via Graph Embedding: The decoder uses two recurrent levels, with a topic-level LSTM producing topic vectors for a word-level LSTM that generates sentences word by word.The topic vector and graph-attention context vector update the word-LSTM gates and states.
- Report Generation via Graph Embedding: A sample dependency parse extracts “minimal,” “patchy,” and “lingula” as attributes of “airspace disease.”These attributes illustrate the finding-specific information available for report generation and evaluation.
- Report Generation via Graph Embedding: The word-level LSTM receives both the topic vector and graph-attention context vector while predicting each sentence.The recurrent-unit formulation explicitly distinguishes word-level gates, cell state, and hidden state.
Quality Evaluation by Graph Matching
The proposed MIRQI metric represents paired reports as finding graphs and evaluates disease mentions and attributes through node-by-node matching. It combines weighted recall and precision into an F1 score.
- Quality Evaluation by Graph Matching: MIRQI processes ground-truth and generated reports with disease-word, negation or uncertainty, and attribute extraction based on dependency graph parsing.The extraction procedure also handles synonyms and variations of disease words.
- Quality Evaluation by Graph Matching: The metric evaluates disease mentions, negation or uncertainty status, and associated attributes rather than relying only on word-level matching.Its graph matching computes true negatives, false positives, and false negatives at the node level.
- Quality Evaluation by Graph Matching: MIRQI recall matches disease mentions and associated attributes node by node between prediction and ground-truth sub-graphs.True positives additionally include correct attribute hits for positive disease mentions.
- Quality Evaluation by Graph Matching: MIRQI precision uses weighted positive and negative matching terms, with wneg = 1 − wpos and wattr weighting attribute contributions.The metric assigns separate weights to positive mentions and attributes.
- Quality Evaluation by Graph Matching: MIRQI-F1 combines MIRQI recall and precision by multiplying MIRQI-r by MIRQI-p.The displayed definition is MIRQI-F1 = MIRQI-r ∗MIRQI-p.
Experiments and Results
Experiments on IU-RR evaluate the graph embedding module for abnormality classification and report generation. The proposed model improves classification AUC and MIRQI report metrics while maintaining or improving conventional language scores.
- Dataset and evaluation: IU-RR contains 3955 reports paired with frontal chest radiographs and optional lateral views; experiments retain cases with both views.
- Dataset and evaluation: The study uses 20 finding keywords as disease categories and graph nodes, with same-organ findings connected and a global node linking organ groups.
- Dataset and evaluation: Classification uses stratified five-fold cross-validation and reports average ROC AUC alongside BLEU, ROUGE, CIDEr, and MIRQI report-generation metrics.
- Multi-label classification: The Densenet+KG model achieves higher or equivalent AUC for most classes, with about 2% average improvement over the otherwise identical baseline.The authors attribute this improvement to graph-based disentangled node features and message passing among correlated classes.
- Report generation: SentSAT+KG outperforms SAT and SentSAT on most language metrics and improves MIRQI metrics by 1.3%-1.8%.The comparison uses attention over graph node features, while comparisons with prior IU-RR systems may be affected by different experimental settings.
- Qualitative results: Qualitative examples show correct positive and negative mentions, alongside occasional false positives and repeated sentences for the same finding.In one example, Airspace opacity and Cardiomegaly are accurately reported, while other finding is falsely mentioned.
Conclusions
The paper proposes a chest abnormality graph with attention and graph convolution to support radiology report generation. Its graph-node features improve classification and reporting, while MIRQI evaluates positive and negative disease mentions; more detailed graph structures remain future work.
- The proposed chest abnormality graph uses attention and graph convolution to learn disentangled node features for classification and report generation.
- MIRQI evaluates the correctness of positive and negative disease mentions in generated reports, complementing conventional language metrics.
- The model is not limited to the current graph structure, and more sophisticated disease relationships are identified as future work.