Source-linked AI summary
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Christy Y. Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing
TL;DR
Generating coherent and accurate medical image reports requires combining visual understanding, medical knowledge, and language modeling. KERP addresses this gap by encoding images into abnormality graphs, retrieving templates, and paraphrasing them with Graph Transformers. It reports state-of-the-art performance across two medical report datasets, along with strong abnormality and disease classification and explainable attention.
Problem
Medical image report generation must bridge visual patterns and language while discovering and using internal medical knowledge structures.
Method
KERP encodes visual features into abnormality graphs, retrieves templates, and paraphrases them using Graph Transformers across multiple graph-structured domains.
Results
KERP achieves state-of-the-art performance on two medical report datasets, with best abnormality and disease classification accuracy and explainable attentive regions.
Takeaways & Limitations
The approach produces structured reports with accurate abnormality prediction and attentive location references supporting interpretative diagnosis.
Takeaways & Limitations
GTR assumes source and target node sets are specified beforehand and exposes a user-defined λ controlling prior versus learned edge weights.
Abstract
from arXiv · showhide
Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit---Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.
Introduction
Medical report generation requires bridging visual content and coherent language while incorporating medical knowledge and preserving accurate terminology. KERP addresses this by structuring report generation around abnormality graphs, retrieval, paraphrasing, and Graph Transformers.
- Long medical reports are more challenging than single-sentence captions because they must bridge visual patterns with linguistic descriptions.
- Existing visual text-generation methods can produce natural sentences yet miss visual grounding and often ignore internal task knowledge structures.
- Medical reports contain detailed findings, impressions, and peripheral information, with terminology and correctness requiring particular care.
- KERP mirrors radiologists’ workflow by encoding image-derived abnormalities, retrieving templates, and paraphrasing them for individual cases.
- Graph Transformer dynamically transforms graph-structured representations across visual features, abnormality graphs, templates, and words while incorporating attention and prior knowledge.
- Experiments on two medical report datasets report state-of-the-art automatic and human-evaluation performance, best abnormality classification, and explainable attentive regions.
Graph Transformer (GTR)
Graph Transformer transforms source and target graph representations through inter-graph and intra-graph message passing, using attention to learn relationships and prior edges to incorporate structure. Its variants support images, graphs, and sequences, enabling KERP’s multi-stage transformations.
- GTR transforms one graph into another, allowing different node counts, feature dimensions, structures, and edge characteristics.
- GTR uses inter-graph message passing to transfer projected source-node features to target nodes and intra-graph message passing to combine neighboring and self features.
- Attention learns source-to-target edge weights when relationships are unavailable, with normalized coefficients representing each source node’s relative importance.
- Learned attention weights are combined with prior edge weights through a weighted sum, with λ controlling their relative importance.
- GTR stacks intra-graph and inter-graph message-passing steps to progressively convert target features into higher-level semantics.
- GTR variants map images to graphs, graphs to graphs or sequences, and graph-sequence inputs to sequence outputs, forming KERP’s building blocks.
Paraphrase (KERP)
KERP models medical report writing as encoding visual findings into an abnormality graph, retrieving templates, and paraphrasing them for individual cases. Its modules use graph transformations to support abnormality-aware, case-specific report generation and disease classification.
- Paraphrase (KERP): KERP follows radiologists’ workflow by encoding visual findings, retrieving report templates, and adjusting them for each case.The process is designed to mimic checking images for abnormalities, following reporting patterns, and modifying template statements when necessary.
- Paraphrase (KERP): The template database contains frequent sentences describing abnormalities, including templates that combine findings such as lung hyperexpansion, diaphragm flattening, and COPD.A disease graph also represents diseases commonly concluded from single or combined abnormalities, such as atelectasis, consolidation, and emphysema.
- Paraphrase (KERP): The Encode module transforms image features into an abnormality graph whose nodes represent medically defined abnormalities and whose features capture attributes such as occurrence, size, and location.Node relationships are represented through edge weights, allowing correlations among abnormal findings to be encoded.
- Paraphrase (KERP): The Retrieve module uses the abnormality graph to generate a sequence of templates through a graph-to-sequence Graph Transformer.Templates are selected as indices from a template sequence using the GTRg2s transformation and Softmax-based selection.
- Paraphrase (KERP): The Paraphrase module rewrites retrieved templates into reports by attending to template words and the encoded knowledge graph.It refines inaccurate or incomplete case information and produces more natural expressions through language modeling.
- Paraphrase (KERP): KERP trains paraphrasing first with ground-truth templates and then with sampled templates to reduce the effect of retrieval-induced disconnection during backpropagation.This training procedure helps the model accommodate templates generated by its own Retrieval module at test time.
Experiments & Results
KERP is evaluated on two medical image report datasets using automatic metrics, human preferences, and classification AUCs. It achieves strong report-generation, abnormality-classification, and qualitative attention-alignment results, while its Paraphrase module improves naturalness beyond retrieved templates.
- Datasets and evaluation: Experiments use IU X-Ray and CX-CHR, with patient-level 7:1:2 training, validation, and testing splits and no patient overlap.IU X-Ray contains 7,470 images; CX-CHR contains 45,598 images from 35,609 patients.
- Datasets and evaluation: KERP is compared with captioning and report-generation baselines using automatic metrics and human evaluation of correctness, fluency, and content coverage.Human evaluation averages preferences from five participants over 100 sampled testing results per method.
- Report generation: KERP outperforms baselines on the reported BLEU and ROUGE-L metrics across datasets, while remaining below HRGR-Agent on CIDEr because HRGR-Agent is optimized with CIDEr-based reinforcement learning.KERP has the highest CIDEr among baselines without reinforcement learning.
- Ablation and error analysis: Compared with KER, KERP’s Paraphrase module improves ROUGE-L and BLEU-n on IU X-Ray and helps transform concise CX-CHR templates into more natural descriptions.The analysis distinguishes smoother common expressions from detection of informative abnormal findings.
- Human and qualitative evaluation: KERP also wins human evaluation on both datasets and produces reports aligned with ground truth and attention regions in qualitative IU X-Ray examples.The visualization links location descriptions to frontal or lateral attention maps and displays predicted abnormality probabilities.
Conclusion
KERP targets accurate and robust medical report generation, while GTR transforms multi-domain graph-structured data through attention.
- KERP performs medical report generation with goals of accuracy and robustness.
- GTR is a generic implementation unit for transforming multi-domain graph-structured data via attention.
- Experiments report state-of-the-art performance on two medical image report datasets.
- KERP generates accurate attribute predictions, a dynamic medical knowledge graph, and explainable location references.
Dataset statistics
The paper presents dataset statistics for the IU X-Ray and CX-CHR datasets in Table 3.
- Table 3 reports detailed statistics for the IU X-Ray and CX-CHR datasets.
- The dataset statistics are organized in Table 3 rather than described in this passage.
- IU X-Ray and CX-CHR are the two datasets covered by the reported statistics.
Visual features extraction
Visual features are extracted from DenseNet representations for both datasets, while Table 3 provides dataset statistics separately.
- DenseNet visual features are extracted for both datasets to compare fairly with all baselines.
- Table 3 is labeled as reporting statistics for CX-CHR and IU X-Ray, with “#” meaning “number of”.
- The extracted representation is a 64 × 64 × 256 feature map from the second dense block’s last convolutional layer.
- Additional lateral and convolutional layers transform features to dimension 256.
Abnormality definition & collection
The abnormality graph represents clinical abnormalities associated with thoracic organs, including abnormal attributes, missing typical attributes, shape changes, and location changes.
- Abnormal findings include abnormal attributes such as bibasilar consolidation.
- They also include absence of typical attributes, such as disappearance of the costophrenic angle.
- Abnormal changes in object shape or location are additional abnormal-finding forms.
- Clinical abnormalities stemming from thoracic organs are represented as nodes in the abnormality graph.
Template definition & collection
The paper constructs abnormality-specific text templates by filtering, manually grouping, and selecting frequent sentences, while specifying graph-weight assumptions and model settings for generation.
- Template definition & collection: Sentences describing each abnormality are collected only when their training-corpus frequency meets a threshold.This filtering step precedes semantic grouping and template selection.
- Template definition & collection: Manually grouped sentences with the same meaning yield one template: the most frequent sentence in each group.Template selection is therefore based on semantic grouping followed by frequency.
- Template definition & collection: The graph model assumes no prior edge weights for the target graph or source-to-target edges.These assumptions apply when generating template and report word sequences.
- Template definition & collection: Uniform prior edge weights are assigned among graph nodes, while learned edge weights receive a weight of 1.0.The passage specifies these values as the edge-weight initialization scheme.
- Training details: GTRi2g uses 3 graph message passing layers and 6 attention heads, whereas GTRg2s and GTRgs2s use 6 layers and 8 heads.The three variants correspond to Encode, Retrieve, and Paraphrase, respectively.
- Training details: All hidden features and embeddings have dimension 256, and provided prior-edge coefficients are set to 0.9.Retrieve and Paraphrase share word embeddings and the projection matrix Ww to word distributions.