Source-linked AI summary
Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, Xiaojun Chang
TL;DR
Radiology report generation is limited by visual and textual bias and by fixed knowledge graphs whose scope may not fit individual cases. DCL dynamically combines general and retrieved case-specific knowledge, then uses contrastive objectives to improve representations and graph accuracy. Across IU-Xray and MIMIC-CXR, it outperforms or matches prior state-of-the-art methods on language-generation and clinical-efficacy metrics.
Problem
Radiology report generation faces severe visual and textual bias, while fixed general-knowledge graphs may not provide the appropriate knowledge scope for individual cases.
Method
DCL dynamically updates a general-knowledge graph with specific knowledge extracted from retrieved reports and applies contrastive learning to visual, textual, and graph representations.
Results
DCL outperforms or matches previous state-of-the-art methods on descriptive-accuracy and clinical-correctness metrics across IU-Xray and MIMIC-CXR.
Takeaways & Limitations
Dynamic graphs that integrate general and case-specific knowledge provide a practical approach for contrastive-learning-based radiology report generation.
Takeaways & Limitations
Retrieved reports may differ from ground truth, introducing knowledge noise that can guide the model toward inaccurate sentences.
Abstract
from arXiv · showhide
Automatic radiology reporting has great clinical potential to relieve radiologists from heavy workloads and improve diagnosis interpretation. Recently, researchers have enhanced data-driven neural networks with medical knowledge graphs to eliminate the severe visual and textual bias in this task. The structures of such graphs are exploited by using the clinical dependencies formed by the disease topic tags via general knowledge and usually do not update during the training process. Consequently, the fixed graphs can not guarantee the most appropriate scope of knowledge and limit the effectiveness. To address the limitation, we propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning, named DCL. In detail, the fundamental structure of our graph is pre-constructed from general knowledge. Then we explore specific knowledge extracted from the retrieved reports to add additional nodes or redefine their relations in a bottom-up manner. Each image feature is integrated with its very own updated graph before being fed into the decoder module for report generation. Finally, this paper introduces Image-Report Contrastive and Image-Report Matching losses to better represent visual features and textual information. Evaluated on IU-Xray and MIMIC-CXR datasets, our DCL outperforms previous state-of-the-art models on these two benchmarks.
1. Introduction
DCL addresses visual and textual bias in radiology report generation by dynamically adapting medical knowledge graphs and applying contrastive learning. It reports state-of-the-art or matched performance across descriptive-accuracy and clinical-correctness metrics on IU-Xray and MIMIC-CXR.
- Automatic radiology reporting aims to generate coherent, informative reports while reducing radiologists’ workload and improving diagnosis interpretation.
- Medical report generation faces severe visual and textual bias, which can prevent models from describing specific crucial abnormalities.
- Fixed graphs built from general knowledge may be inappropriate for individual cases and fail to provide the most appropriate scope of knowledge.
- DCL dynamically integrates general knowledge with case-specific knowledge extracted from retrieved reports to update graph nodes and relations for each image.
- On IU-Xray and MIMIC-CXR, DCL achieves or matches state-of-the-art performance in descriptive accuracy and clinical correctness metrics.
- Contrastive learning improves visual and textual representations while also supporting dynamic-graph accuracy.
2. Related Work
Prior work combines radiology report generation with medical knowledge graphs and contrastive learning, but fixed graphs may not fit individual cases. DCL instead updates graph knowledge from retrieved reports and uses contrastive learning for retrieval and representation improvement.
- Medical Report Generation Meets Knowledge Graph: Medical report generation systems commonly incorporate knowledge graphs to emphasize abnormal terminology, disease keywords, or clinical relationships.
- Medical Report Generation Meets Knowledge Graph: The widely used graph from [46] is fixed, so DCL dynamically injects knowledge extracted from retrieved reports for each case.
- Medical Report Generation Meets Knowledge Graph: DCL’s dynamic graph combines general and case-specific knowledge so the appropriate knowledge scope can be activated for report generation.
- Contrastive Learning: Contrastive learning has been introduced into report generation to improve visual and textual representation expressiveness.
- Contrastive Learning: In DCL, contrastive learning also improves dynamic-graph accuracy by training the model to retrieve semantically similar reports.
3. Methodology
DCL combines a pre-constructed medical knowledge graph with image-specific updates and contrastive objectives for radiology report generation. Its graph modules encode relational structure and integrate graph representations with visual features before decoding.
- 3.2. Dynamic Graph: The dynamic graph starts from a 28-entity general-knowledge structure and is updated with image-specific knowledge extracted from similar reports.Retrieved reports provide RadGraph triplets that add entities and relations to the graph in a bottom-up manner.
- 3.1. Background: DCL uses a Transformer-based encoder-decoder framework with a ViT image encoder and Transformer report decoder.The image encoder produces visual vectors, while the decoder generates reports autoregressively from visual features.
- 3.3. Contrastive Learning: DCL jointly optimizes report generation, image-report contrastive, and image-report matching losses.The multimodal encoder used for matching is restricted to training to improve representation learning.
- 3.2. Dynamic Graph: Relational self-attention uses the adjacency matrix as a visible mask so graph nodes primarily propagate information through linked nodes.Entity and level embeddings initialize node representations before structural message passing.
- 3.2. Dynamic Graph: Graph attention uses visual features as queries and learned graph representations as keys and values to produce dynamic graph-enhanced visual features.The enhanced visual features and graph features use [CLS] tokens to aggregate visual and graph information.
4. Experiments
DCL is evaluated on IU-Xray and MIMIC-CXR using descriptive-accuracy, clinical-efficacy, ablation, and qualitative analyses. Results show strong benchmark performance and support dynamic graph construction, contrastive objectives, and retrieved specific knowledge.
- Datasets, Evaluation Metrics and Settings: The experiments use 2,069/296/590 IU-Xray cases for training, validation, and testing, while MIMIC-CXR contains 368,960 images and 222,758 reports.RadGraph supplies relation-specific triplets for updating the dynamic structure on MIMIC-CXR.
- Main Results: DCL achieves state-of-the-art descriptive accuracy, outperforming other systems on CIDEr and ROUGE-L while matching them on BLEU-4 and METEOR.The comparison covers existing radiology-reporting systems on IU-Xray and MIMIC-CXR.
- Ablation Study: Adding general knowledge to the base model increases CIDEr by 15.2% and BLEU-4 by 1.1%, while adding specific knowledge further raises CIDEr from 0.535 to 0.557.The dynamic graph combines general knowledge with specific knowledge extracted from retrieved Top-3 similar reports.
- Ablation Study: Ablation analysis attributes representation gains to Image-Report Contrastive learning aligning similar image-report pairs, while pretrained ViT and SciBert parameters substantially affect performance.SciBert also supplies pretrained embeddings for medical terminology.
- Case Study: The qualitative case study shows retrieved triplets linking sternotomy with cabg and consolidation with effusion, while DCL predicts a consolidation-related pneumonia sentence that R2Gen does not.Entities absent from the pre-constructed graph motivate dynamically constructed nodes.
5. Conclusion and Discussion
DCL combines a bottom-up dynamic graph with contrastive learning for radiology report generation. Experiments support its effectiveness, while retrieved-report mismatch and graph noise remain limitations that can produce inaccurate sentences.
- Approach: DCL integrates retrieved specific knowledge with general knowledge through bottom-up dynamic graph construction for radiology report generation.Contrastive learning is then used to improve visual and textual representations.
- Effectiveness: Experiments on IU-Xray and MIMIC-CXR show that DCL can outperform or match existing state-of-the-art methods in language-generation and clinical-efficacy metrics.
- Limitation and Future Work: Retrieved reports may differ from ground truth, introducing knowledge noise during dynamic graph construction that may guide inaccurate sentence generation.The authors plan a specific objective to improve graph accuracy and predicted-report quality.