Source-linked AI summary
Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models
Yingshu Li, Yunyi Liu, Zhanyu Wang, Zailong Chen, Lingqiao Liu, Lei Wang, Luping Zhou
TL;DR
RRG systems have become more fluent, but report-level supervision mismatches the disease-grounded structure needed for clinically faithful generation. The paper introduces graph-supervised hierarchical alignment that separates disease-specific correspondence from global coherence, with the graph used only during training. Across three benchmarks, it improves conventional and clinical metrics, and its 3B model surpasses several larger 7B/13B systems.
Problem
Report-level objectives mismatch radiology reports’ disease-grounded findings, weakening precise clinical correspondence despite improved language fluency.
Method
Graph-Supervised Hierarchical Clinical Alignment uses a training-time clinical knowledge graph to decompose supervision into disease-centric and global semantic alignment.
Results
Across MIMIC-CXR, IU-Xray, and COV-CTR, the method improves conventional and clinical metrics, with its 3B model surpassing several 7B/13B systems.
Takeaways & Limitations
The results suggest that restructuring supervision can be more effective for RRG than increasing model size.
Abstract
from arXiv · showhide
Radiology report generation (RRG) has recently benefited from large language models, which substantially improve report fluency. However, clinically faithful generation remains challenging because current supervision is still imposed mostly at the report level. This creates a granularity mismatch: radiology reports are composed of disease-grounded findings, while existing methods are trained mainly with whole-report objectives. To address this problem, we propose Graph-Supervised Hierarchical Clinical Alignment, which reformulates image-report supervision as a hierarchical clinical alignment problem. Our method structures this alignment as a disease-conditioned process, where supervision is decomposed into two levels: Disease-Centric Alignment for fine-grained disease-specific correspondence, and Global Clinical Semantic Alignment for report-level semantic coherence. A clinical knowledge graph is used as a training-time-only structural prior that defines disease-specific supervision units and their clinical relationships, introducing no additional overhead at inference. Because standard contrastive alignment could produce false negatives when studies share overlapping pathologies, we combine instance-conditioned discriminative matching with disease-conditioned soft regularization, enabling fine-grained yet clinically consistent cross-modal representations. Experiments on MIMIC-CXR, IU-Xray, and COV-CTR show that our method consistently improves performance on both conventional and clinical metrics. Notably, our 3B model surpasses several prior systems with larger 7B/13B backbones, suggesting that improving supervision structure, rather than increasing model size, can be more effective for RRG.
1 Introduction
RRG has become more fluent with LLMs, but clinical faithfulness remains limited by report-level supervision that mismatches disease-grounded findings. The proposed framework addresses this mismatch by combining disease-specific correspondence with global report coherence using a training-time clinical graph.
- LLMs and MLLMs improve report fluency, shifting RRG’s bottleneck toward supervision granularity rather than language-generation capacity.
- Whole-report objectives entangle multiple disease findings, weakening precise alignment between clinically meaningful semantics and visual evidence.
- The framework decomposes supervision into Global Clinical Semantic Alignment for holistic consistency and Disease-Centric Alignment for fine-grained clinical correspondence.
- A clinical knowledge graph defines disease-specific supervision units and relationships during training, while instance matching is softened for studies sharing findings.
- The knowledge graph is removed after training, leaving inference without additional graph computation.
- Across three benchmarks, the 3B model achieves state-of-the-art or competitive conventional and clinical results while outperforming several 7B/13B baselines.
2 Related Work
Prior RRG work improves architectures, cross-modal matching, or clinical priors, but generally retains holistic or sample-level supervision. The remaining gap is disease-conditioned alignment that is simultaneously fine-grained and clinically structured.
- Non-LLM RRG methods mainly enhance visual extraction, generation architectures, or auxiliary clinical knowledge rather than redesigning image-report supervision.
- Contrastive and matching-based approaches improve cross-modal consistency but typically operate at holistic or sample-level granularity.
- Sentence-level alignment introduces finer granularity but lacks disease-structured guidance.
- Graph-based and knowledge-guided methods incorporate disease knowledge through pretraining, decoding, prompting, or retrieval, often adding inference-time components.
3 Method
The method reformulates token-level RRG supervision as hierarchical clinical alignment, using a knowledge graph during training to define disease-specific supervision units and two complementary alignment levels.
- Graph-Supervised Hierarchical Clinical Alignment uses a clinical knowledge graph to define disease-specific supervision units during training.
- The framework decomposes image-report alignment into Disease-Centric Alignment for disease-conditioned correspondence and Global Clinical Semantic Alignment for report-level coherence.
3.1 Radiology Report Generation
The base model encodes images into visual tokens, projects them into the LLM space, and uses them for autoregressive report generation. Additional hierarchical alignment objectives regularize representations beyond fluency-only training.
- Each image is encoded into visual tokens, projected into the LLM embedding space, and used to condition autoregressive report generation.
- The model trains on image-report pairs with visual conditioning input and ground-truth reports.
- The generation objective supports fluent output but does not explicitly align visual evidence with clinically meaningful semantics.
- Hierarchical clinical alignment objectives are added during training to regularize the learned representation space.
3.2 Graph-Supervised Hierarchical Clinical Alignment
The framework decomposes image-report alignment into disease-conditioned subproblems using graph-defined clinical units, then aligns disease-level visual and textual evidence. Its complementary objectives combine discriminative matching, soft regularization for shared findings, and intra-modal semantic consistency.
- Clinical knowledge graph and disease-conditioned probing: Graph nodes represent clinically meaningful disease queries that probe observation-level evidence from image and text representations.The graph is built over 14 CheXpert observation categories, with edges encoding anatomical and semantic relatedness.
- Clinical knowledge graph and disease-conditioned probing: Graph self-attention restricts message passing to clinically related nodes before cross-attention produces disease-level visual and textual representations.The refined disease queries probe modality-specific features through multi-head cross-attention.
- Instance-conditioned disease matching: ICDM uses same-sample visual-textual disease features as positive pairs, enforcing disease-specific cross-modal discrimination while preserving instance identity.Its image-to-text and text-to-image objectives operate over disease-node features within a mini-batch.
- Disease-conditioned node regularization: DCNR uses disease-conditioned soft targets so samples sharing a finding contribute positive signal, reducing false negatives caused by overlapping pathologies.The soft targets derive from 14 CheXpert labels and are applied through KL-divergence regularization.
- Intra-modal semantic consistency: IMSC preserves disease-aware semantic structure within each modality, complementing ICDM's discrimination and DCNR's relaxed supervision.It encourages same-finding samples to remain close while suppressing similarity to samples with different findings.
3.3 Global Clinical Semantic Alignment
Global Clinical Semantic Alignment complements fine-grained disease correspondence by aligning image and report representations in a shared report-level semantic space. Instance-conditioned matching preserves paired-study discrimination, while disease-conditioned soft supervision reduces false negatives among clinically similar studies.
- Global Clinical Semantic Alignment: GCSA aligns image and report representations in a shared report-level semantic space because disease-level correspondence alone cannot ensure coherent clinical narratives.The global branch constructs contextualized visual token representations before computing image-report similarity.
- Instance-Conditioned Semantic Matching: ICSM encourages each image to align with its corresponding report while remaining distinguishable from other samples.A symmetric text-to-image objective is defined analogously.
- Disease-conditioned semantic regularization: DCSR uses full-study CheXpert label profiles to keep studies with similar disease compositions closer while preserving the matched pair as the dominant target.It extends disease-conditioned soft supervision from individual nodes to the global semantic space.
- Disease-conditioned semantic regularization: DCSR complements ICSM by preserving report-level discrimination while reducing false negatives among clinically similar studies.Its final loss averages symmetric image-to-text and text-to-image directions.
3.4 Overall Training Objective and Inference
The overall objective combines token-generation cross-entropy with global and graph-supervised alignment losses. Alignment is used only during training, adding less than 10% trainable parameters over the 3B baseline while preserving the base model's inference architecture.
- Overall training objective: The overall training objective combines token-generation cross-entropy with global clinical semantic alignment and graph-supervised disease-centric alignment.The coefficients λ1 and λ2 control the strengths of the two alignment terms.
- Inference architecture and overhead: Less than 10% additional trainable parameters are introduced over the 3B baseline, and all alignment objectives are used only during training.The graph-related modules are removed before inference.
- Inference architecture and overhead: The base model retains the same inference architecture with no additional graph computation or inference-time overhead.Only the RRG backbone remains at inference.
4 Experiments
Experiments span MIMIC-CXR, IU-Xray, and COV-CTR, evaluating conventional, clinical, and disease-wise metrics. The method outperforms prior approaches across these settings, including larger-backbone systems and CT report generation.
- Datasets: MIMIC-CXR contains 377,110 images and 227,835 reports, while IU-Xray contains 7,470 images and 3,955 reports.COV-CTR adds 728 lung CT scans paired with diagnostic reports.
- Evaluation Metrics and Settings: The evaluation reports BLEU, METEOR, ROUGE-L, RadGraph F1, BERTScore, RadCliQ, GREEN, and RateScore.A frozen LLaMA3-3B is paired with a Swin Transformer visual encoder.
- Main Results: On MIMIC-CXR, the 3B model surpasses R2GenGPT (7B), Bootstrapping (13B), Multi-Grained, EKAGen, and KiUT.These comparisons span larger backbones, sentence-level contrastive learning, and disease-knowledge methods without supervision restructuring.
- Main Results: On IU-Xray, the reduced model without disease-conditioned objectives outperforms most prior methods, while CheXbert pseudo labels provide additional gains.Disease labels are unavailable in the original IU-Xray setting.
- Main Results: The method achieves the best RadGraph F1, BERTScore, RadCliQ, GREEN, and RateScore, and also outperforms prior methods on COV-CTR.These results indicate gains in clinically meaningful semantics and cross-modality generalization.
- Disease-wise Analysis: Disease-wise AUROC improves on key CheXpert findings including Edema, Atelectasis, Pneumothorax, and Pleural Effusion.The comparison covers 14 CheXpert categories.
4.4 Ablation Study
The ablations show that hierarchical supervision improves both report-level coherence and disease-level precision, with the full model outperforming larger or partial configurations. Graph structure, ontology breadth, and disease-alignment weighting further affect performance.
- Component ablation: The full model achieves the best performance on all metrics, surpassing the larger 7B backbone as objectives are progressively added to the 3B baseline.The ablation supports structured clinical supervision over backbone scaling alone.
- Component ablation: Global-only alignment leaves RadCliQ at 1.177, whereas node-only alignment reaches 1.155; the full model is best on all metrics.The partial configurations trade report-level correspondence against finding-level precision, showing that both hierarchy levels are needed.
- Component ablation: B@4: 0.126→0.133, a 5.6% relative gain, after adding ICSM brings the 3B baseline close to the 7B model.ICSM provides explicit report-level alignment beyond a conventional projection layer.
- Component ablation: Disease-conditioned soft regularization reduces false negatives by allowing studies sharing the same finding to contribute soft positive signal.This addresses the limitations of strict one-to-one targets in medical contrastive learning.
- Graph design: Extending the 14-disease CheXpert graph to 25 diseases yields consistent lexical and clinical improvements, indicating that broader ontologies can be absorbed.Random and all-connected graph variants are identified as comparison designs in the graph ablation.
- Hyperparameters: Increasing the Disease-Centric Alignment weight 𝜆2 to 2 improves the model further, reinforcing the importance of disease-specific supervision units.The hyperparameter study evaluates 𝜆1 and 𝜆2.
4.5 Qualitative Analysis
Qualitative examples show progressively finer clinical descriptions as alignment is added, while attention maps indicate focus on disease-relevant anatomical regions. The full model therefore improves finding-level specificity and disease-conditioned visual grounding.
- Generated reports: The baseline misses abnormalities, global alignment recovers coarse edema, and the full model adds disease-specific details such as mild cardiac enlargement.The progression illustrates increasing finding-level precision.
- Visual interpretability: The full model focuses attention on clinically relevant regions, including the cardiac silhouette for cardiomegaly and costophrenic angle for pleural effusion.These heatmaps support disease-conditioned alignment rather than reliance on language priors.
5 Conclusions
The paper identifies supervision granularity, rather than language-generation capacity, as the central RRG challenge and proposes hierarchical clinical alignment to address it. The framework combines disease-level correspondence with report-level coherence while using the knowledge graph only during training, and improves generation quality and clinical faithfulness across three benchmarks.
- Conclusion: The central RRG challenge is the granularity of supervision rather than language-generation capacity.Reports contain disease-grounded findings that require appropriately structured supervision.
- Conclusion: Graph-Supervised Hierarchical Clinical Alignment combines Disease-Centric Alignment for disease-level correspondence with Global Clinical Semantic Alignment for report-level coherence.The knowledge graph specifies supervision factorization during training rather than inference.
- Conclusion: The framework improves generation quality and clinical faithfulness across MIMIC-CXR, IU-Xray, and COV-CTR without inference-time graph computation.The graph is used only as a training-time structural prior.