Source-linked AI summary
CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation
Adonay Demewez Gebremedhin, Wessam Shehieb, Sara Alansari, Mohamad Alansari, Muzammal Naseer, Sajid Javed, Naoufel Werghi
TL;DR
Longitudinal chest X-ray interpretation needs localized evidence to explain how findings change across sequential examinations, but this connection remains under-explored. CheXGround addresses it with temporally enhanced anatomical ROI tokens, phrase alignment, and global temporal image context, improving clinical language quality, temporal reasoning, and localization accuracy across evaluated tasks.
Problem
Localized visual evidence remains under-explored as support for longitudinal chest X-ray interpretation, where models must represent both anatomy and temporal change.
Method
CheXGround represents paired studies with temporally enhanced anatomical ROI tokens, aligns them with localized report phrases through TRPA pretraining, and combines them with global temporal image tokens.
Results
Across single-study and longitudinal VQA, findings generation, grounded VQA, temporal grounded VQA, and anatomical grounding, CheXGround consistently improves clinical language quality, temporal reasoning, and localization accuracy over recent baselines.
Takeaways & Limitations
Organizing longitudinal evidence at the anatomical level is a strong representation for grounded radiology language modeling.
Abstract
from arXiv · showhide
Recent radiology multi-modal language models have made substantial progress in chest X-ray report generation, visual question answering, and temporal reasoning. While longitudinal chest X-ray interpretation compares sequential examinations to describe change, visual grounding aims to connect clinical language with localized image evidence. Although longitudinal modeling and visual grounding have each advanced radiology language models, how localized visual evidence can support longitudinal interpretation remains under-explored. We introduce CheXGround, a region-grounded longitudinal chest X-ray language model that represents paired studies through corresponding anatomical regions. CheXGround extracts anatomical regions from current and prior radiographs, encodes them as temporally enhanced Region-of-Interest (ROI) tokens, and combines them with global temporal image context during generation. To connect these region tokens with clinical text, we propose Temporal Region--Phrase Alignment, a pretraining objective that aligns temporal anatomical representations with localized report phrases. We evaluate CheXGround on single-study and longitudinal Visual Question Answering (VQA), longitudinal findings generation, temporal grounded VQA, and anatomical grounding. Across these tasks, CheXGround improves clinical language quality, temporal reasoning, and localization accuracy over recent baselines. Our results suggest that organizing longitudinal evidence at the anatomical level is a strong representation for grounded radiology language modeling. Project page: https://adonaydem.github.io/chexground-website
1 Introduction
CheXGround addresses the challenge of supporting longitudinal chest X-ray interpretation with localized anatomical evidence. It combines temporally aligned anatomical ROI representations, phrase alignment, and global image context for grounded reasoning and generation.
- Motivation: Longitudinal interpretation must connect a finding to its anatomical location and temporal change, because clinically important differences may be subtle and region-specific.Examples include worsening basilar opacity, resolving right lower-lobe consolidation, and newly visible support devices.
- Design rationale: CheXGround uses dual vision–language interleaving and dense grounded supervision to integrate clinical answers with localized visual evidence during reasoning and generation.This contrasts with approaches that separate question answering from coordinate prediction or ground evidence only after generation.
- Approach: CheXGround represents paired studies with corresponding fixed anatomical regions and passes temporally compared ROI tokens to the language model.The design supports reasoning over shared chest anatomy rather than only global image tokens.
- Approach: Temporal Region–Phrase Alignment pretraining aligns temporally contextualized anatomical ROI representations with localized clinical phrases.TRPA combines ROI-level contrastive learning with phrase-composition targets so related phrases can share localized evidence.
- Evaluation: The model combines global temporal image tokens with phrase-aligned anatomical ROI tokens for grounded VQA, progression reasoning, and longitudinal report generation.The evaluation covers diverse visual grounding tasks and reports consistent gains in clinical language quality and grounding accuracy over baselines.
2 Related Work
Radiology vision-language research has expanded from report generation and VQA to localized grounding with boxes, masks, regions, and anatomy-centered supervision. However, existing grounded radiology methods remain largely non-temporal, motivating CheXGround’s focus on anatomical evidence across examinations.
- Radiology vision-language models: Radiology vision-language models have progressed across report generation, question-conditioned reasoning, contrastive image–report alignment, and conversational interaction.Reported approaches include recurrent, transformer, hierarchical, co-attention, memory-based, knowledge-informed, graph-based, and retrieval-based models.
- Visual grounding: General grounding MLLMs support segmentation, region references, pixel-level reasoning, spatial reasoning, and spatio-temporal grounding.Radiology-specific methods extend grounding through boxes, masks, multi-modal inputs, anatomy-centered scene graphs, abnormality boxes, and grounded VQA datasets.
- Research gap: Existing grounded radiology works connect clinical text with localized evidence but remain largely non-temporal.CheXGround instead models anatomical evidence across examinations.
3 Methodology
CheXGround represents paired chest X-rays with global temporal image tokens and anatomically matched ROI tokens, then aligns those regions with localized report phrases. Its temporal ROI processing, phrase-composition targets, and dual visual streams support region-level longitudinal reasoning in language generation.
- Problem formulation and architecture: CheXGround combines global temporal image tokens with fixed anatomical ROI tokens representing localized evidence from paired studies.The model compares corresponding anatomical regions across current and prior radiographs before passing ROI tokens to the language model.
- Anatomical ROI extraction: Anatomical ROI extraction uses detector-predicted boxes and ROIAlign to obtain localized multiscale feature grids for each anatomical slot.The detector is trained on Chest ImaGenome box annotations, and each slot preserves anatomical identity across studies.
- Spatial and temporal ROI encoding: Spatial ROI encoding combines local within-region processing with global cross-anatomical context, while causal temporal attention operates independently within each anatomical slot.The causal mask lets current ROI tokens condition on prior counterparts while preventing prior tokens from accessing current evidence.
- Temporal Region–Phrase Alignment pretraining: Temporal Region–Phrase Alignment pretraining aligns temporally contextualized ROI representations with clinically relevant localized report phrases.TRPA uses pooled current ROI features, phrase encodings, ROI-to-phrase distributions, phrase-to-region attention, and image–report contrastive alignment.
- Temporal Region–Phrase Alignment pretraining: Phrase-composition targets soften hard anatomical assignments by distributing supervision to semantically related phrases while retaining annotated localization as the primary signal.This addresses sparse, compositional radiology language and allows related clinical phrases to share anatomical evidence.
- CheXGround region-grounded language modeling: During language modeling, a global stream provides temporally aligned image context and an ROI stream provides TRPA-enhanced anatomical tokens, with TAC and TRF handling temporal fusion.The architecture uses dual temporal alignment because one-way current-to-prior fusion was insufficient for modeling radiographic change in ablations.
4 Experiments
CheXGround is evaluated across VQA, grounded VQA, longitudinal findings generation, anatomical grounding, and related adaptations, using clinical, temporal, and localization metrics. It achieves strong performance across these settings, while ablations attribute localization gains mainly to TRPA and temporal gains mainly to dual-TAC.
- Evaluation setup: CheXGround is evaluated on VQA, grounded VQA, longitudinal findings generation, anatomical grounding, abnormality grounding, and temporal progression classification.The primary task families include VQA, Grounded VQA, longitudinal findings generation, and anatomy grounding.
- Implementation details: CheXGround uses RAD-DINO, Meditron-7B, frozen encoders and base language-model parameters, and trains projectors, temporal modules, added tokens, and LoRA adapters.Training uses AdamW, cosine learning-rate scheduling, and two NVIDIA A100 80GB GPUs.
- Visual and grounded question answering: CheXGround improves grounded VQA by +9.9 mIoU and +15.1 Recall@0.5 over Libra-ft temporally.It also achieves F1reg/F1temp×reg of 62.4/48.8, versus 43.7/37.7 for MedGemma-1.5-4B-ft and 46.8/28.1 for Libra-ft.
- Anatomical grounding: CheXGround achieves 55.4 mIoU, 64.5 Acc50, and 22.2 Acc75 on anatomical grounding, the best performance across all reported metrics.The task uses coordinate-style queries curated from Chest ImaGenome.
- Longitudinal findings generation: CheXGround improves over Libra by +1.4 BLEU-4, +2.4 ROUGE-L, and +2.8 F1temp in longitudinal findings generation.The task-tuned CheXGround-ft reaches 43.57 ROUGE-L, 45.37 METEOR, 61.38/46.41 CheXbert-F1micro/macro, 37.7 RadGraph-F1, and 40.85 F1temp.
- Ablation studies: TRPA provides the strongest localization gain, adding +4.5/+3.5 Acc50 over R+S/R+D, while dual-TAC remains the larger temporal driver.Matched TRPA contrasts improve language, clinical, grounding, and temporal metrics.
5 Conclusion
CheXGround represents paired chest X-ray studies through corresponding anatomical regions, combining temporal ROI tokens with global temporal context and phrase alignment. Across evaluated tasks, it improves clinical language quality, temporal reasoning, and localization accuracy over recent baselines.
- Conclusion: CheXGround represents current and prior chest X-ray studies through corresponding anatomical regions and phrase-aligned temporal ROI tokens.The model combines these tokens with global temporal image context for language generation.
- Conclusion: Across single-study VQA, longitudinal VQA, findings generation, grounded VQA, temporal grounded VQA, and anatomical grounding, CheXGround improves over recent baselines.The reported improvements span clinical language quality, temporal reasoning, and localization accuracy.
- Conclusion: The findings highlight anatomy-level temporal grounding as an effective framework for longitudinal radiology language modeling.