Source-linked AI summary
CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs
Naren Akash, Arihanth Tadanki, Jayanthi Sivaswamy
TL;DR
Chest-radiograph retrieval remains understudied for cases with multiple findings. CheXtriev uses anatomy-aware graphs to represent radiographs and reports stronger retrieval results than global and local approaches.
Problem
Chest-radiograph retrieval, important to evidence-based clinical decision-making, remains understudied for cases with multiple findings.
Source: §1 · pp. 1–2Method
Results
+61.80% FO/HF mean AP versus AnaXNet; CheXtriev also had higher AP than global baselines across all nine investigated findings.
Source: §4 · p. 7Takeaways & Limitations
The authors conclude that anatomy-aware contextualization benefits case retrieval, particularly for less prevalent findings.
Source: §5 · p. 9
Abstract
from arXiv · showhide
We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev.
1 Introduction
CheXtriev addresses the gap in chest-radiograph retrieval by learning anatomy-aware representations that combine information from specific regions with the interplay between anatomical location and findings. The work is motivated by the clinical value of retrieving relevant past cases and the difficulty of identifying subtle abnormalities in radiographs.
1 Introduction
- MIR supports case-based clinical reasoning by helping clinicians refine diagnoses, plan treatment, and learn from relevant past errors and corrective actions.The broader motivation is that clinical expertise accumulates through experience.Source: §1 · p. 1
- Chest radiographs complicate retrieval because overlapping anatomy, low contrast, and subtle abnormalities make interpretation error-prone, with reported error rates up to 30% and radiologist disagreement up to 56%.MIR for cases with multiple findings remains understudied despite advances in multi-label computer vision.Source: §1 · pp. 1–2
- Existing retrieval methods can lose disease and region information, ATH still relies on global representations, and prior anatomy-focused work has not addressed retrieval itself.Deep hashing methods include DRH, OSDH, and SH-EBM; MMDL and X-TRA incorporate reports, while anatomy has been explored in classification, change detection, and report generation [16].Source: §1 · p. 2
- CheXtriev fuses informative details from specific anatomical regions to capture the interplay between location and radiological findings, and uses anatomy-aware saliency to support transparency.The approach targets visual-content retrieval and grounds its contextualization in evidence-based anatomy [17].Source: §1 · p. 2
2 Method
CheXtriev represents chest radiographs through anatomically defined regions, then models their spatial relationships to create anatomy-aware embeddings for similarity retrieval. It also provides region-level saliency maps to interpret image similarity.
2.1 Thoracic Radioanatomic Region Representation Extraction
- Eighteen anatomical regions are represented using features extracted after the global average pooling layer of a fixed pretrained ResNet50.The regions include lung zones, hilar structures, costophrenic angles, mediastinum, cardiac silhouette, and trachea.Source: §2.1 · p. 3
2.2 Modeling Inter-Region Relationships with Graph Transformer
- The graph transformer combines projected region features and learned location embeddings with edge-aware attention to model context-specific relationships between anatomical regions.Learned edge features are added to attention keys, and neighboring-region features are aggregated using both node and edge information.Source: §2.2 · pp. 3–4, §2 · p. 3
- The model uses shared-source gated residuals to aggregate hierarchical global-local features, then normalizes multi-label classification features into dense vectors for similarity search.The hierarchical representation is intended to capture fine-grained information at different levels of granularity.Source: §2.2 · p. 4
2.3 Efficient Retrieval and Visual Interpretability for Similarity
- FAISS flat indexing performs exact, non-parametric nearest-neighbor retrieval by inner-product search, ranking database radiographs by cosine similarity to the query.The method uses encoded dense embeddings in a pre-built vector database.Source: §2.3 · p. 4
- Occlusion-based saliency maps identify retrieved-image regions important to similarity by measuring cosine-similarity drops after predefined anatomical regions are occluded.Regions causing larger drops are treated as more influential, while regions causing minimal change are considered less important.Source: §2.3 · p. 5
3 Experiments
Experiments use MIMIC-CXR-JPG images with Chest ImaGenome anatomical-region annotations and evaluate finding-based retrieval on the official test split. Retrieval is assessed with average precision, hit ratio, and reciprocal rank.
3.1 Data
- The study uses 226,473 training, 1,863 validation, and 3,191 test frontal images from MIMIC-CXR-JPG, with Chest ImaGenome annotations and official MIMIC-CXR splits.It evaluates nine findings, treating an image as positive when any region is positive, and queries each positive test image against the remaining test set.Source: §3.1 · p. 5
3.2 Experimental Settings
- Images are resized to 224 × 224; the graph transformer uses two layers, eight attention heads, and dimension 64, with Adam at 1e −4 and early stopping.Training runs for at most 30 epochs with batch size 16, gradient accumulation every 8 epochs, learning-rate reduction factor 0.1, gradient clipping 0.5, and four RTX-2080 Ti GPUs.Source: §3.2 · p. 5
- Average Precision (AP) measures ranking quality, Hit Ratio (HR) retrieval effectiveness, and Reciprocal Rank (RR) retrieval efficiency.The metrics assess, respectively, the average position of relevant cases, the proportion of retrieved cases that are relevant, and the reciprocal rank of the first relevant case.Source: §3.2 · p. 5
4 Results and Discussion
CheXtriev improves chest-radiograph retrieval over global and local approaches, with particularly large gains for less prevalent or difficult-to-detect findings. Ablations and saliency analysis support the value of anatomy-aware spatial context and graph-based modeling.
CheXtriev variants
- V1–V6 consistently improve global MLF by +2.1% to +6.4%, while V4’s local gating lowers mAP by 6.38% and mRR by 6.89% versus V6.The results favor fully connected unique learnable edges over uniform or handcrafted adjacency, and V6/V3 outperform V5/V2, respectively, suggesting that learnable location embeddings improve spatial context.Source: §4 · p. 6
Assessment of anatomy-aware feature extraction
- Compared with V0, graph-transformer variants modestly but consistently improve retrieval of relevant cases, while V6 additionally improves ranking and retrieval precision.The passage reports similar mHR across variants but higher mAP and mRR for V6.Source: §4 · p. 6
Assessment of global vs. local approaches
- CheXtriev’s mAP improvement over global baselines ranges from at least 12% (ResNet-50) to 23% (ATH).Improvements are especially large for less prevalent findings: FO/HF (+82.3% to +94.6%), PTX (+40.6% to +253.9%), CONS (+30.6% to +35.6%), and PN (+14.6% to +20.5%). The weighted mean is higher than the macro mean for all reported metrics.Source: Table 2 · p. 7, §4 · p. 7
- Against AnaXNet, CheXtriev significantly improves mean AP for blind-spot findings, including FO/HF (+61.80%), PE/HO (+16.82%), and CONS (+14.49%).The authors attribute the gains to global edge-aware attention, fully connected learnable continuous edges, location embeddings, and gated residual connections; they also note that classification-optimized features may be less effective for retrieval.Source: §4 · pp. 7–8, Table 2 · p. 7
Interpretability analysis
- For the ECS query, CheXtriev’s first three retrieved images show ECS and saliency maps focused on the cardiac silhouette, while its fourth retrieval is incorrect.Some AnaXNet results lack ECS, and ATH/CNN saliency maps highlight irrelevant regions despite retrieving correct cases; the authors caution that saliency maps may be less meaningful for methods without anatomy-aware features.Source: §4 · p. 8, Figure 2 · p. 8
5 Conclusion
CheXtriev uses anatomy-centered graph representations to improve chest-radiograph retrieval, especially for less prevalent findings, and preliminary saliency analysis suggests a possible route to interpretability.
5 Conclusion
- CheXtriev targets informative anatomical regions, models how findings relate to location, and fuses this context into a richer anatomy-aware representation.The approach is inspired by radiologists’ systematic interpretation of radiographs and grounded in evidence-based anatomy.Source: §5 · p. 9
- CheXtriev’s contextualization yields superior case retrieval that is more accurate, effective, and efficient, particularly for less prevalent findings.Source: §5 · p. 9
- Preliminary anatomy-aware saliency maps may help interpret retrieved results, but require further extension to cover multiple findings.Source: §5 · p. 9