Source-linked AI summary
CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs
Naren Akash, Arihanth Tadanki, Jayanthi Sivaswamy
TL;DR
Chest radiograph retrieval must distinguish subtle abnormalities despite similar global appearances and anatomical superimposition. CheXtriev extracts regional features and models spatial and finding relationships with a graph transformer, producing anatomy-aware retrieval representations. It reports higher retrieval and ranking performance than global and local approaches, especially for less prevalent findings.
Problem
Chest radiograph retrieval must identify subtle disease indicators when global appearances are similar, while multi-label retrieval remains understudied.
Method
CheXtriev extracts features from anatomical regions and uses an edge-aware graph transformer to model spatial context and relationships between regions and findings.
Results
CheXtriev outperforms global and local retrieval approaches across reported retrieval and ranking measures, with especially strong gains for findings associated with radiographic blind spots.
Takeaways & Limitations
Anatomy-aware contextualization improves the accuracy, effectiveness, and efficiency of chest radiograph case retrieval, particularly for less prevalent findings.
Abstract
from arXiv · showhide
We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev.
1 Introduction
Medical image retrieval supports case-based clinical reasoning, but chest radiographs contain subtle, superimposed abnormalities that make relevant-case retrieval difficult. CheXtriev addresses this gap with anatomy-aware representations that combine regional details and anatomical context.
- Medical image retrieval helps clinicians refine diagnoses, plan treatments, and recover from prior errors by identifying relevant past cases.
- Chest radiograph retrieval is difficult because similar global appearances can conceal subtle disease indicators.
- 30% error rates and up to 56% radiologist disagreement have been reported for abnormal chest radiographs.
- Multi-label chest radiograph retrieval remains understudied despite progress in computer vision.
- CheXtriev targets informative anatomical regions and fuses them with location–finding relationships for anatomy-aware retrieval.
2 Method
CheXtriev represents chest radiographs through anatomically defined regions and models their spatial relationships with an edge-aware graph transformer. It aggregates regional context into normalized embeddings for efficient similarity retrieval and produces anatomy-based saliency maps for interpretation.
- Regional representation: CheXtriev extracts features from anatomically defined regions and builds a graph whose nodes are regions and edges encode spatial relationships and finding co-occurrences.
- Regional representation: The method identifies eighteen essential anatomical regions in frontal radiographs for structured representation extraction.
- Regional representation: Each regional representation is extracted with a fixed pretrained ResNet50 after global average pooling.
- Regional representation: Two-layer projection and learned location embeddings distinguish anatomical regions and enrich their representations.
- Inter-region relationship modelling: The graph transformer uses multi-head attention with learnable edge features to capture context-specific relationships between regions.
- Feature aggregation: Shared-source gated residual connections selectively aggregate global and local context into hierarchical multi-level features.
- Retrieval and interpretation: Mean-pooled graph features are trained for multi-label classification and normalized into dense vectors for similarity search.
- Retrieval and interpretation: FAISS performs exact k-nearest-neighbour inner-product search, while region occlusion yields similarity-based saliency maps.
3 Experiments
Experiments use frontal MIMIC-CXR-JPG images with eighteen-region annotations and evaluate retrieval using ranking, effectiveness, and efficiency metrics. The design is assessed through controlled CheXtriev variants.
- Dataset and setup: 226,473 training, 1,863 validation, and 3,191 testing images were selected from MIMIC-CXR-JPG with valid eighteen-region annotations.
- Dataset and setup: Images were resized to 224 × 224, and the graph transformer used two layers, eight attention heads, and model dimension DT = 64.
- Evaluation: Average Precision measures ranking quality, Hit Ratio measures retrieval effectiveness, and Reciprocal Rank measures retrieval efficiency.
- Evaluation: Table 1 evaluates how inter-anatomic region modelling and multi-level gated-residual features affect retrieval and ranking performance.
4 Results and Discussion
CheXtriev’s anatomy-aware graph representation improves chest-radiograph retrieval across variants and comparisons with global and local baselines. Its saliency analysis also indicates more anatomically relevant retrieval behavior for an enlarged cardiac silhouette query.
- CheXtriev variants: CheXtriev variants consistently improve over the mean-pooled baseline, with gains from +2.1% mAP to +6.4% global MLF across V1–V6.The results support fully connected unique learnable edges over uniform edge-sharing or handcrafted adjacency schemes.
- CheXtriev variants: Global gated residual connections outperform local gated residual connections, with V4 showing 6.38% lower mAP and 6.89% lower mRR than V6.The comparison emphasizes global gated residual connections with selective refinement for multi-level features.
- CheXtriev variants: Learnable location embeddings improve performance by capturing spatial context useful for accurate ranking.V6 and V3 outperform V5 and V2, respectively, in the reported variant comparisons.
- Global versus local approaches: CheXtriev achieves higher AP than both global baselines across all nine findings, with AP values ranging from 91.7% for LO to 28.8% for FO/HF.The same trend is reported for HR and RR metrics, with macro-mean and weighted-mean metrics used to assess performance on an unbalanced database.
- Global versus local approaches: Against AnaXNet, CheXtriev improves mean AP for findings associated with chest-radiograph blind spots, including FO/HF +61.80%, PE/HO +16.82%, and CONS +14.49%.The reported comparison concerns lung apices, hilar structures, and inferior lung bases.
- Interpretability analysis: For an ECS query, CheXtriev’s first three retrieved images exhibit ECS and its saliency maps focus on the cardiac silhouette, whereas other methods highlight irrelevant regions or retrieve non-ECS images.The fourth CheXtriev retrieval is incorrect and has saliency diffused through the lung region.
5 Conclusion
CheXtriev is presented as a graph-based, anatomy-aware retrieval model grounded in evidence-based anatomy. Its reported results support more accurate, effective, and efficient retrieval, especially for less prevalent findings, while its saliency analysis offers a preliminary interpretability direction.
- Conclusion: CheXtriev is a graph-based radiograph retrieval model inspired by radiologists’ systematic interpretation and grounded in evidence-based anatomy.The model targets informative anatomical details, models location–finding interplay, and fuses them into an anatomy-aware representation.
- Conclusion: CheXtriev reports more accurate, effective, and efficient case retrieval, particularly for less prevalent findings.The conclusion attributes this result to contextualizing anatomical regions and radiological findings.
- Conclusion: Anatomy-aware saliency maps may support interpretability of retrieved results, although further extension is needed to cover multiple findings.This is described as a preliminary analysis.
Supplementary Material
The supplementary material documents retrieval-performance tables and test-dataset statistics for anatomy-annotated frontal chest radiographs. These materials cover top-3 and top-10 comparisons and the Chest ImageGenome test split.
- Performance tables: Top-3 retrieval performance is compared on MIMIC-CXR against CNN, ATH, and AnaXNet, with asterisks marking p < 0.05.The table caption identifies the comparison as top-3 retrieval performance.
- Performance tables: Top-10 retrieval performance is likewise compared on MIMIC-CXR against CNN, ATH, and AnaXNet, with asterisks marking p < 0.05.The table caption identifies the comparison as top-10 retrieval performance.
- Dataset statistics: The Chest ImageGenome test dataset contains frontal PA or AP radiographs with valid bounding-box annotations for 18 anatomical regions.Its split follows the official MIMIC-CXR data splits.