Source-linked AI summary
ARC-CT: Anatomy-Routed Contrastive Vision-Language Learning for 3D Chest CT
Huseyin Umut Isik, Mehmet Alp Ozaydin, Sila Kurugol, Şeyda Ertekin
TL;DR
ARC-CT addresses diluted localized evidence and false negatives in global 3D chest CT contrastive learning. It routes representations through organ masks and softens contrastive targets using report-derived labels, achieving strong mask-free performance with a compact backbone.
Problem
Global volume pooling can dilute small, focal abnormalities, while one-hot contrastive loss incorrectly separates scans sharing abnormalities.
Method
ARC-CT combines anatomy-masked query routing, a label-Jaccard soft target, and organ-level alignment using automatically generated masks and report-derived labels.
Results
0.86 mask-free macro AUC across 18 abnormalities is achieved on CT-RATE with a compact 3D ResNet-18 backbone, outperforming comparable baselines and larger transformer models.
Takeaways & Limitations
ARC-CT provides focused visual grounding and strong in-domain abnormality classification without bounding-box supervision or mask-based inference.
Takeaways & Limitations
ARC-CT places fourth on external RAD-ChestCT, and its single-dataset organ masks and text supervision have not been tested under protocol shift.
Abstract
from arXiv · showhide
Contrastive vision-language learning uses paired chest CT volumes and radiology reports to learn abnormality classifiers without manually annotated labels. However, two characteristics of chest CT challenge conventional global contrastive learning. First, many critical abnormalities are small or anatomically localized, and pooling an en- tire volume into a single embedding may dilute their visual evidence. Second, the standard contrastive objective treats every other scan in a batch as a negative. Because many chest CTs share abnormalities, this objective incorrectly pushes co-positive pairs apart. We propose Anatomy-Routed Contrastive Learning for 3D Chest CT (ARC-CT), a region-aware framework that addresses these limitations using only la- bels extracted from reports by an LLM, with no manual annotations or bounding boxes. ARC-CT combines three components: (1) an Anato- myQFormer localizing evidence via queries constrained by automatically generated organ masks; (2) a label-Jaccard soft InfoNCE objective in- tegrating the standard one-hot target with the label-set overlap of each pair, which reduces false-negative penalties between studies that share clinical findings; and (3) an organ-level alignment loss connecting mask- pooled visual features to organ-specific report text extracted offline with a large language model. ARC-CT achieves a 0.86 mask-free macro AUC across 18 abnormalities using a compact 3D ResNet-18 backbone. Over- all, ARC-CT outperforms both comparable efficient baselines and sev- eral larger transformer models. Our code and weights are available at https://github.com/arc-ct/arc-ct.
1 Introduction
ARC-CT addresses two chest CT challenges: localized abnormalities can be diluted by whole-volume pooling, and one-hot contrastive learning can treat clinically similar scans as false negatives. It introduces anatomy-routed evidence localization and label-aware contrastive supervision without manual spatial annotations.
- Motivation: Localized findings can be diluted when an entire 3D volume is pooled against thousands of silent voxels.The problem affects findings such as nodules and patches of consolidation.
- Motivation: One-hot contrastive loss incorrectly pushes scans sharing abnormalities apart because every other batch scan is treated as a negative.This creates false negatives among co-positive studies.
- Contributions: ARC-CT uses role-typed AnatomyQFormer queries constrained by organ regions to localize evidence without bounding-box or region-proposal supervision.Its anatomy queries are masked to organ regions, while pathology and global queries provide additional roles.
- Contributions: A label-Jaccard soft target combines one-hot matching with label-set overlap to mitigate false-negative penalties among co-positive scans.The framework also adds per-organ alignment for region-level supervision without runtime language-model inference.
- Results: 0.855 mask-free macro AUC was achieved on CT-RATE using a 3D ResNet-18 trained on 47K volumes, outperforming same-scale baselines and larger transformer models.The result is a three-seed mean.
2 Method
ARC-CT trains a compact 3D chest CT vision-language model in two stages, routing query attention through anatomy masks during training and using label-aware global and organ-level alignment objectives. At inference, it averages output queries and classifies abnormalities from text prompts without masks or LLM calls.
- Stage 1: Weakly supervised image pretraining: Stage 1 trains a Kinetics-initialized 3D ResNet-18 on 18 report-extracted abnormality labels before warm-starting Stage 2.Global pooling and asymmetric loss are used during this weakly supervised pretraining stage.
- Stage 2: Region-aware contrastive pretraining: Stage 2 cross-attends image feature maps with AnatomyQFormer anatomy, pathology, and global queries while encoding report text.The feature map has shape [512, 12, 12, 12], and the text tower uses CXR-BERT adapted with LoRA.
- AnatomyQFormer: The AnatomyQFormer uses 30 learned queries whose attention is routed by organ masks, restricting anatomy and pathology queries to relevant regions without regional cropping.Global queries summarize the volume, while pathology queries attend to organs where their findings can occur.
- Label-Jaccard soft InfoNCE: Label-Jaccard soft InfoNCE weights image-report pairs by label overlap while retaining identity matching, reducing penalties for co-positive scans.The mixing weight is α = 0.3, while α = 0 recovers one-hot InfoNCE.
- Per-organ alignment: Per-organ alignment pairs mask-pooled organ image embeddings with corresponding report sentences parsed offline by Qwen3-8B.The alignment loss encourages anatomy queries to encode organ-specific content.
- Query supervision: Focal classes pool query responses over the most active spatial locations, preventing small lesions from being averaged across the whole grid during training.The focal classes include lung nodule, atelectasis, lung opacity, and consolidation.
- Inference: At inference, all queries are averaged and evaluated against positive and negative text prompts, requiring neither segmentation masks nor LLM calls.Adding masks at inference changes macro AUC by only +0.001.
3 Experiments and Results
ARC-CT is evaluated on in-domain and external chest CT benchmarks using prompt-based abnormality classification without manual labels. It achieves strong classification performance, including mask-free macro AUC of 0.855 on CT-RATE and 0.734 on external RAD-ChestCT.
- Datasets and evaluation: The evaluation uses CT-RATE’s 3,039-volume validation split and an external RAD-ChestCT subset of 3,630 volumes.CT-RATE contains 18 abnormality labels, while RAD-ChestCT labels are mapped to the same 18 abnormalities.
- Main results: ARC-CT achieves a mask-free macro AUC of 0.855 on CT-RATE and surpasses MPS-CT, GreenRFM, and larger transformer-based methods.The result is a three-seed mean using a compact 3D ResNet-18 backbone trained at the same data scale as comparable efficient baselines.
- Main results: 0.734 macro AUC is achieved on external RAD-ChestCT without fine-tuning.RAD-ChestCT is used strictly as an external benchmark and is never used for training.
- Main results: ARC-CT leads on accuracy (0.787), F1 (0.809), and positive-class precision (0.455).F1 uses support-weighting, while precision is reported for the positive class only.
- Per-class AUC: ARC-CT exceeds CT-CLIP on every reported abnormality class and matches or beats MPS-CT and GreenRFM on most classes.Its largest gains over MPS-CT occur for lung nodule (+0.06) and lung opacity (+0.05).
4 Discussion and Conclusion
ARC-CT structures visual and report signals with anatomy routing and label-aware contrastive learning, achieving about 0.86 macro AUC on CT-RATE without spatial annotation. Its external performance is constrained by protocol shift, motivating multi-source pretraining and other future extensions.
- Conclusion: 0.86 macro AUC is achieved on CT-RATE using anatomy routing and a label-Jaccard soft target without spatial annotation or extra supervision.The reported advantage is attributed to structuring volume and report signals rather than encoder size or warm-start alone.
- Conclusion: Anatomy routing and label-Jaccard targets address localized evidence dilution and false-negative gradients among co-positive scans.The framework uses automatically generated organ masks rather than manual bounding boxes.
- Limitations: ARC-CT places fourth on external RAD-ChestCT, behind methods trained on more heterogeneous scanner and protocol data.The paper identifies single-dataset training and untested protocol shift as boundaries of the current organ-mask and text-supervision setup.
- Visual grounding: ARC-CT’s Grad-CAM maps sharply localize four findings, whereas CT-CLIP’s activations are diffuse and largely unchanged between findings.The comparison is shown on one CT-RATE validation volume for cardiomegaly, arterial wall calcification, pleural effusion, and consolidation.
- Ablations: The ablation identifies the weakly supervised Stage-1 warm-start as the largest contributor, followed by anatomy routing and the label-Jaccard target.The leave-one-out analysis reports three-seed means, paired-bootstrap 95% confidence intervals, and p < 0.001.
- Retrieval: ARC-CT leads or ties at every image-to-image retrieval cutoff and ranks first on report-to-image recall except at R@100.Retrieval reuses the classification embedding without retrieval-specific tuning.