CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios

Motivation

• CT analysis is clinically valuable but increasingly difficult to perform manually as examination volume grows while radiologist numbers remain relatively stable.CT supports non-invasive detection and treatment planning, and annual U.S. examinations exceed 93 million in 62 million patients.Source: §1 · p. 2

Related work

• Weakly supervised learning reduces dependence on costly, extensive annotations by exploiting limited or imprecise supervision from naturally paired medical data.Prior medical VL work adapted CLIP-style alignment from 2D images to 2D and 3D medical imaging.Source: §1 · p. 2

Motivation

• Global VL alignment collapses information-sparse, anatomically complex CT volumes into one representation, weakening correspondences for clinically salient regions.This limitation arises because entire volumes and reports are treated as single contrastive pairs.Source: §1 · p. 3

Contribution

• CT-GLIP constructs organ-level CT-report pairs to align localized visual components with corresponding textual descriptions and diagnoses.The dataset contains 44,011 fine-grained pairs from 17,702 patients covering 104 organs, using TotalSegmentator and GPT-3.5 Turbo with manual verification.Source: §1 · p. 3
• CT-GLIP highlights grounded VL alignment as a route toward stronger 3D medical VL models for CT analysis and broader clinical adoption.The paper reports more accurate fine-grained visual-textual alignment than existing global alignment methods.Source: §1 · pp. 3–4

Zero-shot evaluation

• CT-GLIP enables zero-shot organ recognition and abnormality detection from natural-language prompts without task-specific fine-tuning.Organ recognition compares organ-level visual embeddings with template-based text embeddings, while abnormality detection compares normal and abnormal descriptions within targeted organ regions.Source: §2.1 · p. 5, §2.2 · pp. 5–6
• 86.9% top-1 accuracy with nnUNet and 85.4% with MiT demonstrate CT-GLIP’s zero-shot organ-recognition performance.Predictions select the highest-similarity organ category among 104 text prompts.Source: §2.2 · pp. 5–6

Ablation studies

• Diagnosis contrastive learning substantially improves zero-shot abnormality detection, with the largest gains in Sensitivity reaching 35.48% for nnUNet and 33.81% for MiT.The reported gains indicate fewer false negatives relative to vanilla CLIP.Source: §2.3 · p. 7
• Anatomy contrastive learning and the abnormality dictionary provide additional abnormality-detection gains beyond diagnosis contrastive learning alone.With nnUNet, anatomy learning adds 1.58% F1 and 0.76% AUC, while the dictionary adds 0.42% F1 and 1.87% AUC; corresponding MiT gains are also reported.Source: §2.3 · p. 7

Fine-tuning evaluation

• CT-GLIP outperforms scratch-trained and vanilla-CLIP baselines for tumor detection and segmentation across CNN- and ViT-based backbones.For MiT, improvements over scratch and vanilla CLIP are 13.1% and 5.1% for segmentation, and 7.7% and 1.6% for detection.Source: §2.4 · pp. 7–8

Overall findings

• CT-GLIP surpasses baseline models across zero-shot organ recognition, abnormality detection, tumor detection, and tumor segmentation evaluations.The study uses 104-way organ recognition, abnormality detection for 16 abnormalities, and tumor tasks across seven cancer types.Source: §3 · p. 8

Dataset and evaluation

• The dataset contains 44,011 grounded CT–report pairs from 17,702 patients and covers 104 organs for full-body evaluation.The evaluation includes zero-shot and fine-tuning protocols spanning organ recognition, abnormality detection, tumor detection, and tumor segmentation.Source: §3 · p. 8, Figure 1 · p. 10

Motivation and approach

• Grounded alignment preserves anatomical correspondences that global CT–report alignment dilutes across sparse voxels and composite radiology reports.The framework ties specific anatomical regions to corresponding findings or diagnoses rather than representing an entire scan and report as one pair.Source: §3 · p. 8

Mechanisms

• Anatomy contrastive learning anchors representations to specific regions, while the abnormality dictionary broadens clinical language coverage through additional contrastive pairs.These components support abnormality detection and handling of real-world clinical terminology.Source: §3 · p. 9

Limitations

• The dataset is much smaller than large 2D vision–language datasets, preventing reliable characterization of 3D grounded-learning scaling behavior.Its training distribution is also dominated by lung, liver, and kidney regions, which may bias representation learning and downstream performance.Source: §3 · p. 9

4.1 Data Preparation of Grounded CT-Report Pairs

• Radiology impression text is paired with CT scans after preprocessing focused on clinically salient findings and anatomical regions.Source: §4.1 · p. 15
• 44,011 organ-level CT–report pairs from 17,702 patients cover 104 organs for grounded cross-modal pretraining.Source: §4.1 · p. 15

4.2 Developing CT-GLIP Framework

• CT-GLIP extracts organ-level visual features by average-pooling encoder feature maps over pseudo-segmented organ masks and matches them with templated organ descriptions.Source: §4.2 · p. 16
• Diagnosis contrastive learning pairs real abnormality descriptions for abnormal organs with generated normal-organ descriptions and expands text diversity using an abnormality dictionary covering 104 organs.Source: §4.2 · pp. 16–17
• The overall objective integrates anatomy contrastive loss, diagnosis contrastive loss, and auxiliary segmentation loss, while training proceeds from pseudo-organ-mask pretraining to grounded contrastive learning.Source: §4.2 · p. 17

4.3–4.4 Evaluation

• Evaluation includes zero-shot recognition of 104 organs and abnormality detection across selected organs, alongside fine-tuning for tumor detection and segmentation on 700 non-contrast CT scans.Source: §4.3 · p. 17, §4.4 · p. 18

• The contributors divided responsibilities across study design, data acquisition and preprocessing, model development, analysis, manuscript preparation, figures, and review.Source: §5.1 · pp. 18–19
• The authors declare no competing interests.Source: §5.2 · p. 19

• Dataset access is restricted by IRB requirements and requires approval plus a data-use agreement; approved access is limited to noncommercial academic purposes.Source: §5.5 · p. 19

Zero-shot Organ Recognition

• Zero-shot organ recognition is a 104-way classification task evaluated by Top-1 accuracy.Source: §A.1 · p. 26

Zero-shot Abnormality Detection

• Zero-shot abnormality detection is a binary classification task assessed using PPV, F1 score, and AUC.Source: §A.1 · p. 26

Tumor Detection

• Tumor detection uses the tumor volume as the score for computing patient-level AUC for tumor presence or absence.Source: §A.1 · p. 26

Tumor Segmentation

• Tumor segmentation evaluates overlap between predicted and ground-truth foreground voxels using a Dice-based formulation.Source: §A.1 · p. 26

• Tables 1 and 2 provide the raw evaluations corresponding to the results shown in Figures 4 and 5, respectively.Source: §A.2 · p. 26, Table 1 · p. 27, Table 2 · p. 27
• Table 1 reports zero-shot organ recognition and abnormality detection using top-1 accuracy, PPV, sensitivity, F1 score, and AUC.ACL, DCL, and A-Dict denote anatomy contrastive learning, diagnosis contrastive learning, and abnormality dictionary, respectively.Source: Table 1 · p. 27
• Table 2 reports downstream fine-tuning results across seven cancer-screening organs and evaluates tumor segmentation with DSC and screening with AUC.The listed organs are pancreas, breast, stomach, colorectum, lung, esophagus, and liver.Source: Table 2 · p. 27