CAPTOR: a vision-language foundation model for Chest-Abdomen-Pelvis Oncologic CT

Motivation

• CAP CT supports oncologic staging, metastatic assessment, and treatment monitoring, amid radiologist shortages and interpretive-error risks under high workload.Scalable AI tools supporting multiple tasks from a single examination are recognized for their workflow-augmentation potential.Source: §1 · p. 1

Research gap

• Existing CT models are either anatomically narrow, lack oncologic-rich training data, or target lesion-level tasks rather than whole-image analysis.This leaves a gap for multimorbidity-rich oncologic CAP imaging, including non-cancerous and incidental findings.Source: §1 · p. 2

Study objective

• CAPTOR is a contrastive vision-language model designed for whole CAP CT examinations, including single- or multiple-series acquisitions.It was compared with five state-of-the-art CT foundation models across medical-condition classification, two-year survival prediction, and image retrieval.Source: §1 · p. 2

2.1 Datasets

• The internal retrospective image-report dataset came from 11 institutions across five countries and was curated predominantly for oncologic CT interpretation.Studies were split at the patient level 80%/20%; public datasets supported downstream experiments and survival benchmarks.Source: §2.1 · p. 2

2.2 CAPTOR Foundation Model – Architecture; 2.3 CAPTOR Foundation Model – Preprocessing

• TotalSegmentator masks center crops on 12 anatomical regions, enabling whole CAP and multiple-series coverage rather than fixed central crops.The model uses a Merlin-based contrastive VLM architecture with Clinical Longformer and an inflated ResNet152 image encoder.Source: §2.2 · p. 2, §2.3 · p. 3, Figure 1 · p. 3

2.4 CAPTOR Foundation Model – Training

• CAPTOR was trained with InfoNCE to pair region-centered crops with report Findings, rotating the anatomical region across study batches.Epoch shuffling exposes all crops and supports complete, including multi-series, CAP examinations without changing the architecture.Source: §2.4 · p. 3

2.5 Evaluation – Medical condition classification

• Condition classifiers used anatomical-crop embeddings and were assessed by macro-AUC across coverage and indication subsets.The study selected 26 conditions from an initial list of 178, built report-derived labels, and also evaluated generalizability on two chest CT datasets.Source: §2.5 · pp. 3–4

2.6 Evaluation – Survival prediction benchmark

• Four TumorImagingBench two-year survival tasks used model-specific preprocessing and k-nearest-neighbor classifiers across 10 random splits.The neighbor count was optimized from 1 to 50 for each model and dataset.Source: §2.6 · p. 4

2.7 Evaluation – Retrieval

• Retrieval was evaluated with Recall@K for text-image matching and MAP@K for image-image retrieval, reported as fold-increase over random.SPECTRE and CT-FM were excluded from text-image retrieval for unavailable projection-layer weights and lack of a text encoder, respectively.Source: §2.7 · p. 4

2.8 Evaluation - Impact of training data volume

• To assess training-volume effects, classifiers were trained on nested 10%, 25%, 50%, and 75% subsets of the training set.CAPTOR-100K used the combined internal and public training set and was compared with internal-only CAPTOR and baselines.Source: §2.8 · p. 4

2.9 Statistical analysis; 2.10 Code availability

• Confidence intervals used patient-level bootstrap resampling, with task-specific pairwise tests and Benjamini-Hochberg correction; preprocessing, training, and inference code and weights are released.Bootstrap analyses used 2,000 resamples for most tasks and 1,000 for retrieval.Source: §2.9 · p. 4, §2.10 · p. 4

3.1 Dataset characteristics

• The internal dataset spanned 11 providers across five countries; the combined full training and testing sets contained 92,302 and 10,794 studies, respectively.The four TumorImagingBench datasets contained 134–421 studies each.Source: §3.1 · pp. 4, 6

3.2 Medical condition classification

• CAPTOR significantly outperformed all baselines on oncologic CAP classification, with AUCs of 0.767 on the CAP subset and 0.784 on the oncological subset.Figure 3 stratifies classification scores by anatomical region and clinical indication.Source: §3.2 · p. 6, Figure 3 · p. 7

3.4 Retrieval

• CAPTOR led text-image retrieval on CAP, oncologic, and chest subsets, while Merlin led on the whole test set, abdomen-pelvis, and non-oncologic subsets.For image-image retrieval, CAPTOR achieved MAP@10 of 1.30 on CAP, 1.41 on oncologic, and 1.23 on chest.Source: §3.4 · p. 7, Table 2 · p. 8

3.3 Survival prediction

• CAPTOR ranked first on three of four survival benchmarks, with AUCs of 0.640, 0.637, and 0.587 on NSCLC-Radiogenomics, C4KC-KiTS, and NSCLC-Radiomics, respectively.It ranked second on Colorectal-Liver-Metastases, and few differences were statistically significant.Source: §3.3 · pp. 6–7, Figure 5 · p. 9

3.5 Impact of training data volume

• Increasing training data generally improved classifier performance, though gains varied across datasets; CAPTOR consistently achieved the highest mean AUC across training fractions versus baselines on whole test, CAP, and oncological subsets.Source: §3.5 · p. 9
• CAPTOR-100K had comparable classification performance overall, while CAPTOR scored numerically higher on CAP and oncologic subsets; CAPTOR-100K improved external non-oncologic chest results.CAPTOR-100K led retrieval on whole-test, chest, AP, and non-oncologic subsets, whereas CAPTOR led on CAP and oncologic subsets.Source: §3.5 · pp. 7, 9, Table 3 · p. 9

4 Discussion

• Across classification, prognostic benchmarks, and retrieval, CAPTOR achieved the highest performance on CAP and oncologic CT subsets.The authors conclude that clinically representative oncologic CAP CT training outperformed region-specific and generalist models in the intended setting.Source: §4 · pp. 9–11
• TotalSegmentator masks center crops on anatomical subregions, yielding independent embeddings that cover full CAP volumes while preserving whole-examination context.This region-centered strategy likely contributed to CAPTOR’s advantage on CAP and oncologic subsets.Source: §4 · pp. 9–10
• CAPTOR performed comparably to, and sometimes slightly better than, CAPTOR-100K despite using substantially fewer examinations, while the larger model had advantages on non-oncologic datasets and retrieval.The authors frame this as a specialization–generalizability trade-off rather than a simple scaling effect.Source: §4 · p. 10
• CAPTOR showed consistently strong image-image and text-image retrieval, particularly on oncologic and CAP examinations, indicating improved image-report cross-modal alignment.The authors present retrieval as complementary evidence that classification gains reflect stronger image representations, not only classifier optimization.Source: §4 · p. 10
• The retrospective design, report-derived labels with possible noise, small external oncologic cohorts, and weaker results on non-oncologic CT constrain validation and generalizability.On CT-RATE and RAD-Chest, generalist models outperformed CAPTOR; the authors call for larger multicenter cohorts and broader-generalizability strategies.Source: §4 · p. 10
• The authors recommend exploring training strategies that retain specialization gains while extending generalizability across broader clinical use cases.They also state that code and model weights will be made publicly available.Source: §4 · p. 11

A.1 Internal dataset

• Series selection prioritized full-coverage, contrast-enhanced acquisitions, with phase and kernel preferences used to break ties within a category.The priority sequence began with CAP, followed by other combinations of chest, abdomen, and pelvis coverage.Source: §A.1 · p. 14

A.2 External dataset series selection criteria

• INSPECT and BIMCVR were excluded from external testing because their reports lacked sufficient detail or their datasets were too limited and nondescriptional.INSPECT reports contained only the Impressions section; BIMCVR was limited in size and descriptive information.Source: §A.2 · p. 14

A.3 More information on oncological datasets

• The four included oncologic cohorts cover NSCLC, renal tumors, and colorectal liver metastases, with cohort sizes of 421, 133, 134, and 194 patients or studies, respectively.The NSCLC-Radiogenomics cohort was filtered to the 133 patients with GTV annotations; C4KC-KiTS was filtered to 134 with 2-year survival data.Source: §A.3 · p. 14
• Two other TumorImagingBench datasets were excluded because they are lesion-level, outside CAPTOR’s whole-examination aim.Source: §A.3 · p. 14

A.4 More information on preprocessing

• CT volumes were reformatted to RAS, resampled to 1.5-mm in-plane resolution and 3-mm slice spacing, intensity-normalized, padded, and cropped to 224x224x160 voxels.The regions used span chest, abdominal and pelvic organs, vasculature and lymph nodes, and musculoskeletal anatomy.Source: §A.4 · p. 14
• Most reports were English; 1,805 Spanish reports (6%) were used as-is for training, while Spanish reports (2% of test reports) were excluded from analyses.Source: §A.4 · p. 14

A.5 Training hyperparameters

• Training used Adam with a 1.25e-6 learning rate, β=(0.9, 0.99), gradient checkpointing, and batch size 18 on one H100 GPU.Source: §A.5 · p. 14

A.6 Baseline Models

• The five baselines span region-specific CT-CLIP and Merlin, broader SPECTRE and COLIPRI, and vision-only CT-FM.CT-CLIP and Merlin were trained on non-oncologic chest and abdomen-pelvis data, respectively; SPECTRE spans multiple regions but is largely non-oncologic, and CT-FM uses CT volumes without reports.Source: §A.6 · pp. 14–15
• CAPTOR and Merlin embeddings came from the second-to-last layer; embeddings for the model comparisons were extracted using official documentation and code.Source: §A.6 · p. 15

A.7 Label extraction - Prompt and conditions

• Label extraction used zero-shot prompting rather than few-shot examples to avoid added variability and potential reductions in reproducibility across datasets.Source: §A.7 · p. 15
• The prompt classifies listed conditions from reports into detected, not-detected, or not-mentioned fields, with no more than one True status per condition.Source: §A.7 · p. 15

A.8 XGBoost parameters and details on classification method

• Identical XGBoost hyperparameters across models and conditions were intended to make performance differences primarily reflect embedding quality rather than downstream optimization.The binary logistic classifier used 100 trees, depth 3, learning rate 0.1, class weighting, L1/L2 regularization, and row and feature subsampling.Source: §A.8 · p. 16
• Baseline embeddings were taken from the series relevant to each condition, such as chest embeddings for chest conditions and AP embeddings for AP conditions.Source: §A.8 · p. 16

A.9 Further explanation on generalization evaluation

• Models were tested on CT-RATE and RAD-ChestCT using supplied labels and training/testing splits; RAD-ChestCT labels were collapsed across locations to indicate whether each abnormality appeared anywhere.CT-RATE labels were extracted with fine-tuned RadBERT-RoBERTa-4m, while RAD-ChestCT used rule-based labels.Source: §A.9 · p. 16

A.10 Further explanation on retrieval metrics

• For image-image retrieval, images counted as relevant when they shared at least one positive condition label, and MAP@K was computed over abnormal volumes as queries.The random baseline was estimated by Monte Carlo simulation; its behavior across K and subsets varies with relevant-label density.Source: §A.10 · pp. 16–17

B.1 Detailed internal dataset characteristics; B.3 Label proportions

• Table 5 summarizes internal-dataset characteristics by country, and Table 6 reports condition proportions across subsets of the combined test set.Source: §B.1 · p. 17, §B.3 · p. 17

B.2 Mistral performance

• Figure 7 displays Mistral Large 3, GPT-5.4, and GPT-4o mini classification performance on CT-RATE against labels extracted with fine-tuned RadBERT-RoBERTa-4m.Source: §B.2 · p. 17

B.4 Statistical significance results by condition

• Table 7 counts conditions where CAPTOR’s difference from each baseline is statistically significant.The supplied passage does not report the table’s counts.Source: §B.4 · p. 17
• Figure 7 compares per-condition classification F1-scores for Mistral Large 3, GPT-5.4, and GPT-4o mini on CT-RATE.Scores are evaluated against CT-RATE’s original labels, extracted using a fine-tuned RadBERT-RoBERTa-4m on 1,000 reports.Source: Figure 7 · p. 17

B.5 Computation times - oncologic datasets

• Table 8 reports computation times on Colorectal-Liver Metastases, which the authors say were consistent across the other datasets.Tests ran on an Apple MacBook Pro with an Apple M4 Pro chip and 24 GB of unified memory.Source: §B.5 · p. 17

• The complete CAPTOR-100K results extend the main-text comparisons of CAPTOR in Figures 3–5 and Table 2 to the larger training-data variant.Source: §B.6 · p. 17

B.7 External datasets with original labels

• CAPTOR-100K significantly outperformed CAPTOR on CT-RATE and RADChest-CT, while COLIPRI and SPECTRE outperformed both models except that CAPTOR-100K was not significantly outperformed on RADChest-CT.Figure 11 reports classification macro-AUC for abnormality labels using each dataset’s original labels.Source: §B.7 · p. 17, Figure 11 · p. 24