Auditable CT Phenotyping Through Report-derived Radiological Observations

Method

• ACT builds a normalized 5,120-dimensional concept-anchored CT embedding by weighting F2LLM directions for 376,194 report-derived atomic observations with native image-text similarities.The model was trained on 38,317 patients from CT-RATE and Merlin, and the shared observation bank supports annotation, phenotyping, audit and restriction.Source: Introduction · p. 2, Figure 1 · p. 3

Zero-shot annotation

• ACT achieved the highest mean abnormality-annotation AUROC among five complete-cohort 2D and 3D vision-language baselines in every evaluated cohort.Evaluation covered 25,183 held-out patients across CT-RATE, PMBB, RSNA-2023 and INSPECT.Source: Introduction · p. 3, Figure 1 · p. 3

Transfer and phenotyping

• ACT’s concept-anchored CT representation exceeded CT-CLIP across 221 CTPA-derived EHR phenotypes under both zero-shot scoring and matched linear probing.CTPA was absent from image pretraining, testing transfer to an unseen acquisition domain.Source: Introduction · p. 4

Audit

• ACT’s probe-observation audit revealed leading directions lacking evidence for the coded phenotype, including directions shared across phenotypes with unrelated clinical targets.The audit tests whether predictive probes depend on phenotype-related named observations rather than proxies for site, acquisition or care process.Source: Introduction · pp. 2, 4

Clinical restriction

• Phenotype-specific observation-bank restriction redirected selected probes toward phenotype-related observations and achieved a mean held-out test AUROC of 0.751 across 86 eligible phenotypes.ACT therefore makes its radiological observation directions actionable for restricting phenotype predictors.Source: Introduction · p. 4

ACT’s native 3D model annotates concepts across diverse CT cohorts

• 0.749 mean AUROC on CT-RATE: ACT’s native model achieved the highest zero-shot finding-annotation performance among five vision-language baselines across four cohorts.Mean AUROC was 0.689 and 0.683 on chest and abdominal PMBB and 0.675 on RSNA-2023; ACT won most individual findings in every cohort.Source: ACT’s native 3D model annotates… · p. 4, Figure 2 · p. 5

ACT’s radiological observations form an anatomically structured reference space

• 0.565 Spearman correlation: distances in ACT’s 5,120-dimensional observation space increased with RadLex ontology separation, and external PMBB observations mapped into corresponding fixed-atlas regions.Mean distance increased from 1.061 at one ontology hop to 1.285 at five; PMBB and reference centroids were direct neighbours in four finding-by-anatomy families.Source: ACT’s radiological observations form… · p. 7, Figure 3 · p. 10

ACT’s concept-anchored CT representation supports CTPA-EHR phenotyping beyond image pretraining

• 0.651 versus 0.572 zero-shot and 0.709 versus 0.662 with linear probing: ACT’s concept-anchored representation outperformed CT-CLIP across 221 CTPA phenotypes.Paired macro-AUROC differences were 0.079 and 0.047, respectively, with confidence intervals excluding zero; ACT was higher for 182 phenotypes zero-shot and 190 with linear probing.Source: ACT’s concept-anchored CT… · p. 9, Figure 4

ACT’s probe-observation audit reveals candidate proxy directions

• 97 observations occupied 221 rank-1 probe positions, with aortic and coronary calcification ranking first for 20 phenotypes including osteoporosis, urinary tract infection and major depressive disorder.Vascular calcification or atherosclerosis formed a lexical majority for 97 probes, while pulmonary air-space disease dominated 54; these are candidate proxy directions rather than patient-level evidence.Source: ACT’s probe-observation audit reveals… · pp. 11, 13

ACT’s observation-bank restriction redirects selected probes toward phenotype-related observations

• 0.751 versus 0.741 mean AUROC: restricting the observation bank improved or preserved held-out performance across 86 eligible phenotypes while redirecting probes toward phenotype-related observations.Restricted AUROC was higher in 55 phenotypes and lower in 31; displayed probes retained findings such as pleural effusion for CHF, collapsed IVC for hypovolemia and contrast extravasation for shock.Source: ACT’s observation-bank restriction… · p. 14, Figure 6 · p. 15

Overall findings

• ACT’s native model achieved the highest mean AUROC in every cohort, and exceeded CT-CLIP across 221 EHR phenotypes on unseen CTPA under zero-shot scoring and matched linear probing.The native model was trained on 38,317 patients, mined 376,194 observations, and was evaluated in 25,183 held-out patients; CTPA was unseen during image pretraining.Source: Discussion · p. 16

Methodological contribution

• ACT anchors report observations as directions in a fixed text-embedding space, spanning the full vocabulary without concept annotation and enabling readable, constrainable predictors unlike prior inspection tools.The reference atlas remained intact for external observations, supporting interpretation of probe directions through radiological vocabulary.Source: Discussion · p. 16

Probe audit

• Only 97 distinct strings ranked first across 221 phenotype probes, with aortic and coronary calcification leading 20 probes, including osteoporosis, urinary tract infection and major depressive disorder.Recurring shortcut families included vascular calcification, pulmonary air-space disease, drains and stents, and biliary dilatation, whereas some bronchiectasis, pneumonia, coronary atherosclerosis and pulmonary-collapse profiles were clinically aligned.Source: Discussion · p. 17

Observation-bank restriction

• Clinician-restricted probes redirected evidence toward phenotype-related observations in 86 eligible phenotypes without costing held-out classification, using a median bank containing under half a percent of observations.Examples included pleural effusion for CHF and a collapsed IVC for hypovolemia, showing that predictor semantics can be separated from accuracy.Source: Discussion · pp. 16–17

Limitations

• The study’s audit is observational and global: it does not demonstrate confounder use or removal in any individual patient, and external evidence came from limited datasets with some descriptive comparisons.The alignment score omits scan-specific similarity and patient-level normalization; confirming shortcuts requires error-conditioned comparison, scan-level presence checks and confounder intervention.Source: Discussion · p. 17

Ethics statement

• The retrospective study used de-identified or controlled-access datasets under institutional approvals, waivers or consent procedures, applicable agreements, and research-only governance.Model output was not used in clinical care.Source: Paired CT scan, radiology report and… · p. 18, Ethics statement · pp. 24–25

Paired CT scan, radiology report and linked EHR datasets

• 72,640 paired CT–report rows from 38,317 patients combined CT-RATE and Merlin, while INSPECT supplied encounter-linked diagnosis codes mapped to 1,692 hierarchical phenotypes.CT-RATE contributed 47,146 rows and Merlin 25,494; examinations without linked INSPECT visits were excluded rather than labelled all-negative.Source: Paired CT scan, radiology report and… · p. 18

Native volume-report model and observation bank

• 376,194 report-derived observation phrases formed ACT’s fixed vocabulary, whose F2LLM embeddings linked the native 768-dimensional image space to a 5,120-dimensional concept-anchored CT representation.Impressions supported contrastive pretraining, whereas findings supplied the observation vocabulary; downstream phenotype probes acted on the normalized concept-anchored representation.Source: Paired CT scan, radiology report and… · p. 18, Native volume-report model and… · pp. 19–20
• ACT used a DINOv2-based volumetric vision encoder and CLIP-style report tower trained with symmetric InfoNCE on paired CT volumes and impressions.The image encoder processed axial slices with Transformer fusion, while the report tower produced a shared 768-dimensional embedding for contrastive training.Source: Native volume-report model and… · p. 19

Component evaluations

• Component evaluations measured zero-shot finding annotation, finding-conditioned retrieval, lesion-volume association, qualitative high-score examples, and observation-bank structure using held-out and external cohorts.Baselines included Merlin, CT-CLIP, MedSigLIP, BiomedCLIP, and OpenAI CLIP; retrieval used pooled Recall@k with fixed random-ranking floors.Source: Component evaluations · pp. 20–21

CTPA-EHR phenotype experiments

• CTPA-EHR experiments retained 221 phenotypes with at least 50 held-out positive scans, fitting frozen multilabel linear probes and auditing their alignment with named observation directions across 20 initializations.The labels were encounter-derived weak phecode labels rather than adjudicated CTPA findings, and audit scores represented global semantic alignment rather than individual-scan observation contributions.Source: CTPA-EHR phenotype experiments · pp. 22–23
• Clinician-defined rules retained non-empty direct or associated observation banks for 86 phenotypes, recomputing each restricted representation before semantic aggregation and scan-wise normalization.Fifty-four phenotypes had direct evidence and 32 had associated evidence; 135 phenotypes matched neither rule and were excluded from restriction analyses.Source: CTPA-EHR phenotype experiments · pp. 23–24

Statistical analysis

• Uncertainty was quantified with patient-clustered bootstrap confidence intervals for annotation, retrieval, and overall probing comparisons, while probe-alignment variation used 20-fit summaries and restriction analyses remained descriptive.Shared resamples yielded paired model differences, and multiplicity was unadjusted except for Holm adjustment in the RadLex distance analysis.Source: Statistical analysis · p. 24

Data availability

• CT-RATE, Merlin, RSNA-2023 and INSPECT data are available through Hugging Face, AIMI, Kaggle or Redivis, subject to repository, competition or research-use agreements.INSPECT imaging and report data are available through AIMI, with the linked EHR release available through Redivis; these data are for non-commercial research.Source: Data availability · p. 25
• Individual-level PMBB images, reports, report-derived labels and the full observation bank remain restricted to eligible Penn collaborations with institutional approval and a data-access agreement.Eight illustrative observation strings are reported in Supplementary Table 9.Source: Data availability · p. 25
• Aggregate numerical data underlying the figures will be provided as Source Data with the published article.Source: Data availability · p. 25

Code availability

• ACT’s public release covers preprocessing, model training and inference, observation-bank construction, probing, auditing, statistical analysis, and figure generation, while patient-level PMBB materials and row-level INSPECT data remain unavailable.The release also includes a versioned software environment, prompts, phenotype-rule manifests, academic model weights, and planned redistributable CT-RATE and Merlin observation-bank artifacts; source-data licences and use terms apply.Source: Code availability · p. 25

A.1 Architecture of ACT’s native 3D volume-report model

• ACT aligns CT-volume and radiology-impression embeddings with a symmetric contrastive objective trained on 72,640 paired scan-report rows from 38,317 patients.Volumes are resized to 160 × 224 × 224, encoded slice-wise with DINOv2 ViT-B, aggregated by a Transformer, and paired with a causal text Transformer.Source: Figure 1 · p. 35

A.2 Uncertainty estimation for supplementary AUROC tables

• Zero-shot finding AUROC confidence intervals use 1,000 patient-clustered bootstrap resamples, quantifying patient-sampling variability conditional on fixed predictions.Shared resamples are reused across eligible models and findings, with multiplicity assigned to all associated scans.Source: §A.2 · p. 35
• RSNA-2023 uncertainty analysis reconstructs the recorded patient split and series ordering, pairing 629 reconstructed patient clusters with the retained 943 × 9 outcome matrix.The reconstruction was checked against all 54 displayed model-finding point AUROCs but did not independently recover patient identity.Source: §A.2 · pp. 35–36
• CTPA phenotype-probe intervals quantify variation across 20 matched probe fits and, for overall ACT–CT-CLIP comparisons, across 1,000 shared patient-clustered bootstrap resamples.Each resample recomputes macro AUROC across 221 phenotypes for fixed models in the linear-probe regime.Source: §A.2 · p. 36
• Observation-bank restriction intervals use 1,000 shared patient-clustered bootstrap resamples to quantify held-out patient-sampling variability for fixed full-bank and rule-restricted probes.The analysis recomputes one AUROC per probe and omits phenotype-level and aggregate hypothesis tests.Source: §A.2 · p. 36

A.3 Selection rules for supplementary audit displays

• Shared-direction panels select observations by rank-1 identity alone, displaying strings shared across phenotypes without using interpretation categories, AUROC thresholds, clinical filters, or alignment magnitudes.The most widely shared string covered 20 phenotypes; a distinct biliary ductal dilatation string covered 15, and the panels illustrate candidate proxy hypotheses rather than prevalence estimates.Source: §A.3 · p. 36
• Display-relevance review classifies selected observations as direct target evidence or clinically related context, while preserving the conservative screen and showing four phenotype profiles with ranks 1–5.Neither the rule-based screen nor the display-relevance review was independently adjudicated by a radiologist.Source: §A.3 · p. 36
• Paired audits compare natural full-bank top-five ranks with the five highest bootstrap-mean-weight observations from restricted probes across eight manually reviewed phenotypes.Displayed profiles retain raw observations without near-duplicate removal, uncertainty filtering, or manual reranking.Source: §A.3 · pp. 36–37

B.1 Native-model annotation and retrieval

• Supplementary Tables 1–2 report zero-shot finding annotation on the CT-RATE test set for two model panels using AUROC with 95% percentile confidence intervals.Estimates use 1,000 shared patient-cluster bootstrap resamples; bold marks the largest point estimate across model panels for each finding, and n+ counts positive test scans.Source: Table 1 · p. 37, Table 2 · p. 37
• Supplementary Tables 3–8 extend zero-shot finding annotation to PMBB chest, PMBB abdomen, and RSNA-2023 abdominal-trauma datasets across two model panels per dataset.Values are AUROC with 95% percentile confidence intervals from 1,000 shared patient-cluster bootstrap resamples, with n+ denoting positive test scans.Source: Table 3 · p. 38, Table 4 · p. 39, Table 5 · p. 39
• Supplementary Fig. 2 evaluates ACT’s native 3D model and five baselines using concept-to-image and image-to-concept cross-modal retrieval across four corpora.Concept-to-image uses Recall@1/5/10 in 64-candidate pools, while image-to-concept uses Recall@1/3/5 after per-finding z-scoring; error bars are shared 95% patient-cluster bootstrap confidence intervals.Source: Figure 2 · p. 43
• The retrieval evaluation uses random floors of 1.6%, 7.8%, and 15.6% for concept-to-image Recall@1/5/10 and reports corpus-specific image-to-concept sample sizes.The image-to-concept sample sizes are 2,226, 8,843, 11,646, and 265 scans for CT-RATE, chest PMBB, abdominal PMBB, and RSNA-2023, respectively.Source: Figure 2 · p. 43

B.2 Observation-bank composition and cross-corpus neighbours

• Supplementary Fig. 3 qualitatively summarizes the lexical content of ACT’s report-derived observation bank across its 16 largest keyword-assigned categories.Word frequency is represented by word-cloud size, category counts are shown, colours are decorative, and gastrointestinal is abbreviated as GI.Source: Figure 3 · p. 44
• Supplementary Table 9 lists nearest affirmative and explicitly negated same-finding observations for eight representative PMBB queries spanning four chest and four abdominal observations.Neighbours come from CT-RATE and Merlin and are ranked by Euclidean distance between ℓ2-normalized 5,120-dimensional F2LLM observation vectors before PCA or UMAP.Source: Table 9 · p. 45
• The representative-neighbour analysis excludes exact matches, hedged, mixed-polarity, technical, and degree-only statements, and does not estimate corpus-wide alignment accuracy.Affirmative and negated neighbours are descriptive labels; explicit negation is not an independently verified negative clinical label.Source: Table 9 · p. 45

• Cystic pancreatic lesions are identified in the abdomen.Source: hepatic mass · p. 46
• Lesions are reported in the pancreatic tail and head, with sizes ranging from 24 mm to 5.2 × 4.6 cm.Source: hepatic mass · p. 46
• The reports note maintained pancreatic duct involvement of the cystic lesions.Source: hepatic mass · p. 46

Complete CTPA-EHR phenotype performance

• Matched linear-probe performance is reported for all 221 INSPECT phenotypes.The table compares model point estimates phenotype by phenotype.Source: Table 10 · p. 47
• Mean AUROC values and two-sided 95% t-based confidence intervals summarize 20 matched probe fits.The intervals quantify fit variation rather than patient-sampling uncertainty.Source: Table 10 · p. 47
• For each phenotype, n+ gives the positive count among 2,612 test scans, with the remaining scans classified as negative.Bold marks the larger model point estimate per phenotype.Source: Table 10 · p. 47

B.4 Complete observation-bank restriction performance

• 0.751 versus 0.741 mean AUROC favored rule-restricted over full-bank probes across 86 phenotypes, with higher values for 55 and lower values for 31.Comparisons were descriptive, based on held-out test AUROCs with shared patient-cluster bootstrap resampling.Source: Table 11 · p. 54

B.5 Probe-observation audit: discordance and nonspecific context

• 20 phenotypes shared rank-1 calcific atherosclerotic changes, spanning 13 phecode groups and including osteoporosis, urinary tract infection, and major depressive disorder.Their 100 top-five slots contained 22 distinct observations, with a median of 14 shared top-25 observations.Source: Figure 4 · p. 57
• 15 phenotypes shared rank-1 extensive biliary-system and common-duct dilatation across 13 phecode groups, including constipation, urinary retention, and toxic exposure.Their 75 top-five slots contained 34 distinct observations, with a median of 9 shared top-25 observations.Source: Figure 4 · p. 57
• 36% and 23% observed top-five agreement in columns 2–5 exceeded 5% and 7% chance levels for the calcification and biliary-dilatation panels, respectively.The figure omitted alignment magnitudes because unnormalized probe weights prevented score-magnitude comparisons across phenotypes.Source: Figure 4 · p. 57
• Full-bank profiles for bronchiectasis, pneumonia, coronary atherosclerosis, and pulmonary collapse distinguished direct target evidence from clinically related or qualified context.The audits retained natural full-bank ranks and separately colored direct evidence versus related context; pneumonia’s uncertain rank-1 observation remained light blue.Source: Figure 5 · p. 59
• Paired audits showed refined top-five changes for cardiomegaly, osteoporosis NOS, other aneurysm, and acute pulmonary embolism or infarction, selected for higher refined AUROC.Each audit compared the full-bank top five with the refined top five using mean ±1 s.d. and individual fit estimates.Source: Figure 6 · p. 60
• Paired audits likewise examined diseases of pancreas, lung cancer, osteoarthrosis, and diaphragmatic hernia, selected for lower refined AUROC.Each audit compared the full-bank top five with the refined top five using mean ±1 s.d. and individual fit estimates.Source: Figure 7 · p. 61