PaperlayerView exact revision on arXiv

Source-linked AI summary

Pillar-0: A New Frontier for Radiology Foundation Models

Kumar Krishna Agrawal, Longchao Liu, Long Lian, Michael Nercessian, Natalia Harguindeguy, Yufu Wu, Peter Mikhael, Gigin Lin, Lecia V. Sequist, Florian Fintelmann, Trevor Darrell, Yutong Bai, Maggie Chung, Adam Yala

arXiv:2511.17803v1cs.CVcs.AI

TL;DR

Radiology foundation models and evaluation benchmarks remain limited in representing full volumetric scans and routine clinical tasks. Pillar-0 pairs a volumetric radiology model with RATE, and ranks best on 87.2% (319/366) of evaluated tasks.

  • Problem

    Existing radiology foundation models and benchmarks have limited coverage of general-purpose tasks and full-fidelity, clinically representative evaluation.

    Source: §1 · p. 2

  • Method

    Pillar-0 combines a volumetric radiology foundation model using multiple intensity windows and multi-scale attention with RATE, which extracts labels for 366 findings from radiology reports.

    Source: §2.0 · p. 4, §1 · p. 4

  • Results

    Across 366 evaluated tasks, Pillar-0 ranked best in 87.2% (319/366), outperforming four comparison models.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Pillar-0 and RATE provide an open, clinically grounded foundation for reproducible comparison and further development of radiology systems.

    Source: §1 · p. 4

    Pretraining data came from one tertiary academic center, and its scanner, protocol, demographic, and disease-prevalence diversity may not represent other health systems.

    Source: §3 · p. 11

Abstract

from arXiv · show

Radiology plays an integral role in modern medicine, yet rising imaging volumes have far outpaced workforce growth. Foundation models offer a path toward assisting with the full spectrum of radiology tasks, but existing medical models remain limited: they process volumetric CT and MRI as low-fidelity 2D slices, discard critical grayscale contrast information, and lack evaluation frameworks that reflect real clinical practice. We introduce Pillar-0, a radiology foundation model pretrained on 42,990 abdomen-pelvis CTs, 86,411 chest CTs, 14,348 head CTs, and 11,543 breast MRIs from a large academic center, together with RATE, a scalable framework that extracts structured labels for 366 radiologic findings with near-perfect accuracy using LLMs. Across internal test sets of 14,230 abdomen-pelvis CTs, 10,646 chest CTs, 4,906 head CTs, and 1,585 breast MRIs, Pillar-0 establishes a new performance frontier, achieving mean AUROCs of 86.4, 88.0, 90.1, and 82.9, outperforming MedGemma (Google), MedImageInsight (Microsoft), Lingshu (Alibaba), and Merlin (Stanford) by 7.8-15.8 AUROC points and ranking best in 87.2\% (319/366) tasks. Pillar-0 similarly outperforms all baselines in an external validation on the Stanford Abdominal CT dataset, including Merlin (82.2 vs 80.6 AUROC). Pillar-0 extends to tasks beyond its pretraining, such as long-horizon lung cancer risk prediction, where it improves upon the state-of-the-art Sybil by 3.0 C-index points on NLST, and generalizes with gains of 5.9 (MGH) and 1.9 (CGMH). In brain hemorrhage detection, Pillar-0 obtained a >95 AUROC when using only 1/20th of the data of the next most sample efficient baseline. Pillar-0 and RATE together provide an open, clinically rigorous foundation for building high-performance radiology systems, enabling applications that were previously infeasible due to computational, data, and evaluation constraints.

1. Main

Rising imaging volumes outpace radiology workforce growth, while existing models and benchmarks do not capture the full complexity of clinical volumetric imaging. Pillar-0 and RATE address this gap with a radiology foundation model and clinically grounded evaluation framework.

  • Pillar-0 achieved mean AUROCs of 90.11 (head CT), 88.0 (chest CT), 86.4 (abdomen-pelvis CT), and 82.9 (breast MRI), improving 7.8–15.8 over the next best model and leading 319/366 tasks.Pretraining included 42,990 abdomen-pelvis CTs, 86,411 chest CTs, 14,348 head CTs, and 11,543 breast MRIs.Source: Figure 1 · p. 3, §1 · p. 4
  • RATE uses radiologist-curated labels for 366 findings and open LLMs such as Qwen3 to extract structured findings from reports with near-perfect accuracy.It provides an efficient way to build labeled datasets for reproducible model comparisons.Source: §1 · p. 4
  • On Stanford external validation, Pillar-0 outperformed the tested foundation models and, using Merlin’s training data, scored 82.2 versus Merlin’s 80.6 despite fewer forms of supervision.Source: §1 · p. 4
  • Finetuned Pillar-0 improved lung-cancer risk prediction beyond Sybil, with C-index gains of 5.9 (81.0 to 86.9) at MGH and 1.9 (81.5 to 83.4) at CGMH.The task is outside Pillar-0’s pretraining distribution.Source: Figure 1 · p. 3, §1 · p. 4
  • Pillar-0 reached >95 AUROC for brain hemorrhage detection using only 1/20 of the data required by the next most sample-efficient baseline.The case study used the RSNA Brain CT challenge dataset.Source: Figure 1 · p. 3, §1 · p. 4
  • The authors released open-source tools for evaluation, volume preprocessing, pretraining, finetuning, and inference, along with all pretrained models.Source: Figure 1 · p. 3, §1 · p. 4

2. Results

Pillar-0 combines multi-window volumetric inputs, efficient long-context processing, and LLM-aligned pretraining; RATE evaluates vision models on full-fidelity volumes using clinical tasks. The supplied results also identify limitations in existing radiology benchmarks.

2.0. Overview of Pillar-0 and Core Innovations

  • Pillar-0 uses radiology-inspired multi-window tokenization, Atlas multi-scale attention, and asymmetric contrastive learning to align volume representations with Qwen-8B text representations.The multiple intensity-range windows aim to preserve tissue properties across channels, while Atlas processes large spatial context.Source: §2.0 · p. 4, Figure 2 · p. 5
  • Using a large LLM text encoder rather than a smaller model boosts performance and strengthens the correlation between pretraining loss and downstream results.The passage describes the stronger loss-performance correlation as a more predictable scaling signal; Figure 2 characterizes it as a signal for clinical utility in pretraining experiments.Source: §2.0 · p. 4, Figure 2 · p. 5
  • 4.6-point AUROC gain from multi-windowing accompanies Atlas processing that is 175× faster than ViT-S.Figure 2 attributes the gain to training with multi-windowing and describes Atlas as using hierarchical multi-scale attention for long-context volumes.Source: §2.0 · p. 4, Figure 2 · p. 5

2.1. RATE: Clinically Grounded Evaluation Framework

  • RATE evaluates vision models on full-fidelity medical volumes using authentic clinical tasks derived from real-world radiology practice.It is presented as a unified framework for evaluating any vision model.Source: §2.1 · p. 5
  • Existing benchmarks use 2D JPEG slices or textbook snapshots, often define tasks unlike radiologists’ routine responsibilities, and typically use synthetic or manually annotated labels.The supplied passage identifies these as shortcomings of existing radiology benchmarks.Source: §2.1 · p. 5

A. RATE comparison with existing radiology evaluation frameworks

RATE provides clinically grounded, extensible evaluation using full-resolution radiology volumes and report-derived labels; Pillar-0 achieves dominant overall performance across evaluated modalities on UCSF held-out test sets.

A. RATE comparison with existing radiology evaluation frameworks

  • RATE alone combines full-resolution volumes, clinically grounded tasks labeled from routine practice, and extensibility to any radiology image-report dataset.Existing frameworks are described as limited to fixed datasets without a pathway to expand or adapt tasks.Source: A. RATE comparison with existing… · p. 6, Figure 3 · p. 6
  • Radiologists curated 366 diverse real-world radiologic findings, and open large language models extract scalable binary labels from radiology reports.The findings span the paper’s evaluated modalities.Source: A. RATE comparison with existing… · p. 6
  • RATE-Evals standardizes vision-encoder assessment by freezing each encoder and training one linear classifier per task on RATE labels, then evaluating held-out exams.This protocol measures representation quality and transferability, analogous to CLIP-style linear evaluation [40].Source: A. RATE comparison with existing… · p. 6

2.2. Pillar-0 Obtains Dominant Performance on Internal Test Sets

  • Pillar-0 achieves dominant overall performance across all evaluated modalities on UCSF held-out test sets against four competitive medical-imaging baselines.MedGemma, Lingshu, and MedImageInsight are large 2D models, while Merlin is an open-source 3D model trained on institutional abdomen-pelvis CTs.Source: §2.2 · p. 6

A. Pillar-0 comparison with baselines

Pillar-0 outperforms baseline models on internal and external radiology benchmarks and improves performance in lung cancer risk prediction and brain hemorrhage detection. Ablations attribute its gains to complementary choices in tokenization, architecture, and pretraining.

A. Pillar-0 comparison with baselines

  • Pillar-0 wins on 319/366 findings (87.2%) and leads the closest baseline, MedGemma, by 7.8–15.8 average AUROC points across four internal modalities.Its average AUROCs are 86.4 for abdomen-pelvis CT, 88.0 for chest CT, 90.1 for head CT, and 82.9 for breast MRI.Source: Figure 4 · p. 7, A. Pillar-0 comparison with baselines · p. 8

2.3. Pillar-0 Outperforms Foundation Models on External Test Sets

  • On the Stanford external dataset, Pillar-0 achieves 82.2 average AUROC, outperforming Merlin (80.6), MedGemma (72.6), MedImageInsight (74.9), and Lingshu (72.1).Pillar-0 (Stanford Only) also beats Merlin (82.2 vs 80.6), while Pillar-0 (UCSF + Stanford) reaches 84.9 vs 80.6.Source: §2.3 · p. 8, Table 1 · p. 8

2.4. Pillar-0 Improves the Ability to Predict Future Cancer from Asymptomatic Screening

  • Finetuned Pillar-0 improves Sybil’s 1-year AUROC from 91.5 to 94.5 on NLST, from 85.9 to 90.8 on MGH, and from 95.1 to 96.8 on CGMH.Sybil-1.5 predicts lesion locations and annual lung cancer risk scores 1–6 years after a single low-dose CT; the reported improvements are for 1-year AUROC.Source: §2.4 · p. 9, Figure 5 · p. 9

2.5. Pillar-0 Reduces Downstream Training Data Needs in Brain Hemorrhage Detection by >20×

  • With 2.5% of training data, Pillar-0 reaches 95.0 AUROC for any intracranial hemorrhage, requiring 20–40× fewer samples than baselines to reach that performance.Swin3D-t required 50% of samples to reach 95.0 AUROC; Merlin required 75%, and ResNet3D required 100%.Source: §2.5 · p. 10, Figure 6 · p. 10

2.6. Ablation Study

  • Ablations on the Stanford Merlin CT dataset indicate that Pillar-0’s gains reflect combined contributions from modality-aware tokenization, efficient multi-scale 3D architecture, and asymmetric contrastive pretraining.Multi-windowing yields 82.2 vs 77.6 AUROC with min–max scaling; Atlas inference takes 0.2 vs 38.8 seconds for ViT-S (175× speedup). Using Qwen3 rather than RoBERTa-Base improves AUROC from 76.6 to 82.2 and strengthens loss–RATE correlation (Pearson r = −0.947 vs. −0.256).Source: §2.6 · pp. 10–11

3. Discussion

Pillar-0 and RATE advance volumetric radiology modeling and clinically grounded evaluation, with reported gains across diagnostic, risk-prediction, and sample-efficiency tasks. The discussion also identifies limits in data diversity, report-derived labels, and pretraining supervision.

Discussion

  • Pillar-0 establishes a new performance frontier across abdomen-pelvis CT, chest CT, head CT, and breast MRI, outperforming leading 2D and 3D models and generalizing across institutions.It also outperforms Merlin on its own in-domain dataset, including when pretrained on the same data.Source: §3 · p. 11
  • On NLST, Pillar-0 sets a new state of the art for long-horizon lung cancer risk prediction and generalizes across external cohorts; for brain hemorrhage detection, it matches or exceeds full-data baselines using 10% of training data.The passage characterizes this as reducing data requirements by more than an order of magnitude.Source: §3 · p. 11
  • Ablations attribute Pillar-0’s gains to multi-window tokenization, an efficient multi-scale 3D backbone, and asymmetric contrastive pretraining with a strong text encoder, producing predictable, clinically aligned scaling behavior.The discussion links its tokenization, architecture, and pretraining to leveraging spatial context and intensity patterns in volumetric CT and MRI.Source: §3 · p. 11
  • Pillar-0’s sample efficiency, external generalization, and downstream initialization utility could support specialist models where annotations are limited, while releasing models, code, and RATE lowers computational and data barriers.The discussion presents these capabilities as enabling radiology AI applications that were previously infeasible.Source: §3 · p. 11
  • Pretraining used 218,217 CTs and MRIs from one tertiary academic center and an 89M-parameter vision encoder; broader datasets and greater model capacity may improve generalizability.Scanner vendors, protocols, demographics, and disease prevalence may not capture diversity across healthcare systems.Source: §3 · p. 11
  • RATE’s binary report-derived labels can inherit omissions and label noise, do not assess localization, segmentation, or temporal evolution, and Pillar-0 omits report-generation and other clinical-context supervision.The passage notes that radiology reports may omit normal or incidental findings and that missing mentions treated as negative labels can introduce noise.Source: §3 · pp. 11–12

A. Methods · A.0. Pillar-0 · A.0.1. Development Dataset

The development dataset comprises adult UCSF CT and breast MRI exams collected from 2001–2025, curated by modality, and divided patient-wise into training, validation, and test splits.

A.0.1. Development Dataset

  • The dataset included 71,510 abdomen-pelvis CT, 107,923 chest CT, 24,042 head CT, and 14,742 breast MRI exams.Source: §A.0.1 · p. 13
  • Exams were performed in adult patients at UCSF between 2001 and 2025.Source: §A.0.1 · p. 13
  • Exams were retrieved through UCSF’s institutional radiology database and Automated Image Retrieval platform.Source: §A.0.1 · p. 13
  • For CT, biopsies and non-reportable exams were excluded, and redundant series were reduced by acquisition time and slice thickness before random selection when multiple series remained.Source: §A.0.1 · p. 13
  • For breast MRI, series descriptions were filtered to retain T1 fat-saturation, T2 fat-saturation, and peak contrast-enhanced series, with acquisition time distinguishing post-contrast series.Source: §A.0.1 · p. 13
  • Patient-wise training, validation, and test splits were created, with exam counts by modality and split summarized in Table 2.Source: §A.0.1 · p. 13, Table 2 · p. 13

A.0.2. Pillar-0 Training Recipe … A.5. Sample Efficiency Case Study

These sections describe Pillar-0’s full-volume training recipe and RATE’s report-to-label evaluation framework, then detail validation and downstream case-study protocols. They emphasize efficient high-resolution modeling, modality-specific image processing, and extensible evaluation, while the supplied passages primarily report methods and study setups rather than final benchmark comparisons.

A.0.2. Pillar-0 Training Recipe

  • Atlas enables full-resolution 3D training with 32k–256k tokens per volume, while modality-specific windowing preserves contrast across CT and breast MRI.The architecture uses fine patches to retain lesions a few millimeters in size; CT channels use anatomically motivated windows, while MRI windows span the 1st–99th intensity percentiles. Vision-language pretraining updates the vision encoder and projection layers against a frozen Qwen3-Embedding-8B text encoder.Source: §A.0.2 · pp. 15–16, Table 7 · p. 15

A.1.1. RATE

  • RATE converts radiology reports into binary labels for expert-curated clinical questions and supports new modalities and query sets.It uses Qwen3-30B-A3B-FP8 to answer Yes/No questions and creates quality-control records; it can also extract verbatim findings text after removing comparisons for vision-language pretraining.Source: §A.1.1 · pp. 16–17

A.1.2. Using RATE to evaluate any vision encoder

  • RATE-Evals assesses frozen vision encoders by training linear probes on embeddings to predict RATE labels, with missing report mentions treated as negative by default.It also supports masking to restrict analysis to explicitly labeled samples and provides extensible dataset and model templates.Source: §A.1.2 · p. 17

A.1.3. Baselines

  • RATE-Evals supports Pillar-0 and external baselines including MedGemma, MedImageInsight, Merlin, and Lingshu, whose implementations pool or encode volume and slice features differently.The described baselines include multimodal models and models trained with contrastive or diagnosis-prediction objectives; volumetric inputs are represented as 2D slices by some implementations.Source: §A.1.2 · p. 17, §A.1.3 · pp. 17–18

A.2. Internal Validation

  • Internal evaluation used a UCSF held-out test set spanning four modalities and 366 RATE-derived findings; radiologists agreed with extracted labels in the reviewed sample.Board-certified radiologists reviewed 20 randomly sampled reports per modality (80 total), with 100% agreement; findings without positive test examples were excluded from performance estimates.Source: §A.2 · p. 18

A.3. External Validation

  • External evaluation used 5,137 held-out Stanford abdominal-pelvis CT exams and 202 RATE findings, comparing Pillar-0 with baselines and two Pillar-0 variants.One variant was pretrained only on the Merlin training corpus; the other continued Pillar-0 pretraining on Merlin for 14 epochs.Source: §A.3 · p. 18

A.4. Lung Cancer Case Study

  • The lung cancer case study finetuned Pillar-0 on NLST data and added year-by-year risk prediction and a DETR-based bounding-box head.Training combined Sybil survival and attention-regularization losses with DETR regression and matching losses; MGH and CGMH cohorts were also included.Source: §A.4 · p. 18

A.5. Sample Efficiency Case Study

  • The sample-efficiency study compared models on patient-wise RSNA-2019 splits using identical CT preprocessing, including an 11-channel multi-window volume.Models trained for up to 10,000 steps with shared optimization settings; RadImageNet processed slices individually, while the other model implementations used their respective volumetric feature pipelines.Source: §A.5 · p. 19

A.6. RAVE: Unified, Efficient Radiology Data Processing · A. Sybil vs Sybil 1.5 · B. Comparing Pillar-0 to MedGemma

Pillar-0 wins most Pillar-0–MedGemma comparisons across four imaging modalities, while RAVE addresses volume-data storage and preprocessing constraints. The Sybil-1.5 tables cover held-out NLST subgroup performance and RSNA-2019 sample efficiency.

A.6. RAVE: Unified, Efficient Radiology Data Processing

  • RAVE uses HEVC compression of DICOM series and NIfTI volumes to fit large medical-volume datasets in limited NVMe storage while preserving detail.The framework exploits slice-to-slice redundancy to compress datasets comprising millions of slices and tens of terabytes.Source: §A.6 · p. 19
  • RAVE preprocessing standardizes volumes through isotropic resampling, fixed-dimension spatial normalization, and multi-windowing to produce GPU-ready tensors.The paper also lists a suite of released open-source tools.Source: §A.6 · p. 19, Table 8 · p. 19

A. Sybil vs Sybil 1.5

  • Table 9 reports Sybil-1.5 held-out NLST subgroup AUROC at 1, 3, and 5 years and C-index, with 95% confidence intervals capped at 100.The supplied passage gives the table's scope but not its subgroup values or comparative outcomes.Source: Table 9 · p. 23
  • Table 10 reports RSNA-2019 sample-efficiency results as validation-set AUROC across training-data fractions.The supplied passage identifies the evaluation measure and dataset but does not provide numeric results.Source: Table 10 · p. 23

B. Comparing Pillar-0 to MedGemma

  • Pillar-0 wins 190/210 (90.5%) abdomen-pelvis CT tasks and 85/92 (92.4%) chest CT tasks against MedGemma.The figures report the head-to-head comparison on UCSF RATE-Evals tasks.Source: Figure 7 · p. 24, Figure 14 · p. 31
  • Pillar-0 wins 28/29 (96.6%) head CT tasks and 33/35 (94.3%) breast MRI tasks against MedGemma.The figures report the head-to-head comparison on UCSF RATE-Evals tasks.Source: Figure 17 · p. 34, Figure 18 · p. 35

C. Performance on full set of RATE-Evals tasks on UCSF Abdomen-Pelvis CT test set

Table 11 compares model AUCs across findings on the UCSF abdomen-pelvis CT test set. Pillar-0 scores exceed listed alternatives on several renal and splenic findings, although performance varies by task.

C. Performance on full set of RATE-Evals tasks on UCSF Abdomen-Pelvis CT test set

D. Performance on full set of RATE-Evals tasks on UCSF Chest CT test set

Among the 19 displayed UCSF Chest CT findings, Pillar-0 achieved the highest AUC on 15 and tied for the highest on one, while trailing other models on ECMO and arthritis.

D. Performance on full set of RATE-Evals tasks on UCSF Chest CT test set

E. Performance on full set of RATE-Evals tasks on UCSF Head CT test set

Table 13 reports model performance as AUC × 100 on the UCSF Head CT test set, with supplied results covering multiple radiologic findings.

E. Performance on full set of RATE-Evals tasks on UCSF Head CT test set

  • On UCSF Head CT, five reported AUC × 100 values span 56.4–86.2 for brain tumors, 70.4–92.1 for edema, 79.8–95.2 for hydrocephalus, and 68.8–93.8 for ICH.Source: Table 13 · p. 43
  • For suspected intracranial metastasis, the five reported AUC × 100 values are 65.9, 72.7, 73.7, 60.2, and 90.2.Source: Table 13 · p. 43
  • For brain atrophy, the five reported AUC × 100 values are 82.6, 79.5, 80.8, 72.9, and 85.5.Source: Table 13 · p. 43

F. Performance on full set of RATE-Evals tasks on UCSF Breast MRI test set

Table 14 reports model AUC × 100 for radiologic findings on the UCSF Breast MRI test set.

F. Performance on full set of RATE-Evals tasks on UCSF Breast MRI test set

G. Performance on full set of RATE-Evals tasks on Merlin Abdomen-Pelvis CT test

Table 15 compares finding-level AUCs (×100) across models on the Merlin abdomen-pelvis CT test; Pillar-0 scores highest on several renal, pancreatic, splenic, and other findings, but not uniformly.

Renal findings

  • Pillar-0 scored 98.9 for renal lacerations versus Merlin’s 95.2, 81.1 for simple renal cysts versus 68.1, and 79.2 for renal hypodensities versus 66.2.Values are AUC × 100.Source: Table 15 · p. 46, set · p. 47

Pancreas-related findings

  • Pillar-0 scored 97.5 for ductal pancreatic carcinomas versus Merlin’s 95.9, 78.1 for pancreatic tumors versus 76.6, and 78.0 for chronic pancreatitis versus 73.4.Values are AUC × 100.Source: Table 15 · p. 46, set · p. 49

Splenic findings

  • Pillar-0 scored 95.2 for splenic infarcts versus Merlin’s 88.9 and 91.2 for splenic lymphoma versus 84.3, but scored 89.3 for splenic lacerations versus Merlin’s 93.2.Values are AUC × 100.Source: Table 15 · p. 46, set · p. 50

Other findings

  • Pillar-0 scored 99.1 for bladder rupture versus Atlas’s 88.0 and 99.2 for primary sclerosing cholangitis versus Atlas’s 95.8.Values are AUC × 100.Source: Table 15 · p. 46, set · p. 49
  • Pillar-0 scored 46.9 for angiosarcoma versus Merlin’s 60.2 and 62.4 for retroperitoneal liposarcoma versus MedImageInsight’s 77.2.Values are AUC × 100.Source: Table 15 · p. 46, set · p. 49

Related papers

Loading 2511.17803v1…