Source-linked AI summary
Pillar-0: A New Frontier for Radiology Foundation Models
Kumar Krishna Agrawal, Longchao Liu, Long Lian, Michael Nercessian, Natalia Harguindeguy, Yufu Wu, Peter Mikhael, Gigin Lin, Lecia V. Sequist, Florian Fintelmann, Trevor Darrell, Yutong Bai, Maggie Chung, Adam Yala
TL;DR
Radiology foundation models are limited by low-fidelity volumetric processing and evaluations that do not reflect routine clinical practice. Pillar-0 combines full-volume modeling with RATE, achieving leading performance across modalities and downstream tasks while releasing an open system.
Problem
Existing radiology models process volumetric scans inadequately and lack evaluation frameworks grounded in real clinical tasks.
Method
Pillar-0 is a volumetric radiology foundation model paired with RATE, which extracts clinically grounded labels for 366 findings from radiology reports.
Results
Pillar-0 ranked first on 319 of 366 internal tasks and outperformed all baselines in external Stanford abdominal CT validation, including Merlin at 82.2 versus 80.6 AUROC.
Takeaways & Limitations
Pillar-0 and RATE provide an open, clinically grounded platform for developing radiology systems across modalities and tasks.
Takeaways & Limitations
Pretraining data came from a single tertiary academic center, so scanner, protocol, demographic, and disease-prevalence diversity may not represent other healthcare systems.
Abstract
from arXiv · showhide
Radiology plays an integral role in modern medicine, yet rising imaging volumes have far outpaced workforce growth. Foundation models offer a path toward assisting with the full spectrum of radiology tasks, but existing medical models remain limited: they process volumetric CT and MRI as low-fidelity 2D slices, discard critical grayscale contrast information, and lack evaluation frameworks that reflect real clinical practice. We introduce Pillar-0, a radiology foundation model pretrained on 42,990 abdomen-pelvis CTs, 86,411 chest CTs, 14,348 head CTs, and 11,543 breast MRIs from a large academic center, together with RATE, a scalable framework that extracts structured labels for 366 radiologic findings with near-perfect accuracy using LLMs. Across internal test sets of 14,230 abdomen-pelvis CTs, 10,646 chest CTs, 4,906 head CTs, and 1,585 breast MRIs, Pillar-0 establishes a new performance frontier, achieving mean AUROCs of 86.4, 88.0, 90.1, and 82.9, outperforming MedGemma (Google), MedImageInsight (Microsoft), Lingshu (Alibaba), and Merlin (Stanford) by 7.8-15.8 AUROC points and ranking best in 87.2\% (319/366) tasks. Pillar-0 similarly outperforms all baselines in an external validation on the Stanford Abdominal CT dataset, including Merlin (82.2 vs 80.6 AUROC). Pillar-0 extends to tasks beyond its pretraining, such as long-horizon lung cancer risk prediction, where it improves upon the state-of-the-art Sybil by 3.0 C-index points on NLST, and generalizes with gains of 5.9 (MGH) and 1.9 (CGMH). In brain hemorrhage detection, Pillar-0 obtained a >95 AUROC when using only 1/20th of the data of the next most sample efficient baseline. Pillar-0 and RATE together provide an open, clinically rigorous foundation for building high-performance radiology systems, enabling applications that were previously infeasible due to computational, data, and evaluation constraints.
1. Main
Pillar-0 addresses limitations in volumetric radiology modeling and evaluation with a general-purpose foundation model and RATE. It achieves dominant performance across modalities and downstream tasks while supporting open scientific use.
- RATE uses radiologist-curated findings and LLM-extracted labels to provide scalable, clinically grounded evaluation across 366 radiologic tasks.
- 87.2% (319/366) of evaluated tasks ranked Pillar-0 first, with mean AUROCs of 90.11, 88.0, 86.4, and 82.9 across head CT, chest CT, abdomen-pelvis CT, and breast MRI.
- Pillar-0 outperformed Merlin on external Stanford abdominal CT validation, including 82.2 versus 80.6 AUROC after retraining with Merlin’s training data.
- Pillar-0 improved lung cancer risk prediction beyond its pretraining distribution, increasing concordance index by 5.9 points at MGH and 1.9 points at CGMH over Sybil.
- >95 AUROC for brain hemorrhage detection was achieved using only 1/20 of the data required by the next most sample-efficient baseline.
- The released system includes open-source evaluation, preprocessing, training, finetuning, inference tools, and pretrained models.
2. Results
Pillar-0 combines modality-specific tokenization, efficient volumetric attention, and report-supervised pretraining to address the computational and representational challenges of 3D radiology.
- Core innovations: Multi-window tokenization preserves clinically relevant contrast by converting full-resolution CT and MRI volumes into modality-specific multi-channel inputs.
- Core innovations: Atlas uses multi-scale attention to process large volumetric contexts efficiently, reaching 175× the speed of a comparable vision transformer.
- Core innovations: Asymmetric contrastive pretraining aligns volume embeddings with a larger frozen LLM text encoder to distill radiology report supervision.
- RATE: RATE evaluates vision models on full-fidelity medical volumes using authentic clinical tasks derived from routine radiology practice.
A. RATE comparison with existing radiology evaluation frameworks
RATE addresses major shortcomings of radiology benchmarks by combining full-resolution volumetric inputs, clinically grounded labels, and extensible evaluation procedures.
- RATE is the only compared framework described as combining full-resolution volumes, clinically grounded routine-practice labels, and extensibility to radiology image-report datasets.
- RATE uses an LLM to extract answers from unstructured radiology reports, enabling scalable generation of clinically grounded binary labels.
- RATE-Evals freezes each pretrained encoder and trains a linear classifier per task, measuring representation quality and transferability on held-out exams.
A. Pillar-0 comparison with baselines
Pillar-0 outperforms competing radiology foundation models across internal and external benchmarks, while extending to lung cancer risk prediction and data-efficient hemorrhage detection. Its gains are associated with modality-aware tokenization, efficient 3D architecture, and clinically aligned representation learning.
- Internal evaluation: Pillar-0 wins 319/366 findings (87.2%) on UCSF internal tests, with average AUROCs of 86.4–90.1 across CT modalities and 82.9 for breast MRI.It exceeds the closest baseline by 7.8–15.8 AUROC points.
- External evaluation: Pillar-0 reaches 82.2 average AUROC on the external Stanford abdominal CT dataset, exceeding Merlin’s 80.6 despite Merlin being developed on that dataset.Using only Stanford data, Pillar-0 also exceeds Merlin 82.2 vs 80.6; UCSF-plus-Stanford pretraining reaches 84.9 vs 80.6.
- Downstream applications: Finetuned Pillar-0 improves lung cancer risk prediction over Sybil across NLST, MGH, and CGMH cohorts and across evaluated time horizons and metrics.On NLST, 1-year AUROC increases from 91.5 to 94.5; on MGH, it increases from 85.9 to 90.8.
- Downstream applications: Pillar-0 achieves 95.0 AUROC for brain hemorrhage detection using 2.5% of training data, while the best baseline requires 50% to reach the same performance.This corresponds to a 20- to 40-fold improvement in sample efficiency.
- Ablation study: Ablations link performance gains to multi-window tokenization, multi-scale 3D attention, and a high-capacity text encoder aligned with RATE performance.Multi-windowing raises Merlin RATE-Evals AUROC from 77.6 to 82.2, while Atlas reduces inference time from 38.8 to 0.2 seconds per sample versus ViT-S.
3. Discussion
Pillar-0 combines volumetric modeling and clinically grounded evaluation to improve radiology foundation-model performance and downstream transfer. Its main scope boundaries are single-center pretraining, report-derived labels, and contrastive-only supervision.
- Discussion: Pillar-0 establishes a performance frontier across CT and MRI, outperforming leading 2D and 3D medical models and generalizing across institutions.The model also serves as an initialization for downstream development and supports tasks beyond radiologist-performed interpretation.
- Discussion: RATE provides scalable, reproducible benchmarking by extracting structured labels for hundreds of radiologic findings from clinical reports.The framework aligns evaluation with radiologist-prioritized findings and diagnoses.
- Discussion: Pillar-0’s tokenization, architecture, and pretraining leverage 3D spatial context and intensity patterns for diagnostic and long-horizon risk tasks.The authors report reduced downstream data requirements, strong external generalization, and open release of models, code, and RATE.
- Discussion: Pretraining uses data from one tertiary academic center, limiting coverage of scanner vendors, protocols, demographics, and disease prevalence across healthcare systems.The authors identify larger and more diverse datasets as important for improving generalizability.
- Discussion: RATE inherits report-supervision limitations, including omitted findings, incomplete appearance descriptions, and potential errors from treating missing mentions as negative.The framework also currently relies on binary labels and contrastive pretraining rather than full report generation or additional clinical context.
A. Methods
The methods assemble large UCSF CT and breast MRI datasets from institutional imaging systems and characterize their demographic and scanner distributions. The development data span multiple modalities and are organized into modality-specific splits.
- A. Methods: 71,510 abdomen-pelvis CT, 107,923 chest CT, 24,042 head CT, and 14,742 breast MRI exams were identified at UCSF between 2001 and 2025.The cohort comprised adult patients and was retrieved using institutional radiology and image-retrieval platforms.
- A. Methods: The pretraining dataset summary reports exam counts per split separately for abdomen-pelvis CT, chest CT, head CT, and breast MRI.The dataset is organized by modality rather than as a single pooled imaging cohort.
- A. Methods: Dataset characterization includes age, gender, and manufacturer distributions for head CT, abdomen-pelvis CT, chest CT, and breast MRI.The reported denominators differ by modality and reflect patient or exam totals.
A.0.2. Pillar-0 Training Recipe
Pillar-0’s training recipe combines full-resolution volumetric encoding, radiology-specific intensity preprocessing, modality-specific vision-language pretraining, and RATE-based clinical evaluation. RATE-Evals then measures frozen representation transfer with linear probes.
- Architecture: Atlas uses multi-scale attention to make full-resolution, long-context 3D volume modeling computationally feasible while preserving fine lesions and whole-field anatomical coverage.Its attention complexity scales as O(Nlog N) rather than O(N^2).
- Radiology-specific tokenizer: Multi-window CT tokenization converts Hounsfield-unit volumes into channels emphasizing lung, soft tissue, mediastinal, and bone structures.Each window is independently clipped and rescaled to [0, 1].
- Radiology-specific tokenizer: Breast MRI uses foreground-histogram windows spanning the 1st–99th intensity percentiles, followed by linear rescaling to [0, 1].This adaptive strategy is applied separately to each complementary MRI series.
- Pretraining pipeline: Single-modality vision-language pretraining updates the vision encoder and projection layers while keeping the 8B Qwen3 text encoder frozen.The asymmetric contrastive objective matches volumes with their reports and treats other batch combinations as negatives.
- Clinical evaluation: RATE converts reports into binary labels from expert-curated clinical questions, while RATE-Evals benchmarks frozen encoders using linear probes.The pipeline supports quality control, extensible task definitions, and clinically grounded captions from report findings.
- Clinical evaluation: Evaluation summarizes per-question AUROC after class-balanced linear-probe training, with missing report mentions treated as negative by default.A masking mode can instead restrict training and analysis to explicitly labeled samples.
- Evaluation and release: Baseline comparisons use identical CT preprocessing, including 256 × 256 × 128 volumes and 11-channel multi-window inputs.The released checkpoints and code support reproducible modality-specific pretraining and evaluation.
A.6. RAVE: Unified, Efficient Radiology Data Processing
RAVE addresses storage and preprocessing bottlenecks for large-scale radiology volumes by compressing source data and producing standardized GPU-ready tensors. The surrounding evaluation materials include sample-efficiency and head-to-head benchmark artifacts.
- RAVE: Unified, Efficient Radiology Data Processing: RAVE compresses DICOM series and NIfTI volumes with HEVC to reduce the storage burden of millions of medical-image slices.The system targets datasets spanning tens of terabytes that are impractical to keep on local NVMe storage.
- RAVE: Unified, Efficient Radiology Data Processing: RAVE standardizes isotropic resampling, spatial normalization, and multi-windowing into GPU-ready tensors.These operations complement compression by preparing consistent model inputs.
- Released tools and evaluation artifacts: The released-tooling summary includes RAVE and related open-source components, while the accompanying tables report sample-efficiency results and subgroup performance.The supplied table captions identify validation AUROC across training fractions and subgroup metrics for Sybil-1.5.
- Comparing Pillar-0 to MedGemma: Pillar-0 wins 190/210 abdomen-pelvis CT RATE-Evals tasks against MedGemma, compared with 20/210 wins for MedGemma.The head-to-head figures divide this comparison across seven parts.
- Comparing Pillar-0 to MedGemma: Pillar-0 wins 85/92 chest CT tasks, 28/29 head CT tasks, and 33/35 breast MRI tasks against MedGemma.The corresponding win rates are 92.4%, 96.6%, and 94.3%, respectively.
C. Performance on full set of RATE-Evals tasks on UCSF Abdomen-Pelvis CT test set
On the UCSF abdomen-pelvis CT test set, Pillar-0 shows strong performance across a broad set of RATE-eval findings, often exceeding the comparison models.
- 98.7 AUC is achieved for a reported abdomen-pelvis finding, exceeding 87.2 for MedImageInsight and 85.1 for Merlin.
- Pillar-0 reaches 99.5 AUC for a reported finding, above 86.9 for MedGemma and 74.4 for MedImageInsight.
- Pillar-0 records 98.2 AUC for a reported finding, compared with 93.1 for MedGemma and 88.7 for LingShu.
- 99.9 AUC is achieved by Pillar-0 for splenic abscesses, compared with 93.5 for MedGemma and 92.2 for MedImageInsight.
D. Performance on full set of RATE-Evals tasks on UCSF Chest CT test set
On the UCSF chest CT test set, Pillar-0 attains higher AUCs than the listed comparison models for several reported thoracic findings, though not every task.
- 87.4 AUC is reported for a chest finding, above 79.8 for MedImageInsight and 79.1 for MedGemma.
E. Performance on full set of RATE-Evals tasks on UCSF Head CT test set
On the UCSF head CT test set, Pillar-0 reports high AUCs across tumor, edema, hydrocephalus, hemorrhage, and device-related findings.
- 90.2 AUC is reported for a head-CT finding, above 73.7 for MedImageInsight and 72.7 for MedGemma.
- Pillar-0 reaches 99.9 AUC for deep-brain-stimulation-device detection, exceeding 97.2 for MedGemma and 96.9 for MedImageInsight.
F. Performance on full set of RATE-Evals tasks on UCSF Breast MRI test set
On the UCSF breast MRI test set, Pillar-0 reports AUCs for postoperative changes, metastases, and tumor morphology findings, with higher values on the listed tasks than the comparison models.
- 88.0 AUC is achieved for spiculated mass margins, above 70.7 for MedImageInsight and 70.1 for MedGemma.
- 85.5 AUC is achieved for tumor involvement of the skin, compared with 63.4 for MedImageInsight and 63.2 for MedGemma.
- 84.8 AUC is reported for irregular mass shape and satellite-mass detection, exceeding 67.8 and 70.1 for MedImageInsight, respectively.
- 84.8 AUC is reported for a breast-MRI finding, above 77.0 for MedImageInsight and 74.2 for MedGemma.
G. Performance on full set of RATE-Evals tasks on Merlin Abdomen-Pelvis CT test
Table 15 reports AUC × 100 for Atlas, MedGemma, MedImageInsight, Merlin, and Pillar-0 across a broad set of abdomen-pelvis CT findings. Pillar-0 achieves the highest listed value for several findings, although performance varies across tasks.
- Pillar-0 reaches the highest listed AUC for several findings, including renal abscesses, renal lacerations, simple renal cysts, and splenic infarcts.The reported Pillar-0 values are 97.8 for renal abscesses, 98.9 for renal lacerations, 81.1 for simple renal cysts, and 95.2 for splenic infarcts.
- 99.2 AUC is reported for primary sclerosing cholangitis with Pillar-0, compared with 95.8 for Atlas, 47.8 for MedGemma, 69.5 for MedImageInsight, and 89.7 for Merlin.
- Performance is not uniformly highest: Atlas leads Pillar-0 on retroperitoneal liposarcoma, 64.5 versus 62.4, and on adrenal adenomas, 71.0 versus 70.6.
- Pancreas-related findings: Pillar-0 also leads on selected pancreatic findings, with AUCs of 97.5 for ductal pancreatic carcinomas and 78.0 for chronic pancreatitis.