PaperlayerView exact revision on arXiv

Source-linked AI summary

Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset

Louis Blankemeier, Ashwin Kumar, Joseph Paul Cohen, Jiaming Liu, Longchao Liu, Dave Van Veen, Syed Jamal Safdar Gardezi, Hongkun Yu, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Robbie Holland, Cesar Truyts, Christian Bluethgen, Yufu Wu, Long Lian, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hung Weng, Edson Amaro Junior, Neera Ahuja, Jason Fries, Nigam H. Shah, Greg Zaharchuk, Marc Willis, Adam Yala, Andrew Johnston, Robert D. Boutin, Andrew Wentland, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, Akshay S. Chaudhari

arXiv:2406.06512v2cs.CVcs.AI

TL;DR

Abdominal CT interpretation is burdensome, while existing medical VLMs are generally limited to 2D images and short reports. Merlin is a 3D vision-language foundation model trained from CT scans, EHR data, and radiology reports, and it was evaluated across 6 task types and 752 tasks. The model showed generalization across institutions and anatomies and outperformed several alternative model classes, although report generation still under-reported positive findings.

  • Problem

    Existing medical VLMs are generally limited to 2D images and short reports, despite the need to interpret large and complex abdominal CT studies.

  • Method

    Merlin is a 3D vision-language foundation model trained with volumetric CT scans, EHR diagnosis codes, and radiology reports using a multistage pretraining framework without additional manual annotations.

  • Results

    Merlin generalized across institutions and anatomies and outperformed 2D VLMs, CT foundation models, and other cited baselines across its evaluated tasks.

  • Takeaways & Limitations

    Merlin provides a task-agnostic foundation for diagnostic, prognostic, and research capabilities supporting clinicians across multiple medical imaging tasks.

  • Takeaways & Limitations

    Merlin-generated reports generally had correct structure but tended to under-report positive findings, leaving room for improvement in report generation.

Abstract

from arXiv · show

The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings for abdominal CT interpretation, we introduce Merlin, a 3D VLM that learns from volumetric CT scans, electronic health record data and radiology reports. This approach is enabled by a multistage pretraining framework that does not require additional manual annotations. We trained Merlin using a high-quality clinical dataset of paired CT scans (>6 million images from 15,331 CT scans), diagnosis codes (>1.8 million codes) and radiology reports (>6 million tokens). We comprehensively evaluated Merlin on 6 task types and 752 individual tasks that covered diagnostic, prognostic and quality-related tasks. The non-adapted (off-the-shelf) tasks included zero-shot classification of findings (30 findings), phenotype classification (692 phenotypes) and zero-shot cross-modal retrieval (image-to-findings and image-to-impression). The model-adapted tasks included 5-year chronic disease prediction (6 diseases), radiology report generation and 3D semantic segmentation (20 organs). We validated Merlin at scale, with internal testing on 5,137 CT scans and external testing on 44,098 CT scans from 3 independent sites and 2 public datasets. The results demonstrated high generalization across institutions and anatomies. Merlin outperformed 2D VLMs, CT foundation models and off-the-shelf radiology models. We also release our trained models, code, and dataset, available at: https://github.com/StanfordMIMI/Merlin.

1 Main

Abdominal CT interpretation is time-consuming and difficult to automate because scans contain many slices and structures, while existing medical VLMs largely target 2D images and do not use the full range of clinical data. Merlin addresses these gaps with a 3D vision-language model trained on CTs, EHR codes, and radiology reports.

  • Clinical need: Abdominal CT exams can contain more than 300 slices and require examination of numerous anatomical structures, making interpretation time-consuming.The reported interpretation time can reach 20 minutes per exam.
  • Current limitations: Existing medical imaging algorithms commonly use unimodal imaging data and require manually curated labels or task-specific retraining.This limits reuse across tasks and increases annotation demands.
  • Opportunity: Clinical data provide paired medical images, radiology reports, and EHR diagnosis codes that can supervise vision-language models without additional data labeling.Radiology reports are generated during routine clinical care and can provide human-annotated supervision.
  • Current limitations: Most radiology VLMs focus on 2D modalities, while many approaches process 3D volumes slice-by-slice or in chunks rather than modeling the full volume.Volumetric anatomical structures can change rapidly across all three dimensions.
  • Proposed approach: Merlin is a 3D abdominal CT vision-language foundation model trained with paired CT images, EHR diagnosis codes, and radiology reports.The reported training dataset includes 6,387,231 images from 15,331 CTs, 1,839,559 codes, and 6,036,645 report tokens.
  • Evaluation: The study evaluates Merlin across 6 task types and 752 tasks, including zero-shot classification and retrieval, disease prediction, report generation, and 3D organ segmentation.The evaluations cover diagnostic, prognostic, and quality-related tasks.

2 Results

Merlin performs across zero-shot classification, retrieval, disease prediction, report generation, and segmentation tasks, with strengths in large-scale phenotype analysis, external validation, and label-scarce segmentation. Its performance depends on training design and data scale, while report generation and full-data segmentation retain important limitations.

  • Zero-shot Findings Classification: 0.741 average F1 on internal and 0.647 on external data for 30 zero-shot findings, outperforming 2D baselines in both settings.External performance remained strongest for coarse-grained or salient findings and declined for subtle, fine-grained findings.
  • Zero-shot Findings Classification: 45.0% and 29.0% higher F1 than supervised baselines with 10% and 100% pretraining data, respectively, in supervised zero-shot findings classification.
  • Phenotype Classification: 0.812 macro-average AUROC across 692 phenotypes, with AUROC above 0.85 for 258 phenotypes and above 0.9 for 102.Performance covered diseases across multiple organ systems, including the liver, kidneys, ureters, and gastrointestinal tract.
  • Phenotype Classification: Increasing training data improved phenotype-classification performance, while larger models and smaller convolutional receptive fields generally performed better across model variations.
  • Radiology Report Generation: Merlin consistently outperformed RadFM on RadGraph-F1, BERT Score, ROUGE-2, and BLEU across anatomical sections and full-report findings, but tended to under-report positive findings.The generated reports generally placed findings in the correct anatomical sections, while examples showed missed findings such as cholelithiasis.
  • 3D Semantic Segmentation: With 10% of training cases, Merlin exceeded nnUNet by 4.7% in average Dice score across 20 organs, but nnUNet performed slightly better with 100% of the data.Merlin improved over nnUNet on 12 of 20 organs in the 10% setting, including a 41.0% prostate Dice improvement.

3 Discussion

Merlin is a 3D vision-language foundation model for abdominal CT that integrates imaging with EHR diagnoses and radiology reports. It supports broad clinical evaluation, generalizes across external datasets, and identifies scaling, resolution, batch-size, anatomy, and compute constraints for future development.

  • Model scope: Merlin integrates CT scans, EHR diagnoses, and radiology reports to support diagnostic, prognostic, and research tasks beyond bespoke classification.The model’s evaluated capabilities include findings and phenotype classification, retrieval, disease prediction, report generation, and 3D segmentation.
  • Supervision strategy: Training with both EHR diagnoses and radiology reports outperforms using either source alone, while multi-task training outperforms staged training.Report-only contrastive pretraining accounts for most gains, and adding EHR diagnoses provides a modest additional improvement.
  • Baseline comparisons: Merlin outperforms finetuned 2D VLMs, lifted 2D-to-3D VLMs, 3D vision-only models, and recent CT embedding foundation models.The comparison holds in both data-scarce and fully supervised settings; full-volume processing and language alignment are identified as relevant distinctions.
  • Generalization: Over 44,000 external CT scans across multiple sites and anatomies show consistent performance under differences in demographics, acquisition parameters, and reporting practices.These results support generalization across distributional shifts between training and testing cohorts.
  • Future improvements: Larger pretraining datasets improve performance, but future scaling may require architecture updates and should be assessed with task-specific scaling curves.Higher image resolution and larger batch sizes are also proposed directions, with resolution gains bounded by the original scan resolution and signal-to-noise trade-offs.
  • Limitations: Single-GPU training may limit generalizability because task performance could increase with greater compute, while public abdominal-CT baselines remain limited.Report generation and segmentation also require further adapter, language-model, and decoder optimization.

4 Methods

Merlin is trained as a 3D vision-language model using abdominal CT volumes, structured EHR phenotypes, and sectioned radiology reports. Its evaluation spans non-adapted and adapted tasks using internally and externally separated datasets.

  • Merlin uses abdominal CT images together with structured EHR and unstructured radiology reports as supervision signals.
  • The dataset was collected from consecutive abdominal CT examinations and includes linked CT studies, radiology reports, and hospital EHR information.
  • CT preprocessing reorients scans, resamples them to 1.5 mm in-plane resolution and 3 mm slice spacing, clips Hounsfield units, and crops volumes to 224 × 224 × 160.
  • EHR ICD-9 and ICD-10 codes are mapped to 1,692 hierarchical phenotypes with positive labels propagated through the phenotype tree.
  • Merlin uses an inflated 3D ResNet152 image encoder and a clinical Longformer text encoder with a 4,096-token context length.
  • Training combines phenotype classification and contrastive report learning, with comparisons between joint multitask training and staged EHR-then-report training.
  • Evaluation includes zero-shot findings classification, phenotype classification, cross-modal retrieval, 5-year disease prediction, report generation, and 3D segmentation.

Contributions

The paper evaluates Merlin across a broad set of diagnostic, prognostic, retrieval, generation, and segmentation tasks. The figures examine scaling, architectural alternatives, training strategies, and external generalization.

  • Zero-shot findings classification: Zero-shot findings classification compares Merlin with 2D baselines across internal, external, and fracture-detection datasets and examines scaling and pretraining ablations.
  • Phenotype classification: Phenotype classification evaluates AUROC and AUPRC across 692 phenotypes, prevalence groups, model stems, and baseline architectures.
  • Cross-modal retrieval: Cross-modal retrieval measures top-1 recall for CT–report matching, including findings and impression sections, scaling experiments, and training-strategy ablations.
  • Chronic disease prediction: Five-year prediction compares Merlin and baseline variations for chronic disease onset using full and data-scarce downstream training settings.
  • Report generation: Report generation compares Merlin with RadFM across report sections and the full report using four metrics and annotated examples.
  • 3D semantic segmentation: Three-dimensional segmentation compares model variations across 20 abdominal organs using average Dice scores, reduced-data training, and qualitative examples.
  • Alternative architectures: Alternative architecture experiments compare 2D, lifted 2D-to-3D, and 3D vision-only baselines on findings classification and phenotype prediction.
  • External validation: External validation evaluates abdominal and chest CT performance across 44,098 scans from external sites and compares Merlin with chest CT foundation models.

A.1 Dataset Details

The supplementary tables summarize the pretraining data splits and the characteristics of the internal and external datasets.

  • Table 1 summarizes the pretraining dataset splits.
  • Table 2 reports internal dataset characteristics for 18,321 patients, including age as mean ± standard deviation and other variables as percentages.
  • Table 3 reports external dataset characteristics for 5,804 patients, using the same age and percentage conventions.

B.1 Zero-shot Classification

Merlin’s zero-shot classification evaluations compare directly trained 3D modeling with pooled 2D and lifted 2D-to-3D baselines. Pooling and finetuning materially affect baseline performance, but Merlin remains stronger than the evaluated lifted alternatives.

  • Evaluation scope: The zero-shot evaluations include 30 findings and ablations of initialization, staged versus multitask training, and report-text handling.The reported evaluation setup includes both internal and external clinical datasets, plus vertebral-fracture evaluation on VerSe.
  • Pooling strategies: Top-k pooling improved zero-shot F1 over average pooling for off-the-shelf BiomedCLIP and OpenCLIP, with gains declining as k increased.At k=1, BiomedCLIP improved by 32.0% and OpenCLIP by 29.0%.
  • 2D finetuning: Finetuning increased BiomedCLIP’s average-pooling F1 by 52.0% but decreased OpenCLIP’s by 2.1%.After finetuning, average pooling yielded the highest F1 among the evaluated pooling strategies.
  • 2D-to-3D comparisons: 2D-to-3D OpenCLIP achieved the highest baseline F1 of 0.632, followed by ResnetCLIP at 0.614 and BiomedCLIP at 0.587.These results were reported for zero-shot findings classification.
  • 2D-to-3D comparisons: OpenCLIP’s 2D-to-3D F1 was 17.0% lower than Merlin’s on zero-shot findings classification.The comparison used the best-performing 2D-to-3D baseline, OpenCLIP.

Counterfactual Analysis

Merlin’s counterfactual analysis modifies latent representations to test whether phenotype predictions rely on expected anatomical features. The qualitative examples show prediction-reducing changes localized to the relevant pathology.

  • Method: Latent Shift counterfactuals modify a low-dimensional representation so Merlin’s prediction decreases while reconstructed CT volumes remain near the data manifold.The modified volumes are inspected for changes in features driving classification.
  • Results: For pleural effusion, the counterfactual reduced the effusion in the left lung, indicating use of the expected localized feature.The analysis examined whether the model relied on expected features rather than shortcuts.
  • Results: For splenomegaly, the counterfactual reduced spleen size, providing a corresponding anatomical feature change.Both examples qualitatively investigate features used by Merlin for image classification.

EHR Phenotypes Task

The EHR phenotype evaluation examines how 2D and lifted 2D-to-3D VLMs behave under finetuning and limited-data conditions. Lifted models show larger few-shot gains, while retrieval remains robust for Merlin as pools grow.

  • Few-shot phenotype classification: At 10% training data, finetuning 2D VLMs produced 2.6% higher AUROC and 4.9% higher AUPRC across phenotypes.ResnetCLIP improved more substantially than finetuned OpenCLIP and BiomedCLIP, with 11.0% AUROC and 13.0% AUPRC gains.
  • Few-shot phenotype classification: At 10% training data, 2D-to-3D lifted VLMs improved phenotype AUROC by 6.8% and AUPRC by 15.0%.These results suggest an advantage for lifted VLMs in few-shot settings.
  • Cross-modal retrieval: Retrieval performance generally decreased as pools expanded from 64 to 256 and 1,024 examples, while relative model performance remained consistent.The evaluation covered in-distribution findings and shorter, out-of-distribution impressions.
  • Cross-modal retrieval: Merlin achieved recall above 0.3 from a retrieval pool of 1,024 examples and outperformed 2D and lifted baselines.The extended retrieval study compared findings and impressions across larger pools.
  • Adapted tasks: The broader evaluation also includes five-year disease prediction and radiology report generation comparisons.The supplied table captions identify multi-disease prediction and report-section generation as adapted tasks.

C.1 Additional Alternative Architecture Comparison

Alternative-architecture experiments compare Merlin with fully finetuned 2D, lifted 2D-to-3D, and 3D vision-only models, as well as Google CT Foundation embeddings. Merlin achieves the strongest reported quantitative performance, though the Google comparison has reproducibility constraints.

  • 2D-to-3D lifted VLMs: Merlin had 18.0% higher F1 than lifted VLMs on findings-based disease classification and 12.0% higher AUROC on EHR phenotype classification.The comparison was conducted in the alternative-architecture evaluation shown in Figure 14.
  • 3D vision-only models: 3D vision-only models were 8.1% lower in F1 and 5.3% lower in AUROC than Merlin at 100% training data.Among those baselines, SwinUNETR also underperformed 3D Resnet.
  • Full finetuning: Merlin achieved the best quantitative performance across the fully finetuned alternative-architecture tasks.The reported comparison includes findings-based disease classification and EHR phenotype classification.
  • Comparison caveat: Google CT Foundation’s undisclosed weights, training methodology, and preprocessing limit reproducibility of the embedding comparison.Access was restricted to an API that returned 1,408-dimensional embeddings.
  • Google CT Foundation comparison: Merlin’s embeddings produced 4.1% higher F1 and 5.8% higher AUPRC than Google CT Foundation on findings-based disease classification.Both models were evaluated using lightweight multilayer perceptrons with consistent hyperparameters.

C.2 Image data augmentation

Image augmentation hurt Merlin’s zero-shot findings classification performance, likely because altering abdominal CT images disrupted precise alignment with radiology reports. Weak and strong augmentation strategies were evaluated against training without augmentation.

  • Augmentation significantly reduced zero-shot findings classification performance compared with Merlin trained without augmentation.The evaluation covered 30 abdominal CT findings.
  • For many diseases, stronger augmentation progressively degraded performance.The comparison included weak and strong augmentation variants designed to preserve the original image signal.
  • Augmentation may disrupt critical image elements and misalign them with corresponding radiology reports, causing the model to learn noise rather than meaningful signal.

C.3 External Validation Experiments

Merlin was externally validated for zero-shot findings classification across internal and external abdominal CT datasets and consistently outperformed alternative architectures. Performance remained higher despite demographic, scanner, protocol, and reporting differences across sites.

  • Internal and external performance: Merlin outperformed the second-best model by 17.2% on the internal test set.The comparison used zero-shot findings classification and F1 scores.
  • Internal and external performance: Merlin outperformed the second-best model by 34.4% at out-of-state site #1, while its performance drop was 12.4%.All 2D-to-3D models experienced a 25.5% external-performance drop relative to the internal test set.
  • Internal and external performance: Merlin outperformed the second-best model by 15.7% at in-state site #2 and 8.89% at in-state site #3.
  • Distribution shifts: External datasets differed from the internal test set in patient age, demographics, slice thickness, scanner manufacturer, acquisition parameters, and radiologist reporting.The external cohorts generally had higher mean patient age and varied in scanner and protocol distributions.
  • Distribution shifts: Reporting-style variation may explain slight external performance differences, while scanner-level and protocol-level variation contributed to dataset differences.Site #2 was dominated by Canon scanners, whereas the internal set had an approximately 60/40 GE Healthcare–Siemens split; Site #3 used mostly 120 kVp and GE Healthcare scanners.
  • Figure overview: Figure 17 compares F1 scores across one internal and three external sites, with error bars representing 95% confidence intervals.The figure evaluates alternative architectures on zero-shot findings classification.
Loading 2406.06512v2…