PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2308.02463v5cs.CVcs.CL

TL;DR

Radiology foundation models need broader multimodal data, more general architectures, and comprehensive evaluation. This paper builds RadFM using the 16M-scan MedMD dataset, a visually conditioned model, and the five-task RadBench benchmark. The model supports heterogeneous radiologic inputs and outperforms prior methods across RadBench, while remaining below clinically satisfactory performance on some tasks.

  • Problem

    Radiology foundation-model development is limited by insufficient multimodal datasets, general architectures, and comprehensive benchmarks.

  • Method

    The paper constructs MedMD, trains a visually conditioned autoregressive model on 2D and 3D scans with text, fine-tunes on RadMD, and evaluates it with RadBench.

  • Results

    RadFM outperforms previous methods across all five RadBench tasks and supports multiple 2D or 3D image inputs interleaved with text.

  • Takeaways & Limitations

    The work demonstrates a complete radiology foundation-model development process spanning data collection, model design, training, and evaluation.

  • Takeaways & Limitations

    Rationale diagnosis and report generation remain far from practically satisfactory, and no evaluated model exceeds a human-rating score of 3.

Abstract

from arXiv · show

In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

4 Evaluation

The supplied passage provides author-contribution and contact metadata rather than evaluation findings.

  • The passage marks the authors as making equal contributions.
  • The passage lists email addresses for the authors.
  • The passage contains no substantive evaluation result or methodological claim.

1 Introduction

The paper targets three barriers to radiology foundation models: limited multimodal data, fragmented architectures, and inadequate comprehensive benchmarks. It introduces RadFM with MedMD, RadBench, and a unified architecture for diverse radiologic inputs and tasks.

  • Research challenges: Medical foundation-model development lacks a large, diverse dataset spanning text, signals, 2D images, 3D images, genomics, and other modalities.
  • Research challenges: Existing medical systems commonly use task-specific architectures rather than one model supporting diverse clinical tasks and modalities.
  • Research challenges: Current benchmarking relies on limited task-specific test sets, leaving no comprehensive high-quality measure across clinical tasks.
  • Proposed system: RadFM learns from paired radiology scans and text to address a wide spectrum of clinical radiology tasks.
  • Dataset: 16M 2D and 3D radiology scans in MedMD include textual descriptions across 17 medical systems and over 5000 diseases.
  • Model: RadFM is a visually conditioned autoregressive generator integrating natural language with 2D or 3D scans, pretrained on MedMD and fine-tuned on 3M curated radiologic samples in RadMD.
  • Evaluation: RadBench evaluates modality recognition, diagnosis, visual question answering, report generation, and rationale diagnosis using automatic and human assessment.
  • Contributions: The study presents a radiology foundation-model demonstration, a large multimodal dataset, and a comprehensive benchmark for monitoring progress.

2 Dataset

This section constructs MedMD and its radiology-focused subset RadMD, then introduces RadBench for evaluating radiology foundation models across diverse clinical tasks. The datasets combine 2D and 3D scans, textual or instructional supervision, and manually verified evaluation cases.

  • MedMD: The dataset combines interleaved image-language data with visual instruction datasets for multimodal training.Interleaved data links images to surrounding paper text, while instruction data pairs inputs with responses.
  • Dataset sources: RP3D adds expert-reviewed 3D radiology cases containing captions, modality labels, radiological rationales, and visual question-answering pairs.Its cases may contain multiple images from the same or different modalities and are organized into several task-specific formats.
  • RadMD: RadMD is a 3M-sample radiology subset of MedMD created for supervised visual instruction tuning across formats, modalities, tasks, and over 5000 diseases.Filtering removes non-radiologic and 2D-only sources, addresses inconsistent report styles, and balances normal and abnormal diagnosis cases.
  • RadBench: RadBench evaluates radiology foundation models on modality recognition, disease diagnosis, visual question answering, report generation, and rationale diagnosis.The benchmark draws on 13 diverse datasets and includes manually verified samples for visual question answering, report generation, and rationale diagnosis.
  • RadBench tasks: Modality recognition reformulates identifying an image modality as prompt-based visual question answering scored with accuracy and F1.Predictions are matched against CT, MRI, Ultrasound, PET, X-ray, and angiography labels using difflib.SequenceMatcher.

3 Building Generalist Foundation Model for Radiology

RadFM unifies interleaved medical text with 2D or 3D scans in a generative framework, using a general pre-training objective and task-specific prompting across radiologic problems.

  • Unified Learning Paradigm: MedMD and RadMD samples pair language containing image placeholders with sets of 2D or 3D medical scans.Visual inputs may be 2D tensors or 3D volumes, while text and images form interleaved prompts.
  • Unified Learning Paradigm: RadFM models the probability of each text token conditioned on preceding interleaved images and language, training with negative log-likelihood.The model predicts the next token using visual and textual context preceding that token.
  • Training Objectives: Per-token weights emphasize medically important UMLS terms and skip image-placeholder tokens in interleaved datasets.PMC-Inline receives higher weights for medical-related words, while image placeholders contribute no loss.
  • Training Objectives: Visual instruction datasets separate user instructions from responses so training focuses on generating correct answers.The instruction includes the question and image placeholders; the response is the text the model must produce.
  • Task Formulation: The prompting framework covers modality recognition, disease diagnosis, VQA, report generation, and rationale diagnosis.These formulations range from binary or open-ended diagnosis to reports and explanations containing radiological features.
  • Architecture: The architecture combines a 2D/3D-capable visual encoder, a perceiver that aggregates scans, and an LLM that generates free-form text.The perceiver maps arbitrary numbers of visual patch tokens to a fixed length for fusion, while a 3D ViT supports both input types.

4 Evaluation

RadBench evaluates radiology foundation models across classification and open-ended generation, combining automatic metrics with radiologist ratings to address limitations of text-overlap evaluation.

  • Benchmark Design: RadBench assesses modality recognition, disease diagnosis, VQA, report generation, and rationale diagnosis.Metrics and datasets are selected according to whether tasks have fixed answer choices or open-ended responses.
  • Automatic Evaluation: Fixed-answer tasks use community metrics such as ACC and F1, while open-ended tasks use BLEU, ROUGE, UMLS metrics, and BERT score.The metric choice follows the task format, distinguishing predefined answers from generated responses.
  • Automatic Evaluation: UMLS_Precision and UMLS_Recall measure overlap of medical-related words extracted from reference and generated text.The metrics use UMLS to identify medically relevant terms and count overlapping terms as true positives.
  • Human Evaluation: Because automatic metrics may miss semantic accuracy, the study adds professional human evaluation by radiologists.Three radiologists rate generated answers on a 0-to-5 scale using an evaluation interface.
  • Human Evaluation: Human rating is limited to open-ended tasks, with 400 randomly sampled RP3D-series test cases per task.Modality recognition and disease diagnosis use automatic metrics because their fixed answers are considered less ambiguous.
  • Baselines: Baseline comparisons include OpenFlamingo, Med-Flamingo, MedVInT, and GPT-4V, with 3D baselines evaluated using selected slices.GPT-4V is used only for human evaluation, while other models receive central slices of 3D volumes because they lack native 3D support.

5 Results

Across RadBench, RadFM improves over foundation-model baselines on modality recognition, diagnosis, VQA, report generation, rationale diagnosis, and human-rated generation quality.

  • 5.1.1 Modality Recognition: 49.47% ACC is reported for OpenFlamingo and Med-Flamingo on modality recognition, while RadFM outperforms competing models by a substantial margin.The task requires models to explicitly name modalities such as CT or MRI.
  • 5.1.2 Disease Diagnosis: 80.62% ACC is achieved by RadFM on disease diagnosis, compared with existing models near 50% ACC.The authors describe the judgment-format baseline performance as nearly random and characterize RadFM as more suited to clinical applications.
  • 5.1.3 Medical Visual Question Answering: 31.77% UMLS_Precision and 24.93% UMLS_Recall are reported for RadFM across the VQA test set, improving over MedVInT’s 14.55% and 14.56%.The comparison spans five VQA datasets, including contextual cases and real 3D scans.
  • 5.1.4 Report Generation: 22.49% UMLS_Precision and 12.07% UMLS_Recall are achieved by RadFM for report generation, versus 11.16% and 2.84% for existing models.The task evaluates generated reports with multiple automatic metrics, especially medical terminology overlap.
  • 5.1.5 Rationale Diagnosis: RadFM is the only model reported to effectively respond to rationale diagnosis, improving BLEU by 8.56% and ROUGE by 15.46%.Its UMLS_Precision and UMLS_Recall improvements are reported as 23.16% and 10.96%, respectively.
  • 5.1.6 Human Rating: RadFM receives higher human scores on all three generative tasks and is preferred by clinicians in comparisons with other models, including GPT-4V.For GPT-4V comparisons, 22 of 1200 testing cases were dismissed because GPT-4V refused to answer for safety reasons.

5.2 Transferring Ability to Public Benchmarks

RadFM’s learned representations can be transferred to public benchmarks through further fine-tuning, improving diagnosis and text-generation performance.

  • Transfer Learning: Fine-tuning RadFM improves diagnosis results and text-generation quality across public datasets.The image encoder initializes diagnosis models, while the whole model is fine-tuned for VQA and report generation.

5.3 Generalization to Unseen Classes in PadChest

RadFM is evaluated zero-shot on PadChest disease classes that were absent from training, using balanced yes/no induction prompts. Other foundation-model baselines perform at random on this task.

  • RadFM is evaluated zero-shot on unseen PadChest disease classes using prompts asking whether each disease appears in an image.Each disease class is absent from training, and the task is reformulated as induction.
  • The PadChest test set balances ‘yes’ and ‘no’ responses across disease classes.
  • 50% accuracy is obtained by all other foundation-model baselines, indicating random performance on the unseen-class task.

5.4 Qualitative Results

Qualitative examples show that RadFM can answer medical VQA consistently, identify diseases in reports, and provide diagnoses with related reasoning. The examples also expose errors in fine-grained tumor discrimination, missing anatomical laterality, and nonspecific case reasoning.

  • Medical VQA: RadFM answers medical VQA questions in a consistent format, but can confuse closely related tumor types.It misclassifies an ectopic ACTH-producing tumor as a primary lung neoplasm in one challenging example.
  • Report generation: RadFM can identify underlying diseases in generated radiology reports, although reports may omit left-right anatomical location.
  • Rationale diagnosis: RadFM diagnoses pulmonary embolism and subarachnoid haemorrhage accurately while producing reasoning that remains general rather than case-specific.

6 Discussion

RadFM is designed to unify radiologic tasks across 2D and 3D scans, multiple images, and interleaved text. On RadBench, it outperforms prior methods, while limitations remain in long-form generation, 3D-data coverage, evaluation metrics, and metadata completeness.

  • Results: RadFM can answer varied medical VQA questions, but it may confuse closely related tumor types in difficult cases.One example misclassified an ectopic ACTH-producing tumor as a primary lung neoplasm.
  • Evaluation: RadBench evaluates modality recognition, disease diagnosis, visual question answering, report generation, and rationale diagnosis using 2D and 3D images.The benchmark also introduces UMLS_Precision and UMLS_Recall for medical-text evaluation.
  • Results: RadFM outperforms previous methods across all five RadBench tasks, with human VQA scores of 2.87 versus 2.13 for GPT-4V.The reported advantage is especially pronounced for medical VQA, report generation, and rationale diagnosis.
  • Model capabilities: RadFM integrates 2D or 3D scans, multiple images, and interleaved text within one architecture.The design targets clinical cases requiring heterogeneous scans and patient-background information.
  • Limitations: Long-sentence generation remains below practical clinical standards, despite quantitative and human-evaluation improvements over prior work.None of the evaluated models exceeded a human-rating score of 3, defined as moderately accurate.
  • Limitations: The dataset is dominated by 2D images, some 3D metadata such as imaging spacing is missing, and current automatic metrics do not fully reflect true performance.Missing spacing prevents precise statements such as tumor size, while human evaluation is accurate but costly and difficult to scale.

7 Related Work

Earlier medical foundation-model work is constrained by small or narrowly focused multimodal datasets and architectures that primarily support 2D images. The paper positions RadFM as addressing the missing combination of heterogeneous inputs and broad clinical-task support.

  • Overview: Medical foundation-model research remains comparatively nascent and is organized around dataset construction and model training.The related work contrasts this stage with stronger progress in natural-scene multimodal modeling.
  • Dataset construction: Medical multimodal datasets lag behind natural-scene resources in scale and diversity.MIMIC-CXR contains 224K chest X-ray image-report pairs, while PMC-OA contains 1.6M image-caption pairs.
  • Dataset construction: Existing medical datasets are largely limited to 2D images and provide medical reports mainly for chest X-rays.These constraints omit broader 3D imaging and modality diversity needed for generalist medical models.
  • Model training: Most prior medical foundation models support only 2D images, and none simultaneously support 2D, 3D, multi-image, and interleaved-text inputs.The paper identifies this heterogeneous-input combination as a target for RadFM.

8 Conclusion

The paper presents an end-to-end process for building a radiology foundation model, from data collection through evaluation. RadFM processes heterogeneous radiologic inputs and is reported to outperform open-source multimodal foundation models.

  • Contributions: The work covers data collection, problem formulation, model design, training, and evaluation as one foundation-model-building process.The authors describe this as a complete workflow for medical foundation-model development.
  • Model capabilities: RadFM processes multiple 2D or 3D image inputs interleaved with text.This input format is presented as more compatible with practical radiology use.
  • Results: The authors report that RadFM significantly surpasses the latest open-source multimodal foundation models.The paper states that corresponding data, code, and models will be released.

9 Data availability

The paper points readers to Table 9 for the datasets used during pre-training.

  • Data availability: Table 9 lists the datasets used for pre-training.
Loading 2308.02463v5…