Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

Challenges

• Medical foundation models face shortages of multimodal training data, task-specific architectures, and comprehensive benchmarks beyond limited task-specific tests.Source: §1 · p. 3

Dataset construction

• MedMD contains 16M 2D and 3D radiology scans, spans 17 medical systems, and covers more than 5000 diseases with high-quality textual descriptions.Source: §1 · p. 3

Model design

• RadFM is a visually conditioned autoregressive text-generation model that integrates natural language with 2D or 3D scans, pre-trained on MedMD and fine-tuned on 3M curated radiology samples in RadMD.Source: §1 · p. 3

RadBench evaluation

• RadBench manually verifies samples for visual question answering, diagnosis, and report generation; the paper also reports rationale-diagnosis evaluations and adaptation to public benchmarks.Source: §1 · p. 3, Figure 1 · p. 4
• RadFM showed significant benefits across RadBench tasks and outperformed public models including Open-flamingo, MedVInT [55], MedFlamingo [31], and GPT-4V.The comparison includes automatic and human evaluations.Source: §1 · p. 3, Figure 1 · p. 4

2.1 Medical Multimodal Dataset (MedMD)

• MedMD combines 16M image-text pairs, including 15.5M 2D images and 500K 3D scans, and covers over 5,000 diseases.The authors augment existing datasets to address limitations in scan dimensionality, modality diversity, and report quality.Source: §2.1 · p. 4, §2 · p. 4

2.1.1 Interleaved Dataset

• PMC-Inline contributes 11M 2D radiology images by linking figures to inline references in paper text, creating interleaved images and contextual text.This differs from figure-caption collections such as PMC-OA by using references in the main body of papers.Source: §2.1.1 · p. 5

2.1.2 Visual-language Instruction Tuning Dataset

• The instruction-tuning datasets include 103K clinical case reports, 1.1M contextual VQA pairs, and peer-reviewed 3D Radiopaedia scans converted into caption, modality, rationale, and VQA formats.MedPix cases provide clinical findings and diagnostic results alongside case-level and scan-level annotations.Source: §2.1.2 · p. 7

2.1.3 Radiology Multimodal Dataset (RadMD)

• RadMD is a 3M-image radiology subset of MedMD for supervised visual instruction tuning, filtered to remove non-radiology data and emphasize 3D cases.Filtering also removes patient age or structure-size information lacking image spacing or background, and balances normal and abnormal diagnosis cases.Source: §2.1.3 · p. 7

2.2 Radiology Evaluation Benchmark (RadBench)

• RadBench covers five tasks and 13 diverse datasets; eight annotators manually verified its visual question answering, report generation, and rationale diagnosis samples.The benchmark contains 14,419 modality-recognition, 112,256 disease-diagnosis, 7,329 visual-question-answering, 2,098 report-generation, and 1,000 rationale-diagnosis samples.Source: §2.2 · pp. 7, 9

Disease diagnosis.

• The benchmark tasks assess modality recognition, disease diagnosis, medical VQA, report generation, and rationale diagnosis, with diagnosis prompts testing over 5,000 diseases using balanced yes/no cases.Modality recognition and disease diagnosis use prompt-based generation matched to closed answer lists; rationale diagnosis assesses whether models provide radiologic reasons as well as disease predictions.Source: §2.2 · p. 9, Disease diagnosis · p. 10

3.1 A Unified Learning Paradigm

• Each example interleaves text with a set of 2D or 3D scans, and training predicts each text token from preceding text and image content.Token weights can emphasize selected text or skip special tokens.Source: §3.1 · p. 10
• For interleaved data, UMLS-extracted medical terms receive higher loss weights and image placeholders are excluded; instruction data trains responses rather than user instructions.The interleaved-data weighting is described for PMC-Inline; for instruction data, the language is divided into instructions and responses.Source: §3.1 · p. 11
• Task prompts support modality recognition and disease diagnosis, including balanced positive and negative disease sampling and open prompts that request disease names.Modality prompts use annotated scans; diagnosis data provide binary disease labels.Source: §3.1 · p. 11
• The instruction-response paradigm also covers visual question answering, report generation, and rationale diagnosis that pairs a disease label with radiological features.Questions may concern spatial relationships or reasoning, while report-generation prompts request long captions or reports from one or more scans.Source: §3.1 · pp. 11–12

3.2 Architecture Detail

• RadFM combines a 3D ViT for visual encoding, a Perceiver that maps variable-length scan tokens to a fixed length, and an LLM for free-form responses.2D images are padded by replicating slices, allowing the encoder to process both 2D and 3D inputs while sharing the rest of the model.Source: §3.2 · pp. 12–13
• Visual embeddings replace image placeholders within the text sequence, which a decoder-only LLM processes using its self-attention layers for multimodal fusion.The Perceiver aggregates visual representations before fusion with text.Source: §3.2 · p. 13

3.3 Training Procedure

• Training first pre-trains on MedMD, including noisy PMC-Inline and PMC-OA data alongside cleaner VQA, captioning, and diagnosis datasets, then fine-tunes on RadMD’s over 3M radiologic images.The two stages use identical training settings and differ in their data, progressing from general medical to radiology-specific.Source: §3.3 · pp. 13–14

3.3.1 Training Details

• The 14B-parameter model uses a 12-layer 3D ViT, a six-layer Perceiver, and MedLLaMA-13B; training spans eight epochs, with the LLM frozen for the first epoch.Images are resized according to 2D or 3D format, and depth is rounded to a multiple of four; training uses 32 NVIDIA A100 GPUs with FSDP, mixed precision, and gradient checkpointing.Source: §3.3.1 · p. 14

4.1.1 Five Tasks

• RadBench evaluates modality recognition, disease diagnosis, medical VQA, report generation, and rationale diagnosis using task-appropriate metrics.Table 3 lists ACC, F1, BLEU, ROUGE, BERT-Sim, UMLS_Precision, and UMLS_Recall among the reported metrics.Source: §4.1.1 · p. 14, Table 3 · p. 18

4.1.2 Machine Rating

• For fixed-choice tasks, the study uses F1 and accuracy; for open-ended tasks, it uses BLEU, ROUGE, and BERT-sim, plus UMLS-based precision and recall.UMLS_Precision and UMLS_Recall measure overlap in medical terms extracted from reference and generated responses; automatic metrics may miss semantic accuracy, motivating radiologist evaluation.Source: §4.1.2 · pp. 14–15

4.1.3 Human Rating

• Three radiologists rated open-ended responses from 0 to 5, with 400 randomly sampled RP3D-series test cases per task, totaling 1,200 cases.Human ratings covered medical VQA, report generation, and rationale diagnosis; modality recognition and disease diagnosis were evaluated automatically because their answers are fixed.Source: §4.1.3 · p. 15

4.1.4 Publicly Accessible Foundation Model Baselines

• The study compares RadFM with OpenFlamingo, MedVInT [55], Med-Flamingo [31], and GPT-4V, using zero- and few-shot settings for OpenFlamingo and Med-Flamingo.For models without 3D input support, the authors used central volume slices for automated evaluation and radiologist-selected slices for GPT-4V; GPT-4V was evaluated only under human rating.Source: §4.1.4 · pp. 15–16

4.2 Evaluation on Public Benchmarks

• The model can serve as a pre-training model adapted to specific tasks by fine-tuning on instructions fitting each training set, with results compared against task-specific state-of-the-arts.The datasets include VinDr-Mammo [33] for mammography diagnosis, CXR14 for chest X-ray diagnosis, LDCT for 3D diagnosis, and BraTS2019, where T1CE is used to classify high- versus low-grade glioma.Source: §4.2 · p. 16
• The cited datasets are VQA-RAD (3,515 questions, 517 answers), SLAKE [27] (642 images, 14K questions), PMC-VQA [55] (149K images, 227K questions), and IU-Xray (7,470 image-report pairs).The authors manually cleaned PMC-VQA’s test set and retested the SOTA MedVInT [55] checkpoint on it. IU-Xray reports are generated from a single-view image.Source: §4.2 · pp. 16–17
• On PadChest, the study tests generalization by excluding seen classes and rare classes, leaving 163 unseen radiographic finding classes for prediction.The dataset contains 160,000 chest X-rays from 67,000 patients and 174 finding labels.Source: §4.2 · p. 17

5 Results

• RadBench evaluation covers five tasks, including modality recognition, disease diagnosis, report generation, and visual question-answering across radiologic modalities and anatomical regions.Figure 7 provides examples of inputs and outputs for the five evaluation tasks.Source: §5 · p. 18, Figure 7 · p. 19
• RadFM is compared with foundation-model baselines in zero-shot and few-shot settings, followed by fine-tuning experiments to assess its performance.The comparisons are reported in Tables 4–8.Source: §5 · p. 18
• Zero-shot evaluation on PadChest tests generalization to unseen classes, and PadChest was not used during model training.Source: §5 · p. 18

5.1.2 Disease Diagnosis

• RadFM achieved 80.62% ACC on disease diagnosis, while existing models scored nearly 50% when prompted to judge whether a patient had a disease.The task resembles multi-label classification across more than 5,000 distinct categories.Source: §5.1.2 · pp. 19–21, Table 4 · p. 20

5.1.3 Medical Visual Question Answering (VQA)

• RadFM improved UMLS_Precision from 14.55% to 31.77% and UMLS_Recall from 14.56% to 24.93% across medical VQA datasets, compared with MedVInT.MedVInT performed better on its in-domain PMC-VQA test set but generalized poorly to contextual PMC-CaseReport and 3D RP3D-VQA.Source: Table 4 · p. 20, §5.1.3 · pp. 21–22

5.1.1 Modality Recognition

• OpenFlamingo and Med-Flamingo scored 49.47% ACC on modality recognition, while RadFM outperformed competitors on the open-ended task.Models had to explicitly name the recognized modality, such as CT or MRI.Source: §5.1.1 · p. 19, Table 4 · p. 20

5.1.5 Rationale Diagnosis

• On rationale diagnosis, RadFM was the only model effective, outperforming traditional BLEU and ROUGE scores by 8.56% and 15.46%, and improving UMLS_Precision and UMLS_Recall by 23.16% and 10.96%, respectively.Source: Table 5 · p. 21, §5.1.5 · p. 23

5.1.4 Report Generation

• RadFM improved report-generation UMLS_Precision from 11.16% to 22.49% and UMLS_Recall from 2.84% to 12.07% over existing models.Report generation requires emphasizing clinically significant image observations.Source: Table 5 · p. 21, §5.1.4 · p. 23

5.1.6 Human Rating

• RadFM's outputs were preferred by clinicians in all model comparisons, and it surpassed GPT-4V even though GPT-4V received four selected 2D slices for 3D cases.The four slices were selected by radiologists using reference answers, making the questions easier than evaluation on the original 3D volumes used for RadFM.Source: §5.1.6 · p. 24, Figure 9 · p. 25

5.2 Transferring Ability to Public Benchmarks

• RadFM improves diagnosis and text-generation automatic metrics across diverse medical imaging modalities after task-specific fine-tuning.For diagnosis, the image encoder initializes both 2D and 3D models; for VQA and report generation, the whole model is fine-tuned on each dataset.Source: §5.2 · p. 25

5.3 Generalization to Unseen Classes in PadChest

• For unseen PadChest diseases, RadFM is prompted with “Is {disease} shown in this image”; other foundation-model baselines achieve random 50% accuracy across classes on the balanced test set.The task is framed as induction, and each disease prompt is randomly selected; only RadFM results are plotted.Source: §5.3 · p. 25

5.4 Qualitative Results

• In medical VQA, RadFM generally answers questions accurately and consistently, but it mistakes an ectopic ACTH-producing tumor for a primary lung neoplasm in a fine-grained abnormality query.The example illustrates difficulty distinguishing tumor types.Source: §5.4 · p. 25
• In report generation, RadFM identifies underlying diseases and sometimes performs exceptionally well, but its reports may omit anatomical laterality such as left or right.Source: §5.4 · p. 25
• For pulmonary embolism and subarachnoid haemorrhage cases, RadFM gives accurate free-form diagnoses and related radiologic reasoning, though its reasoning remains general rather than input-specific.Source: §5.4 · p. 25

6.1 RadFM is the first foundation model that unified 2D and 3D radiologic images.

• RadFM unifies 2D and 3D clinical images in one training structure and is supported by MedMD, a dataset of 16M scans with text descriptions or labels.Source: §6.1 · p. 26

6.2 RadFM unifies the medical tasks with a generative model.

• RadFM formulates diverse medical tasks as multimodal question-answering and answers arbitrary questions or instructions zero-shot, without exemplars, surpassing MedVInT on medical VQA.Source: §6.2 · pp. 26–27

6.3 RadFM supports multiple images as input.

• RadFM supports multiple-image input, and its training flow accommodates the dataset’s largely multi-image data for clinical scenarios such as longitudinal follow-up.Source: §6.3 · pp. 28–29

6.4 A general evaluation benchmark for radiology foundation models.

• RadBench covers five tasks across 2D and 3D images and introduces UMLS_Precision and UMLS_Recall, tailored to medical-domain performance.Source: §6.4 · pp. 29–30

6.5 The superiority of RadFM.

• RadFM outperforms previous methods across all five RadBench tasks; its average human-evaluation score on medical VQA is 2.87 versus GPT-4V’s 2.13.The discussion highlights medical VQA, report generation, and rationale diagnosis as challenging tasks.Source: §6.5 · pp. 30–31

6.6 Clinical impact.

• RadFM is described as the first medical foundation model to support 3D, multiple scans, and interleaved text per query, combining imaging with patient-background information.Source: §6.6 · p. 31

6.7 Limitations.

• RadFM remains far from clinical usefulness, with underdeveloped long-form generation, limited 3D data, and automatic metrics that do not fully reflect performance.Human evaluation is costly and not scalable, while no model scored above 3, the threshold described as moderately accurate.Source: §6.7 · p. 31
• Internet-sourced 3D images lack metadata such as imaging spacing, preventing size statements, while studying components of the 16M-pair, 14B-parameter model is prohibitively costly.Source: §6.7 · p. 31

Dataset Construction

• Medical datasets remain limited: MIMIC-CXR has 224K chest X-ray reports, PMC-OA has 1.6M image-caption pairs but often represents 3D scans as 2D slices, and VQA datasets cover 2D images.MTB contains approximately 0.8M images interleaved with text but is not open-source.Source: §7 · p. 32
• These data limitations leave medical foundation models focused on narrow modalities, with PubMed Central captions posing a domain gap and Med-PaLM M relying mostly on X-rays.The passage cites LLaVA-Med [23] and MedVInT [55] for PubMed Central captions, and Med-PaLM M [44] for a dataset composed mostly of X-rays.Source: §7 · p. 32

Model Training

• Most medical foundation models support only 2D images, and none simultaneously handles 2D and 3D inputs, multiple images per case, and images interleaved with text.The paper frames these heterogeneous input forms as necessary for closer alignment with clinical practice.Source: §7 · p. 32

• The model significantly surpasses the latest open-source multimodal foundation model, according to the authors.Source: §8 · p. 32
• The model processes multiple 2D or 3D image inputs interleaved with text, which the authors say better fits practical usage.Source: §8 · p. 32
• The paper constructs a complete medical foundation model-building process spanning data collection, problem formulation, model design, training, and evaluation, and a medical multimodal database it describes as the largest.Source: §8 · p. 32
• The authors plan to release the data, code, and models to promote development of medical foundation models.Source: §8 · p. 32

• Table 9 lists the datasets used for pre-training.Source: §9 · p. 32