BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

Motivation

• Specialist biomedical models struggle to integrate diverse patient data and provide comprehensive real-world clinical support, motivating versatile generalist systems.Precision medicine and patient-centered care require analysis of multiple data types and patient-specific details.Source: Introduction · p. 2

Generalist framework

• BiomedGPT’s unified framework processes visual, textual, and serialized tabular inputs and supports varied downstream clinical tasks, with lightweight fine-tuning enabling additional applications.Figure 1 describes multimodal and text-only use cases in which the model generates answers to clinical queries.Source: Introduction · p. 2, Figure 1 · p. 3

Model motivation and contribution

• BiomedGPT is a fully transparent, open-source, lightweight generalist vision-language model that performs diverse biomedical tasks across modalities using natural-language instructions.It is reported as 3,088 times smaller than Med-PaLM M, which has 562 billion parameters.Source: Introduction · p. 4

Pretraining and instruction tuning

• The model’s pretraining corpus contains 592,567 images, approximately 183 million text sentences, 46,408 object-label pairs, and 271,804 image-text pairs.An instruction-tuned variant, Instruct-BiomedGPT, was developed to improve instruction following.Source: Introduction · p. 4

Evaluation scope and limitations

• BiomedGPT was evaluated across 25 datasets through fine-tuning, also demonstrated zero-shot multimodal question answering, and received satisfactory human-evaluation results in radiology tasks.The authors stress that safety, equity, bias, and other limitations require substantial improvement before real-world clinical deployment.Source: Introduction · p. 4

Benchmark overview

• BiomedGPT exhibits superior performance on the SLAKE visual question answering dataset despite using significantly fewer parameters than its counterparts.Figure 2 also presents comparisons with prior state-of-the-art results and Med-PaLM M across multiple biomedical metrics.Source: Figure 2 · p. 5, Introduction · p. 6

Pre-training using large and diverse datasets

• BiomedGPT was pre-trained with masked modeling and supervised learning on 14 freely available datasets spanning diverse modalities, then fine-tuned across clinically motivated classification, language, captioning, and VQA tasks.Three model scales—BiomedGPT-S, BiomedGPT-M, and BiomedGPT-B—were introduced to assess performance across model sizes and support multitask adaptation.Source: Pre-training using large and diverse… · p. 6, Fine-tuning for downstream tasks · p. 6

BiomedGPT is lightweight but competitive in multimodal tasks

• 86.1% overall accuracy on SLAKE surpassed BiomedCLIP’s 85.4%, while BiomedGPT-B achieved 73.2% and 85.2% weighted F1 on VQA-RAD and SLAKE, respectively, with 98.5% fewer parameters than Med-PaLM M; captioning also exceeded benchmarks across reported metrics.On PathVQA, BiomedGPT-B scored 56.9% weighted F1, only 0.4% below Med-PaLM M; on PEIR GROSS, improvements were 8.1 percentage points in ROUGE-L, 0.5 points in METEOR, and 89.8 points in CIDEr.Source: BiomedGPT is lightweight but… · p. 7, Figure 3 · p. 8

BiomedGPT enables accurate medical image classification

• BiomedGPT outperformed previous SOTAs on 7 of 9 biomedical image-classification datasets after 5-epoch fine-tuning, including 5 of 7 best MedMNIST-Raw accuracies and higher average performance than MedViT and BiomedCLIP.On SZ-CXR and MC-CXR, accuracies were 97.0% and 89.7%, improving over LightTBNet by 6.0% and 0.8%; BiomedGPT-B also outperformed all Med-PaLM M versions on CBIS-DDSM.Source: BiomedGPT enables accurate medical… · p. 9
• BiomedGPT’s scale increased MC-CXR accuracy from 75.9% for the small model to 82.8% for the medium and 89.7% for the base model, although performance saturated on some datasets.The medium and base models each improved by 6.9% over the preceding scale on MC-CXR, while saturation became more pronounced after extreme image downscaling.Source: BiomedGPT enables accurate medical… · p. 9

BiomedGPT is able to understand and summarize clinical text

• BiomedGPT-B outperformed BioGPT and LLaVA-Med on treatment suggestion, in-hospital mortality, and clinical-trial matching, while remaining close to leading systems on MedNLI and text summarization.Treatment suggestion accuracy was 50.0% ± 5.3%, clinical-trial matching accuracy was 85.2% ± 1.5%, and MIMIC-III summarization reached 30.7% ROUGE-L versus Med-PaLM M’s 29.5%.Source: BiomedGPT is able to understand and… · pp. 10–11

BiomedGPT can perform zero-shot prediction on new data

• Instruction tuning improved mean alignment accuracy from 79.2% for BiomedGPT-B to 95% for Instruct-BiomedGPT-B, whose zero-shot accuracy reached 54.7% ± 5.7%, exceeding GPT-4V’s 53.0% ± 6.7%.BiomedGPT also demonstrated zero-shot disease diagnosis and X-ray report generation performance matching Med-PaLM M and LLaVA-Med, although diagnostic accuracies remained under 60%.Source: BiomedGPT can perform zero-shot… · pp. 11–12

Human evaluation of BiomedGPT for radiology tasks

• Radiologist evaluation gave BiomedGPT an average radiology-VQA score of 1.75 versus 1.17 for GPT-4V and 1.4 for LLaVA-Med, with superior performance in four of five question categories.For 192 generated findings, BiomedGPT-B had an 8.3% significant-error rate, comparable to approximately 6% human observer variability, and a 7.0% significant-omission rate.Source: Human evaluation of BiomedGPT for… · pp. 13–14
• BiomedGPT-generated radiology summaries were preferred 48% versus 52% for reference summaries, with no significant preference and equal 4.4/5 average correctness; medically adverse items occurred in 6.0% of both systems.BiomedGPT summaries exceeded references in completeness above 3 (81.0% versus 66.0%) and correctness above 3 (90.0% versus 86.0%), while average completeness was 3.9 versus 4.0.Source: Human evaluation of BiomedGPT for… · pp. 14–15

• BiomedGPT achieves competitive transfer learning across vision, language, and multimodal domains through unified pre-training on diverse biomedical modalities and tasks.The authors report that this framework supports high-performance models while revealing limitations that motivate future improvements.Source: Discussion · p. 16
• Biomedical multimodal data remain limited and imbalanced because annotation is costly, radiology dominates available datasets, and detailed image annotations are scarce.The authors expect better visual-semantic mappings as more biomedical data are curated and open-sourced (Fig. 6).Source: Discussion · p. 16
• Generated medical text still lacks comprehensive factuality evaluation because CIDEr and ROUGE-L measure agreement, while domain-specific factual metrics remain uncommon outside radiology.The discussion anticipates analogous factual-integrity metrics for pathology and other biomedical domains.Source: Discussion · p. 16
• Adding modalities can cause negative transfer: excluding image data improved language-only downstream tasks, despite successful extension to 3D images.The authors propose controllable strategies such as mixture-of-experts learning to mitigate modality interference.Source: Discussion · p. 16, Figure 6 · p. 18
• Larger models improve zero-shot and fine-tuned performance, but scaling increases fine-tuning, training-speed, and memory costs, while prompt tuning caused large performance degradation.The scaling-performance relationship is supported across the reported analyses, whereas prompt tuning adds small-scale parameters to condition-frozen models but underperformed.Source: Discussion · p. 17
• BiomedGPT’s zero-shot text comprehension, especially versus GPT-4V, remains limited by model scale and a single encoder that entangles modality representations.Fine-tuning is promising for mitigating these limitations, even with a smaller-scale model, while more refined training strategies are needed.Source: Discussion · p. 17

Methods

• BiomedGPT follows modality-agnostic, task-agnostic, and modality-and-task-comprehensive principles by discretizing data into patches or tokens.The design adapts unified-model ideas from ViT and language models and provides BiomedGPT-S, -M, and -B variants with 33M, 93M, and 182M parameters.Source: Methods · p. 19, BiomedGPT architecture · p. 19

BiomedGPT architecture

• BiomedGPT uses a BERT-style encoder and GPT-style autoregressive decoder to map modalities into a unified semantic space.It adds layered normalization, head-wise scaling, decoupled absolute position embeddings, and one-dimensional text and two-dimensional image relative position biases.Source: BiomedGPT architecture · p. 19

Unifying input/output

• All modalities and task outputs are represented through a shared finite token vocabulary containing 59,457 tokens.The vocabulary includes 50,265 language tokens, 1,000 location tokens, and 8,192 vision tokens; images use 16 × 16 patch sequences and inputs are truncated to 512 tokens.Source: Unifying input/output · pp. 19–20
• 93.0% chest X-ray classification accuracy gave RadGPT-B a 7.6% improvement over a ResNet-50 baseline, although liver CT required scaling for comparable performance.The experiment tested transfer from X-ray pre-training to chest X-ray, breast ultrasound, and liver CT datasets.Source: Unifying input/output · p. 20
• 15.2% accuracy loss on SLAKE-CT VQA and 8.1% on DermaMNIST show that modality comprehensiveness is critical for efficient cross-domain transfer.RadGPT-B required twice as many fine-tuning epochs as the all-modality BiomedGPT-B baseline, while liver CT required model scaling to match the baseline.Source: Unifying input/output · p. 20

Natural language as a task instructor

• Natural-language instructions unify vision-only, text-only, and vision-language tasks, including masked modeling, image infilling, object detection, captioning, VQA, classification, summarization, and inference.Instructions are handcrafted for most tasks, VQA uses its text input directly, and downstream instructions can be customized for new tasks.Source: Natural language as a task instructor · p. 21
• Excluding MLM reduced performance across all downstream tasks, whereas excluding MIM mainly hurt image-centric and multimodal tasks and could improve text-centric tasks.MLM was especially important for text-centric tasks and supported image-only tasks that generate labels through text-token dictionaries.Source: Natural language as a task instructor · pp. 21–22
• Excluding object detection notably reduced image classification and radiology captioning performance, supporting diverse task pre-training for generalization.Effects on other datasets were relatively minor, likely because object-detection samples were limited and weakly connected to language-only tasks.Source: Natural language as a task instructor · p. 22

Model pre-training

• BiomedGPT uses autoregressive seq2seq learning over linguistic and visual tokens, including subwords, image codes, and location tokens.The model masks 15% of subword tokens, generates object-detection location tokens with Pix2Seq, and uses sparse VQGAN image codes for masked image modeling.Source: Model pre-training · p. 22
• Biomedical image preprocessing removes trivial backgrounds, crops objects, resizes images to 256 × 256, and feeds 128 × 128 centers into pretrained VQ-GAN.Vision-language tasks follow the same tokenization flow, while fine-tuning also uses seq2seq learning with task-specific datasets.Source: Model pre-training · p. 22
• 8:2:1:1 is the pretraining-data mixture ratio for multimodal, text-only, vision-only, and object-detection samples, emphasizing vision-language interaction.Modality sampling balances each batch to reduce feature shift from inherent modality imbalance.Source: Model pre-training · pp. 22–23
• 1e−4 is the peak learning rate, with linear decay, 0.01 warmup, 0.1 dropout, 0.01 weight decay, and 0.1 stochastic depth.Optimization uses AdamW with β1 = 0.9, β2 = 0.999, and ε = 1e−8; training uses mixed precision on 10 NVIDIA A5000 GPUs.Source: Model pre-training · pp. 22–23
• Pretraining uses masked modeling, object detection, and image-text matching to align biomedical concepts across diverse modalities.BiomedGPT continues training from OFA’s pretrained checkpoints using a curated multimodal biomedical dataset.Source: Model pre-training · p. 23

• BiomedGPT fine-tunes only its existing structure, using instructions that mirror pre-training rather than adding components such as output layers or decoders.This preserves consistency and efficiency during model adaptation.Source: Model fine-tuning and inference · p. 23
• Hyperparameters for long-context generation are selected from training-set statistics, while inference uses the best validation checkpoint or cross-validation procedures when official splits are unavailable.For datasets without official splits, the last-epoch checkpoint is used and mean and standard deviation are reported.Source: Model fine-tuning and inference · p. 23
• Approximately 16× faster validation results from trie-based beam search, which restricts candidate tokens to valid labels and improves classification decoding efficiency and accuracy.The prefix tree prevents invalid-label generation by assigning −∞ logits to disallowed tokens.Source: Model fine-tuning and inference · p. 23

Instruction-tuning

• Open-vocabulary instruction tuning lets BiomedGPT determine responses independently during training and inference without a predefined answer set.The method followed LLaVA-Med’s data-curation approach while diverging from traditional VQA.Source: Model instruction-tuning and zero-shot… · p. 24

Zero-shot evaluation

• Zero-shot trials applied dataset-specific prompts across VQA-RAD, disease diagnosis, tuberculosis detection, and report generation, with evaluation settings aligned to Med-PaLM M where applicable.Datasets included RSNA, MedMNIST v2, BreastMNIST, DermaMNIST, OCTMNIST, PathMNIST, and MIMIC-CXR.Source: Model instruction-tuning and zero-shot… · p. 24

Zero-shot prediction

• Instruction-tuned large language models showed significant discrepancies from fine-tuned models in preliminary zero-shot studies, highlighting in-context learning and text comprehension as future priorities.The studies explored upper bounds for in-context learning and potential integration into BiomedGPT to improve reasoning.Source: Model instruction-tuning and zero-shot… · p. 24

Model extension

• BiomedGPT was extended to 3D medical imaging classification through pre-training and fine-tuning, using a pre-trained 3D VQGAN and learnable 3D visual encoder.The VQGAN tokenizes 3D images for masked image modeling, and the extension is reported in Extended Data Table 5 and Supplementary Table 4.Source: Model extension · p. 24
• For non-text generation tasks such as segmentation, additional decoders, including a mask decoder, can be introduced.Source: Model extension · p. 25

Hardware and training software

• Ten 24-GB NVIDIA A5000 GPUs supported multi-GPU pre-training through PyTorch DDP, fairseq, CUDA 12.2, and Python 3.7.4.Masked image modeling used cropped images converted into visual-token sequences based on a pre-trained VQGAN model.Source: Computing hardware and software · p. 25

Data and modeling libraries

• The open-source pipeline used Pillow, timm, torchvision, OpenCV, ftfy, einops, NumPy, and Pandas for image, text, tensor, data-processing, and analysis workflows.Images were converted to base64 strings, while ftfy handled broken Unicode and NumPy/Pandas supported data collection, preprocessing, and analysis.Source: Computing hardware and software · p. 25

Evaluation software

• Evaluation used pycocotools, pycocoevalcap, and torchmetrics to calculate NLP metrics including ROUGE-L and CIDEr and other reported metrics.The study states that all experiments and analyses can be replicated using the outlined open-source libraries.Source: Computing hardware and software · p. 25

Classification and inference metrics

• Accuracy and F1 score evaluate medical image classification, VQA, and natural language inference, with F1 addressing class imbalance through precision and recall.Higher accuracy and F1 scores indicate better performance.Source: Evaluation metrics · pp. 25–26
• Weighted F1 averages class-specific F1 scores by class frequency for VQA, whereas F1-macro gives every CBIS-DDSM image-classification class equal weight.These variants provide different treatments of class imbalance.Source: Evaluation metrics · pp. 25–26

Text-generation metrics

• ROUGE-L evaluates image captioning and summarization by longest-common-subsequence overlap, with higher scores indicating greater similarity to reference text.The metric is recall-oriented and reflects how well generated text captures salient reference content.Source: Evaluation metrics · p. 26

Captioning metrics

• METEOR and CIDEr supplement captioning evaluation using word overlap and n-gram similarity, respectively.CIDEr weights n-grams by saliency and rarity; human captions typically score near 90 on its 0–100 scale.Source: Evaluation metrics · pp. 26–27

Datasets

• BiomedGPT’s pre-training and fine-tuning datasets span multiple modalities and downstream tasks, with dataset split details provided for training, validation, and testing.Extended Data Fig. 1 summarizes modality distributions and dataset splits, while Extended Data Tables 1 and 2 provide experimental and dataset details.Source: Figure 1 · p. 34, Table 1 · p. 39, Table 2 · p. 40

Model architecture

• BiomedGPT combines bidirectional encoder blocks with autoregressive decoder blocks, and its input resolution can be adjusted for fine-tuning and inference.The model uses different numbers of blocks across model scales and varying image resolutions during pre-training.Source: Figure 2 · p. 35

Model components

• Its key components include head-scale multi-head attention, separate positional and input correlations, learnable relative position bias, and trie-based beam search that accelerated validation by approximately 16×.Trie-based search masks invalid tokens and is also applied during fine-tuning validation.Source: Figure 3 · p. 36, Extended Data · p. 37

Prompt tuning

• Prompt tuning substantially underperformed full-model fine-tuning, with best-result accuracy reductions of 32.3%, 54.6%, and 32.6% on colon pathology, blood cell microscope, and chest X-ray datasets.Performance generally improved with longer prompts, but remained substantially below fine-tuning despite additional training epochs.Source: Figure 4 · p. 37

Zero-shot evaluation

• Additional zero-shot evaluations cover five disease diagnosis tasks and compare BiomedGPT with Med-PaLM M, while noting that no single model consistently leads across all four report-generation metrics.BiomedGPT generates responses without label candidates or a trained linear classifier; TBLightNet is identified as the fine-tuned state-of-the-art TB detector.Source: Figure 5 · p. 38

Supplementary evaluation details

• The Extended Data also specifies pre-training task instructions, zero-shot VQA-RAD question types, and 3D medical image classification results reported using accuracy and F1-Macro.These materials are provided in Extended Data Tables 3–5.Source: Table 3 · p. 41, Table 4 · p. 41, Table 5 · p. 42

Supplementary Information

• Supplementary ablations examine how pretraining-task and dataset diversity affect BiomedGPT, including object-detection exclusion, MLM inclusion, domain-specific datasets, and chest-X-ray continual pretraining.The supplementary figure describes both the ablation configurations and the domain-specific pretraining setup used before fine-tuning on other domains.Source: Figure 2 · p. 44
• Supplementary evaluations cover treatment suggestion, clinical-trial matching, and in-hospital mortality prediction, with zero-shot BioMistral [1] and LLaMA-3-instruct compared against fine-tuned models.The comparison includes 4,680 treatment-suggestion cases and 7,079 clinical-trial-matching cases; the mortality task is also included.Source: Figure 3 · p. 45
• Qualitative supplementary examples illustrate BiomedGPT’s hits and misses in disease diagnosis, object recognition, VQA, and radiology report generation.Radiology examples include generated findings and impressions compared with reference reports, with radiologist-identified errors and omissions highlighted.Source: Figure 5 · p. 47, Figure 6 · p. 48, Figure 7 · p. 49
• Clinical-trial matching examples show BiomedGPT producing both eligible and ineligible judgments, including a COVID-19 case whose prediction conflicts with the stated reference.One example labels a patient eligible, while another predicts ineligibility although the reference says eligible; a separate example’s reference is ineligible.Source: Figure 9 · p. 51

Supplementary reference

• Supplementary tables report reference performance comparisons on non-difference MIMIC-Diff-VQA questions and fine-tuned BiomedGPT results across MedMNIST modalities.The MIMIC-Diff-VQA comparison uses exact-match accuracy against MMQ [5], EKAID [6], and PLURAL [7].Source: Table 1 · p. 52, Table 2 · p. 52

Supplementary Information

• The supplementary VQA human-evaluation materials define categories for abnormality, presence, location, type, and level, with example questions addressing abnormalities, evidence, position, type, and level.Source: Table 3 · p. 53, Supplementary Information · p. 53
• Additional experimental details cover 3D BiomedGPT extensions, task-specific hyperparameters, image-captioning ablations, dataset splits, preprocessing sources, and modality distribution in instruction-tuning data.The 3D extension uses ACS Convolutions [11], ADNI data for MRI fine-tuning, LIDC data, and 3D VQGAN pretraining based on BRaTS 2021 [12] and LUNA16 [13] following [14].Source: Figure 1 · p. 43, Table 4 · p. 53, Table 5 · p. 54