PaperlayerView exact revision on arXiv

Source-linked AI summary

BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, James Zou, Quanzheng Li, Hongfang Liu, Lichao Sun

arXiv:2305.17100v4cs.CLcs.AI

TL;DR

Biomedical AI models are often specialized by task or modality, limiting their flexibility for diverse biomedical data. BiomedGPT addresses this gap with an open-source, lightweight vision-language foundation model that achieves competitive performance across biomedical domains, including 86.1% accuracy on SLAKE VQA versus 85.4% for the prior state of the art.

  • Problem

    Most biomedical AI models are specialized for particular tasks and modalities, limiting their ability to integrate diverse data and patient-specific details.

  • Method

    BiomedGPT is an open-source, lightweight generalist vision-language foundation model that unifies biomedical modalities and tasks through a transformer-based architecture and diverse pre-training.

  • Results

    BiomedGPT achieved competitive transfer-learning performance across vision, language, and multimodal domains, including 86.1% SLAKE VQA accuracy versus 85.4% for the prior state of the art.

  • Takeaways & Limitations

    The results support a unified pre-training framework for practical biomedical AI spanning diverse modalities and tasks.

  • Takeaways & Limitations

    BiomedGPT’s text comprehension, especially compared with GPT-4V, is not fully established, and robust zero-shot understanding remains challenging in complex medical applications.

Abstract

from arXiv · show

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners, and patients. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation, and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.

Introduction

BiomedGPT is introduced as an open-source, lightweight generalist vision-language foundation model for diverse biomedical tasks across modalities. It uses cross-disciplinary pre-training and task-specific fine-tuning to achieve broad evaluation coverage while remaining computing-friendly.

  • Multimodal capabilities: BiomedGPT processes visual and textual inputs, can incorporate tabular data through serialization, and supports versatile downstream visual-language tasks.Additional clinical tasks can be added through lightweight, task-specific fine-tuning.
  • Model motivation and contribution: BiomedGPT is a generalist vision-language foundation model designed to perform diverse biomedical tasks across modalities using natural language instructions.It is presented as an alternative to biomedical systems specialized for a single task, discipline, or modality.
  • Model motivation and contribution: BiomedGPT is described as the first fully transparent generalist medical AI model comprehensively evaluated on publicly accessible datasets and by medical professionals.The study evaluates transfer learning across 25 datasets through fine-tuning and uses recognized metrics from prior literature.
  • Performance and efficiency: BiomedGPT achieves superior performance on the SLAKE VQA dataset despite having significantly fewer parameters than larger models.The scale-related comparison highlights performance advantages at a substantially smaller parameter count.

Results

BiomedGPT achieved strong results across biomedical VQA, imaging classification, clinical prediction, treatment suggestion, and summarization tasks, often outperforming established or substantially larger baselines. Performance generally improved with model scale, while instruction tuning substantially enhanced alignment accuracy and human evaluations showed competitive generation quality.

  • VQA: 86.1% overall accuracy on SLAKE surpassed BiomedCLIP’s previous state-of-the-art 85.4%.BiomedGPT also recorded 88.0% accuracy on closed-ended VQA questions.
  • VQA: 73.2% and 85.2% weighted F1 scores on VQA-RAD and SLAKE represented increases of 22.5% and 0.02%, respectively, over Med-PaLM M.These results were achieved by BiomedGPT-B despite its much smaller size than Med-PaLM M (12B).
  • Imaging classification: 97.0% and 89.7% accuracies on SZ-CXR and MC-CXR improved by 6.0% and 0.8%, respectively, over LightTBNet.BiomedGPT-B also outperformed Med-PaLM M versions with 12B, 84B, and 584B parameters on CBIS-DDSM using F1-macro.
  • Model scaling: 75.9%, 82.8%, and 89.7% MC-CXR accuracies for small, medium, and base models showed gains of 6.9% at each successive scale increase.The results also indicated performance saturation on several datasets, although the supplied passage does not specify them.
  • Clinical tasks: Over 15% higher in-hospital mortality accuracy than BioGPT and LLaVA-Med was achieved by BiomedGPT-B.BiomedGPT variants also outperformed baselines on clinical-trial matching, where BiomedGPT-B achieved 85.2% ± 1.5% mean accuracy.
  • Generation and human evaluation: 52.3% and 42% ROUGE-L scores on MedQSum and HealthCareMagic were close to leading models’ 53.2% and 44.7% despite BiomedGPT-B being smaller.In human evaluations, BiomedGPT averaged 1.75 across 52 samples, while GPT-4V and LLaVA-Med averaged 1.17 and 1.4.

Discussion

BiomedGPT demonstrates competitive cross-domain transfer through unified pre-training across diverse biomedical modalities and tasks, while revealing limitations in data quality, generated-text evaluation, modality expansion, scaling, and zero-shot text comprehension.

  • Overall assessment: BiomedGPT achieves competitive transfer learning across vision, language, and multimodal domains by integrating diverse biomedical modalities and tasks within unified pre-training.The study also identifies limitations that suggest directions for improvement.
  • Data limitations: Expensive, time-consuming annotation and compromised public-dataset quality make high-quality multimodal biomedical training data difficult to obtain.These challenges are especially pronounced for biomedical image-text pairs.
  • Evaluation limitations: Generated-text evaluation remains limited because CIDEr and ROUGE-L assess agreement with gold standards but do not ensure factual accuracy.F1-RadGraph was introduced to qualitatively assess factual correctness and completeness.
  • Modality expansion: Extending BiomedGPT to video and time-series data may broaden its capabilities, but additional modalities can cause negative transfer and impair performance.The framework has already been extended to 3D images by introducing a 3D image encoder.
  • Scaling and zero-shot limitations: Larger models improve zero-shot and fine-tuned performance, but increase fine-tuning, training-speed, and memory challenges, while BiomedGPT’s zero-shot text comprehension remains limited.The language backbone is resource-constrained but expandable, and prompt tuning was explored to improve efficiency.

Methods … Natural language as a task instructor

BiomedGPT is a lightweight unified biomedical vision-language model that uses modality- and task-agnostic token representations with natural-language instructions to support diverse vision-only, text-only, and vision-language tasks. Its architecture combines BERT-style encoding and GPT-style autoregressive decoding, while ablations examine modality transfer and the contributions of pre-training tasks.

  • Methods: BiomedGPT follows unified-model principles by being modality-agnostic, task-agnostic, and comprehensive across modalities and tasks.Patches and tokens unify inputs and outputs using ideas from vision transformers and language models.
  • BiomedGPT architecture: BiomedGPT combines a BERT-style encoder over corrupted text with a GPT-style left-to-right autoregressive decoder.Both components use transformer multi-head attention, and each layer adds three normalization operations to improve pre-training convergence efficiency and stability.
  • Unifying input/output: Diverse modalities are represented with tokens from a unified finite vocabulary, avoiding task-specific output structures.Images and objects use discretization methods, while text outputs use BPE tokens.
  • Unifying input/output: 256 × 256 images are sparsely encoded as sequences of 16 × 16 patches, reducing image-representation sequence length while preserving patch correlation.Object descriptors discretize target-side objects, and BPE encodes text outputs including object labels and summarizations.
  • Unifying input/output: Cross-domain transfer was evaluated by fine-tuning RadGPT on blood-cell microscopy and dermoscopy classification and MRI-only or CT-only VQA datasets.Results were compared with the all-modality BiomedGPT-B benchmark using accuracy.
  • Natural language as a task instructor: Natural-language instructions specify tasks without task-specific modules, covering vision-only, text-only, and vision-language abstractions.VQA is excluded from handcrafted instructions because its task is fully specified by text inputs.
  • Natural language as a task instructor: Pre-training instructions support masked image modeling, image infilling, and object detection, including recovery of masked patches and generation of object bounding boxes.For masked image modeling and image infilling, the instruction is “What is the image in the middle part?”.
  • Natural language as a task instructor: Downstream instructions distinguish image classification, text summarization, and natural language inference, and BiomedGPT is extendable for custom tasks.The instructions ask what an image describes, request a text summary, or assess whether one text implies another.

Model pre-training

BiomedGPT is pretrained with seq2seq learning over linguistic and visual tokens, using task-specific tokenization and balanced multimodal sampling. Training uses AdamW with scheduled optimization, regularization, stochastic depth, mixed precision, and three model scales.

  • Pre-training objective: BiomedGPT uses autoregressive seq2seq learning, minimizing token-level negative log-likelihood over input sequences.The input sequence may contain both linguistic and visual tokens.
  • Multimodal tokenization: 15% of subword tokens are masked for masked language modeling, while object detection uses Pix2Seq-conditioned location tokens.Subwords come from a BPE tokenizer; visual inputs include image codes and location tokens.
  • Optimization and regularization: AdamW pretraining uses β_1 = 0.9, β_2 = 0.999, ε = 1e−8, a peak learning rate of 1e−4, and a 0.01 warmup ratio.The learning rate follows a linear decay scheduler.
  • Optimization and regularization: Dropout is 0.1, weight decay is 0.01, and stochastic depth is 0.1 in the encoder and decoder except convolution blocks.These settings provide regularization during pretraining.
  • Data balancing and compute: 8:2:1:1 sampling emphasizes vision-language interaction and balances modalities within each pretraining batch.Models are pretrained on 10 NVIDIA A5000 GPUs using mixed precision.
  • Data balancing and compute: 87, 32, and 9 hours are required for the base, medium, and small-scale models, respectively, on the stated hardware.The reported durations correspond to approximately 10 NVIDIA A5000 GPUs with mixed precision.

Model fine-tuning and inference

BiomedGPT uses fine-tuning to adapt pretrained model weights to new biomedical data without the extra components required by most previous biomedical models. During inference, beam search improves generation quality, while a prefix tree addresses classification-task issues caused by unrestricted vocabulary search and invalid labels.

  • Model fine-tuning: Fine-tuning adapts a pre-trained model’s weights to new data and has been widely effective in natural language processing, computer vision, and medical AI.The passage characterizes fine-tuning as a form of transfer learning.
  • Model fine-tuning: Unlike most previous biomedical models, BiomedGPT does not require adding and training extra components such as a linear output layer or decoder.
  • Inference: Beam search improves generation quality during inference but can cause unnecessary vocabulary search and invalid labels in classification tasks.
  • Inference: A beam search strategy incorporating a prefix tree, or trie, is applied to address these classification-task challenges by restricting the search.

Model instruction-tuning and zero-shot prediction

BiomedGPT was instruction-tuned to improve question understanding using an open-vocabulary approach rather than predefined answer sets. Zero-shot studies used task-specific prompts, while instruction-tuned large language models differed substantially from fine-tuned models.

  • Instruction-tuning: Instruction-tuning targeted improved question understanding and enabled open-vocabulary prediction without a predefined answer set during training or inference.The method followed LLaVA-Med’s data-curation approach and diverged from traditional VQA.
  • Zero-shot prediction: Zero-shot VQA-RAD used the dataset’s original questions as prompts, whereas disease-diagnosis trials used the template “Does the patient have <disease> given the image?”.The disease-diagnosis evaluation datasets were curated from the RSNA Pneumonia Detection Challenge (2018).
  • Zero-shot prediction: Preliminary studies found significant discrepancies between instruction-tuned large language models and fine-tuned models.The studies explored the upper bounds of in-context learning and potential integration of advanced language backbones into BiomedGPT.

Model extension

BiomedGPT was initially developed for 2D visual and text data, but its capabilities were extended to additional biomedical tasks and modalities. The extension includes 3D medical imaging classification and can support segmentation through additional decoders.

  • BiomedGPT initially processed visual data, specifically 2D images, and text data.
  • 3D medical imaging classification was added through both pre-training and fine-tuning stages.
  • Segmentation can be supported by introducing additional decoders, such as a mask decoder.

Computing hardware and software

The study used Python and open-source libraries for all experiments and analyses. Pre-training used ten 24-GB NVIDIA A5000 GPUs with PyTorch DistributedDataParallel and fairseq.

  • Python 3.7.4 was used for all experiments and analyses, with replication enabled through open-source libraries.
  • Pre-training used ten 24-GB NVIDIA A5000 GPUs configured for multi-GPU training with PyTorch DistributedDataParallel.The software stack included PyTorch 1.8.1, CUDA 12.2, and fairseq 1.0.0.

Evaluation metrics

BiomedGPT is evaluated with task-specific metrics spanning classification, VQA, natural language inference, captioning, and summarization. Accuracy and F1 variants assess classification-oriented tasks, while ROUGE-L, METEOR, and CIDEr assess generated text and image-caption quality.

  • Classification and question answering: Accuracy evaluates medical image classification, VQA, and natural language inference, while F1 accounts for class imbalance through precision and recall.F1 is the harmonic mean of precision and recall.
  • Classification and question answering: Weighted F1 averages per-class VQA F1 scores after weighting each class by its frequency of occurrence.This weighting supports convenient comparison with state-of-the-art approaches.
  • Classification and question answering: Macro-average F1 independently computes each class’s F1 and averages classes equally for CBIS-DDSM image classification, without accounting for class imbalance.Higher accuracy or F1 indicates better performance.
  • Text generation: ROUGE-L evaluates image captioning and text summarization by measuring longest-common-subsequence overlap between candidate and reference text.A higher ROUGE-L score indicates greater sequence similarity and typically better capture of reference salient points.
  • Text generation: METEOR and CIDEr provide additional captioning metrics, with CIDEr using saliency- and rarity-weighted n-gram matching against reference captions.CIDEr ranges from 0 to 100, and human captions typically score near 90.

Extended Data

The Extended Data documents BiomedGPT’s dataset statistics, model architecture, key components, search procedure, comparative and zero-shot evaluations, and supplementary experimental resources. It also provides tables detailing experiments, datasets, task instructions, VQA-RAD question types, and 3D medical image classification performance.

  • Datasets: Extended Data Fig. 1 summarizes pre-training modality distributions and training, validation, and test sample counts for downstream fine-tuning datasets.Further data-split details are provided in Supplementary Table 7.
  • Model and decoding: Extended Data Figs. 2–3 describe BiomedGPT’s configuration, architecture, head-scale multi-head attention, positional correlations, relative position bias, and trie-based beam search.Trie-based search masks invalid tokens and was also used during validation, yielding an approximately 16× speed increase in the reported experiments.
  • Comparative evaluation: Extended Data Fig. 4 compares BiomedGPT-B with MedPaLM M on CBIS-DDSM and reports prompt-tuning results on three image-classification datasets.Prompt tuning learns soft prompts or extra task-specific parameters instead of copying the entire pre-trained model for each downstream task.
  • Zero-shot evaluation: Extended Data Fig. 5 illustrates zero-shot classification, linear-probing transfer learning, and zero-shot generation, including zero-shot performance on five disease-diagnosis tasks.BiomedGPT can generate responses without additional label candidates or a trained linear classifier.
  • Supplementary resources: Extended Data Tables 1–5 provide fine-tuned results across 25 experiments, dataset details, pre-training output instructions, VQA-RAD question types, and 3D medical image classification metrics.The 3D classification results are reported in terms of accuracy and F1-Macro.

Supplementary Information · Supplementary reference

The supplementary information documents BiomedGPT’s data composition, ablations, task-specific evaluations, qualitative predictions, human-evaluation categories, three-dimensional extension, and training details. It also provides examples and supplementary comparisons across biomedical tasks.

  • Supplementary Information: Instruction-tuning data span multiple modalities, with the modality distribution reported in Supplementary Fig. 1.The figure specifically characterizes the modalities represented in BiomedGPT’s instruction-tuning data.
  • Supplementary Information: Pretraining ablations examine dataset and task diversity, including object detection exclusion, MLM inclusion, domain-specific datasets, and chest X-ray continual pretraining.The supplementary ablation description also states that the chest X-ray-pretrained model is later fine-tuned on other domains.
  • Supplementary Information: Supplementary figures compare zero-shot BioMistral and LLaMA-3-instruct with fine-tuned models for treatment suggestion, clinical-trial matching, and in-hospital mortality prediction.The reported sample sizes are n=4680 for treatment suggestion and n=7079 for clinical-trial matching.
  • Supplementary Information: Qualitative examples cover disease diagnosis, object recognition, VQA, chest X-ray report generation, and clinical-trial matching, including both correct and incorrect predictions.The clinical-trial examples include eligible and ineligible judgments, while radiology examples show generated impressions and highlighted errors or omissions.
  • Supplementary Information: Supplementary tables report non-difference-question accuracy on MIMIC-Diff-VQA and fine-tuned MedMNIST results against SOTA approaches by modality.The tables compare BiomedGPT with MMQ, EKAID, and PLURAL on MIMIC-Diff-VQA and list SOTA approaches for each MedMNIST modality.
  • Supplementary Information: Human-evaluation VQA categories include abnormality, presence, location, type, level, and view.These categories are defined by questions concerning abnormalities and their visual attributes.
  • Supplementary Information: The 3D extension uses ADNI data for AIBL and MIRIAD MRI evaluation, 1,050 LIDC training samples, and ACS Convolutions for processing 3D volumes.ADNI includes 1,216 normal-control images and 1,565 Alzheimer’s-disease images.
  • Supplementary Information: Additional supplementary tables provide task-specific fine-tuning settings, an IU X-RAY image-captioning hyperparameter ablation, and dataset split and preprocessing references.The captioning ablation notes that the maximum target length is too small and inappropriate.
Loading 2305.17100v4…