PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Generalist Biomedical AI

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, Vivek Natarajan

arXiv:2307.14334v1cs.CLcs.CV

TL;DR

Biomedical AI needs to handle diverse modalities and tasks beyond today’s predominantly unimodal specialist systems. This paper introduces MultiMedBench and Med-PaLM M, demonstrating competitive performance across 14 tasks, broader zero-shot capabilities, and encouraging radiologist-evaluated report generation while identifying validation and data-access challenges.

  • Problem

    Most biomedical AI systems are unimodal and single-task, while comprehensive multimodal benchmarks for developing generalist biomedical AI are limited.

  • Method

    The paper curates MultiMedBench and fine-tunes Med-PaLM M, a unified multimodal generative model, on biomedical language, imaging, and genomics tasks.

  • Results

    Med-PaLM M is competitive with or exceeds SOTA on all MultiMedBench tasks, with evidence of zero-shot generalization, positive transfer, and multimodal medical reasoning.

  • Takeaways & Limitations

    The results support the potential of generalist biomedical AI systems for applications spanning scientific discovery to care delivery.

  • Takeaways & Limitations

    Further work is needed to expand biomedical data access and rigorously validate performance and safety in real-world applications.

Abstract

from arXiv · show

Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system. Med-PaLM M is a large multimodal generative model that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. Med-PaLM M reaches performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. We also report examples of zero-shot generalization to novel medical concepts and tasks, positive transfer learning across tasks, and emergent zero-shot medical reasoning. To further probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist evaluation of model-generated (and human) chest X-ray reports and observe encouraging performance across model scales. In a side-by-side ranking on 246 retrospective chest X-rays, clinicians express a pairwise preference for Med-PaLM M reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.

1 Introduction

Biomedical AI is often limited by unimodal, single-task systems, motivating a unified model that handles diverse modalities and tasks. The paper introduces MultiMedBench and Med-PaLM M, which performs competitively across the benchmark while showing early evidence of broader generalization and reasoning.

  • Motivation: Most biomedical AI systems are unimodal and single-task, limiting their ability to integrate complementary medical information or support interactive clinical use.Examples include mammography systems that cannot incorporate records, MRI, or literature and cannot explain predictions or learn from physician feedback.
  • Contributions: MultiMedBench is an open-source multimodal benchmark spanning 14 biomedical tasks across language, imaging, and genomics.Tasks include question answering, visual question answering, image classification, radiology report generation and summarization, and genomic variant calling.
  • Contributions: Med-PaLM M is a flexible multimodal sequence-to-sequence model that interleaves biomedical information and uses a unified generative training framework.It is designed to handle multiple biomedical tasks with the same model weights.
  • Results: Med-PaLM M reaches performance competitive with or exceeding SOTA on all MultiMedBench tasks, surpassing specialists by large margins in some cases.It exceeds prior SOTA by over 8% micro-F1 on MIMIC-CXR report generation and over 10% on BLEU-1 and F1 for Slake-VQA.
  • Results: Experiments provide evidence of benefits from scale, zero-shot generalization to novel medical concepts and tasks, positive transfer, and zero-shot multimodal medical reasoning.Radiologist evaluation of generated chest X-ray reports also showed encouraging performance across model scales.
  • Limitations: The study identifies large-scale biomedical data access, real-world validation, and safety implications as substantial remaining challenges.These are presented as limitations and directions for future research.

2 Related Work

Foundation and multimodal models motivate unified biomedical systems that can adapt across tasks and modalities. Existing biomedical efforts provide important foundations but generally require task-specific downstream adaptation, while benchmarks for generalist biomedical AI remain limited.

  • Foundation models: Foundation models use large-scale pretraining and can adapt to many downstream tasks through in-context learning or few-shot fine-tuning.Their generative capabilities also support human-AI interaction and collaboration.
  • Multimodal foundation models: Multimodal foundation models combine modalities and often use language supervision or a unified generative output space.Med-PaLM M builds on pretrained vision-language models and biomedical domain fine-tuning.
  • Generalist models: Generalist models aim to address many tasks with the same weights, extending the multitask and prompting capabilities demonstrated by language-only models.This framing distinguishes generality across tasks from modality-specific specialization.
  • Biomedical applications: Biomedical multimodal efforts include Geneformer, BiomedGPT, and LLaVA-Med, but these pretrained systems require further task-specific data and fine-tuning for downstream applications.Med-PaLM M is described as directly trained to jointly solve many biomedical tasks without further parameter updates.
  • Benchmarks: Existing generalist biomedical benchmarks have been limited, with BenchMD covering many datasets and modalities but focusing primarily on classification.The paper positions MultiMedBench as also including generative medical tasks.

3 MultiMedBench: A Benchmark for Generalist Biomedical AI

MultiMedBench is a multimodal, multitask benchmark designed to develop and evaluate generalist biomedical AI. It spans clinically relevant tasks, biomedical modalities, output formats, and more than one million samples.

  • Benchmark scope: MultiMedBench contains 14 individual tasks drawn from 12 de-identified open-source datasets for generalist biomedical AI evaluation.The benchmark covers medical questions, radiology reports, pathology, dermatology, chest X-ray, mammography, and genomics.
  • Task coverage: The benchmark spans five task types, including question answering, report generation and summarization, visual question answering, image classification, and genomic variant calling.These task types combine language-only and multimodal evaluation.
  • Modality coverage: MultiMedBench covers seven biomedical data modalities, including text, radiology, pathology, dermatology, mammography, and genomics.Radiology includes CT, MRI, and X-ray data.
  • Data resources: The benchmark contains over 1 million samples in total.Its datasets are de-identified and open source.
  • Evaluation goals: Language-only tasks assess medical knowledge comprehension, recall, and manipulation, while multimodal tasks assess visual understanding and multimodal reasoning.The multimodal set includes VQA, image classification, chest X-ray report generation, and genomic variant calling.

4 Med-PaLM M: A Proof of Concept for Generalist Biomedical AI

Med-PaLM M adapts the PaLM-E multimodal architecture to biomedical tasks through unified instruction tuning on MultiMedBench. It combines language and vision encoders with interleaved multimodal prompts and supports diverse biomedical outputs using shared model parameters.

  • Model foundations: Med-PaLM M inherits pretrained-model architectures and general domain knowledge while being adapted to biomedical data.The methods describe pretrained PaLM and Vision Transformer components as foundations for the model.
  • Model foundations: PaLM-E provides the base architecture by processing interleaved text, vision, and sensor inputs in a single multimodal prompt.It also supports zero-shot multimodal reasoning and few-shot in-context learning.
  • Model scale: The study evaluates PaLM-E combinations ranging from PaLM 8B with ViT 4B to PaLM 540B with ViT 22B.These combinations correspond to 12B, 84B, and 562B total parameter variants.
  • Biomedical adaptation: Med-PaLM M is developed by fine-tuning and aligning PaLM-E to the biomedical domain using MultiMedBench.The resulting models encode multimodal inputs and perform question answering, report generation and summarization, image classification, and genomic variant calling.
  • Instruction tuning: Instruction tuning presents task-specific instructions, relevant context, and questions within a unified generative framework.A text-only one-shot exemplar is added for most tasks to improve instruction following and output formatting.
  • Model training: The model is fine-tuned end-to-end with all parameters updated across a mixture of MultiMedBench tasks.Training uses empirically selected task mixture ratios and model-size-specific hyperparameters.
  • Multimodal prompting: Multimodal prompts interleave image tokens with textual context such as clinical history, view orientation, and study reason.During fine-tuning, inputs contain at most one image, although the model can process multiple images at inference.

5 Evaluation

The evaluation examines Med-PaLM M’s generalist performance, zero-shot generalization, task transfer, and radiology-report quality across model scales and expert review settings.

  • Evaluation purposes: Med-PaLM M was evaluated across all MultiMedBench tasks and model scales to assess generalist capabilities and scaling effects.Performance was compared with prior specialist systems and a generalist model without biomedical finetuning.
  • Evaluation purposes: The evaluation probes zero-shot generalization to novel medical concepts, unseen tasks, and multimodal chain-of-thought reasoning.The study also examines whether jointly training diverse tasks produces positive task transfer.
  • Novel medical concepts: Tuberculosis detection used the Montgomery County chest X-ray set, containing 138 frontal images with expert annotations of lung abnormalities.The model had been trained on other data, and the task was formulated as two-choice question answering.
  • Novel medical concepts: Zero-shot multimodal reasoning generated both a class prediction and an image-finding report from a text-only exemplar without its corresponding image.The hand-crafted exemplar and dummy image placeholder make the setup zero-shot rather than one-shot.
  • Novel tasks and transfer: Novel task generalization was tested with two-view chest X-ray report generation, although training used only single-view inputs.Positive task transfer was examined by removing MIMIC-CXR classification tasks from one 84B training mixture and comparing report generation.
  • Radiology evaluation: Radiology-report quality was assessed by four thoracic radiologists using side-by-side rankings and independent error and omission annotations on 246 cases.The evaluation used a single image per study and included calibration on a separate 25-case pilot set.

6 Results

Med-PaLM M performs competitively across MultiMedBench with shared weights, generalizes to novel biomedical settings, and shows encouraging but scale-dependent radiology-report results.

  • Benchmark performance: Across MultiMedBench, Med-PaLM M exceeded prior SOTA on 5 of 12 comparable tasks while remaining competitive on the rest.These results used the same model weights without task-specific architecture customization or optimization.
  • Benchmark performance: Med-PaLM M outperformed the previous best PaLM results by a large margin on all three medical question answering datasets in the same few-shot setting.Med-PaLM 2 remained higher-performing as the SOTA comparison for these tasks.
  • Benchmark performance: Med-PaLM M improved over PaLM-E 84B on all 14 tasks, often by a significant margin.PaLM-E 84B was used as a generalist baseline without biomedical domain finetuning.
  • Scaling: Scaling from 12B to 562B substantially benefits language reasoning and reasoning-intensive multimodal tasks, while image classification and chest X-ray report generation show diminishing gains.For chest X-ray report generation, the 84B model roughly matched or slightly exceeded the 562B model on most metrics.
  • Zero-shot generalization: Med-PaLM M competitively detected tuberculosis against a specialized ensemble despite using a zero-shot text-only exemplar rather than dataset-specific training.The open-ended question-answering formulation did not report AUC because normalized class probabilities were unavailable.
  • Zero-shot reasoning: Large models identified the major tuberculosis lesion in the correct location, whereas the 12B model failed to generate a coherent visually conditioned response.Expert review still found omissions and errors in the generated reports.
  • Novel task generalization: Med-PaLM M achieved zero-shot performance comparable to single-view report generation on a novel two-view task despite training only on single-view inputs.The model achieved SOTA results on clinical efficacy metrics for the two-view setup.

7 Discussion

Med-PaLM M demonstrates broad biomedical generalization, while the discussion identifies domain data, scaling, integration, and validation constraints that shape practical deployment.

  • Generalist performance: Med-PaLM M generalizes across medical modalities and tasks, reaching near or above prior state of the art across a diverse array.The authors describe it as the first demonstration of a generalist biomedical AI system with this breadth.
  • Benchmarks and data: MultiMedBench remains limited by approximately 1 million cumulative samples and missing modalities and tasks such as transcriptomics and proteomics.The authors identify these limitations as barriers to further benchmark development.
  • Benchmarks and data: Large-scale multimodal datasets are needed to jointly learn and align modality-specific encoders with the decoder across more biomedical data types.The discussion presents dataset scale as a key barrier to broader multimodal coverage.
  • Finetuning and specialization: Domain-specific biomedical finetuning is critical for strong biomedical performance, as Med-PaLM M substantially outperforms PaLM-E across model scales.The authors suggest distribution shift between medical and non-medical data may explain this difference.
  • Scaling and architecture: Scaling multimodal generalist models is challenging because medical data scarcity and weak modality encoders can bottleneck tasks requiring combined modalities.The discussion specifically identifies vision encoding as a likely bottleneck for mammography and dermatology classification.
  • Radiology evaluation: Radiologists preferred Med-PaLM M reports over human-generated references in up to 40.50% of cases, with clinically significant errors comparable to prior human-report studies.The evaluation suggests encouraging performance on a challenging multimodal task, while potential clinical utility remains future-facing.
  • Alternative approaches: Alternative multimodal approaches may be more applicable depending on data availability, pretrained models, compute, and application scenarios.Examples include frozen encoders with adapter layers and language models interfacing with specialized systems.
  • Real-world deployment: Real-world use requires matching or exceeding specialized models or reaching clinically applicable performance, alongside careful safety and equity validation.These considerations are explicitly described as beyond the scope of the work.

8 Perspective on Generalist Biomedical AI

The authors frame generalist biomedical AI as a milestone that could complement specialist systems and support discovery and cross-domain assistance.

  • Perspective: A single model reaching near or above state of the art across diverse biomedical tasks is a noteworthy milestone for generalist biomedical AI.The authors compare this future to collaboration among generalist and specialist clinicians.
  • Perspective: Generalist biomedical AI could provide near zero-shot insight, integrate knowledge across biomedical areas, and offer assistance across many fields.These are presented as potential application frontiers rather than established deployments.

9 Conclusion

Generalist biomedical AI may support next-generation learning health systems by integrating multimodal medical data and adapting to clinical contexts. The authors present Med-PaLM M as an important step, while emphasizing the need for further development and rigorous validation.

  • Conclusion: Generalist biomedical AI systems may underpin next-generation learning health systems and make healthcare more accessible, efficient, equitable, and humane.The passage conditions this vision on effectively assimilating multimodal medical data at scale and rapidly adapting to new clinical contexts.

Data Availability

The study uses open-source, de-identified datasets, but the underlying large language models cannot be open sourced.

  • Data availability: MultiMedBench comprises de-identified datasets that are all open source.The paper presents an overview of these datasets in Table 1.
  • Data availability: The large language models used in the study will not be open sourced.The authors state that they provide comprehensive methodological details and build on previously detailed models so similar approaches can be tried with other LLM classes.

Appendix

The appendix details MultiMedBench datasets and tasks, Med-PaLM M training, task-specific performance analyses, human evaluation, and example outputs.

  • Datasets and tasks: The appendix describes the datasets and tasks included in MultiMedBench.
  • Training procedure: It explains the training procedure for Med-PaLM M.
  • Performance analysis: Task analyses cover language-only question answering, radiology report summarization, medical image classification, visual question answering, and chest X-ray report generation.
  • Evaluation and examples: The appendix also reports human evaluation of model-generated chest X-ray reports and examples from MultiMedBench tasks.

A.1 MultiMedBench

This section presents a comprehensive overview of MultiMedBench, including its datasets, preprocessing, task setups, and coverage across biomedical tasks.

  • MultiMedBench is described through detailed dataset, data-preprocessing, and task-setup information.
  • Figure A.1 summarizes the benchmark across its various biomedical tasks.
  • The benchmark overview provides the foundation for evaluating generalist biomedical AI across multiple task settings.

A.1.1 Language-only datasets

The language-only component uses multiple-choice medical question-answering datasets, while the radiology summarization component draws on reports from MIMIC-III.

  • MultiMedQA: MultiMedQA contributes the MedQA, MedMCQA, and PubMedQA multiple-choice medical question-answering datasets.
  • MultiMedQA: These question-answering tasks use language only and do not require interpreting additional modalities.
  • MultiMedQA: The training set contains 10,178 MedQA questions and 182,822 MedMCQA questions.
  • Radiology report summarization: MIMIC-III contains 79,790 radiology reports spanning CT and MRI across seven anatomic regions.
  • Radiology report summarization: A total of 78,875 reports were selected for the radiology report summarization dataset based on criteria including report length.

A.1.2 Multimodal datasets

The multimodal evaluation spans dermatology, mammography, chest radiography, and other biomedical datasets, combining classification, generation, and visual-language tasks. Med-PaLM M is competitive across these tasks, while dataset artifacts, scaling limits, and subjective evaluation constrain interpretation.

  • Dermatology: PAD-UFES-20 provides 2,298 smartphone-captured skin-lesion images collected under varying resolutions, sizes, and lighting conditions.
  • Dermatology: The dermatology setup includes six lesion types and associates images with up to 21 patient clinical features for classification.
  • Mammography: VinDr-Mammo contains 5,000 mammography studies and 20,000 grayscale images, while CBIS-DDSM contains 2,620 scanned film studies with lesion-level annotations.
  • Chest radiography: MIMIC-CXR supports chest X-ray report generation and binary classification using 377,110 images from 227,835 studies involving 65,379 patients.
  • Classification performance: All Med-PaLM M size variants exceeded prior SOTA on VinDr-Mammo macro-AUC, while Med-PaLM M 562B achieved 79.09% macro-AUC on MIMIC-CXR versus 81.27% for ParallelXNet.
  • Interpretation and limitations: Med-PaLM M showed positive task transfer and competitive multimodal performance, but MIMIC-CXR references to prior studies or absent views can induce hallucinations in single-image inputs.
Loading 2307.14334v1…