PaperlayerView exact revision on arXiv

Source-linked AI summary

Med-Flamingo: a Multimodal Medical Few-shot Learner

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec

arXiv:2307.15189v1cs.CVcs.AI

TL;DR

Medical multimodal models generally require sizeable downstream fine-tuning, despite scarce data and the need to combine visual and textual information. Med-Flamingo adapts OpenFlamingo-9B through medical image-text pre-training and evaluates few-shot generative VQA with clinician ratings, achieving the best average clinical rank and enabling rationale generation. The work is an early proof of concept rather than a clinically safe system.

  • Problem

    Medical applications often require multimodal reasoning, but existing models typically need sizeable downstream datasets even when labeled medical data are scarce.

  • Method

    Med-Flamingo continues pre-training OpenFlamingo-9B on paired and interleaved medical image-text data, then evaluates few-shot generative VQA with Visual USMLE and clinician ratings.

  • Results

    1.67 average clinical evaluation rank across three generative medical VQA datasets, versus 2.33 for the best prior model, with up to 20% improvement over prior models.

  • Takeaways & Limitations

    Med-Flamingo enables multimodal medical few-shot adaptations, including complex question answering and rationale generation.

  • Takeaways & Limitations

    Med-Flamingo is a proof of concept not intended or safe for clinical use; hallucinations and occasional low-quality generations were observed.

Abstract

from arXiv · show

Medicine, by its nature, is a multifaceted domain that requires the synthesis of information across various modalities. Medical generative vision-language models (VLMs) make a first step in this direction and promise many exciting clinical applications. However, existing models typically have to be fine-tuned on sizeable down-stream datasets, which poses a significant limitation as in many medical applications data is scarce, necessitating models that are capable of learning from few examples in real-time. Here we propose Med-Flamingo, a multimodal few-shot learner adapted to the medical domain. Based on OpenFlamingo-9B, we continue pre-training on paired and interleaved medical image-text data from publications and textbooks. Med-Flamingo unlocks few-shot generative medical visual question answering (VQA) abilities, which we evaluate on several datasets including a novel challenging open-ended VQA dataset of visual USMLE-style problems. Furthermore, we conduct the first human evaluation for generative medical VQA where physicians review the problems and blinded generations in an interactive app. Med-Flamingo improves performance in generative medical VQA by up to 20\% in clinician's rating and firstly enables multimodal medical few-shot adaptations, such as rationale generation. We release our model, code, and evaluation app under https://github.com/snap-stanford/med-flamingo.

1 INTRODUCTION

Medical AI could benefit from multimodal in-context learning, but medical data complexity and existing models’ lack of multimodal few-shot design make this challenging. Med-Flamingo addresses this gap with a medically adapted generative vision-language model evaluated through open-ended VQA and clinician ratings.

  • In-context learning uses a few labeled examples in the prompt without tuning model parameters.This capability could support rare-case handling, rationale generation, and customization of medical AI outputs.
  • Medical in-context learning is difficult because medical data are complex and multimodal, while tasks are diverse.
  • Existing multimodal medical models were not designed or tested for multimodal in-context learning with few-shot examples.
  • Med-Flamingo is a medical foundation model based on Flamingo that ingests interleaved images and text to generate conditioned text.Its preparation included an interleaved image-text dataset derived from more than 4K medical textbooks and authoritative sources.
  • The study evaluates open-ended generative medical VQA using a new Visual USMLE dataset and clinician assessment of blinded generations.The evaluation targets clinical usefulness rather than only post-hoc scoring of answer options.
  • 1.67 average clinical evaluation rank, versus 2.33 for the best prior model, with up to 20% improvement over prior models.The model also generated rationales for complex medical questions, while training-data availability and task complexity may limit performance.

1. Multimodal pre-training on medical literature

Med-Flamingo is pre-trained on paired and interleaved medical image-text data, then used for few-shot generative VQA. The study introduces medical datasets and evaluates multimodal few-shot capabilities.

  • Med-Flamingo is initialized from an OpenFlamingo checkpoint and continued-pre-trained on paired and interleaved medical image-text data from publications and textbooks.
  • 2. Few-shot generative VQA: The study evaluates few-shot generative VQA using two existing medical VQA datasets and a novel Visual USMLE-style dataset.
  • The paper presents a multimodal medical few-shot learner supporting applications such as rationale generation and conditioning on retrieved multimodal context.

2 RELATED WORKS

Medical language models have expanded specialized medical modeling, while newer vision-language systems address multimodal clinical information. Med-Flamingo is positioned as a generative medical VLM alongside existing encoder-only and generative approaches.

  • Medical language models such as BioBERT, ClinicalBERT, PubMedBERT, BioLinkBERT, DRAGON, BioMedLM, BioGPT, and Med-PaLM target specialized medical tasks.These models can match or surpass general-purpose LLMs on tasks such as medical question answering.
  • Medical applications often require multimodal information, including joint analysis of X-ray images and radiology reports.
  • Existing medical VLMs include encoder-only BiomedCLIP and MedVINT, while the paper focuses on a generative VLM and compares it with MedVINT.
  • The MTB dataset categorizes medical textbook titles into 49 manually created categories or an “other” category.

3 MED-FLAMINGO

Med-Flamingo adapts OpenFlamingo-9B to medical multimodal in-context learning through continued pre-training on paired and interleaved image-text data. Its objective models medical text conditioned on preceding visual and language tokens.

  • 3 MED-FLAMINGO: Med-Flamingo continues pre-training the OpenFlamingo-9B checkpoint to adapt it to the medical domain.The base model uses frozen LLaMA-7B language and CLIP ViT/L-14 vision components; Med-Flamingo results from continued medical pre-training.
  • 3.1 DATA: The model jointly trains on paired image-text data from PMC-OA and interleaved image-text data constructed from medical textbooks.The MTB textbook dataset contains approximately 0.8M images and 584M tokens; PMC-OA contributes 1.3M training image-caption pairs.
  • 3.2 OBJECTIVES: The language-modelling objective predicts each language token from preceding language and visual tokens.The medical setting models image-text data only, excluding videos.
  • 3.2 OBJECTIVES: The joint objective combines losses over paired and interleaved datasets with λ = 1.Dp denotes the paired dataset and Di the interleaved dataset.
  • 3.3 TRAINING: Training updated 1.3B parameters while keeping roughly 7B parameters frozen, for a total model size of 8.3B parameters.The model was pre-trained for 2700 steps with a total batch size of 400 on 8x 80GB NVIDIA A100 GPUs.

4 EVALUATION

The evaluation compares generative VQA systems on existing medical datasets and a challenging open-ended Visual USMLE benchmark. It combines automatic metrics with clinician ratings of blinded generations and addresses dataset leakage through deduplication.

  • 4 EVALUATION: The evaluation compares MedVINT and OpenFlamingo in zero-shot, fine-tuned, or few-shot settings where applicable.MedVINT lacks a fine-tuned Visual USMLE baseline because that dataset has no separate training split.
  • 4.1 AUTOMATIC EVALUATION: Visual USMLE contains 618 open-ended USMLE-style questions combining images, case vignettes, and potentially laboratory tables.The benchmark spans multiple specialties and uses four shots because its problems and answers are comparatively long.
  • Evaluation metrics: Generative answers are assessed with clinical evaluation score, BERT similarity, and exact-match metrics.Clinical evaluation is provided by three medical doctors, while exact-match is described as noisy and conservative.
  • 4.2 HUMAN EVALUATION: The human evaluation app shows raters the image, question, correct answer, and randomized blinded generations for scoring from 0 to 10.The app was implemented in Streamlit for clinical experts to evaluate generated answers.
  • 4.3 DEDUPLICATION AND LEAKAGE: Deduplication removed 194 highly similar images from the 6,700-image PVQA test set after detecting pre-training leakage.Similarity was assessed using Vision Transformer image embeddings and FAISS nearest-neighbor search.

5 RESULTS

Med-Flamingo is evaluated on generative medical VQA using clinician-centered human assessment alongside automated metrics. Across datasets, few-shot prompting produces clinically preferred generations, while pathology and lengthy Visual USMLE answers expose evaluation and performance challenges.

  • Human evaluation asks clinical experts to score blinded open-ended generations from 0 to 10 for clinical usefulness.
  • Conventional VQA datasets: On VQA-RAD, Med-Flamingo few-shot improves the clinical evaluation score by ∼20% over the best baseline.The dataset uses custom splits to address leakage, and automated metrics are relatively aligned with clinical preference.
  • Conventional VQA datasets: PathVQA has the lowest clinical evaluation performance across models, revealing a pathology performance deficit that automated classification-based metrics can overestimate.The authors relate this deficit to limited pre-training on large-scale, fine-grained pathology image datasets.
  • Visual USMLE: Exact-match is constantly 0 on Visual USMLE because the correct answers are lengthy paragraphs.
  • Visual USMLE: On Visual USMLE, Med-Flamingo few-shot produces the clinically most preferable generations, although its automated scores are lower than those of zero-shot counterparts.The authors hypothesize that summarizing long vignettes and answers for few-shot prompts may favor shorter responses that differ more from reference answers.
  • Across datasets: Across evaluation datasets, Med-Flamingo few-shot achieves the best average clinical-evaluation rank of 1.67, ahead of OpenFlamingo zero-shot at 2.33.
  • Qualitative analysis: Few-shot prompting supports rationale generation, but multimodal rationales may be non-robust when the model answers incorrectly.The examples include rationales that visually guide readers toward medically relevant objects.

6 DISCUSSION

Med-Flamingo is presented as an early proof-of-concept for medical multimodal few-shot learning, with future improvements expected from scaling, data cleaning, and preference alignment. The work is not intended or safe for clinical use.

  • Rationale generation and retrieved multimodal context are identified as opportunities for explainability and grounding in verified sources.
  • Med-Flamingo is not intended or safe for clinical use because hallucinations were observed and untuned generations may occasionally be low quality.

A.1 ADDITIONAL DETAILS FOR MTB DATASET

The MTB image analysis clustered image embeddings into 100 groups, manually annotated the groups, discarded unclear clusters, and visualized the remaining frequencies.

  • MTB image embeddings were clustered into 100 groups and manually reviewed to assign annotations.
  • Noisy or unclear image clusters were discarded before displaying the remaining cluster frequencies.
  • Figure 7 shows the distribution of manually annotated image clusters in the MTB dataset.

A.2 ADDITIONAL DETAILS FOR VISUAL USMLE DATASET

The Visual USMLE dataset’s specialty-topic distribution is visualized using categories defined for title classification, with rare categories merged into Other for visualization.

  • Table 4 lists 49 categories plus Other for visualizing the MTB dataset in Figure 3.
  • Figure 8 shows the distribution of specialty topics in the Visual USMLE dataset, classified using categories from Table 4.
Loading 2307.15189v1…