Med-Flamingo: a Multimodal Medical Few-shot Learner

• In-context learning uses a few labeled examples in the prompt without tuning model parameters.This capability could support rare-case handling, rationale generation, and customization of medical AI outputs.Source: §1 · p. 1
• Medical in-context learning is difficult because medical data are complex and multimodal, while tasks are diverse.Source: §1 · p. 1
• Existing multimodal medical models were not designed or tested for multimodal in-context learning with few-shot examples.Source: §1 · pp. 1–2
• Med-Flamingo is a medical foundation model based on Flamingo that ingests interleaved images and text to generate conditioned text.Its preparation included an interleaved image-text dataset derived from more than 4K medical textbooks and authoritative sources.Source: §1 · p. 2
• The study evaluates open-ended generative medical VQA using a new Visual USMLE dataset and clinician assessment of blinded generations.The evaluation targets clinical usefulness rather than only post-hoc scoring of answer options.Source: §1 · p. 2
• 1.67 average clinical evaluation rank, versus 2.33 for the best prior model, with up to 20% improvement over prior models.The model also generated rationales for complex medical questions, while training-data availability and task complexity may limit performance.Source: §1 · p. 2

• Med-Flamingo is initialized from an OpenFlamingo checkpoint and continued-pre-trained on paired and interleaved medical image-text data from publications and textbooks.Source: Figure 2 · p. 3, §1 · p. 3

2. Few-shot generative VQA

• The study evaluates few-shot generative VQA using two existing medical VQA datasets and a novel Visual USMLE-style dataset.Source: Figure 2 · p. 3, §1 · p. 4

• The paper presents a multimodal medical few-shot learner supporting applications such as rationale generation and conditioning on retrieved multimodal context.Source: §1 · p. 4

• Medical language models such as BioBERT, ClinicalBERT, PubMedBERT, BioLinkBERT, DRAGON, BioMedLM, BioGPT, and Med-PaLM target specialized medical tasks.These models can match or surpass general-purpose LLMs on tasks such as medical question answering.Source: §2 · p. 4
• Medical applications often require multimodal information, including joint analysis of X-ray images and radiology reports.Source: §2 · p. 4
• Existing medical VLMs include encoder-only BiomedCLIP and MedVINT, while the paper focuses on a generative VLM and compares it with MedVINT.Source: §2 · p. 4
• The MTB dataset categorizes medical textbook titles into 49 manually created categories or an “other” category.Source: Figure 3 · p. 4

3 MED-FLAMINGO

• Med-Flamingo continues pre-training the OpenFlamingo-9B checkpoint to adapt it to the medical domain.The base model uses frozen LLaMA-7B language and CLIP ViT/L-14 vision components; Med-Flamingo results from continued medical pre-training.Source: §3 · pp. 4–5

3.1 DATA

• The model jointly trains on paired image-text data from PMC-OA and interleaved image-text data constructed from medical textbooks.The MTB textbook dataset contains approximately 0.8M images and 584M tokens; PMC-OA contributes 1.3M training image-caption pairs.Source: §3.1 · p. 5

3.2 OBJECTIVES

• The language-modelling objective predicts each language token from preceding language and visual tokens.The medical setting models image-text data only, excluding videos.Source: §3.2 · p. 5
• The joint objective combines losses over paired and interleaved datasets with λ = 1.Dp denotes the paired dataset and Di the interleaved dataset.Source: §3.2 · p. 5

3.3 TRAINING

• Training updated 1.3B parameters while keeping roughly 7B parameters frozen, for a total model size of 8.3B parameters.The model was pre-trained for 2700 steps with a total batch size of 400 on 8x 80GB NVIDIA A100 GPUs.Source: §3.3 · p. 5

4 EVALUATION

• The evaluation compares MedVINT and OpenFlamingo in zero-shot, fine-tuned, or few-shot settings where applicable.MedVINT lacks a fine-tuned Visual USMLE baseline because that dataset has no separate training split.Source: §4.1 · p. 6

4.1 AUTOMATIC EVALUATION

• Visual USMLE contains 618 open-ended USMLE-style questions combining images, case vignettes, and potentially laboratory tables.The benchmark spans multiple specialties and uses four shots because its problems and answers are comparatively long.Source: §4.1 · p. 6

Evaluation metrics

• Generative answers are assessed with clinical evaluation score, BERT similarity, and exact-match metrics.Clinical evaluation is provided by three medical doctors, while exact-match is described as noisy and conservative.Source: §4.1 · p. 6

4.2 HUMAN EVALUATION

• The human evaluation app shows raters the image, question, correct answer, and randomized blinded generations for scoring from 0 to 10.The app was implemented in Streamlit for clinical experts to evaluate generated answers.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 DEDUPLICATION AND LEAKAGE

• Deduplication removed 194 highly similar images from the 6,700-image PVQA test set after detecting pre-training leakage.Similarity was assessed using Vision Transformer image embeddings and FAISS nearest-neighbor search.Source: §4.3 · p. 7

• Human evaluation asks clinical experts to score blinded open-ended generations from 0 to 10 for clinical usefulness.Source: §5 · pp. 7–8

Conventional VQA datasets

• On VQA-RAD, Med-Flamingo few-shot improves the clinical evaluation score by ∼20% over the best baseline.The dataset uses custom splits to address leakage, and automated metrics are relatively aligned with clinical preference.Source: §5 · p. 8, Table 1 · p. 8
• PathVQA has the lowest clinical evaluation performance across models, revealing a pathology performance deficit that automated classification-based metrics can overestimate.The authors relate this deficit to limited pre-training on large-scale, fine-grained pathology image datasets.Source: §5 · pp. 8–9, Table 2 · p. 9

Visual USMLE

• Exact-match is constantly 0 on Visual USMLE because the correct answers are lengthy paragraphs.Source: Table 3 · p. 9, §5 · p. 9
• On Visual USMLE, Med-Flamingo few-shot produces the clinically most preferable generations, although its automated scores are lower than those of zero-shot counterparts.The authors hypothesize that summarizing long vignettes and answers for few-shot prompts may favor shorter responses that differ more from reference answers.Source: §5 · p. 9

Across datasets

• Across evaluation datasets, Med-Flamingo few-shot achieves the best average clinical-evaluation rank of 1.67, ahead of OpenFlamingo zero-shot at 2.33.Source: §5 · p. 9

Qualitative analysis

• Few-shot prompting supports rationale generation, but multimodal rationales may be non-robust when the model answers incorrectly.The examples include rationales that visually guide readers toward medically relevant objects.Source: Figure 5 · p. 8, §5 · p. 9

• Rationale generation and retrieved multimodal context are identified as opportunities for explainability and grounding in verified sources.Source: §6 · p. 10
• Med-Flamingo is not intended or safe for clinical use because hallucinations were observed and untuned generations may occasionally be low quality.Source: §6 · p. 10

• MTB image embeddings were clustered into 100 groups and manually reviewed to assign annotations.Source: §A.1 · p. 14
• Noisy or unclear image clusters were discarded before displaying the remaining cluster frequencies.Source: §A.1 · p. 14
• Figure 7 shows the distribution of manually annotated image clusters in the MTB dataset.Source: Figure 7 · p. 14

• Table 4 lists 49 categories plus Other for visualizing the MTB dataset in Figure 3.Source: Table 4 · p. 15
• Figure 8 shows the distribution of specialty topics in the Visual USMLE dataset, classified using categories from Table 4.Source: Figure 8 · p. 15