PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review

Iryna Hartsock, Ghulam Rasool

arXiv:2403.02469v2cs.CVcs.LG

TL;DR

Medical VLM research must integrate visual and textual healthcare data while addressing the specialized requirements of report generation and VQA. This review synthesizes datasets, architectures, training strategies, evaluation metrics, and recent models, concluding that clinical validation, collaboration, and responsible deployment remain essential.

  • Problem

    Healthcare data are multimodal, but medical VLM development is constrained by limited diverse and representative datasets for complex clinical scenarios.

  • Method

    The review analyzes medical VLM architectures, pre-training and fine-tuning strategies, datasets, evaluation metrics, and models for report generation and VQA.

  • Results

    The review covers 17 publicly available vision-language datasets and 15 recent medical VLMs, including 14 publicly available models.

  • Takeaways & Limitations

    Clinical validation and adoption require collaboration between medical experts and AI/ML researchers to align models with clinical needs and ethical deployment.

  • Takeaways & Limitations

    Limited ML-ready diverse and representative medical datasets restrict comprehensive VLM training for diverse and rare clinical scenarios.

Abstract

from arXiv · show

Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.

1 Introduction

Medical AI increasingly requires multimodal learning because healthcare data combine text, images, and tabular information. This review focuses on medical VLMs for report generation and visual question answering, covering datasets, architectures, training, and evaluation.

  • Healthcare data span clinical text, medical images, and tabular information, motivating models that integrate multiple modalities.
  • VLMs analyze visual and textual data to support more holistic understanding of patient information and clinical tasks.
  • This review targets recent medical VLM advances for medical report generation and visual question answering.
  • The review examines 17 publicly available vision-language datasets, evaluation metrics, and 15 recent medical VLMs, including 14 publicly available models.

2 Machine Learning (ML) - A Brief Review

This section reviews foundational machine-learning concepts underlying medical VLMs, including neural networks, NLP, CV, tokenization, embeddings, recurrent models, and Transformers. It also introduces CNNs and ViTs as visual-processing architectures.

  • Neural Networks (NNs): Neural networks use interconnected layers, weighted connections, activation functions, and loss minimization to learn mappings from inputs to outputs.
  • Neural Networks (NNs): Backpropagation computes loss gradients with respect to model weights so optimization algorithms can update learnable parameters.
  • Natural Language Processing (NLP): NLP analyzes, processes, and generates human language through tasks such as named entity recognition and text summarization.
  • Tokenization: Tokenization splits text into smaller units, while WordPiece and BPE construct subword vocabularies using different merging strategies.
  • Token Embeddings: Token embeddings convert tokens into numerical vectors that capture semantic relationships; biomedical embeddings adapt this representation to clinical terminology.
  • Recurrent Neural Networks: RNNs model sequential and temporal dependencies by feeding layer outputs back into inputs and retaining information in hidden states.
  • Transformers: Transformers use encoder and decoder components to process embedded, positionally encoded tokens, but long sequences remain challenging.
  • Computer Vision: CNNs extract local visual patterns through convolution, whereas ViTs represent images as patch sequences with positional encodings and a global-information token.

3 Vision-Language Models (VLMs)

Multimodal learning is motivated by real-world tasks that require combining heterogeneous data sources. In cancer care, fusing imaging, genomic, pathology, and clinical information may support several stages of care.

  • Cancer care can combine radiology images, genomic data, digitized histopathology slides, and clinical reports for diagnosis, treatment planning, and surveillance.

3.1 Model Architecture

Medical VLM architectures differ in how they fuse visual and textual information and whether they decode outputs. Single-stream models favor parameter efficiency, while dual-stream and encoder-decoder designs support richer interactions or generation at greater computational cost.

  • VLMs are categorized as single-stream or dual-stream according to how visual and textual modalities are fused.
  • Single-Stream Models: Single-stream models concatenate modality features early and process them in one module, sharing parameters across modalities for efficiency.
  • Dual-Stream Models: Dual-stream models encode visual and textual inputs separately before attention-based fusion, enabling complex cross-modal dependencies.
  • Dual-Stream Models: Dual-stream architectures generally require more computation than single-stream architectures because they use separate processing streams and late fusion.
  • VLMs are also classified as encoder-only or encoder-decoder models depending on whether learned cross-modal representations are passed to a decoder.
  • Encoder-only models support efficient representation learning but may be limited for intricate outputs, whereas encoder-decoder models generate complex outputs with higher computational load.

3.2 Model Training

Medical VLM training typically combines broad image-text pre-training with task-specific fine-tuning. Strategies include supervised, human-feedback, instruction-based, curriculum, and self-supervised learning, using objectives such as contrastive, masked, and image-text matching tasks.

  • Transfer Learning: Transfer learning adapts pre-trained VLM parameters to downstream tasks using smaller task-specific datasets.The original architecture may be modified with task-specific layers while retaining pre-training knowledge.
  • Curriculum Learning: Curriculum learning presents training examples in an ordered progression from simpler to more complex cases.LLaVa-Med applies this strategy during training.
  • Self-Supervised Learning: Self-supervised learning lets VLMs generate labels from data, reducing dependence on costly large-scale annotation.It formulates tasks from inherent data structure to learn representations across modalities.
  • Pre-Training: Pre-training pairs images or videos with textual descriptions and uses multiple objectives to learn representations for downstream tasks.Common objectives include contrastive learning, masked language and image modeling, and image-text matching.
  • Fine-Tuning: Combining contrastive, masked-prediction, and task-specific losses provides versatile multimodal representations.After pre-training, VLMs are commonly fine-tuned on smaller datasets using supervised, human-feedback, or instruction-based approaches.

3.3 Parameter-Efficient Fine-Tuning (PEFT)

Parameter-efficient fine-tuning adapts VLMs by freezing most pre-trained parameters and adding or updating task-specific components. The review describes adapters, LoRA, prompt tuning, and prefix token tuning as representative approaches.

  • PEFT: PEFT freezes the model’s parameters while updating newly added task-specific layers or adapters.It is particularly useful when labeled data for the target task is limited.
  • LoRA: LoRA adapts pre-trained models by fine-tuning two smaller low-rank matrices instead of the full weight matrix.Medical VLMs including Visual Med-Alpaca use LoRA to support efficient task adaptation.
  • Prompt Tuning: Prompt tuning learns continuous vector representations that serve as input hints for task adaptation.Qwen-VL and InstructBLIP are examples of VLMs using prompt tuning.
  • Prefix Token Tuning: Prefix token tuning adds independently trainable task-specific vectors to initial input tokens while keeping the remaining model parameters frozen.VL-T5 uses different prefixes for questions from different datasets.

3.4 In-Context Learning

In-context learning adapts VLM behavior through context rather than parameter updates. Prompt engineering can guide report generation, while retrieval-augmented generation adds external knowledge during retrieval and generation.

  • In-Context Learning: In-context learning uses VLMs for inference while keeping model parameters and any PEFT or LoRA adapters frozen.Adaptation occurs through the supplied context rather than fine-tuning.
  • Prompt Engineering: Prompt engineering uses task-specific instructions or examples to tailor a pre-trained model’s output.RAMM can be prompted to generate a radiology report for a specific image.
  • Retrieval-Augmented Generation: Retrieval-augmented generation combines retrieval and generation prompts to incorporate external knowledge into multimodal tasks.Medical VLMs use RAG for tasks such as VQA, including RAMM.

3.5 Downstream Tasks

Medical VLMs support downstream tasks requiring joint understanding of visual and textual data, especially report generation and visual question answering. These tasks range from producing comprehensive diagnostic summaries to answering open-ended or closed-ended questions about medical images.

  • Overview: Multimodal downstream tasks use knowledge acquired during VLM pre-training for applications requiring joint visual-textual understanding.
  • Report Generation: Report generation creates comprehensive summaries of visual medical data, such as abnormalities and their locations in radiology images.Automating reports can reduce the workload associated with report writing.
  • Visual Question Answering: VQA requires models to interpret medical images or videos together with questions and produce relevant answers.Questions may concern the presence of medical conditions or abnormalities.
  • Visual Question Answering: Medical VQA includes open-ended responses and closed-ended formats such as multiple choice, yes/no, or numeric ratings.The task may be formulated as classification, generation, or both.
  • Other Downstream Tasks: Other vision-language tasks include referring expression comprehension and visual commonsense reasoning.These tasks involve locating image regions or answering and justifying image-based questions.

4.1 Medical Datasets for VLMs

Medical VLM development relies on diverse image-text and question-answer datasets spanning radiology, pathology, anatomy, and multimodal clinical concepts. The review catalogs datasets used for pre-training, fine-tuning, and evaluating report generation and VQA.

  • Medical VLM datasets contain image-caption or image-question-answer pairs used for pre-training, fine-tuning, and evaluating report generation and VQA.The review presents these datasets comparatively in Table 1.
  • Radiology datasets: ROCO contains 81,825 radiology images and 6,127 out-of-class images, each accompanied by captions and UMLS-related metadata.Its radiology images include CT, ultrasound, X-ray, fluoroscopy, PET, mammography, MRI, angiography, and PET-CT.
  • Radiology datasets: MIMIC-CXR provides 377,110 chest X-rays paired with 227,835 free-text radiology reports from de-identified clinical studies.MIMIC-CXR-JPG converts the images to JPG and associates 227,827 reports with pathology labels derived using NegBio and CheXpert.
  • Specialized supervision: Several datasets target specialized supervision, including 38,003 image-NLE pairs in MIMIC-NLE and 1,162 radiologist-verified image-text bounding-box pairs in MS-CXR.MIMIC-NLE also contains 44,935 image-diagnosis-NLE triplets.
  • VQA datasets: VQA resources include SLAKE, VQA-RAD, PathVQA, VQA-Med 2019, and VQA-Med 2020, covering bilingual anatomy questions, radiology, pathology, modalities, and abnormalities.SLAKE contains 14,028 QA pairs, PathVQA 32,799, VQA-Med 2019 15,292, and VQA-Med 2020 5,000.

4.2 VLM Evaluation Metrics

Medical VLM evaluation combines benchmark datasets with language-overlap, semantic, clinical, classification, and human-assessment metrics. Report-generation metrics assess textual agreement or uncertainty, while clinical and human measures address diagnostic content and error severity.

  • Evaluation begins by selecting benchmark datasets and metrics tailored to the specific vision-language task.For report generation, MIMIC-CXR and Open-I are prevalent benchmark datasets.
  • Text-overlap metrics: BLEU and ROUGE quantify overlap between generated and reference text using n-gram precision, clipped counts, brevity adjustment, or longest common subsequences.BLEU scores range from 0 to 1, with higher scores indicating greater agreement; ROUGE scores also range from 0 to 1, with higher similarity preferred.
  • Text-overlap metrics: ROUGE-L measures the longest common subsequence between generated text Y and reference text X using precision, recall, and β.Its overall score averages instance-level scores.
  • Semantic metrics: METEOR incorporates overlapping 1-grams, fluency, meaning, and a penalty based on aligned chunks; scores range from 0 to 1.Higher METEOR values indicate better alignment with the reference text.
  • Semantic metrics: Perplexity measures average uncertainty in predicting each word, with values from 1 to +∞ and lower values indicating more accurate, confident language modeling.BERTScore instead uses token embedding similarities between generated and reference text.
  • Clinical and human evaluation: Clinical evaluation includes RadGraph F1, CheXpert or CheXbert label-based accuracy, precision, recall, and F1, alongside expert human scoring of report errors.RadGraph F1 compares clinical entities and relations represented as graph nodes and edges.

4.3 Medical Models

The review surveys publicly available medical VLMs for VQA and report generation, emphasizing their encoders, multimodal fusion, pre-training objectives, and task adaptations. Architectures range from CLIP-style contrastive models to retrieval-based and instruction-tuned systems.

  • The reviewed medical VLMs are organized chronologically and focus mainly on recently introduced open-source or publicly available models for VQA and report generation.Their comparative overview is provided in Table 2.
  • Report-generation models: MedViLL combines ImageNet-trained ResNet-50 visual features with BERT-based textual features and positional information for cross-modal report generation.On Open-I, it achieves BLEU-4 0.049, perplexity 5.637, accuracy 73.4%, precision 0.512, recall 0.594, and F1 0.550.
  • Contrastive models: PubMedCLIP and BiomedCLIP use CLIP-style contrastive learning with biomedical image-caption corpora and domain-specific text encoders.PubMedCLIP uses ROCO, whereas BiomedCLIP uses the 15 M-pair PMC-15 dataset and adapts through METER.
  • VQA models: RepsNet, RAMM, and related systems combine visual and textual encoders with multimodal fusion for VQA and automated medical report interpretation.RAMM uses Swin Transformer, PubMedBERT, a six-layer multimodal Transformer, and pre-training on MIMIC-CXR, ROCO, and PMCPM.
  • Generative models: UniXGen unifies report generation with view-specific X-ray generation by tokenizing chest X-rays through VQGAN.VQGAN combines encoding, vector quantization through learnable codebook vectors, and decoding.
  • Instruction-tuned models: Visual Med-Alpaca routes image inputs through specialized modules, converts visual information into text prompts, and fine-tunes LLaMA-7B on 54,000 curated medical QA pairs.Supported modules include DePlot for plots and charts and Med-GIT for radiology images.

4.3.9 Contrastive X-ray-Report Pair Retrieval based Generation (CXR-RePaiR-Gen)

Medical VLMs in this group support retrieval-augmented report generation, conversational radiology, instruction-following, surgical VQA, and multimodal few-shot learning. Their designs combine domain-specific encoders with retrieval, prompting, co-attention, masking, or task-specific fine-tuning.

  • CXR-RePaiR-Gen: CXR-RePaiR-Gen retrieves impressions from CXR-PRO using image-text similarity and supplies them as context to an LLM for radiology report generation.The model uses ALBEF with ViT-B/16 and a six-layer BERT text encoder, and is designed to mitigate hallucinated references.
  • LLaVa-Med: LLaVa-Med adapts LLaVA to biomedical instruction following by projecting CLIP or BiomedCLIP visual features into tokens for an LLaMA-7B or Vicuna language model.Its curriculum uses 600,000 PMC-15 image-caption pairs transformed into instruction-following data.
  • Conversational models: XrayGPT transforms MedCLIP visual features into tokens and uses assistant and doctor queries to guide conversational chest-radiograph analysis.The assistant query defines the system as a healthcare virtual assistant, while the doctor query guides response generation.
  • Specialized VQA: CAT-ViL DeiT targets surgical VQA with answer localization through a ResNet-18, BERT-based text encoder, and co-attention fusion module.Its design emphasizes interaction between visual and textual features in surgical scenarios.
  • Multimodal pre-training: MUMC aligns visual and textual features with cross-attention and pre-trains using contrastive learning, masked language modeling, image-text matching, and masking 25% of image patches.It is fine-tuned and evaluated on VQA-RAD, SLAKE, and PathVQA.
  • Few-shot learning: Med-Flamingo adapts Flamingo for medical few-shot learning using textbook segments containing text and images plus PMC-OA image-caption pairs.Its newly curated MTB dataset comprises 4,721 multimodal textbook segments.
  • Conversational radiology: RaDialog combines BioViL-T visual features with BERT alignment and instruction tuning across eight radiology tasks, achieving F1 0.397 on binary CheXpert QA and 0.403 on complete CheXpert QA.The tasks include report generation, explanation, QA, summarization, correction, and reformulation.

5 Challenges and Potential Future Directions

Medical VLMs face challenges involving data scarcity, evaluation, hallucinations, knowledge retention, and clinical adoption. Proposed directions include retrieval, federated learning, specialized metrics, modality alignment, continual learning, and collaboration between clinicians and AI researchers.

  • Data availability and privacy: Limited diverse and representative medical datasets restrict comprehensive VLM training and the ability to handle rare clinical scenarios.
  • Data availability and privacy: Retrieval-augmented approaches could expand model context with relevant information and improve robustness to new and unforeseen medical cases.
  • Data availability and privacy: Federated learning could address data scarcity while protecting privacy by training across institutions without sharing patient data.
  • Evaluation: Traditional metrics may miss clinically relevant distinctions in medical reports and open-ended queries, motivating specialized evaluation metrics for medical RG and VQA.
  • Reliability and knowledge retention: Hallucinations undermine reliability because outputs may not be grounded in images or established knowledge, with potential diagnostic and treatment consequences.
  • Reliability and knowledge retention: Catastrophic forgetting can arise from excessive fine-tuning, while continual learning and adapters may help retain knowledge across clinical tasks.
  • Clinical validation and adoption: Clinical validation and adoption require collaboration between medical experts and AI/ML researchers, alongside trust, clinical alignment, and ethical deployment.
Loading 2403.02469v2…