PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical Visual Question Answering: A Survey

Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, Zongyuan Ge

arXiv:2111.10056v3cs.CVcs.AI

TL;DR

Medical VQA lacks the mature, general-domain evidence base needed for clinically relevant image-question answering. This survey synthesizes public datasets and published approaches, then identifies clinical challenges and future directions. It covers eight datasets and 45 approach papers while emphasizing dataset scale, image-encoder design, interpretability, generalizability, and safe LLM integration.

  • Problem

    Medical VQA requires specialized investigation because clinical questions, medical knowledge, fine-grained lesions, and expert annotation differ from general-domain VQA.

  • Method

    The survey reviews eight public datasets and 45 medical VQA approach papers, organizing their data sources, task designs, methods, techniques, and clinical challenges.

  • Results

    The review finds that medical datasets are generally smaller and more synthetic than general VQA datasets, while research emphasizes pretrained image encoders and task-oriented techniques.

  • Takeaways & Limitations

    The field’s supported research priorities include broader question coverage, medical feature integration, evidence verification, modality balance, and clinical workflow integration.

  • Takeaways & Limitations

    Medical VQA evidence is limited by unresolved evidence-annotation choices and costly radiologist annotation, while current datasets mainly cover radiology and pathology.

Abstract

from arXiv · show

Medical Visual Question Answering~(VQA) is a combination of medical artificial intelligence and popular VQA challenges. Given a medical image and a clinically relevant question in natural language, the medical VQA system is expected to predict a plausible and convincing answer. Although the general-domain VQA has been extensively studied, the medical VQA still needs specific investigation and exploration due to its task features. In the first part of this survey, we collect and discuss the publicly available medical VQA datasets up-to-date about the data source, data quantity, and task feature. In the second part, we review the approaches used in medical VQA tasks. We summarize and discuss their techniques, innovations, and potential improvements. In the last part, we analyze some medical-specific challenges for the field and discuss future research directions. Our goal is to provide comprehensive and helpful information for researchers interested in the medical visual question answering field and encourage them to conduct further research in this field.

1. Introduction

Medical VQA applies vision-language question answering to clinical images, with potential roles in decision support, professional efficiency, patient engagement, and consultation. This survey reviews its datasets and methods while identifying challenges for clinically useful systems.

  • Medical VQA combines computer vision and natural language processing to answer free-form questions about medical images.
  • Systems may support clinical decision-making, patient engagement, professional efficiency, second opinions, and online consultation.
  • Medical VQA is technically challenging because expert annotation is costly, lesions require fine-grained analysis, and questions demand medical knowledge.
  • The field expanded after the first medical VQA challenge in 2018, motivating this first comprehensive survey.
  • The survey reviews publicly available datasets and published approaches, then discusses medical-specific challenges and future directions.

2. Datasets and performance metrics

The survey catalogs eight public medical VQA datasets and their construction, question categories, and modalities. It highlights diverse dataset designs, including expert-checked caption-derived QA, clinician-generated radiology questions, and visual dialogue.

  • Datasets: Eight publicly available medical VQA datasets are identified, covering radiology and pathology and summarized by source, quantity, QA creation, and question categories.
  • VQA-Med-2018: VQA-Med-2018 generates QA pairs from captions through rule-based generation followed by two-pass expert review for semantic correctness and clinical relevance.
  • VQA-RAD: VQA-RAD collects free-form and template questions directly from clinicians and manually validates and classifies the resulting QA pairs.
  • VQA-Med-2019: VQA-Med-2019 addresses modality, plane, organ system, and abnormality, combining classification tasks with answer generation.
  • RadVisDial: RadVisDial provides radiology visual dialogue from MIMIC-CXR, including silver-standard synthetic dialogues and a gold-standard dataset.

Positional

The survey compares medical VQA datasets, methods, and challenges across question categories and clinical requirements. It emphasizes limited scale, synthetic QA creation, annotation constraints, and the need for broader, more reliable systems.

  • Dataset discussion: Medical VQA datasets range from 315 to 91,060 images and from 1 to 10 QA pairs per image across chest X-ray, CT, MRI, and pathology.
  • Dataset discussion: Compared with general VQA, medical datasets are smaller, use synthetic QA creation more often, and emphasize image findings rather than object relationships or external knowledge.
  • Dataset discussion: Medical dataset construction is constrained by the absence of a large-scale source like Microsoft COCO and by costly professional annotation.
  • Future directions: Only radiology and pathology are currently represented substantially, leaving ophthalmology and dermatology as potential areas for future VQA datasets.

3. Methods

The survey reviews medical VQA approaches through literature-search strategies, framework patterns, encoders, and data-related design constraints. Most approaches follow joint embedding, while some exploit restricted question types by omitting language features.

  • Literature Review: The review identified 45 published papers describing 46 medical VQA approaches, using competition overviews and citation searches.The collection included 32 competition work notes and 13 conference or journal papers.
  • Mainstream Framework: 39 of 46 approaches use a joint-embedding framework with image encoding, question encoding, feature fusion, and answer prediction.The framework commonly uses CNN image encoders and neural classifiers or recurrent language generators.
  • Alternative Framework: Seven approaches omit question features because their datasets restrict questions to abnormality-focused Yes/No or What types.These approaches achieved outstanding performance in VQA-Med-2020 and VQA-Med-2021 according to the survey.
  • Architecture Diversity: The reviewed medical VQA architectures are less diverse than general-domain VQA, with no compositional models adopted so far.The survey identifies other general-domain frameworks as potential directions for medical VQA.
  • Image Encoder: 29 of 46 teams directly used ImageNet-pretrained image encoders despite domain mismatch, because medical datasets contain few images and labels.The survey describes this practice as non-reasonable but workable under limited medical pretraining data.
  • Image Encoder: Medical VQA image encoders increasingly explore CheXpert pretraining, extra datasets, contrastive learning, multi-task pretraining, and meta-learning.MEVF combines MAML and CDAE to initialize image-encoder weights under limited data.
  • Image Encoder: Auxiliary data for image pretraining includes captioning, unlabeled-image, and segmentation datasets, while reviewed encoders remain CNN classification models.The survey notes that reliance on classification encoders restricts detection-based applications.

3.3. Language Encoder

Medical VQA language encoders range from recurrent networks and template matching to Transformer-based models. The survey reports a shift toward BERT-family encoders while noting that lightweight processing can suit restricted tasks.

  • Encoder Distribution: LSTM appears in 18 of 46 approaches, Bi-LSTM in 5 of 46, GRU in 3 of 38, and Transformer models in 12 of 46.Transformer models include BERT and BioBERT.
  • Transformer Adoption: All five top teams in VQA-Med-2019 used BERT, and participating teams increasingly selected BERT or BioBERT over recurrent encoders.The survey associates this shift with Transformer models and BERT pretraining despite corpus differences.
  • Lightweight Encoding: Seven teams used keyword or template matching instead of deep learning, suggesting light language encoders can be practical when question categories are limited.The survey attributes this choice to the effectiveness of matching in their tasks.

3.4. Fusion Algorithm

Fusion combines visual and language features through attention and pooling mechanisms. Across the reviewed approaches, attention is common, direct concatenation is widespread but average-performing, and specialized methods report improvements.

  • Fusion Overview: The fusion stage models relationships between visual and language features and commonly uses attention mechanisms or pooling modules.It is described as the core component of VQA methods.
  • Attention Mechanisms: 23 of 46 approaches apply attention mechanisms during fusion, often using question features to select answer-related image regions.Stacked Attention Networks query image regions multiple times to infer answers progressively.
  • Multimodal Pooling: Direct concatenation is used by 10 of 46 approaches and shows average performance, while attention-based pooling is used by 8 of 46.The winners of VQA-Med-2018 and VQA-Med-2019 used attention with MFB pooling.
  • Multimodal Pooling: Multi-modal pooling methods reduce the computational cost of high-dimensional outer products through techniques such as MCB, MFB, and MFH.MCB embeds image and text features into higher-dimensional vectors and performs efficient Fourier-space convolution with multiplications.
  • Innovative Fusion: QC-MLB and MedFuseNet are the only two reviewed works proposing innovative fusion algorithms, and both show performance improvement.QC-MLB uses multiglimpse attention, while MedFuseNet applies image attention and image-question co-attention.

3.5. Answering Component

Medical VQA systems predominantly use classification outputs, while generation or switching strategies are selected according to the answer-length distribution and answer-space size.

  • Output Strategy: 33 of 46 approaches use classification, 8 use generation, and several combine both through a switching strategy.Classification is advantageous for small answer spaces but becomes difficult as the answer space grows.

3.6. Other Techniques

Medical VQA methods also use task-specific strategies and architectural techniques beyond basic components, trading potential performance gains against allocation and generalization risks.

  • Task-specific strategies: Sub-task strategies divide VQA into question- or modality-specific branches selected by an additional classification module.They are especially used when questions have distinct categories or modalities.
  • Task-specific strategies: These multiple-model approaches may improve single-model effectiveness but risk inappropriate model allocation from classification errors.
  • Other techniques: Global Average Pooling replaces final fully connected layers with averaged feature maps to produce better image representations.
  • Other techniques: Other techniques include Embedding-based Topic Model, Question-Conditioned Reasoning, and Image Size encoder.

3.7. Performance Comparison

Performance patterns differ across medical VQA datasets, but stronger image encoding and attention-based fusion emerge as recurring features of high-performing approaches.

  • Dataset-specific patterns: State-of-the-art approaches rely on different factors across datasets, including generative output, language encoders, image-encoder ensembles, and image pre-training.The comparison identifies image-encoder enhancement as essential for state-of-the-art performance in most medical VQA datasets.
  • Fusion strategies: 0.576 average accuracy was achieved by attention-based fusion approaches in VQA-Med-2019, compared with 0.522 for approaches without attention.
  • Fusion strategies: Attention-based fusion approaches averagely outperform approaches without attention over most datasets.

3.8. Overall Discussion

The survey finds rapid methodological diversity in medical VQA, with image encoding prioritized in technical research, while cross-dataset generalizability and interpretability remain limited.

  • Overall discussion: 45 medical VQA approach papers were reviewed, comprising 32 challenge work notes and 13 conference or journal papers.Challenge work notes commonly apply pretrained deep-learning components, whereas technical papers explore further solutions.
  • Overall discussion: Technical papers focus more on image encoders, using meta-learning, contrastive learning, pre-training, and auxiliary data to enhance system components.
  • Overall discussion: Acquiring a generalized image encoder is identified as a high-priority and specific task in medical VQA research.
  • Generalizability: Only 5 of 13 technical papers evaluated approaches on multiple datasets, and only one evaluated both medical and general VQA datasets.The survey proposes multiple-dataset evaluation as a future standard.
  • Interpretability: Five of 13 technical papers visualized models, but none used SLAKE annotations to evaluate visualization despite available segmentation and bounding-box annotations.

4. Medical VQA v.s. General VQA

Medical and general VQA share visual-language goals but differ in applications, objectives, datasets, methods, domain knowledge, and evaluation requirements.

  • Application: Medical VQA targets clinical decision support, telemedicine, and patient empowerment, whereas general VQA also supports search, virtual assistants, navigation, and robotics.
  • Objective: Medical VQA is designed to assist diagnosis, treatment, and decision-making through medical image understanding and abnormality localization.
  • Datasets: Medical VQA datasets are smaller and less diverse in question categories because data sources and expert annotation are limited.
  • Method: Medical VQA method development is hindered by dataset complexity, while general VQA methods span multi-task learning, logical reasoning, and environmental interaction.
  • Domain knowledge: Medical VQA requires specialized medical concepts, terminology, and context to interpret questions and images.
  • Evaluation: Medical VQA evaluation emphasizes highly accurate and reliable answers, with metrics potentially focusing on medical relevance, precision, and recall.

5. Challenge and Future Works

The survey identifies medical VQA challenges spanning question diversity, medical knowledge, evidence verification, generalizability, multimodal inputs, interpretability, and clinical integration. It proposes expanded data sources, specialized evaluation, external knowledge, multi-image modeling, and workflow-focused research.

  • 5.1. Question Diversity: Medical VQA questions vary across modality, plane, organ system, abnormality, presence, positional reasoning, color, size, attributes, and counting.
  • 5.1. Question Diversity: Future datasets should reflect practical requirements by using textbooks, real clinical conversations, and questions extending beyond visible image content.
  • 5.1. Question Diversity: Medical VQA requires domain knowledge about anatomy, procedures, diseases, imaging modalities, treatments, and clinical practice.
  • 5.1. Question Diversity: Imbalanced question categories can mislead overall evaluation, motivating category-specific, weighted, and language-based metrics.
  • 5.2. Additional Inputs: Future systems should incorporate EHR data and support multiple images because clinical decisions may depend on structured variables, scan planes, and sequential slices.
  • 5.3. Interpretability: Interpretability remains underdeveloped: few medical VQA works address it, and quantified benchmarks for evidence alignment are still needed.
  • 5.3. Evidence Verification: Evidence verification must accommodate text-based explanations and expensive expert annotation, while existing annotated medical datasets may provide useful terminology.
  • 5.4. Generalizability: Generalizability is threatened by distribution shifts involving patients and imaging devices, while language-encoder and cross-dataset shifts remain understudied.

6. Conclusion

The survey reviews medical VQA datasets and approaches, then organizes the field’s main challenges and future research directions. Its coverage emphasizes broader questioning, medical features, evidence verification, modality bias, and clinical workflow benefit.

  • The survey covers 8 medical VQA datasets and 45 papers on medical VQA approaches.
  • It identifies challenges involving comprehensive question categories, medical features, answer evidence, modality bias, and workflow benefits.
  • Future work should examine real-world implementation, conversations between medical professionals and non-professionals, and meaningful additional data.
Loading 2111.10056v3…