Source-linked AI summary
MMBERT: Multimodal BERT Pretraining for Improved Medical VQA
Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, CV Jawahar
TL;DR
Medical VQA is constrained by domain-specific images and costly expert annotation, while existing datasets are small. MMBERT learns medical image-text representations through masked language modeling with image features, then fine-tunes for VQA. It achieves state-of-the-art results on VQA-Med 2019 and VQA-RAD and provides attention maps, although its evaluation does not account for plausible differential diagnoses.
Problem
Small medical VQA datasets and costly expert annotation limit supervised learning for diagnostically relevant image-question answering.
Method
MMBERT is pretrained with masked language modeling on medical image-caption pairs using image features, then fine-tuned on medical VQA datasets.
Results
MMBERT achieves state-of-the-art performance on VQA-Med 2019 and VQA-RAD, outperforming previous approaches including ensemble or dedicated models.
Takeaways & Limitations
The model provides attention maps that highlight image regions contributing to predictions, supporting model interpretability.
Takeaways & Limitations
The quantitative evaluation does not account for abnormalities that could also be differential diagnoses for a human expert.
Abstract
from arXiv · showhide
Images in the medical domain are fundamentally different from the general domain images. Consequently, it is infeasible to directly employ general domain Visual Question Answering (VQA) models for the medical domain. Additionally, medical images annotation is a costly and time-consuming process. To overcome these limitations, we propose a solution inspired by self-supervised pretraining of Transformer-style architectures for NLP, Vision and Language tasks. Our method involves learning richer medical image and text semantic representations using Masked Language Modeling (MLM) with image features as the pretext task on a large medical image+caption dataset. The proposed solution achieves new state-of-the-art performance on two VQA datasets for radiology images -- VQA-Med 2019 and VQA-RAD, outperforming even the ensemble models of previous best solutions. Moreover, our solution provides attention maps which help in model interpretability. The code is available at https://github.com/VirajBagal/MMBERT
1. INTRODUCTION AND RELATED WORK
Medical VQA aims to answer diagnostically relevant questions about medical images, but limited, costly annotations motivate self-supervised pretraining. MMBERT pretrains on medical image-caption data before VQA fine-tuning and reports strong results with interpretable attention maps.
- Medical VQA seeks to answer diagnostically relevant natural-language questions about medical images for professionals and patients.
- Small datasets and costly expert annotation make supervised learning difficult in medical VQA, motivating self-supervised pretraining.
- Attention maps show the model attending to grey matter, white matter, and cerebrospinal fluid when predicting “MR-T2 Weighted.”
- MMBERT is pretrained on medical images and captions with masked language modeling, then fine-tuned for VQA.
- Existing methods did not use large multimodal medical datasets to learn better image and text representations.
- MMBERT achieves new state-of-the-art results on VQA-RAD and improves VQA-Med 2019 Accuracy by 5% over the previous state of the art.
2. METHOD
MMBERT addresses medical-domain representation learning with multimodal self-attention and masked language modeling over image-caption pairs. Image and text features are jointly encoded, medical keywords are masked, and the pretrained model is later adapted for VQA.
- Motivation: General-domain pretrained weights and word co-occurrences may transfer poorly to medical images and text, motivating medical representations learned from scratch.
- Self-Attention: Self-attention models intra- and intermodality relationships by weighting value vectors using scaled query-key dot products.
- Self-Attention: Multi-head attention performs parallel self-attentions and concatenates their outputs to represent different subspaces and positions.
- Architecture: MMBERT combines five ResNet152 image features with caption token embeddings in a four-layer BERT-like model with twelve attention heads.
- Pretraining: Masked language modeling predicts masked medical keywords using both caption context and accompanying image features.
- Fine-tuning: During fine-tuning, pretrained weights are adapted to each medical VQA dataset, and token representations are averaged before classification.
3. EXPERIMENTS AND RESULTS
Experiments evaluate MMBERT across VQA-Med 2019 and VQA-RAD, comparing pretrained and category-specific settings with quantitative and qualitative analyses. The results show strong benchmark performance alongside interpretable attention maps, while highlighting a limitation of single-image abnormality evaluation.
- Datasets and settings: VQA-Med 2019 contains four main question categories plus Yes/No, while VQA-RAD includes 315 images and 3,515 questions.VQA-Med 2019 provides 3,200 training images, 500 validation images, and 500 test images; VQA-RAD has 11 question types.
- Datasets and settings: MMBERT experiments compare a general model, category-specific Exclusive models, and non-pretrained category-specific models.Dedicated-category settings first predict the question category using a BERTBASE classifier at inference.
- Quantitative results: MMBERT Exclusive achieves state-of-the-art overall Accuracy and BLEU on VQA-Med 2019, surpassing the three-model CGMVQA ensemble.MMBERT General also outperforms the ensemble on Abnormality and Yes/No, and beats single dedicated CGMVQA models in every category except Modality.
- Quantitative results: 7.2 and 9 points are the Accuracy and BLEU gains from pretraining over MMBERT NP in the reported comparison.The passage reports these gains respectively for Accuracy and BLEU, without specifying the category in the excerpt.
- Qualitative analysis: MMBERT attention maps generally highlight clinically relevant regions, including skull contents for Organ and Yes/No and the longitudinal fissure for Plane questions.The Modality example attends to soft tissue and fluid but misses the Doppler color cue; an Abnormality example predicts modality, organ, and abnormality together.
- Qualitative analysis: Quantitative evaluation does not account for differential diagnoses that may be plausible from a single medical image.Experts often use multiple slices, planes, and other evidence when diagnosing abnormalities.
4. CONCLUSION
MMBERT is pretrained on ROCO with masked language modeling using image features, then fine-tuned for medical VQA. It achieves state-of-the-art results and produces attention maps that identify image regions used for prediction.
- MMBERT is pretrained on the ROCO dataset with masked language modeling using image features for medical VQA.
- MMBERT is fine-tuned on VQA-RAD and VQA-Med 2019, achieving new state-of-the-art results on both datasets.
- Qualitative results show that MMBERT attends to image regions relevant to its predictions.
5. COMPLIANCE WITH ETHICAL STANDARDS
The study used retrospective human-subject data from open-access sources. Ethical approval was not required under the licenses attached to those datasets.
- The research study was conducted retrospectively using human-subject data.
- The data came from open-access sources provided by Pelka et al., Ben Abacha et al., and Lau et al.
- Ethical approval was not required, as confirmed by the licenses attached to the open-access data.