Source-linked AI summary
Overcoming Data Limitation in Medical Visual Question Answering
Binh D. Nguyen, Thanh-Toan Do, Binh X. Nguyen, Tuong Do, Erman Tjiputra, Quang D. Tran
TL;DR
Medical VQA lacks the large labeled datasets required by traditional VQA methods. The paper combines CDAE and MAML for image feature extraction, achieving state-of-the-art results on VQA-RAD for both close-ended and open-ended questions.
Problem
Medical VQA lacks large-scale labeled training data, while VQA-RAD contains only 315 images.
Method
The framework combines CDAE to leverage unlabeled medical images with MAML to learn meta-weights that quickly adapt to VQA.
Results
The proposed framework achieves state-of-the-art results on VQA-RAD for both close-ended and open-ended questions.
Takeaways & Limitations
Combining pretrained MAML and CDAE weights for image feature extraction addresses medical VQA's limited labeled-data setting.
Abstract
from arXiv · showhide
Traditional approaches for Visual Question Answering (VQA) require large amount of labeled data for training. Unfortunately, such large scale data is usually not available for medical domain. In this paper, we propose a novel medical VQA framework that overcomes the labeled data limitation. The proposed framework explores the use of the unsupervised Denoising Auto-Encoder (DAE) and the supervised Meta-Learning. The advantage of DAE is to leverage the large amount of unlabeled images while the advantage of Meta-Learning is to learn meta-weights that quickly adapt to VQA problem with limited labeled data. By leveraging the advantages of these techniques, it allows the proposed framework to be efficiently trained using a small labeled training set. The experimental results show that our proposed method significantly outperforms the state-of-the-art medical VQA.
1 Introduction
Medical VQA is constrained by scarce, noisy labeled data, motivating a framework that combines unlabeled medical images with meta-learning for rapid adaptation.
- Medical VQA aims to answer questions consistently with the visual content of medical images.
- VQA-RAD contains only 315 manually constructed images, limiting direct application of powerful deep learning models.
- Large unlabeled medical-image collections may provide domain-matched representations that adapt more readily than ImageNet-pretrained weights.
- The framework uses MAML and CDAE to initialize image-feature-extraction weights for medical VQA.
2 Literature Review
Prior medical VQA methods commonly adapt general-domain models, while this work motivates meta-learning and denoising auto-encoding to use unlabeled images and improve robustness to noise.
- Existing medical VQA approaches commonly apply general VQA attention mechanisms and ImageNet-pretrained VGG or ResNet models.
- MAML learns a meta-model whose weights can be quickly adapted to new tasks with few training images.
- Auto-encoders extract high-level features without labels, enabling use of unlabeled medical images.
- CDAE is proposed to exploit unlabeled datasets while making feature extraction robust to noise in medical images.
3 Methodology
The framework initializes medical VQA image features with MAML and CDAE, then fine-tunes the complete model end-to-end. It combines task-adaptive initialization, denoising reconstruction, attention-based multimodal fusion, and VQA classification.
- Framework overview: MAML and CDAE pretrained weights initialize the MEVF image feature extraction component before end-to-end VQA fine-tuning.The figure identifies MEVF as the image feature extraction component and states that its weights are initialized by MAML and CDAE.
- MAML initialization: MAML learns meta-model weights by adapting to k-shot n-way tasks and updating shared parameters across a meta-batch.Each task is split into training and validation sets, and adapted parameters are computed before the shared meta-model is updated with SGD.
- CDAE initialization: CDAE encodes noisy medical images into latent representations, decodes them into reconstructions, and minimizes reconstruction error against the original images.Gaussian noise is added to the original image, while convolutional and deconvolutional layers form the encoder-decoder architecture.
- VQA architecture: Questions are encoded by a 1024-D LSTM, while MEVF produces two 64-D visual vectors concatenated into an 128-D enhanced image feature.Each question is trimmed or padded to 12 words, and each word uses a 600-D embedding combining GloVe with a VQA-RAD augmenting embedding.
- VQA architecture: An attention mechanism combines image and question features into a joint representation for a multi-class answer classifier, trained with VQA and reconstruction losses.The full model is fine-tuned end-to-end on VQA-RAD using a multi-task objective that incorporates CDAE reconstruction.
4 Experiments
The experiments evaluate image-feature extraction and initialization strategies on VQA-RAD, including training from scratch, pretraining followed by finetuning, and comparisons with ImageNet-pretrained baselines. Combining MAML and CDAE with finetuning achieves the strongest reported performance, including substantial gains over the best baseline.
- Dataset: VQA-RAD contains 315 images and 3,515 questions, with 451 questions reserved for testing.Questions span 11 categories and include both close-ended and open-ended formats.
- Experimental setup: The study compares MAML, CDAE, their combination MEVF, ImageNet-pretrained VGG, and training either from scratch or after pretraining and finetuning.The comparisons focus on alternative image-feature extraction methods while retaining the VQA framework settings described for each experiment.
- Ablation Study: Pretraining followed by finetuning significantly improves MAML and CDAE performance over training from scratch using only VQA-RAD.This comparison directly tests whether initialization from the proposed pretraining procedures helps under limited labeled data.
- Ablation Study: MEVF, which combines pretrained MAML and CDAE weights before finetuning, delivers the best performance among the tested image-feature extraction methods.The result is reported for VQA-RAD open-ended and close-ended test questions.
- Ablation Study: Close-ended question accuracy exceeds open-ended accuracy for MAML, CDAE, and MEVF, while finetuning gains are more pronounced for open-ended questions.The paper attributes this pattern to open-ended questions requiring detailed, longer answers and more information from the input images.
- Comparison with the state of the art: The proposed framework significantly outperforms baseline SAN and BAN frameworks, with BAN attention improving over the best BAN baseline by 16.3% on open-ended and 8.6% on close-ended VQA.Baseline image features use VGG or ResNet pretrained on ImageNet and finetuned on VQA-RAD.
5 Conclusion
The paper concludes that combining MAML and CDAE for image-feature extraction addresses limited labeled data in medical VQA. CDAE uses large-scale unlabeled images, while MAML learns weights that adapt quickly to VQA, yielding state-of-the-art VQA-RAD results.
- Conclusion: The framework combines meta-learning MAML and denoising auto-encoder CDAE for medical VQA image-feature extraction.CDAE leverages unlabeled images, while MAML learns meta-weights quickly adaptable to VQA.
- Conclusion: The method establishes new state-of-the-art results on VQA-RAD for both close-ended and open-ended questions.