Source-linked AI summary
Simple Baseline for Visual Question Answering
Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus
TL;DR
Visual question answering must answer specific questions about images, sometimes requiring knowledge or reasoning beyond image content. This paper carefully implements iBOWIMG, a bag-of-words and CNN-feature baseline that concatenates both modalities for answer classification. On COCO VQA, it achieves comparable performance to several recurrent approaches while enabling interpretable analysis, though comparisons exclude methods using non-standard dataset setups.
Problem
Visual question answering requires answering specific questions that can involve broader knowledge and reasoning than image captioning.
Method
iBOWIMG uses naive bag-of-words question features and GoogLeNet image features, concatenates them, and predicts answer classes with softmax.
Results
iBOWIMG achieves comparable performance to several recurrent neural network-based approaches on the COCO VQA dataset.
Takeaways & Limitations
The baseline captures correlations between informative question words, image contents, and answers while supporting interpretable analysis of those contributions.
Takeaways & Limitations
Performance comparisons exclude methods evaluated with non-standard VQA dataset versions or train-test splits.
Abstract
from arXiv · showhide
We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to predict the answer. When evaluated on the challenging VQA dataset [2], it shows comparable performance to many recent approaches using recurrent neural networks. To explore the strength and weakness of the trained model, we also provide an interactive web demo and open-source code. .
1 Introduction
Visual question answering extends image understanding by requiring answers to specific questions that may demand visual content, knowledge, or reasoning. The paper revisits a simple bag-of-words and image-feature baseline, showing that careful implementation can rival recurrent approaches while remaining interpretable.
- Motivation: Visual QA requires answering specific image questions, including questions involving knowledge or reasoning beyond visible content.Unlike captioning, which can select easy relevant descriptions, visual QA must answer the particular question posed.
- Prior approaches: Recent VQA models commonly combine recurrent question features with CNN image features, with some adding visual attention mechanisms.These approaches typically concatenate LSTM-derived question features and visual features before classification.
- Prior approaches: The earlier BOWIMG baseline outperformed LSTM models on a synthesized dataset but performed worse on the larger COCO VQA dataset.This motivated a careful reimplementation under the newer benchmark setting.
- Contribution: The iBOWIMG implementation concatenates question and image features, applies softmax classification, and provides an interactive demo and open-source code.The framework is intentionally simple, while the demo is intended to expose model strengths and weaknesses.
2 iBOWIMG for Visual Question Answering
The iBOWIMG model treats visual question answering as answer classification using bag-of-words question features and GoogLeNet image features. It embeds the question, concatenates both modalities, and predicts the answer class with softmax.
- Task formulation: Visual QA is commonly simplified to classification, with each distinct training-set answer treated as a final class.The general pipeline combines question-word features and image features before predicting an answer class.
- Model: iBOWIMG uses naive bag-of-words for text and deep GoogLeNet features for the image.The question is converted to a one-hot vector and transformed through a word-embedding layer.
- Model: The embedded question feature and CNN image feature are concatenated and passed to softmax as a multi-class logistic regression model.The paper describes the implementation as requiring no more than 10 lines of Torch code.
3 Experiments
The experiments evaluate iBOWIMG on COCO VQA under standard benchmark protocols and show that this simple baseline performs comparably to much more complex recurrent models. Further analyses expose how question words and image features contribute to predictions and identify informative words and regions.
- Dataset and evaluation: 248,349 train2014 and 121,512 val2014 question-answer pairs cover 123,287 images in the COCO VQA training data.Each question has ten annotated answers, which are reduced by majority voting to one training answer.
- Dataset and evaluation: The final model trains on train2014 plus all val2014 images after tuning on a 70%/30% split of val2014 images.Questions sharing an image remain in the same split to reduce potential overfitting.
- Dataset and evaluation: Test-dev supports unlimited submissions for debugging and validation, whereas test-standard is used for model comparison with limited submissions.The paper reports results on both sets, with Table 1 covering test-dev and Table 2 covering test-standard.
- Benchmark performance: iBOWIMG performs comparably to recent recurrent neural network approaches, with DPPnet about 1.5% better.The compared methods, aside from the provided IMG, BOW, and BOWIMG baselines, use deep or recurrent neural networks.
- Training details: A higher learning rate and separate weight clipping for word embeddings and softmax improve training, while BOW alone approaches 48% accuracy without image content.The result indicates that the word model is important to baseline accuracy.
- Understanding the Visual QA model: The model decomposes each answer response into question-word and image contributions, allowing predictions based on either information source to be ranked separately.The softmax weights are split into Mw for word features and Mv for visual features, with rw = Mwxw and rv = Mvxv.
- Understanding the Visual QA model: Question words usually dominate predictions, although image features can correct them: “playing baseball” combines 2.01 image points with 8.66 word points.Words-only alternatives were “playing wii” (10.62), “eating” (9.97), and “playing frisbee” (9.24).
- Understanding the Visual QA model: Because the model uses two linear transformations, it can rank individual word importance and use CAM heatmaps to identify informative image regions.Examples link “doing” to “texting,” “eating” to “hot dog,” and highlight cellphone and hot-dog regions.
4 Interactive Visual QA Demo
The iBOWIMG model powers an interactive web demo for real-time visual question answering on a given image.
- The web demo lets people type questions about a given image and receive the system’s most probable answers.The demo is powered by iBOWIMG, with image deep features extracted beforehand.
5 Concluding Remarks
On COCO visual question answering, the simple baseline achieves comparable performance to several recurrent neural network approaches. Its predictions rely on correlations between question words, image contents, and answers, while moving beyond these correlations toward reasoning remains future work.
- The iBOWIMG implementation achieves comparable performance to several recurrent neural network-based approaches on COCO visual question answering.
- The baseline captures correlations between informative question words and answers, and between image contents and answers.
- Moving beyond memorized correlations toward actual reasoning and understanding of the question and image remains a goal for future research.