Source-linked AI summary
Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
Vahid Kazemi, Ali Elqursh
TL;DR
Visual question answering seeks to test image understanding through natural-language questions and answers, but prior approaches left room for stronger baselines. This paper uses a simple LSTM–residual-network model with soft attention and reports the best results on VQA 1.0 and VQA 2.0 among the cited prior work.
Problem
VQA evaluates whether models understand image content through targeted natural-language questions, providing a more specific evaluation target than generic image captions.
Method
The model encodes questions with an LSTM, extracts image features with a deep residual network, applies soft attention, and classifies among frequent answers.
Results
The model outperforms previously reported results on VQA 1.0 and VQA 2.0, achieving 64.6% on VQA 1.0 test-standard and 59.67% on VQA 2.0 validation.
Takeaways & Limitations
The results support using this architecturally simple model as a strong VQA baseline and show that training details matter.
Takeaways & Limitations
The classifier considers only the 3000 most frequent answers, covering 92% of validation answers; other answers are ignored during training.
Abstract
from arXiv · showhide
This paper presents a new baseline for visual question answering task. Given an image and a question in natural language, our model produces accurate answers according to the content of the image. Our model, while being architecturally simple and relatively small in terms of trainable parameters, sets a new state of the art on both unbalanced and balanced VQA benchmark. On VQA 1.0 open ended challenge, our model achieves 64.6% accuracy on the test-standard set without using additional data, an improvement of 0.4% over state of the art, and on newly released VQA 2.0, our model scores 59.7% on validation set outperforming best previously reported results by 0.5%. The results presented in this paper are especially interesting because very similar models have been tried before but significantly lower performance were reported. In light of the new results we hope to see more meaningful research on visual question answering in the future.
1. Introduction
VQA tests whether models understand image content by answering targeted natural-language questions. The paper presents a simple architecture that surpasses prior reported results on VQA benchmarks when trained carefully.
- Motivation: VQA asks models to generate natural-language answers from an image and a natural-language question.The task probes understanding of objects, interactions, states, and properties rather than object recognition alone.
- Motivation: Narrow questions make VQA easier to evaluate than image captioning because they require specific answers.The paper argues that generic captions may not reveal what a model understands, whereas targeted questions can force specificity.
- Contribution: A relatively simple architecture, trained carefully, bests state of the art on visual question answering.The authors emphasize that the model is simple compared with recent approaches.
- Contribution: 64.6% accuracy on VQA 1.0 test-standard improves over state of the art by 0.4%.The result is reported without using additional data.
- Contribution: The model encodes questions with LSTMs, computes image features with a deep residual network, applies soft attention, and classifies among frequent answers.Attention produces multiple image-feature glimpses conditioned on the LSTM state.
- Contribution: The paper attributes the result to getting training details right despite using a model similar to previously tried systems.This motivates the paper’s role as a strong baseline for future VQA research.
2. Related work
Related work established CNNs, attention, and LSTMs as important components for visual and language modeling. These ideas had already been applied to VQA, including both similar and more complex attention architectures.
- Neural network foundations: CNNs advanced computer vision by learning deep image representations and outperforming hand-designed features on ImageNet.Subsequent work explored deeper and very deep neural networks.
- Neural network foundations: Attention enables neural networks to extract localized features from input data.The paper identifies attention as one of the few ideas found advantageous across domains.
- Neural network foundations: LSTMs have been widely used across tasks including machine translation and speech recognition.The paper uses LSTMs as part of its VQA model.
- VQA-related work: The proposed model resembles stacked attention networks, yet the paper reports a 5.8% improvement over their VQA 1.0 result.More complex and expensive attention models had also been explored, but their advantage was described as unclear in light of this paper’s results.
3. Method
The method treats VQA as fixed-answer classification, combining ResNet image features with an LSTM question representation and multiple attention glimpses before classification.
- VQA is formulated as predicting the most likely answer from a fixed set given an image and natural-language question.
- 3.1. Image embedding: A pretrained residual CNN computes the image representation, which is a 14 × 14 × 2048 tensor before final pooling.
- 3.2. Question embedding: Tokenized question embeddings are processed by an LSTM, whose final state represents the question.
- 3.3. Stacked attention: Multiple attention distributions use the question state and spatial image features to produce weighted-average image glimpses.
- 3.4. Classifier: The image glimpses and final LSTM state are concatenated and passed through nonlinear classifier layers to produce answer-class probabilities.
- 3.4. Classifier: The final loss averages log-likelihoods over all correct answers.
4. Experiments
The model is evaluated on open-ended VQA using the provided accuracy metric across unbalanced VQA 1.0 and more balanced VQA 2.0, with predefined train, validation, and test splits.
- VQA 1.0 contains 204,721 MS COCO images, 614,163 questions, and 6,141,630 answers.
- Experiments primarily train on the training split and report validation results, with additional train-and-validation training for test-set comparison.
- VQA 2.0 contains 658,111 questions and 6,581,110 answers and is more balanced because each question has two images yielding different answers.
- The evaluation uses the open-ended VQA challenge accuracy metric.
- An answer is considered correct when at least three annotators agree, and accuracy averages scores over all 10 choose 9 ground-truth subsets.
5. Results
The results show that careful training and architectural choices substantially improve this simple VQA baseline, which outperforms prior results on both VQA 1.0 and VQA 2.0.
- 5.1. Baselines: Applying l2 normalization, dropout, and soft-attention significantly improves model accuracy on VQA 1.0 validation.The authors also report that l2 normalization improves learning dynamics and reduces training time, while dropout helps avoid over-fitting.
- 5.1. Baselines: Different word embedding sizes and LSTM configurations were not major factors, although a larger embedding with a smaller LSTM seemed best.The explored configurations included embedding sizes of 100, 300, and 500, plus multiple LSTM sizes and depths.
- 5.1. Baselines: Sampling loss produces significantly worse results and longer training time than averaging the log-likelihood of correct answers.The authors contrast their finding with previous work that used sampling loss.
- 5.1. Baselines: Stacked attention marginally improves results, whereas a two-layer classifier significantly outperforms a single-layer classifier.Adding more classifier layers or increasing classifier width did not appear to improve results.
- 5.2. Comparison to state of the art: 59.67% accuracy on the VQA 2.0 validation set is about 0.5% higher than the best previously reported results.The same model was trained on the VQA 2.0 training set and evaluated on its validation set.
6. Conclusion
The paper presents a simple VQA baseline that outperforms previously reported results on both VQA 1.0 and VQA 2.0. Its performance is attributed to getting implementation details right despite close similarity to earlier models.
- 6. Conclusion: The proposed baseline outperforms previously reported results on both VQA 1.0 and VQA 2.0 datasets.The model is architecturally simple and very similar in essence to models tried previously.