Source-linked AI summary
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
Damien Teney, Peter Anderson, Xiaodong He, Anton van den Hengel
TL;DR
VQA performance varies substantially with architecture and hyperparameter choices, motivating a careful study of a relatively simple joint-embedding model. The paper evaluates these choices extensively and reports a high-performing model whose success is associated with several implementation decisions, while cautioning that ablation effects are not necessarily additive and benchmark metrics can overstate progress.
Problem
VQA is a challenging multimodal task, and the effects of architecture and hyperparameter choices on deep neural network performance are limited and sometimes unpredictable.
Method
The paper conducts extensive experiments around a joint question-image embedding and multi-label answer classifier, comparing alternative architectures and hyperparameters.
Results
The model significantly outperforms other approaches reported at the time, with gains associated with sigmoid outputs, soft targets, gated activations, bottom-up image features, and pretrained answer representations.
Takeaways & Limitations
VQA performance is highly dependent on implementation details, and the model provides a strong basis for future incremental development and multimodal reasoning research.
Takeaways & Limitations
The cumulative effects of multiple modifications are not necessarily additive, complicating the search for optimal architectures and hyperparameters.
Abstract
from arXiv · showhide
This paper presents a state-of-the-art model for visual question answering (VQA), which won the first place in the 2017 VQA Challenge. VQA is a task of significant importance for research in artificial intelligence, given its multimodal nature, clear evaluation protocol, and potential real-world applications. The performance of deep neural networks for VQA is very dependent on choices of architectures and hyperparameters. To help further research in the area, we describe in detail our high-performing, though relatively simple model. Through a massive exploration of architectures and hyperparameters representing more than 3,000 GPU-hours, we identified tips and tricks that lead to its success, namely: sigmoid outputs, soft training targets, image features from bottom-up attention, gated tanh activations, output embeddings initialized using GloVe and Google Images, large mini-batches, and smart shuffling of training data. We provide a detailed analysis of their impact on performance to assist others in making an appropriate selection.
1. Introduction
The paper presents a relatively simple joint-embedding VQA model whose performance depends strongly on carefully selected architecture and hyperparameter choices. Extensive empirical exploration identifies several practical design choices associated with high performance.
- Model and objective: The model jointly embeds the input question and image, then predicts answers with a multi-label classifier over candidate answers.This general approach is augmented with technical innovations and extensive architecture and hyperparameter exploration.
- Key design choices: Sigmoid outputs represent multiple correct answers per question instead of using a single-label softmax.The design matches VQA annotations, where questions may have several correct or synonymous answers.
- Key design choices: Soft target scores cast VQA as regression over candidate answers rather than traditional classification.The targets preserve graded agreement among human annotators.
- Key design choices: Gated tanh activations are used in all nonlinear layers.This is one of the implementation choices identified among the model’s high-performing characteristics.
- Key design choices: Bottom-up attention supplies region-specific image features instead of traditional grid-like CNN feature maps.The features focus on specific image elements and are treated as object-centered representations.
- Key design choices: Pretrained answer representations, large mini-batches, and smart training-data shuffling are also identified as successful choices.The paper presents these choices as part of the model characteristics associated with high performance.
2. Background
VQA combines visual understanding, language comprehension, and sometimes common-sense reasoning, making it a challenging multimodal task. Background work emphasizes joint-embedding systems, large datasets, object-centric visual features, and careful implementation.
- Task and datasets: VQA requires answering a text question about an image by combining visual concepts with language and sometimes broader knowledge.The task spans computer vision, natural language processing, and artificial intelligence.
- Task and datasets: Language priors and rote question-answer learning motivated VQA v2, which was introduced to improve evaluation of visual understanding.Earlier benchmark behavior made competing methods harder to evaluate and compare effectively.
- Task and datasets: Visual Genome provides fine-grained scene annotations and diverse questions used for object-centric image features and overlapping answer supervision.Its annotations support Faster R-CNN training, while only questions with answers overlapping VQA v2 are used directly.
- Prevailing methods: The prevailing VQA pipeline classifies candidate answers with a joint-embedding neural network trained end-to-end from questions, images, and answers.The sparse training signal and large input space motivate large-scale datasets.
- Prevailing methods: The proposed work shows that a relatively simple joint-embedding model can achieve strong performance through careful implementation and hyperparameter selection.The reported improvements accumulate through choices such as gated activations, regression outputs, and smart shuffling.
3. Proposed model
The proposed VQA model jointly embeds questions and images with question-guided attention, then predicts soft scores for multiple candidate answers. Its performance depends on carefully selected architectural choices, including bottom-up image features, sigmoid outputs, pretrained answer representations, and gated nonlinear layers.
- Model architecture: The model uses a joint RNN/CNN embedding of the question and image with question-guided, one-glimpse attention over image regions.Question and image representations are combined through element-wise multiplication before classification.
- Question embedding: Questions are tokenized, capped at 14 words, represented with 300-dimensional GloVe-initialized embeddings, and processed by a GRU.Out-of-vocabulary words are initialized with zeros and optimized, while padding remains fixed.
- Image features: Bottom-up attention features use a Faster R-CNN-based ResNet trained on Visual Genome to represent image regions centered on detected objects.The model evaluates fixed K=36 regions and adaptive region counts up to 100, averaging about K=60 regions per VQA v2 image.
- Output classifier: The output vocabulary contains N=3129 candidate answers, and the classifier predicts each candidate’s score with a sigmoid rather than a softmax.Soft target scores represent annotator agreement and permit multiple correct answers per question.
- Classifier pretraining: Candidate-answer output embeddings are initialized from linguistic and visual representations, using word embeddings and averaged ResNet features from 10 Google Images per answer.Visual representations extend coverage to multi-word or rare answers, but may be uninformative for abstract expressions.
- Nonlinear layers: Each learned nonlinear layer uses a gated hyperbolic tangent activation whose gate multiplicatively modulates the intermediate activation.The formulation is related to gating in LSTMs, GRUs, and highway networks.
4. Ablative experiments
The paper evaluates individual architecture, training-data, and hyperparameter choices through extensive single-network and cumulative ablations. Several choices materially affect performance, while others show modest or metric-specific benefits.
- Experimental setup: More than 3,000 GPU-hours were used to compare architecture and hyperparameter alternatives against a reference model.Each experiment varied one reference-model choice, and repeated runs used different random seeds.
- Training data: +0.67% performance gain came from adding 485,000 Visual Genome questions, a modest improvement relative to the added data.Only Visual Genome instances compatible with the VQA v2 answer vocabulary and images were retained.
- Training data: Keeping balanced question pairs in the same mini-batches improved accuracy over pairs, although overall VQA performance was virtually identical.The cumulative ablation confirmed the pairwise advantage more clearly.
- Training data: Retaining questions whose ground-truth answers were outside the selected candidates preserved virtually identical VQA accuracy and slightly benefited accuracy over pairs.The authors attribute the training signal to suppressing scores for selected candidates.
- Question embedding: Randomly initialized word embeddings reduced performance by 0.87%, while shuffled GloVe vectors performed worse than embeddings learned from scratch.The results support word-specific information in pretrained GloVe embeddings.
- Question embedding: Simpler bag-of-words, summation, or averaging baselines remained surprisingly strong but underperformed the reference model; more advanced GRUs also gave lower performance.Backward, bidirectional, and two-layer GRUs were among the tested alternatives.
4.3. Image features
Image representation strongly influences VQA performance: bottom-up attention features work best in the reported model, while conventional ResNet maps perform substantially worse. Feature normalization and attention design also matter.
- Feature types: Bottom-up attention uses region-specific Faster R-CNN features with adaptive K, capped at 100 and averaging about K=60.Fixing K=36 degraded performance only slightly.
- Feature types: 63.15% versus 57.52% was obtained with bottom-up attention features versus standard 14×14 ResNet features.Global average pooling was even worse, while interpolated 7×7 maps reached 59.24.
- Normalization and transfer: Optimal design choices were stable across image-feature types, and L2 normalization was crucial for good performance with the reported architecture and optimizer.This stability included the attention mechanism.
- Attention: Multiple attention heads and sigmoid-normalized attention weights were not beneficial with either ResNet or bottom-up attention features.The reference model instead used one softmax-normalized set of attention weights.
4.5. Output vocabulary
The experiments examine candidate-answer vocabulary, soft targets, sigmoid outputs, and classifier pretraining. The strongest reported choices use soft annotation scores and sigmoid outputs, with pretrained classifier initialization providing answer-specific benefits.
- Output vocabulary: A threshold of ℓ=8–12 answer occurrences yielded a broad optimum of approximately N=2,400–3,800 candidate answers.Higher thresholds, corresponding to smaller vocabularies, still gave reasonable performance with lower coverage.
- Training targets: Soft annotation scores performed significantly better than either binarized target version.The reference model uses these soft scores as ground-truth targets.
- Output activation: Sigmoid outputs performed significantly better than softmax because they use complete annotation data, including questions with multiple correct answers.Both alternatives used cross-entropy loss, but softmax used only the single provided ground-truth answer.
- Classifier pretraining: Pretrained classifier initialization can improve answer recall, especially for answers with few training occurrences.The experiments also found that random initialization could surpass the reference model on overall VQA score but not on accuracy over pairs.
- Classifier pretraining: Pretraining effects were not uniformly positive: some answers’ recall decreased despite the overall benefit for the proposed approach.The authors note that other architectures may be needed to obtain the full benefits.
4.7. General architecture
The general architecture favors gated nonlinear layers, a moderate hidden-state size, element-wise multimodal fusion, and relatively large mini-batches. These choices improve or stabilize performance but impose computational trade-offs.
- Nonlinear layers: Gated tanh layers clearly outperformed gated ReLU, simple ReLU, and tanh activations.Gated layers double learned parameters without increasing hidden-state dimensionality.
- Hidden-state size: 512 hidden units were selected as a reasonable sweet spot among dimensions from 256 to 1280.Larger dimensions could perform better without guarantees, while repeated-run variance increased, likely from overfitting and unstable training.
- Multimodal fusion: Element-wise multiplication combined question and image representations far better than concatenation.The paper did not test the advanced pooling alternatives cited in the discussion.
- Optimization: Midrange mini-batches in {128, 256, 384, 512, 768} outperformed smaller batches, but required substantially more memory and high-end GPUs.The optimal batch-size choice remained stable across architecture variations.
4.9. Training set size
Training performance improves with more VQA data but follows a roughly logarithmic trend, so the gain from a ten-fold data increase is relatively small. Pretrained embeddings and classifiers are especially useful with reduced data, provide complementary benefits, and help represent rare words and concepts.
- Performance improves monotonically with training-data quantity but roughly follows a logarithmic trend.
- 10% of the training data already yields reasonable performance, making the gain from using the full dataset small relative to a ten-fold data increase.
- Pretrained word embeddings and classifiers are especially beneficial with reduced VQA-specific training data, while their advantage shrinks as more VQA data is used.
- Pretrained word embeddings and classifiers each provide benefits of the same order of magnitude, and combining them produces the best performance.
4.10. Ensembling
The paper ensembles multiple instances of the same network, trained with different random seeds, by summing their candidate-answer scores. Performance rises monotonically with ensemble size; even small ensembles significantly outperform a single network, while the final results use 30 networks.
- Ensembling sums candidate-answer scores from identical networks trained with different random seeds.The final answer is the candidate with the highest summed score.
- Performance increases monotonically with ensemble size, and the final best results use an ensemble of 30 networks.
- Ensembles of 2–5 instances already provide a significant performance increase over a single network.
- Training ensemble members is independent and can be parallelized across multiple CPUs or GPUs.
- The reported experiments train on the VQA v2 validation split and evaluate on test-dev, producing higher overall performance than Tables 1 and 2.
5. Cumulative ablations
Cumulative ablations show that removing the proposed contributions steadily decreases performance and identify sigmoid outputs, soft targets, bottom-up attention, gated tanh, pretrained output layers, and smart shuffling as critical components. However, modification effects are not necessarily additive when combined.
- Ablation effects are not necessarily additive, so choices that look promising alone may fail when combined with other optimizations.
- Conversely, options discarded early may become effective after other hyperparameters are tuned.
- Removing each proposed contribution steadily decreases performance in the cumulative ablations.
- The most critical components are sigmoid outputs, soft ground-truth scores, bottom-up attention features, gated tanh activations, pretrained output layers, and smart training-data shuffling.
6. Comparison with existing methods
The best model is a 30-network ensemble with enlarged hidden states and additional VQA v2 training data. It won first place in the 2017 VQA Challenge and surpassed competing methods by a significant margin at the time of writing.
- The best model won first place in the 2017 VQA Challenge.
- The model surpassed all competing methods by a significant margin at the time of writing.
- The best model ensembles 30 networks identical to the reference model, except that hidden-state dimensions increase to 1,500.
- The best model also trains using the VQA v2 validation split, in addition to the training data.
7. Discussion and conclusions
The study finds that VQA performance remains highly dependent on implementation details, while better image features and additional data offer promising avenues for progress. It also emphasizes that conventional accuracy figures can overstate current visual understanding.
- 7. Discussion and conclusions: VQA performance depends strongly on design and implementation details, including both easily implemented and non-trivial choices.The authors distinguish large mini-batches and sigmoid outputs from gated activations, bottom-up attention, and output-classifier pretraining.
- 7. Discussion and conclusions: The paper does not claim breakthrough advances because VQA remains a largely unsolved problem.Its extensive design exploration is presented as a basis for future development and as a source of promising directions.
- 7. Discussion and conclusions: Significant gains remained achievable through region-specific image features produced with bottom-up attention.The paper also reports that gains from additional VQA training data had not reached a clear plateau.
- 7. Discussion and conclusions: Simple bag-of-words question encoding performed almost as well as state-of-the-art recurrent encoders.The authors suggest this may reflect limited information in question word ordering or models’ inability to use language structure effectively.
- 7. Discussion and conclusions: Balanced-pair accuracy drops to the order of 35%, compared with commonly reported per-question accuracy above 70%.The stricter measure requires correct answers for complementary versions of a question about different images.