A Focused Dynamic Attention Model for Visual Question Answering

1 Introduction

• VQA requires computer vision to understand image contents and natural language processing to understand questions and generate answers.The introduction illustrates the task with locating apples in a basket, understanding their relationship, and counting them.Source: §1 · p. 2
• Global image features may miss multiple objects and their spatial configuration, while using all proposed regions may add noise irrelevant to the question.The paper presents both as limitations in modeling image content for VQA.Source: §1 · p. 3
• FDA uses question keywords to select relevant image regions, then combines their features with global image context through an LSTM.Its example selects regions corresponding to “apples” and “basket” before incorporating the whole-image feature.Source: §1 · p. 3
• The paper evaluates FDA on the VQA dataset for both open-ended and multiple-choice tasks.It describes the dataset as the largest VQA benchmark.Source: §1 · p. 3
• FDA achieved state-of-the-art accuracy on the open-ended and multiple-choice VQA tasks.The paper also reports substantial improvement over well-established baselines.Source: §1 · p. 3

2 Related Work

• SAN applies CNN attention to image regions using question-word unigrams, bigrams, and trigrams, but does not use object bounding boxes.The paper characterizes SAN’s attention as less focused than FDA’s.Source: §2 · p. 4
• ABC-CNN uses the question embedding to configure convolutional kernels that produce an attention-weighted image map.FDA instead encodes region features in question-word order and extracts CNN features from cropped regions without handcrafted weights.Source: §2 · p. 4
• Another attention model uses object proposals, whereas FDA also uses question-word order to focus on corresponding bounding boxes.The cited alternative concatenates all region and question-word features before passing them to a two-layer network.Source: §2 · p. 4
• Jiang et al. combine CNN image features and LSTM representations with Compositional Memory units to fuse image and word vectors.Source: §2 · p. 4
• Ma et al. use three CNNs to represent the image, question, and their shared multimodal space before SoftMax answer prediction.Source: §2 · p. 4
• Andreas et al. parse questions with a semantic grammar and compose neural networks from proposed layouts and task-specific modules.The modules address sub-tasks such as counting or locating an object.Source: §2 · p. 4

3 Method Overview

• The overview introduces the motivation and formal problem formulation developed in the following subsections.Source: §3 · p. 5

3.1 Problem Formulation

• VQA is formulated as predicting the best answer from an image and question, commonly as classification over the 1,000 most common training-set answers.The answer set is denoted Ω and the model weights θ.Source: §3.1 · p. 5

3.2 Motivation

• The motivation is that baseline methods gain only modest accuracy from adding image features, prompting attention to question-related regions.The reported increases are 4.98% for open-ended questions and 2.42% for multiple-choice questions.Source: §3.2 · p. 5

4.1 Question Understanding

• An LSTM encodes the question while retaining feature vectors for important words, providing a word-attention mechanism.Source: §4.1 · p. 5

4.2 Image Understanding

• A pretrained CNN extracts visual features from the whole image and specific regions, and an LSTM combines local and global features into a joint representation.Source: §4.2 · p. 6

4.3 Focused Dynamic Attention Mechanism

• FDA matches question words to object labels with word2vec, selects objects scoring above 0.5, and feeds their bounding-box features to an LSTM in question-word order before the whole-image feature.During training, object boxes and labels are ground truth; at test time, precomputed boxes are classified to obtain labels.Source: §4.3 · pp. 6–7

4.4 Multimodal Representation Fusion

• The model combines the final question and image LSTM states by applying Tanh and ReLU, multiplying the representations element-wise, and passing the result through a fully connected network and SoftMax classifier.Source: §4.4 · p. 7

5 Evaluation

• The evaluation section introduces implementation details and comparison with current state-of-the-art methods.Source: §5 · p. 7

5.1 Dataset

• The VQA dataset has 204,721 COCO images, each with three human questions and ten answers; evaluation covers open-ended and 18-choice tasks on test-dev and test-standard.The experiments use the public evaluation server.Source: §5.1 · p. 8

5.2 Baseline Model

• The comparison baseline is an LSTM+CNN VQA model that encodes questions with an LSTM, images with CNN features, and classifies among 1,000 common answers.The authors' implementation uses a two-layer question LSTM and normalized image features.Source: §5.2 · p. 8

5.3 Model Implementation and Training Details

• The implementation embeds question words and uses pre-trained ResNet image features, then encodes each modality with a separate one-layer LSTM before element-wise fusion.The question and image LSTMs have 512-dimensional state vectors.Source: §5.3 · p. 8

5.4 Model Evaluation and Comparison

• FDA beats SAN by around 0.6% on open-ended VQA and D-LSTM by 1.1% on multiple-choice VQA.The comparisons are reported for the VQA test-dev and test-standard splits.Source: Table 1 · p. 9, Table 2 · p. 9, §5.4 · p. 10

5.5 Qualitative Results

• Qualitative examples show question-aligned regions support color answers, shift focus across questions, and simplify complex and counting questions through object identification.The examples include color questions where the predominant image color is not the answer, and cases involving unusual questions, small objects, and occlusion.Source: §5.5 · p. 10, Figure 3 · p. 11, Figure 4 · p. 12

• FDA provided significant performance improvement over baselines on the VQA benchmark for both open-ended and multiple-choice tasks.Source: §6 · p. 14
• FDA fuses question-related regional representations, global context, and question representations through multiple LSTM layers.Source: §6 · p. 14
• By focusing on regions identified for each question, FDA filters irrelevant background information and improves visual representations for answering.Source: §6 · p. 14
• Future work will explore different attention methods for visual information selection and better reasoning models for interpreting relations between visual contents and questions.Source: §6 · p. 14