Source-linked AI summary
Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, Xiaogang Wang
TL;DR
VQA attention methods focus either on free-form image regions or on pre-specified detection boxes, although the paper argues these provide complementary information. It combines both in a dual-branch network with multiplicative feature embedding, and reports outperforming state-of-the-art approaches on two large VQA datasets.
Problem
VQA attention methods focus either on free-form regions, which can attend irrelevant context, or detection boxes, which may omit regions needed for some questions.
Source: §1 · pp. 1–2Method
A dual-branch network co-attends question-related free-form regions and detection boxes using multiplicative embedding of question, whole-image, and detection-box features.
Source: §1 · p. 2Results
The proposed model outperforms state-of-the-art approaches on two large VQA datasets.
Source: §5 · p. 8Takeaways & Limitations
Free-form regions can address questions such as weather when no sky detection box exists, while detection attention can better serve animal questions.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recently, the Visual Question Answering (VQA) task has gained increasing attention in artificial intelligence. Existing VQA methods mainly adopt the visual attention mechanism to associate the input question with corresponding image regions for effective question answering. The free-form region based and the detection-based visual attention mechanisms are mostly investigated, with the former ones attending free-form image regions and the latter ones attending pre-specified detection-box regions. We argue that the two attention mechanisms are able to provide complementary information and should be effectively integrated to better solve the VQA problem. In this paper, we propose a novel deep neural network for VQA that integrates both attention mechanisms. Our proposed framework effectively fuses features from free-form image regions, detection boxes, and question representations via a multi-modal multiplicative feature embedding scheme to jointly attend question-related free-form image regions and detection boxes for more accurate question answering. The proposed method is extensively evaluated on two publicly available datasets, COCO-QA and VQA, and outperforms state-of-the-art approaches. Source code is available at https://github.com/lupantech/dual-mfa-vqa.
1 Introduction
VQA requires understanding both an image and its question, while existing attention approaches focus on either free-form regions or pre-specified detection boxes. The paper proposes combining both mechanisms through a dual-branch network and multiplicative feature embedding.
1 Introduction
- VQA requires understanding both image contents and natural-language questions to infer answers.Unlike image captioning and text-to-image retrieval, it must account for both inputs.Source: §1 · p. 1
- Free-form attention can capture global context and foreground objects, but may focus on partial objects or irrelevant context.Detection-based attention is restricted to pre-specified boxes, which may omit relevant regions such as the sky.Source: §1 · pp. 1–2
- The paper argues that free-form and detection-based attention provide complementary information for answering different question types.Its examples contrast animal questions that may benefit from detection boxes with weather questions better served by free-form regions.Source: §1 · p. 2
- The proposed dual-branch network attends to question-related free-form regions and detection regions in one framework.It uses multiplicative feature embedding to jointly represent the question, whole image, and detection boxes.Source: §1 · p. 2
2 Related work
Related work covers multimodal feature embedding and visual attention for VQA, including methods that attend to free-form image regions or detection boxes. The paper positions its approach as integrating both region types to use their complementary information.
2 Related work
- Earlier VQA embedding approaches combined visual and question features, with methods spanning concatenation, recurrent answer generation, and bilinear pooling.Examples include MCB and MLB bilinear pooling approaches (Fukui et al. 2016; Kim et al. 2017).Source: §2 · p. 2
- Prior attention methods relate questions to image regions through semantic similarity, iterative attention, or joint question-image co-attention.The cited approaches include Xu and Saenko (2016), Yang et al. (2016), and Lu et al. (2016).Source: §2 · p. 2
- Detection-based VQA methods have attended to sets of detection boxes, including 100-box features or iteratively selected top boxes.Examples are Shih, Singh, and Hoiem (2016) and Li and Jia (2016).Source: §2 · p. 2
- The paper identifies a gap in prior attention methods: each focuses on either free-form regions or detection boxes, and proposes integrating both.The stated aim is to use complementary information from the two region types.Source: §2 · p. 2
3 Method
The method co-attends free-form spatial regions and detection boxes, learning each from the question, whole-image features, and detection features. The attended visual features are combined with question representations for answer classification.
3 Method
- The network jointly attends free-form image regions and detection boxes, using multiplicative embeddings of the question, whole image, and detection features in both branches.The two branches have separate embedding parameters; their attended visual features are fused with the question representation.Source: §3 · pp. 2–3, Figure 2 · p. 3
3.1 Input feature encoding
- ResNet-152 encodes each 448 × 448 image as a 2048 × 14 × 14 feature map, while the top 19 Faster-RCNN boxes use 4096-dimensional fc7 features plus detection scores.The resulting detection-box features are 4097-dimensional.Source: §3.1 · p. 3
- A GRU encodes the question, with its final hidden state as the embedding and skip-thought weights initializing the word embeddings.The GRU is fine-tuned from this initialization.Source: §3.1 · p. 3
3.2 Learning to attend free-form region visual features with multiplicative embedding
- The free-form branch uses multiplicative fusion of question, whole-image, and detection features to produce attention over unrestricted image regions.The joint representation is element-wise multiplied and L2-normalized; a 1 × 1 convolution and softmax produce the spatial attention map.Source: §3.2 · p. 4
3.3 Learning to attend detection-box visual features with multiplicative embedding
- The detection branch fuses question, whole-image, and box features to learn attention weights over detection boxes.Its inclusion of whole-image features distinguishes it from the cited detection-attention methods.Source: §3.3 · pp. 4–5
3.4 Learning for answer prediction
- For answer prediction, each attended visual feature is element-wise multiplied with a branch-specific transformed question representation, and the branch encodings are added before linear classification.The two question transformations reflect the distinct information in free-form scene features and foreground-object features.Source: §3.4 · p. 5
4 Experiments
Experiments evaluate Dual-MFA on VQA and COCO-QA, where it improves on reported state-of-the-art comparisons. Ablations examine embedding and attention choices, and visualizations show complementary successes as well as a classification failure.
4.1 Datasets and evaluation metrics
- VQA uses yes/no, number, and other questions; COCO-QA uses object, number, color, and location questions, with accuracy reported for both datasets.COCO-QA also reports WUPS; VQA correctness requires a prediction to match more than three of ten ground-truth answers.Source: §4.1 · p. 5
4.2 Implementation details
- Implementation fixes question length at 26, uses 620-dimensional word embeddings and a 2400-unit GRU, and trains with batch size 300 for 250,000 iterations.The joint embedding dimension is 1200; two attention glimpses are used per branch, and validation with early stopping is performed every 10,000 iterations.Source: §4.2 · p. 5
4.3 Comparison with state-of-the-arts
- On VQA test-std, Dual-MFA improves over MLB from 65.07% to 66.09% for open-ended and from 68.89% to 69.97% for multiple-choice questions.Table 1 reports the test-set comparisons; Dual-MFA also outperforms QRU by 5.33%.Source: §4.3 · pp. 5–6, Table 1 · p. 6
4.4 Ablation study
- Element-wise multiplication beats addition by 3.67% for joint embedding; L2 normalization helps, while addition best fuses the two branch outputs.The embedding comparisons also include unsigned power operation and no normalization; branch-output addition outperforms multiplication and concatenation.Source: §4.4 · pp. 6–7
4.3 Comparison with state-of-the-arts
- On COCO-QA, Dual-MFA improves over HieCoAtt from 65.40% to 66.49% and outperforms QRU by 3.49%.The reported improvement for color questions is 2.99%.Source: §4.3 · p. 6, Table 3 · p. 7
4.4 Ablation study
- Single-branch MFA-R and MFA-D outperform the compared region- and detection-based baselines, while Dual-MFA improves over Dual-MLB by 0.75%.The full model has 82.6M parameters, versus 61.7M for MFA-R and 56.8M for MFA-D; shared language-model and classifier parameters limit the increase.Source: §4.4 · p. 7
4.5 Qualitative evaluation
- Visualizations show cases where both branches or only one branch attend correctly, but the model fails when the correct label is absent from training.The free-form map attends to sky and dry ground where detection boxes are absent; detection attention highlights five giraffes in another example.Source: §4.5 · p. 7
5 Conclusion
The proposed VQA model combines two visual-attention branches with multiplicative feature embedding; evaluations on two large VQA datasets show it outperforms state-of-the-art approaches.
- On two large VQA datasets, the proposed model outperforms state-of-the-art approaches.Source: §5 · p. 8
- Two visual-attention branches select free-form image regions and detection boxes most related to the input question.Source: §5 · p. 8
- A multiplicative embedding scheme fuses question, whole-image, and detection-box features to generate visual-attention weights.Source: §5 · p. 8
- An ablation study demonstrates the effectiveness of the model’s individual components.Source: §5 · p. 8