Source-linked AI summary
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, Jianyong Wang
TL;DR
VQA requires combining image and language understanding, while existing semantic concepts may be limited or unrelated to the question. R-VQA uses detected relation facts with sequential visual and semantic attention, and the authors report state-of-the-art results on two benchmarks.
Problem
Existing VQA methods use image-question fusion or attention, while entity and attribute concepts can be limited in semantic capacity and unrelated to the question.
Source: §1 · pp. 1–2Method
The paper builds the R-VQA dataset and relation detector, then uses sequential visual and semantic attention to incorporate question-related relation facts into VQA.
Source: §1 · p. 2Results
Takeaways & Limitations
The experiments demonstrate the benefit of considering visual relation facts for VQA.
Source: §1 · p. 2The relation detector uses a candidate number k and a simpler framework; the authors identify designing a fine-grained model for better prediction as future work.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Recently, Visual Question Answering (VQA) has emerged as one of the most significant tasks in multimodal learning as it requires understanding both visual and textual modalities. Existing methods mainly rely on extracting image and question features to learn their joint feature embedding via multimodal fusion or attention mechanism. Some recent studies utilize external VQA-independent models to detect candidate entities or attributes in images, which serve as semantic knowledge complementary to the VQA task. However, these candidate entities or attributes might be unrelated to the VQA task and have limited semantic capacities. To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA. Specifically, we build up a Relation-VQA (R-VQA) dataset based on the Visual Genome dataset via a semantic similarity module, in which each data consists of an image, a corresponding question, a correct answer and a supporting relation fact. A well-defined relation detector is then adopted to predict visual question-related relation facts. We further propose a multi-step attention model composed of visual attention and semantic attention sequentially to extract related visual knowledge and semantic knowledge. We conduct comprehensive experiments on the two benchmark datasets, demonstrating that our model achieves state-of-the-art performance and verifying the benefit of considering visual relation facts.
1 INTRODUCTION
VQA requires reasoning across image and language, but existing semantic concepts can be limited and irrelevant to the question. R-VQA adds visual relation facts and sequential visual and semantic attention; the authors report state-of-the-art performance on two benchmarks.
1 INTRODUCTION
- Existing semantic concepts such as entities and attributes provide restricted knowledge, and concepts from other tasks may be irrelevant to VQA.Source: §1 · p. 2
- The authors build a large-scale dataset with relation facts and design a relation detector to predict question-related facts from images.Each R-VQA instance includes an image, a relevant question, and a relation fact semantically similar to the image-question pair.Source: §1 · p. 2
- The proposed framework sequentially combines visual and semantic attention to learn visual relation facts for answering VQA questions.Visual attention extracts image features, semantic attention selects relation facts, and joint knowledge learning combines visual and semantic representations.Source: §1 · p. 2
- The model achieves state-of-the-art performance on two benchmark datasets.Source: §1 · p. 2
2 RELATED WORK
The related work reviews feature-based VQA, attention mechanisms, and semantic facts. It positions sequential visual and semantic attention and relation facts as the paper’s response to limitations in prior approaches.
2.1 Visual Question Answering
- Common VQA methods encode image and question features, then fuse them to predict an answer; bilinear pooling is also studied for multimodal fusion.Source: §2.1 · p. 2
2.2 Attention Methods
- Prior VQA attention methods focus on image regions or combine visual features with multimodal embeddings, whereas this work sequentially combines visual and semantic clues.Source: §2.2 · p. 2
2.3 Semantic Facts
- Relation facts link two entities, and the paper argues they offer richer semantic knowledge than treating image concepts as entities or attributes alone.The authors propose learning relation facts and selecting relevant ones through semantic attention.Source: §2.3 · p. 3
3 PRELIMINARY
The preliminary section frames VQA as answering a question from an image and describes common feature-embedding and attention components. These provide the baseline formulation and framework for the paper’s approach.
3.1 Problem Formulation
- VQA predicts the most likely answer from an image and question, commonly as classification over candidate answer phrases.Source: §3.1 · p. 3
3.2 Common Framework
- Common frameworks extract image features with a CNN and question representations with an RNN such as an LSTM or GRU.The question embedding is the RNN’s final hidden state.Source: §3.2 · p. 3
- Image and question representations are jointly embedded by multimodal pooling and passed to a classifier; visual attention weights image regions by their relevance to the question.The attention-weighted visual representation is formed from image regions.Source: §3.2 · p. 3
4 RELATION FACT DETECTOR
This section builds R-VQA by aligning image-question-answer pairs with relation facts, then trains a detector to predict question- and image-related facts. The merged-data detector outperforms the initial-data version, and models using both image and question information perform better than versions without either modality.
4.1 Data Collection for Relation-VQA
- R-VQA aligns Visual Genome question-answer pairs with entity concepts, entity attributes, or entity relations, selecting the highest-scoring fact after threshold filtering.Response Ranking scores candidate-fact relevance from 0 to 1; human evaluation found higher thresholds yield higher matched accuracy but smaller datasets.Source: §4.1 · pp. 3–4
- 198,889 samples at threshold 0.30 have 90.8% average matched accuracy in R-VQA.The dataset includes 5,833 unique subjects, 2,608 relations, and 6,914 objects.Source: Table 2 · p. 4, §4.1 · p. 4
4.2 Relation Fact Detector
- The detector encodes image content with ResNet-152 and questions with a GRU, maps both into a shared semantic space, fuses them by element-wise addition, and uses separate linear classifiers to predict subject, relation, and object.Source: §4.2 · pp. 4–5
- Before training, synonymous fact elements are merged into simpler forms, with the candidate set retaining 2,000 subjects, 256 relations, and 2,000 objects.The authors report that merging similar elements lets top candidates cover more training data.Source: Table 4 · p. 5, §4.2 · p. 5
- The merged-data detector outperforms the initial-data detector, while versions without image or question information perform worse.The authors identify finer-grained modeling as future work for improving prediction performance.Source: Table 5 · p. 5, §4.2 · pp. 5–6
5 VISUAL QUESTION ANSWERING WITH FACTS
The framework uses question-aware visual attention and fact-aware semantic attention in sequence, then combines their representations to predict an answer.
5.1 Context-aware Visual Attention
- MLB combines question and image context to score 14 × 14 image regions, pool their weighted representations, and fuse the result with question features.The region weights represent semantic relevance to the input question.Source: §5.1 · p. 6
5.2 Fact-aware Semantic Attention
- The detector generates candidate relation facts, and semantic attention uses the context-aware visual representation to select important facts as semantic knowledge.Each fact is represented by concatenating embeddings of its subject, relation, and object.Source: §5.2 · pp. 6–7
5.3 Joint Knowledge Embedding Learning
- The model combines visual and fact-aware semantic representations through transformed element-wise addition, then uses a linear classifier to infer the answer.The two knowledge representations are merged to jointly encode visual and semantic knowledge.Source: §5.3 · p. 7
6 EXPERIMENTS
Experiments compare RelAtt with prior methods on VQA and COCO-QA, then examine component ablations and examples of fact detection and attention.
6.1 Datasets and Evaluation Metrics
- VQA evaluation covers open-ended and multi-choice tasks, while COCO-QA has four question types and counts an answer as correct when it matches the label.VQA correctness requires more than three annotator votes; COCO-QA answers are single words.Source: §6.1 · p. 7
6.2 Implementation Details
- Training uses RMSProp with 200-example mini-batches, a 3 × 10−4 initial learning rate, and dropout probability 0.5 at each linear transformation layer.Validation is performed every 10,000 iterations, with early stopping after five validations without improvement.Source: §6.2 · pp. 7–8
6.3 Comparison with State-of-the-art
- On the VQA dataset, the proposed multi-step semantic attention network RelAtt improves over MLAN from 64.60% to 65.69% on open-ended and from 64.80% to 69.60% on multi-choice tasks.Source: Table 6 · p. 8, §6.3 · p. 8
- 65.15% vs 62.50% on COCO-QA: RelAtt improves over QRU and outperforms AMA by 3.77%.The authors report the comparison on the COCO-QA dataset.Source: Table 7 · p. 8, §6.3 · p. 8
6.4 Ablation Study
- Removing semantic attention reduced accuracy by 0.79%; the ablations also suggest element-wise addition may work better than multiplication for joint knowledge encoding.The first three ablations indicate visual attention provides limited visual information and relation facts can play an important role.Source: Table 8 · p. 8, §6.4 · p. 8
6.5 Case Study
- Testing examples show that the detector generates question-dependent relation facts for the same image, while semantic attention highlights likely facts for answering.The examples include images, questions, predicted answers, fact candidates, and attention weights.Source: §6.5 · p. 9, Figure 5 · p. 9
7 CONCLUSION
The paper proposes learning visual relation facts from images and questions for VQA, combining them with visual knowledge through sequential attention and reporting improved performance over state-of-the-art approaches.
7 CONCLUSION
- The framework learns relation facts from image-question data using a detector trained with the Relation-VQA dataset.The dataset is built to support visual relation facts for VQA.Source: §7 · p. 9
- A multi-step attention model incorporates detected relation facts through sequential visual and semantic attention to fuse visual and semantic knowledge for answering.Source: §7 · p. 9
- Experiments report that the method outperforms state-of-the-art approaches and demonstrate the effectiveness of considering visual semantic knowledge.Source: §7 · p. 9