Source-linked AI summary
Relation-Aware Graph Attention Network for Visual Question Answering
Linjie Li, Zhe Gan, Yu Cheng, Jingjing Liu
TL;DR
Semantically complicated VQA questions require understanding interactions among image objects, not just detecting regions. ReGAT represents images as graphs and uses question-adaptive graph attention over explicit and implicit relations. It reports state-of-the-art results on VQA 2.0 and VQA-CP v2 and compatibility with generic VQA architectures.
Problem
VQA must capture interactive dynamics among image objects to understand actions, locations, and other semantics needed for semantically complicated questions.
Method
ReGAT encodes image regions as graphs and learns question-adaptive representations from explicit spatial or semantic relations and implicit relations using graph attention.
Results
ReGAT achieves state-of-the-art results on both VQA 2.0 and VQA-CP v2 and improves existing VQA architectures as a generic relation encoder.
Takeaways & Limitations
The relation encoder can be infused into state-of-the-art VQA architectures in a plug-and-play fashion.
Abstract
from arXiv · showhide
In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects. We propose a Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations. Two types of visual object relations are explored: (i) Explicit Relations that represent geometric positions and semantic interactions between objects; and (ii) Implicit Relations that capture the hidden dynamics between image regions. Experiments demonstrate that ReGAT outperforms prior state-of-the-art approaches on both VQA 2.0 and VQA-CP v2 datasets. We further show that ReGAT is compatible to existing VQA architectures, and can be used as a generic relation encoder to boost the model performance for VQA.
1. Introduction
VQA must bridge the semantic gap between images and language by modeling objects together with their actions, locations, and interactions. ReGAT addresses this with question-adaptive graph-based relation encoding that augments visual representations for semantically complicated questions.
- Motivation: VQA models must understand objects, environments, actions, and locations to answer semantically complicated image-grounded questions.Object detection alone may not distinguish which properties or actions belong to which objects.
- Motivation: Beyond object detection, scene understanding requires interpreting geometric positions and semantic dependencies between objects.Examples include spatial relations such as motorcycle-next to-car and semantic interactions such as girl-eating-cake.
- Approach: ReGAT encodes images as graphs and uses graph attention to learn explicit spatial or semantic relations alongside implicit object relations.The relation encoder is designed to capture inter-object relations beyond static region detection.
- Approach: Question-conditioned graph attention injects question semantics into relation encoding and focuses on relation types and instances relevant to each question.The resulting features capture interactive visual content while absorbing semantic clues from the question.
- Contributions: ReGAT is presented as a generic relation encoder that improves existing VQA models and achieves state-of-the-art performance on VQA 2.0 and VQA-CP v2.The approach is described as compatible with existing VQA systems rather than replacing their overall architecture.
2. Related Work
Prior VQA and vision methods use multimodal attention, textual semantic knowledge, or explicit and implicit relation modeling, but ReGAT directly encodes object relations into visual representations. It combines explicit and implicit relations with question-adaptive graph attention and is designed to fit generic VQA models.
- Visual Question Answering: Existing VQA systems commonly combine CNN or R-CNN image features, RNN question features, multimodal fusion, and answer prediction.Attention mechanisms are also used to locate image regions relevant to the question.
- Visual Question Answering: Some approaches add high-level semantic information such as attributes, captions, or relation facts, usually converted into textual representations.ReGAT instead encodes object relations directly into the image representation.
- Visual Relationship: Visual relationships include spatial relations and semantic interactions, which have supported tasks including captioning, image search, localization, and segmentation.Explicit relations can construct graphs with labeled edges, while earlier work used them for post-processing or representation learning.
- Visual Relationship: Implicit-relation methods infer object interactions without predefined semantic or spatial labels, often over fully connected image graphs.These methods capture relations through attention or higher-order operations.
- Relational Reasoning: ReGAT combines explicit and implicit relations, uses GAT for differentiated neighborhood importance, and filters question-irrelevant implicit relations.The paper contrasts this with simple GCN aggregation and treating all implicit relations equally.
3. Relation-aware Graph Attention Network
ReGAT constructs relation graphs over detected image objects and uses question-adaptive graph attention to produce relation-aware visual representations for VQA. It combines implicit relations with explicit spatial and semantic relations, then integrates the refined features with question embeddings for answer prediction.
- VQA predicts an answer from an image-question pair, treating the task as classification with a trained model pθ.
- Faster R-CNN detects object regions, while a bidirectional GRU with self-attention encodes the question before relation encoding.Each detected object has visual and bounding-box features; experiments use K = 36, dv = 2048, db = 4, and dq = 1024.
- Implicit Relation: The implicit relation encoder uses a fully connected graph whose learned edge weights represent relations between every pair of image objects.The graph contains K × (K −1) edges, and the weights are learned without prior relation knowledge.
- Explicit Relation: Explicit relation graphs prune unavailable edges and include labeled directed edges plus an identical self-loop for each object.Spatial graphs encode 11 geometric relation categories with symmetric edges but direction-specific predicates, whereas semantic edges encode non-interchangeable subject-predicate-object relations.
- Question-adaptive Graph Attention: Question-adaptive graph attention injects question semantics and assigns higher weights to relation types and instances relevant to each question.The attention mechanism is relation-specific, can account for edge direction and labels, and uses multi-head attention whose outputs are concatenated.
- Multimodal Fusion and Answer Prediction: The encoder produces refined region features endowed with relational information, adds them to the original visual features, and preserves dimensionality for multimodal fusion.The resulting joint representation is formed by a fusion method f with trainable parameters Θ before answer prediction.
4. Experiments
ReGAT is evaluated across VQA 2.0 and VQA-CP v2 using multiple VQA architectures, relation configurations, and ablations. Results show consistent gains from relation encoding, attention, and question-adaptive features.
- Datasets and setup: ReGAT is evaluated on VQA 2.0 and VQA-CP v2, using accuracy as the evaluation metric.Visual Genome is used to pre-train the semantic relation classifier and augment VQA data for test-dev and test-std evaluation.
- VQA 2.0 validation: The relation encoder is integrated with BUTD, MUTAN, and BAN, producing consistent performance gains over baseline models across all three fusion architectures.The experiments vary implicit, semantic, spatial, pairwise, and combined relation configurations.
- VQA-CP v2: 40.30 vs. 39.54: a single relation already achieves state-of-the-art performance on the VQA-CP v2 test split, while combining all relations adds +0.88.This evaluation uses BAN with four glimpses as the baseline under differing train and test answer distributions.
- VQA 2.0 test results: ReGAT surpasses prior single-model results on VQA 2.0 test-dev and test-std, including BAN with eight bilinear attention maps and Pythia with augmented features.The final model uses bilinear attention with four glimpses and does not use the feature augmentation used by Pythia.
- Ablation study: +0.70 for semantic relations and +0.81 for spatial relations: graph attention improves accuracy over the corresponding ablated relation encoder.Question-adaptive features add approximately +0.1 for semantic and spatial relations, and +0.74 for implicit relations; the complete model achieves the highest accuracy.
- Ablation study: Graph attention helps capture object interactions and improves alignment between image regions and questions, while question adaptation further changes the attended regions.Visualizations compare complete ReGAT with ablated models and show top attended regions with their attention weights.
5. Conclusion
ReGAT models multiple object relations with question-adaptive graph attention for VQA. It achieves state-of-the-art results on two datasets and can be integrated into existing VQA architectures.
- ReGAT models explicit and implicit object relations through question-adaptive graph attention.The framework learns relation-aware region representations for visual question answering.
- ReGAT achieves state-of-the-art results on both VQA 2.0 and VQA-CP v2 datasets.
- ReGAT is compatible with generic VQA models and can be infused into existing architectures in a plug-and-play fashion.