Source-linked AI summary
Graph-Structured Representations for Visual Question Answering
Damien Teney, Lingqiao Liu, Anton van den Hengel
TL;DR
VQA methods often overlook structure in scenes and question syntax, limiting joint visual-language reasoning. This paper builds graphs for scene objects and question words, processes them with a graph neural network, and reports improved accuracy across abstract-scenes and balanced-scene evaluations. The authors conclude that structured language processing improves generalization, while substantial gains may require external information or supervision.
Problem
VQA must jointly reason over visual content and language, but conventional representations and question models overlook scene relationships and syntactic structure.
Method
The model builds scene and question graphs, uses parsed dependencies and pretrained word embeddings, and processes the graphs with recurrent graph neural networks and matching.
Results
The method improves reported accuracy from 71.2% to 74.4% on abstract scenes and from 34.7% to 39.1% on balanced-scene pairs, with benefits over sequential LSTM processing.
Takeaways & Limitations
Structured representations and explicit language processing improve VQA generalization without relying entirely on VQA-specific annotations.
Takeaways & Limitations
The best balanced-dataset performance is about 39%, and further significant improvement may require external information at training or test time.
Abstract
from arXiv · showhide
This paper proposes to improve visual question answering (VQA) with structured representations of both scene contents and questions. A key challenge in VQA is to require joint reasoning over the visual and text domains. The predominant CNN/LSTM-based approach to VQA is limited by monolithic vector representations that largely ignore structure in the scene and in the form of the question. CNN feature vectors cannot effectively capture situations as simple as multiple object instances, and LSTMs process questions as series of words, which does not reflect the true complexity of language structure. We instead propose to build graphs over the scene objects and over the question words, and we describe a deep neural network that exploits the structure in these representations. This shows significant benefit over the sequential processing of LSTMs. The overall efficacy of our approach is demonstrated by significant improvements over the state-of-the-art, from 71.2% to 74.4% in accuracy on the "abstract scenes" multiple-choice benchmark, and from 34.7% to 39.1% in accuracy over pairs of "balanced" scenes, i.e. images with fine-grained differences and opposite yes/no answers to a same question.
1. Introduction
The paper frames VQA as joint reasoning over visual and linguistic structure, arguing that graph representations can better capture object relationships and question syntax than conventional sequential or holistic representations.
- VQA combines computer vision, natural language processing, and artificial intelligence to predict answers from images and natural-language questions.
- Clip-art scenes isolate semantic reasoning and vision-language interactions from visual recognition, while enabling controlled manipulation of image data.
- Challenges: Questions range from direct visual queries to complex questions requiring multiple facts or action understanding.
- Challenges: Sparse training data cannot cover the combinatorial diversity of possible objects and concepts, while many methods process questions with LSTMs trained only on question data.
- Graph representations: The proposed model represents scenes and questions as graphs, using object and spatial relations alongside parsed word dependencies and pretrained word embeddings.
- Contributions: 71.2% to 74.4% accuracy was achieved on the abstract scenes multiple-choice benchmark, while balanced-scene pairs improved from 34.7% to 39.1%.
2. Related work
Related VQA systems typically combine CNN image features with recurrent question encodings and attention, whereas this paper emphasizes graph structure and explicit edge features for semantic and syntactic relations.
- Most VQA methods map CNN image features and LSTM or RNN question representations into a joint space before classifying answers.
- Attention mechanisms model interactions between input regions and question content, and this approach interprets analogous weighting as subgraph matching.
- The proposed structured representations contrast with CNN spatial maps and RNN word sequences by modeling relationships between scene objects and question words.
- Unlike typical VQA systems, the method accepts arbitrary relation features, including syntactic dependencies obtained by pre-parsing questions.
- Graph neural networks update node features through repeated message passing, while related set-processing work uses recurrent attention over unordered elements.
3. Graph representation of scenes and questions
The paper constructs separate question and scene graphs: words and dependency types represent language, while object features and relative positions represent scenes.
- Each question word forms a graph node, and each typed edge represents a syntactic dependency between words.
- Each scene object forms a node with visual features, and pairwise relations create a fully connected graph encoding relative spatial relationships.
- The experiments use clip-art scenes with provided object lists to isolate the proposed method from object-detector performance.
- The same method is described as applicable to real images when the object list is replaced by candidate object detections.
- Question and scene node and edge features are projected into a common vector space, using lookup-table embeddings for language and affine projections for scene features.
4. Processing graphs with neural networks
The neural network independently processes scene and question graphs with recurrent message passing, then uses pairwise attention to match their nodes and classify the answer.
- Each graph node uses a GRU over multiple iterations to integrate information from neighboring nodes.
- Neighbor features are pooled into fixed-size representations, with average pooling performing best in the implementation.
- Attention computes matching weights for every question-word and scene-object pair to identify relevant input elements and align the two graphs.
- Weighted pairwise features are summed over scene elements and then question elements, with projections and nonlinearities producing the final prediction.
- The output classifier supports two scores for balanced binary questions or scores over candidate answers in the abstract scenes dataset.
5. Evaluation
The evaluation examines performance on abstract-scenes and balanced VQA datasets, including ablations, confidence calibration, data scaling, and language preprocessing. Results show benefits from graph processing, attention-like matching, parsing, pretrained embeddings, and complementary-scene training.
- Datasets and metrics: The evaluation uses the original abstract-scenes dataset and its balanced extension, with the latter tested using accuracy over complementary-scene pairs.The balanced metric excludes ambiguous pairs and prevents blind models from earning accuracy through question-only guessing.
- Limitations: The authors report that their best balanced-dataset performance is about 39% and suggest significant further improvement will require external information or supervision.They attribute the challenge to the wide range of concepts involved in the questions.
- Ablative evaluation: Graph-based processing benefits from syntactic parsing, pretrained word embeddings, scene edge features, recurrent updates, and learned matching weights.Ablations show performance drops when these components are removed or replaced with uniform matching weights.
- Precision/recall: A strict confidence threshold can filter incorrect answers and yield very high precision at low recall, while soft targets provide a slight abstract-scenes advantage.Soft targets incorporate ambiguity in multiple human-provided answers.
- Effect of training set size: Both parsing and pretrained embeddings improve generalization as training data varies, with complementary effects on the balanced dataset.Performance also grows steadily with training-set size, although the authors note that the effect of larger datasets remains uncertain.
- Abstract-scenes evaluation: The method achieves the top leaderboard position on abstract scenes in both open-ended and multiple-choice settings, with its largest advantage on binary and counting questions.Validation results corroborate the benefits of parsing, pretrained embeddings, and graph processing with a GRU.
6. Conclusions
The paper concludes that graph-structured representations of scenes and questions improve VQA by incorporating syntactic parsing and pretrained word embeddings. Its experiments demonstrate gains on abstract-scene benchmarks, while extension to real images remains future work.
- 6. Conclusions: Graph-structured scene and question representations improve VQA by enabling explicit processing of visual and linguistic structure.The network leverages pretrained word embeddings and syntactic parsing rather than relying only on sequential question processing.
- 6. Conclusions: Syntactic parsing provides a significant advantage over traditional sequential question processing with LSTMs.
- 6. Conclusions: The method improves generalization without relying entirely on VQA-specific annotations.
- 6. Conclusions: The experiments are conducted on clip-art scenes, with real-image extension proposed through object-detector proposals.The future extension would replace scene-graph nodes with proposals from pretrained object detectors.
- 6. Conclusions: Table 2 reports average percentage scores across overall, yes/no, other, and number questions in multiple-choice and open-ended settings.
- 6. Conclusions: Figure 5 visualizes qualitative predictions and question–object matching weights for abstract scenes and balanced pairs.
A. Implementation
The implementation uses 300-dimensional representations, recurrent graph updates, dependency-labeled question edges, normalized inputs, dropout, and Adadelta optimization. Training and evaluation settings vary by dataset and task.
- A. Implementation: The network uses H=300 for node embeddings, edge features, and hidden states.H=200 also reportedly gives very good results with substantially lower training time.
- A. Implementation: Graph node representations are updated for T_Q=T_S=4 recurrent iterations.At least one iteration matters more than the exact number, while question-graph performance nearly saturates after two or more iterations.
- A. Implementation: Word embeddings are initialized from publicly available 300-dimensional GloVe vectors and fine-tuned at one-tenth the learning rate of other weights.
- A. Implementation: Adadelta trains mini-batches of 128 questions until convergence, selecting the epoch with highest validation performance.Training typically lasts 20 epochs on abstract scenes and 100 epochs on balanced data.
- A. Implementation: Question-graph edges use directed Stanford dependency labels supplemented by differently tagged symmetric edges.The parser’s conjunct-dependency propagation produces relatively densely connected graphs.
- A. Implementation: Object nodes combine category, type, expression or pose, and ten scalar human-pose features.Object edges encode signed and inverse absolute X/Y differences plus relative depth-plane position.
- A. Implementation: Inputs are normalized to zero mean and unit variance, and dropout with ratio 0.3 is applied before classification.
- A. Implementation: Complementary balanced-scene pairs are kept in the same minibatch because this noticeably stabilizes optimization.
B. Additional details
The paper explains dataset scope, metric interpretation, confidence evaluation, attention design, and question-type performance patterns. It emphasizes balanced-pair accuracy as the more meaningful metric while noting unresolved explanations for number-question gains.
- B. Additional details: Abstract scenes isolate semantic reasoning and vision-language interaction by removing visual-recognition confounds.The balanced dataset also permits fine-grained scene manipulation and evaluation against dataset biases.
- B. Additional details: The method’s direct real-image extension would construct scene graphs from candidate object detections.
- B. Additional details: Balanced-pair accuracy is more meaningful than individual-scene accuracy because it is harder to satisfy, while scene-level performance nearly saturates.
- B. Additional details: Precision/recall curves expose answer confidence and can reveal method differences hidden by aggregate accuracy.They support filtering uncertain or ambiguous cases through confidence thresholds.
- B. Additional details: Pre-GRU node features perform slightly better for attention because node identity may suffice without neighboring context.
- B. Additional details: The authors cannot draw definitive conclusions about why number questions show the largest performance gains.They suggest competing methods may rely on number-answer biases, whereas the proposed model may better use input information.
- B. Additional details: The paper provides additional qualitative examples in the same format as Figure 5.
C.1. Additional results: abstract scenes dataset
Additional examples illustrate the model answering binary, counting, spatial, color, activity, and relational questions in abstract scenes. The examples include both direct answers and paired alternatives.
- C.1. Additional results: abstract scenes dataset: The examples cover binary questions about activities, attributes, states, relationships, and scene content.Examples include exercising, reading, grass color, dating, depression, and sitting on an armrest.
- C.1. Additional results: abstract scenes dataset: Counting examples predict 0 flowers, 3 clouds, and 2 clouds in different scenes.
- C.1. Additional results: abstract scenes dataset: Spatial questions identify a girl sitting on a sandbox bench and a slide facing sandbox left.
- C.1. Additional results: abstract scenes dataset: The examples answer object, pattern, and color questions with responses including wine toys, floral curtains, and brown couch pillows.
- C.1. Additional results: abstract scenes dataset: Action and relational questions include jumping rope, sitting while watching TV, identifying a baby, and assessing whether a child can climb on a couch.
- C.1. Additional results: abstract scenes dataset: Some displayed answers contain paired alternatives, such as “red black,” “yes no,” and “no yes.”
C.2. Additional results: balanced dataset
The balanced-dataset examples pair repeated questions with differing yes/no answers across closely contrasted scenes. The questions cover object presence, appearance, actions, relationships, and attributes.
- C.2. Additional results: balanced dataset: The examples include contrasting answers about whether a young lady is tempted to pet a dog.The question is answered yes in p00 and no in p16.
- C.2. Additional results: balanced dataset: Action questions vary between yes and no across paired examples, including helping, dropping a football, and jump roping.The examples contain differing answers for running to help, dropping a football, and jump roping.
- C.2. Additional results: balanced dataset: The question about whether a girl is about to kick the soccer ball appears with differing answer sequences.The examples record “no” in p04 and “no yes” in p21.