Source-linked AI summary
Learning Conditioned Graph Structures for Interpretable Visual Question Answering
Will Norcliffe-Brown, Efstathios Vafeias, Sarah Parisot
TL;DR
VQA requires combining image and question understanding, while many existing methods use two-stream features and few capture higher-level semantic and spatial relationships. This paper learns question-conditioned graph structures over detected objects and applies graph convolutions to model relevant interactions, achieving 66.18% accuracy on VQA v2 with interpretable learned graphs.
Problem
Many VQA methods use separate image and question streams, while few exploit higher-level representations that capture semantic and spatial relationships.
Method
The model uses a question-conditioned graph learner over object detections and applies graph convolutions to produce representations influenced by relevant neighboring objects.
Results
66.18% accuracy is reported on the VQA v2 dataset, with promising performance and interpretable learned graph structures.
Takeaways & Limitations
Question-conditioned graph learning provides object representations influenced by relevant neighbors and supports interpretation through learned nodes and edges.
Takeaways & Limitations
The approach depends on object-detector quality, and its fixed number of detected objects can amplify duplicate or missed-object errors.
Abstract
from arXiv · showhide
Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which can capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain promising results with 66.18% accuracy and demonstrate the interpretability of the proposed method. Code can be found at github.com/aimbrain/vqa-project.
1 Introduction
VQA combines visual and language understanding to answer questions about images, but many methods rely on two-stream features and engineered, image-specific graphs. The paper proposes learning interpretable graph structures conditioned on each question.
- VQA answers questions about images by combining Computer Vision and Natural Language Processing.
- Most VQA methods separately embed images and questions before merging the two streams for classification.
- Graph-based VQA methods can model semantic and spatial object interactions, but their graphs are often engineered and image-specific rather than question-specific.
- The proposed graph learner uses question context to learn bounding-box connections, identifying relevant objects and relationships without handcrafted graph structure.
2 Related work
Related work extends convolutional processing to graph-structured data and applies graphs to VQA, but existing formulations face structural or transferability constraints. The proposed architecture uses question-conditioned graph learning to model object interactions in natural images.
- Graph Convolutional Neural Networks: Graph CNNs generalize convolutional processing from regular Euclidean grids to non-regular structures such as meshes and networks.
- Graph Convolutional Neural Networks: Spectral GCNs require the same graph structure across training samples because their filters are defined on the graph Laplacian basis.
- Graph Convolutional Neural Networks: Spatial GCNs use engineered node orderings or patch operators, while graph attention weights can be interpreted as edges between neighboring nodes.
- Graph-based VQA: Graph-based VQA has shown strong performance, but approaches built for abstract scenes are not easily applicable when natural-image scene graphs are unknown beforehand.
- Object-based VQA: Object proposals have improved VQA, enabling later methods to model interactions through implicit or explicit graphs, especially for counting.
3 Methods
The model learns a question-conditioned graph over detected image objects, then applies spatial graph convolutions to capture relevant object relationships before combining the graph and question representations for answer classification.
- Computing model inputs: The model encodes image regions as object descriptors and questions as embeddings, using bounding boxes as nodes in a question-specific graph.Object features come from detector proposals, while a GRU-based RNN produces a single question embedding.
- Graph learner: The graph learner concatenates each visual feature with the question embedding and uses the resulting joint embeddings to construct a self-looped adjacency matrix.The adjacency matrix is defined as A = EE^T, making edge strengths dependent on both visual similarity and question relevance.
- Graph learner: Sparse node neighborhoods retain the m strongest adjacency connections so graph processing focuses on the most relevant question-specific image relationships.The neighborhood of each node is formed by ranking its adjacency values and selecting the strongest connections.
- Spatial graph convolutions: Spatial graph convolutions aggregate neighboring object features using Gaussian-kernel weights based on pairwise pseudo-coordinates such as relative direction and distance.The patch operator produces weighted sums over irregular neighborhoods, with kernels learning how neighboring features influence each convolution.
- Spatial graph convolutions: Adjacency-derived softmax weights further modulate message passing, enabling pairwise visual attention conditioned on the question.The factor αij scales messages between connected vertices according to selected adjacency values.
- Prediction: After L graph-convolution layers, max pooling creates a permutation-invariant graph vector that is multiplied element-wise with the question embedding for MLP-based answer classification.The task uses sigmoid outputs with soft target scores and a multi-label binary cross-entropy loss.
4 Evaluation
The evaluation examines parameter sensitivity, test-set performance against baselines and prior methods, and interpretability through question-specific learned graph structures.
- Experimental setup: The model is evaluated on VQA 2.0 using 3000 possible answer classes and object-detection features from 36 bounding boxes per image.Questions use 300-dimensional pre-trained GloVe embeddings, while each detected box has a 2048-dimensional feature vector.
- Parameter exploration: m = 16 and K = 8 are optimal parameters for the model.Performance drops for m < 16, remains stable for m > 16, and K = 8 is especially optimal for number questions.
- Test-set comparison: The test-set comparison includes ReasonNet, Bottom-Up, Counting module, Attention, and a k-nearest-neighbour graph baseline.The comparison also tests the importance of learning the graph structure by using a kNN graph based on distances between bounding-box centres.
- Test-set results: The model performs close to state of the art and substantially improves on numeric questions except against the counting-specialized method.Bottom-Up and Counting module use variable numbers of detected objects, whereas this method uses a fixed number.
- Interpretability: Learned graph nodes indicate where the network is looking, while edge weights indicate relationships considered relevant to answering the question.These structures support inspection of whether the model focused on appropriate objects and relationships.
- Interpretability: Failure examples show that incorrect focus can result from missing detected objects, while some cases are poorly suited to bounding-box-based models.The examples include a wrong object due to missing purse or face detections and a scene that may not fit bounding-box representations.
5 Discussion
The paper presents a question-conditioned graph representation for VQA and finds that learned graph structures are relevant and interpretable. It also identifies limitations arising from graph parameterization, object detection, fixed object counts, and the VQA dataset and classification formulation.
- The model learns a question-conditioned graph representation and uses graph convolutions to derive image features from relevant neighbours.The learned structure is intended to capture question-specific interactions between detected objects.
- Scalar edge weights may not capture the full complexity of relationships between graph items; vector edges are proposed as a possible improvement.The paper suggests implementing this as an adjacency matrix per convolutional kernel.
- The approach depends heavily on object-detector quality, which can produce duplicate or missing objects and thereby affect model performance.The fixed number of detected objects per image can further enhance this problem.
- 66.18% accuracy is reported on VQA v2, while the paper describes performance on the dataset as still rather limited.The authors link this limitation partly to subjective questions and answers absent from the predefined classification outputs.
A Appendix
The appendix illustrates how learned graph structures support relational question answering and expose failure modes. Successful examples use relations among objects, whereas unsuccessful cases often reflect classification limits or missing and uneven object detections.
- Graph structures support questions requiring relational comparisons, such as checking sheep colours or identifying which woman wears a black top.These examples show the graph comparing multiple relevant objects for one question.
- The classification setting can limit performance when the correct explanation is not among the possible answer outputs.One example identifies a black sheep but cannot provide the requested explanation because that answer is unavailable.
- Missing detections can cause failures, including overlooking an out-of-focus mouse or a mostly hidden person.The detector may select keyboard keys instead of the mouse and omit the background person.
- The model can count visible people yet fail to represent the question’s relational condition, such as whether anyone is on a bridge.Another example shows that uneven bounding-box detections can omit a relevant player from the count.