Source-linked AI summary
Dynamic Graph Attention for Referring Expression Comprehension
Sibei Yang, Guanbin Li, Yizhou Yu
TL;DR
Referring expression comprehension must connect linguistic structure with relationships among image objects, but prior methods often isolate objects or support limited reasoning. DGA predicts a language-guided sequence of reasoning steps and executes it on a multi-modal relation graph. Across common benchmarks, it outperforms existing state-of-the-art methods while producing interpretable visual evidence for complex descriptions.
Problem
Existing approaches often ignore object relationships or linguistic structure and therefore provide limited support for complex referring expressions.
Method
DGA predicts constituent-expression reasoning steps with a differential analyzer and performs multi-step attention-based reasoning over an object-relation graph.
Results
DGA outperforms existing state-of-the-art methods across common benchmark datasets and generates visualizable, interpretable evidence for stepwise grounding.
Takeaways & Limitations
Language-driven graph reasoning provides a way to identify compound objects step by step while exposing visual evidence for the decision process.
Takeaways & Limitations
The method uses predicted constituent expressions and, on the evaluated datasets, extra reasoning steps can introduce noise.
Abstract
from arXiv · showhide
Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.
1. Introduction
Referring expression comprehension requires locating image objects through language-guided visual reasoning, especially when expressions describe objects through multiple relationships. DGA addresses this by predicting reasoning steps from linguistic structure and executing them over an object-relation graph.
- Referring expression comprehension locates an image object from a natural-language description and requires joint understanding of language and visual content.
- The example “the umbrella held by the person in the pink hat” requires three reasoning steps from hat, to person, to umbrella.
- Existing approaches often omit reasoning or support only single-step reasoning, while learned monolithic features are less competitive for complex expressions.
- DGA predicts constituent expressions step by step with a differential analyzer to capture the linguistic structure guiding visual reasoning.
- DGA performs multi-step visual reasoning on a multi-modal relation graph and identifies compound objects through predicted constituent expressions.
- Experiments show DGA surpasses existing state-of-the-art algorithms and generates visualizable evidence for stepwise grounding of complex descriptions.
2. Related Work
Related approaches commonly underrepresent object relationships and linguistic dependencies, limiting their handling of complex or structurally varied expressions. DGA instead combines linguistic structure, visual-object graphs, and attention-based interactions for multi-step reasoning.
- Some methods encode language and vision separately or learn a shared feature space using matching losses such as softmax or triplet loss.
- Prior approaches represent images as independent objects or direct-relational compound objects and encode expressions sequentially, ignoring broader dependencies.
- Template-based modular methods decompose expressions into semantic components, but fixed templates do not cover unrestricted referring-expression forms.
- The proposed framework targets interpretable multi-step reasoning by identifying compound objects corresponding to partial referring expressions.
- DGA encodes dependencies in expressions and images by modeling visual objects as a graph and building cross-modal interactions through attention.
- MattNet decomposes expressions into subject, location, and relationship phrases but cannot perform multi-step reasoning.
3. Dynamic Graph Attention Network
DGA predicts a language-guided visual reasoning process and performs multi-step reasoning over an expression-conditioned graph of visual objects and their relationships.
- Architecture: DGA comprises differential analysis, static graph attention, dynamic graph attention, and matching modules for interpretable multi-step referring-expression reasoning.The differential analyzer predicts the reasoning process; static attention constructs the expression-conditioned graph; dynamic attention updates visual-object features during reasoning.
- Language-Guided Visual Reasoning: The language-guided process recurrently predicts a soft constituent expression for each reasoning step from the whole expression and previous-step output.Words are encoded with a bidirectional LSTM, while each step transforms the overall expression feature and combines it with the previous output.
- Graph Construction: The image graph represents visual objects as nodes and pairwise spatial relationships as directed edges, with node features combining visual and spatial information.Spatial features use normalized object coordinates, width, height, and area; relationships include containment, overlap, and directional relations.
- Static Attention: Expression-guided attention over graph nodes and edge types produces a multi-modal graph whose node features combine image-graph features with language representations.Word weights distinguish entity and relation terms, while attention distributions estimate word relevance to nodes and relationship types.
- Dynamic Attention: Dynamic graph attention uses the predicted reasoning process to attend to nodes and edges and update expression-related visual-object features step by step.At each step, the current constituent expression identifies compound objects corresponding to partial expressions.
- Matching: DGA trains with triplet loss and online hard-negative mining, then selects the proposal with the highest matching score during inference.The loss separates ground-truth and negative proposal scores by a margin.
4. Experiments
Experiments evaluate DGA on three referring-expression benchmarks using ground-truth and detected objects, compare it with existing methods, and analyze its reasoning behavior and components. DGA consistently improves benchmark accuracy and produces interpretable stepwise visual evidence, while ablations show the importance of relationships and appropriately selected reasoning depth.
- Evaluation and Implementation: DGA is evaluated on RefCOCO, RefCOCO+ and RefCOCOg using both ground-truth and automatically detected objects.Accuracy is correct top-object selection for ground-truth objects and requires IoU above 0.5 for detected objects.
- Comparison with the State of the Art: Using VGG-16 features, DGA improves average accuracy over the best existing approach by 2.00%, 3.25% and 2.86% on RefCOCO, RefCOCO+ and RefCOCOg, respectively.With ResNet-101 Faster R-CNN features, average accuracy across all splits increases by approximately 4.03%.
- Qualitative Evaluation: DGA visualizations expose word, node and relationship attention across iterative reasoning steps, together with final matching scores.These visualizations are used to inspect the language information and related objects attended to at each step.
- Qualitative Evaluation: In qualitative examples, DGA progressively combines attended objects into compound referents and assigns the highest matching score to the final compound object.The examples include both tree-structured and chain-structured reasoning processes.
- Ablation Study: Static DGA performs worse because it ignores object relationships and reasoning, while three reasoning steps outperform two and four-step reasoning.Two steps consider only direct relationships, whereas extra steps beyond three may introduce noise on these datasets.
5. Conclusion
The conclusion presents Dynamic Graph Attention Networks for referring expression comprehension through language-guided multi-step reasoning over object relationships. Experiments show that DGA outperforms existing state-of-the-art methods and generates visualizable, interpretable decision evidence.
- Conclusion: DGA performs multi-step reasoning over relationships among image objects, guided by the learned linguistic structure of the referring expression.The method addresses referring expression comprehension by linking language structure with visual-object relationships.
- Conclusion: Experiments on common benchmark datasets show that DGA outperforms existing state-of-the-art methods and produces visualizable, interpretable decision rules.The conclusion reports both performance gains and interpretable results.