Source-linked AI summary
Graph-Structured Referring Expression Reasoning in The Wild
Sibei Yang, Guanbin Li, Yizhou Yu
TL;DR
Grounding referring expressions requires reasoning over linguistic structure and diverse visual content, while many existing approaches neglect that structure or use limited forms of it. The paper proposes SGMN, which reasons over image and language graphs, and the Ref-Reasoning real-world dataset. SGMN outperforms existing methods on Ref-Reasoning and structured methods on common benchmarks.
Problem
Grounding requires jointly reasoning over semantically rich expressions and diverse visual content, while many existing methods neglect linguistic structure or explore it only in limited ways.
Source: §1 · p. 1Method
SGMN represents images as semantic graphs, parses expressions into language scene graphs, and uses neural modules to reason over the graphs under linguistic-structure guidance.
Source: §1 · p. 2Results
Takeaways & Limitations
SGMN provides interpretable visual evidence of its reasoning through a graph attention mechanism.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Grounding referring expressions aims to locate in an image an object referred to by a natural language expression. The linguistic structure of a referring expression provides a layout of reasoning over the visual contents, and it is often crucial to align and jointly understand the image and the referring expression. In this paper, we propose a scene graph guided modular network (SGMN), which performs reasoning over a semantic graph and a scene graph with neural modules under the guidance of the linguistic structure of the expression. In particular, we model the image as a structured semantic graph, and parse the expression into a language scene graph. The language scene graph not only decodes the linguistic structure of the expression, but also has a consistent representation with the image semantic graph. In addition to exploring structured solutions to grounding referring expressions, we also propose Ref-Reasoning, a large-scale real-world dataset for structured referring expression reasoning. We automatically generate referring expressions over the scene graphs of images using diverse expression templates and functional programs. This dataset is equipped with real-world visual contents as well as semantically rich expressions with different reasoning layouts. Experimental results show that our SGMN not only significantly outperforms existing state-of-the-art algorithms on the new Ref-Reasoning dataset, but also surpasses state-of-the-art structured methods on commonly used benchmark datasets. It can also provide interpretable visual evidences of reasoning. Data and code are available at https://github.com/sibeiyang/sgmn
1. Introduction
Grounding referring expressions requires reasoning across expression structure and visual objects, attributes, and relations. The paper proposes SGMN and the Ref-Reasoning dataset, reporting stronger benchmark results for SGMN than existing methods.
1. Introduction
- Grounding is challenging because it requires reasoning over semantically rich expressions and diverse visual content, including objects, attributes, and relations.Source: §1 · p. 1
- Existing methods often neglect linguistic structure or explore limited structures through self-attention, although expression structure provides a layout for reasoning over visual content.Source: §1 · p. 1
- SGMN models images as semantic graphs, parses expressions into language scene graphs, and uses neural modules to reason over the graphs under linguistic-structure guidance.Source: §1 · p. 2
- Ref-Reasoning is a large-scale real-world dataset with semantically rich expressions describing objects, attributes, and direct and indirect relations across varied reasoning layouts.Expressions are generated over image scene graphs using diverse templates and functional programs, with intermediate-step annotations and balancing by reasoning-step counts.Source: §1 · p. 2
- SGMN significantly surpasses existing state-of-the-art algorithms on Ref-Reasoning and outperforms state-of-the-art structured methods on common benchmarks, while providing interpretable visual evidence of reasoning.Source: §1 · p. 2
2. Related Work
Prior grounding methods vary in how they represent expression structure, while dataset work addresses bias and the limits of synthetic visual content. Ref-Reasoning generation uses real-world image scene graphs and structured expression templates.
2.1. Grounding Referring Expressions
- Earlier approaches often learn holistic representations [29]; component-based methods use fixed semantic components [9] [26], while DGA uses self-attention and neglects syntactic information [24].Source: §2.1 · p. 2
- Another approach grounds the referent with a parse tree whose nodes are noun, preposition, or verb words or phrases [3].Source: §2.1 · p. 2
2.2. Dataset Bias and Solutions
- Dataset bias can reward shallow correlations and expression-independent models, while CLEVR-Ref+ uses simple rendered 3D scenes that do not readily generalize to real-world scenes [4] [17].Source: §2.2 · pp. 2–3
- Ref-Reasoning generation uses scene graphs from Visual Genome [14] and GQA [10], with templates and functional programs to create expressions about objects, attributes, and relations.Source: §2.2 · p. 3
3. Approach
SGMN builds compatible graphs for the image and expression, then uses the expression’s structure to guide modular reasoning over visual content. Its modules combine node- and relation-level evidence to identify the referent.
3. Approach
- SGMN represents the image and expression as graphs, then performs structured reasoning over them with neural modules.The language scene graph guides reasoning over the image semantic graph.Source: §3 · p. 3
3.1. Scene Graph Representations
- The language scene graph defines the reasoning layout, and its nodes and edges correspond to subsets of those in the image semantic graph.Source: §3.1 · p. 3
3.1.1 Image Semantic Graph
- The image semantic graph is directed over visual objects; its object features include CNN appearance and normalized box geometry, while edge features combine relative spatial and visual information.Source: §3.1.1 · p. 3
3.1.2 Language Scene Graph
- An external parser converts the expression into a language scene graph whose noun-phrase nodes and relation edges represent objects and how they modify one another.Source: §3.1.2 · pp. 3–4
3.2. Structured Reasoning
- Reasoning follows the language graph’s structure, using AttendNode to match nodes and Merge, AttendRelation, and Transfer to process edge information.Source: §3.2 · p. 4
3.2.1 Reasoning Process
- Inference starts at the zero-out-degree referent node and traverses the reversed graph so modifying nodes are processed before the nodes they modify.Leaf-node attention uses appearance and spatial descriptions; intermediate-node attention combines evidence from connected nodes and edges.Source: §3.2.1 · p. 4
3.2.2 Neural Modules
- AttendNode and AttendRelation produce visual node and edge attention, while Transfer and Merge move or combine attention maps; Norm constrains their scale.Source: §3.2.2 · pp. 5–6
3.3. Loss Function
- Training applies cross-entropy to the final referent attention, and inference selects the object with the highest probability.Source: §3.3 · p. 6
4. Ref-Reasoning Dataset
Ref-Reasoning is generated from real-world scene graphs using templates and functional programs to create expressions with varied reasoning layouts. Its sampling and split construction balance examples by reasoning complexity and expression difficulty.
4. Ref-Reasoning Dataset
- Ref-Reasoning generates expressions over real-world image scene graphs using diverse templates.Source: §4 · p. 6
4.1. Preparation
- The dataset uses normalized Visual Genome scene graphs, with object attributes and semantic, spatial, and comparative relations; templates encode layouts as directed acyclic subgraphs and use one to five nodes.Source: §4.1 · p. 6
4.2. Generation Process
- For each image, generation samples a referent and scene-graph subgraph, selects a matching layout template, fills its parameters, and keeps expressions whose referent is unique.Source: §4.2 · p. 6
- Sampling downweights common node and relation classes and increases multi-order layouts; final splits are sampled by difficulty and number of described nodes.Difficulty is measured by the number of objects in the shortest sub-expression that identifies the referent.Source: §4.2 · pp. 6–7
5. Experiments
SGMN outperforms structured methods on the common benchmarks and leads all Ref-Reasoning test splits. The experiments also describe dataset and implementation choices, and show intermediate attention maps that make its reasoning steps interpretable.
5.1. Datasets
- Ref-Reasoning contains 791,956 expressions across 83,989 images, with 721,164, 36,183, and 34,609 expression-referent pairs for training, validation, and testing, respectively.Its expressions describe objects, attributes, direct and indirect relations, and different reasoning layouts.Source: §5.1 · p. 7
5.2. Implementation and Evaluation
- Accuracy is the fraction of correct referent predictions; image semantic graphs connect each object to its five nearest nodes by normalized center coordinates.Ref-Reasoning uses ResNet-101 Faster R-CNN features, while the common benchmarks follow CMRIN for object features; models use Adam with dataset-dependent learning rates.Source: §5.2 · p. 7
5.3. Comparison with the State of the Art
- CNN+LSTM reaches 75.29% accuracy on direct-description expressions, while DGA and CMRIN perform better on two-, three-, and four-node splits.The passage attributes the direct-description result to the absence of relation reasoning in that split, and the contextual methods' gains to language-guided object representations.Source: §5.3 · p. 7
- SGMN beats the best existing structured method by 0.92%, 2.54%, and 2.96% in average testing accuracy on RefCOCO, RefCOCO+, and RefCOCOg, respectively, and leads every Ref-Reasoning test split.Table 2 compares methods on the three common benchmarks; the Ref-Reasoning result holds across test splits requiring different numbers of reasoning steps.Source: §5.3 · pp. 7–8, Table 2 · p. 8
5.4. Qualitative Evaluation
- Attention maps at language-scene-graph nodes illustrate relational reasoning, including narrowing attention to a chair before identifying a blanket on it.The examples show intermediate attention changes along graph relations, not only a final referent prediction.Source: §5.4 · p. 8, Figure 3 · p. 8
5.5. Ablation Study
- The experiments include an ablation study on Ref-Reasoning, with the best-performing method marked in bold in Table 3.Source: Table 3 · p. 8
6. Conclusion
The paper presents SGMN, which performs graph-structured reasoning over image and expression representations, and proposes Ref-Reasoning for structured expression reasoning. SGMN surpasses prior methods in the reported settings and provides interpretable visual evidence through graph attention.
6. Conclusion
- SGMN performs graph-structured reasoning over constructed representations of the input image and expression using neural modules.Source: §6 · p. 8
- Ref-Reasoning is a large-scale real-world dataset for structured referring expression reasoning.Source: §6 · p. 9
- SGMN significantly outperforms state-of-the-art algorithms on Ref-Reasoning and surpasses state-of-the-art structured methods on common benchmarks, while graph attention provides interpretable visual evidence.Source: §6 · p. 9