Source-linked AI summary
Learning to Compose and Reason with Language Tree Structures for Visual Grounding
Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, Hanwang Zhang
TL;DR
Grounding natural language in images requires handling fine-grained, compositional language, whereas existing approaches rely on holistic language–visual associations. RVG-TREE recursively composes a binary language tree and performs bottom-up visual reasoning, achieving state-of-the-art performance with more explainable reasoning.
Problem
Natural-language grounding must localize image regions from fine-grained, compositional descriptions, but existing solutions neglect the compositional reasoning implied by language.
Method
RVG-TREE recursively parses language into a binary tree, accumulates grounding scores from child subtrees, and uses Gumbel-Softmax to train through discrete decisions.
Results
RVG-TREE achieves state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg, with a transparent visual reasoning process.
Takeaways & Limitations
The model makes complex language composition more compositional and explainable by decomposing it into easier sub-grounding tasks and exposing intermediate grounding results.
Takeaways & Limitations
RVG-TREE does not fully exploit linguistic prior knowledge, and constituency-parser priors are helpful but insufficient.
Abstract
from arXiv · showhide
Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained and compositional language space. However, existing solutions rely on the association between the holistic language features and visual features, while neglect the nature of compositional reasoning implied in the language. In this paper, we propose a natural language grounding model that can automatically compose a binary tree structure for parsing the language and then perform visual reasoning along the tree in a bottom-up fashion. We call our model RVG-TREE: Recursive Grounding Tree, which is inspired by the intuition that any language expression can be recursively decomposed into two constituent parts, and the grounding confidence score can be recursively accumulated by calculating their grounding scores returned by sub-trees. RVG-TREE can be trained end-to-end by using the Straight-Through Gumbel-Softmax estimator that allows the gradients from the continuous score functions passing through the discrete tree construction. Experiments on several benchmarks show that our model achieves the state-of-the-art performance with more explainable reasoning.
1 INTRODUCTION
Natural-language grounding must distinguish a referent from contextual objects using fine-grained compositional reasoning. RVG-TREE recursively builds a binary language tree, accumulates grounding confidence bottom-up, and remains differentiable for end-to-end training.
- Motivation: Grounding natural-language instructions in images extends object detection from fixed vocabularies to open-vocabulary sentences.The task supports applications including visual question answering, visual dialog, and robotic navigation.
- Motivation: Grounding expressions such as “black dog” requires using attributes and spatial relations to distinguish the referent from same-category contextual objects.The example distinguishes a black dog from a golden dog using “black” and “left of the tree.”
- Motivation: Existing compositional models use subject-predicate-object triplets, but this representation is too coarse for longer sentences with adjective clauses.The paper contrasts “person riding bike” with a longer expression involving a dog, tree, and comparative clause.
- RVG-TREE: RVG-TREE recursively decomposes language into semantic constituents and accumulates grounding confidence from simpler lower-layer sub-tasks.For example, “black dog” separates “black” and “dog,” while “on the left of the tree” is decomposed further.
- RVG-TREE: The model dynamically merges adjacent nodes into a binary tree, classifies nodes as score or feature nodes, and performs bottom-up visual reasoning.Score nodes return recursive grounding scores, whereas feature nodes provide visual features for higher-level computation.
- Training and evaluation: RVG-TREE uses Gumbel-Softmax to pass gradients through discrete node merging and score-feature decisions, enabling standard SGD training.Experiments on RefCOCO, RefCOCO+, and RefCOCOg report comparative or better performance with transparent visual reasoning.
2 RELATED WORK
Related work grounds expressions with joint embeddings, generation models, or modular compositions, while language research learns latent tree structures. RVG-TREE applies learned binary trees directly to visual grounding for bottom-up reasoning.
- Grounding methods: Referring-expression grounding localizes an image region from a natural-language description and is also called referring expression comprehension.Its inverse task is referring expression generation.
- Grounding methods: Earlier grounding methods use joint embeddings or CNN-LSTM speaker models, while later systems combine these perspectives with speaker-listener-reinforcer architectures.These approaches model conditional relationships between visual objects and referring expressions.
- Grounding methods: Modular approaches decompose expressions into subject appearance, location, and relationships, then combine module scores using language-dependent weights.Visual attention helps subject and relationship modules focus on relevant image regions.
- Latent tree structures: Natural language has latent hierarchical structure, motivating RVG-TREE’s automatic binary-tree composition and bottom-up accumulation of grounding confidence.The model explicitly incorporates hierarchical structure into visual reasoning.
- Latent tree structures: Prior latent-tree methods include supervised shift-reduce parsing, chart parsing, and Gumbel Tree-LSTM with learned greedy merges.Chart parsing can incur O(n^3) time and space complexity, while Gumbel Tree-LSTM uses Straight-Through Gumbel-Softmax for end-to-end training.
- Vision-language tree methods: Vision-language research has used dependency trees as structural losses and learned layouts for visual question answering, whereas RVG-TREE performs grounding along its learned tree.The cited VQA systems explore latent structure in input questions.
3 RVG-TREE MODEL
The RVG-TREE model is introduced by formally defining natural-language grounding, presenting a walkthrough of the model, and explaining end-to-end neural-network training.
- The model section first formalizes natural-language grounding, then presents RVG-TREE through a walkthrough, and finally explains end-to-end training.
3.1 Problem Definition
The task is to select the image region that best matches a natural-language sentence while exploiting its compositional referent and context. Existing holistic, phrase-level, and triplet-based approaches face difficulties with complex language composition and weakly supervised context localization.
- Task formulation: Grounding selects the ROI maximizing its association score with the input language sentence.The image is represented as ROI features, and the language as a sequence of word embeddings.
- Motivation: Holistic and phrase-level grounding models are suboptimal because they do not fully exploit compositional language structures.The challenge is distinguishing the referent from contextual objects, including objects of the same category.
- Prior formulation: Triplet composition decomposes grounding into referent, context, and relationship subscores, but accurately obtaining these components remains difficult.Its score combines Ss(xi, ys), Sv(xv, yv), and Sp([xi, xv], yp).
- Challenges: Complex expressions expose errors because parser-generated compositions may be linguistically valid yet visually difficult to learn.The paper motivates finer-grained decomposition for expressions containing nested attributes and relationships.
- Challenges: Context localization is weakly supervised because annotating both referent and context is prohibitively expensive, while context may combine multiple regions.The paper identifies representing contexts such as “black and white” or nested spatial descriptions as unresolved.
3.2 RVG-TREE Construction
RVG-TREE constructs a binary language tree bottom-up by recursively scoring and merging adjacent nodes. The discrete greedy merge decisions are addressed with Gumbel-Softmax during training.
- Design motivation: RVG-TREE recursively decomposes language into binary components to support fine-grained visual reasoning.The design is motivated by recursively parsing clauses into pairs such as subject-object, attribute-subject, or preposition-subject.
- Initialization: The model prunes selected determiners and symbols before embedding words as leaf nodes and constructing the tree.The paper reports that pruning does not affect overall performance while increasing speed.
- Bottom-up merging: At each layer, RVG-TREE assigns validity scores to candidate parents and merges the adjacent pair with the largest normalized score.Merged and unmerged nodes continue into the next layer until only the root remains.
- Node representation: Node representations are updated with BiLSTM-derived leaf features and TreeLSTM states as adjacent nodes are merged.TreeLSTM concatenates the children’s hidden states as input hidden states.
- Training: Greedy argmax merging is nondifferentiable, so the model uses the Gumbel-Softmax trick to handle the discrete tree-construction process.The node embedding functions themselves are differentiable.
3.3 Recursive Grounding
RVG-TREE performs grounding recursively over the constructed tree, separating feature propagation from score accumulation. This yields hierarchical reasoning while allowing language components without direct visual referents to contribute appropriately.
- Recursive grounding: Each node combines a score computed locally with the grounding score returned by its score child.The feature child supplies a visual feature, while the score child supplies a recursively accumulated score.
- Recursive grounding: Recursive grounding performs a divide-and-conquer computation in a bottom-up fashion, with O(2N) score calculations for sentence length N.The paper describes this as linear overhead relative to sentence length, though more expensive than holistic or simple compositional methods.
- Score and feature nodes: A score node passes its grounding score upward, whereas a feature node produces a score-weighted visual-region feature for its parent.Every root is a score node, and feature-node weights are normalized by grounding scores.
- Score and feature nodes: Feature nodes tend to contain visual concepts such as colors, while score nodes tend to contain non-visual relationship words.Word-cloud analysis reports that adjectives such as colors are more likely to be feature nodes, whereas relationships such as “behind” and “sitting” are more likely to be score nodes.
- Language feature: Language features use word-level attention to suppress irrelevant words when associating language with individual regions or region pairs.The model uses ys for single-region association and yp for pairwise relationship association.
- Leaf case: At the leaf boundary, the recursive score for an empty sentence is defined as zero.This exit condition prevents recursion from requiring a grounding score for an absent language subsequence.
3.4 RVG-TREE Training
RVG-TREE training combines supervised region discrimination with Straight-Through Gumbel-Softmax to optimize discrete tree construction and node-role decisions. Supervised pre-training provides a stable starting point, while fine-tuning produces more meaningful binary structures.
- Training Objective: The model trains to assign high grounding scores to the ground-truth region and low scores to other regions.Its cross-entropy objective is also described as Maximum Mutual Information training.
- Training Objective: The overall objective uses cross-entropy because alternative large-margin triplet training yielded only marginal gains and required trickier learning-rate adjustment.The authors therefore retain Eq. (15) as the training objective.
- Differentiable Training: Straight-Through Gumbel-Softmax bridges gradients through discrete node merging and score-versus-feature classification decisions.Forward propagation uses argmax samples, while backward propagation uses continuous values; test-time noise is disabled.
- Differentiable Training: Gumbel noise perturbs policy logits, and the temperature parameter gradually produces samples resembling one-hot selections.This supports end-to-end optimization despite discrete decisions.
- Supervised Pre-Training: Because training all tree and node-selection parameters from scratch can be unstable, the model first uses supervised pre-training before fine-tuning.The pre-training stage is intended to provide a fair initial solution for subsequent optimization.
- Supervised Pre-Training: After fine-tuning, the learned tree can organize the referent and context more meaningfully than the expert layout.For example, “human arm” is separated as the referent while its relational description remains context for further parsing.
4 EXPERIMENTS
Experiments evaluate RVG-TREE through ablations, comparisons on three referring-expression benchmarks, qualitative visualizations, and detected-region tests. Results support recursive compositional reasoning and transparent intermediate grounding, while exposing failures on nuanced relationships and imperfect action recognition.
- Experimental setup: Experiments use RefCOCO, RefCOCO+, and RefCOCOg, with analyses covering tree structure, recursive scores, explainability, and detected regions.The datasets contain referring-expression grounding benchmarks; RefCOCOg has richer language and is more challenging.
- Ablative studies: RVG-TREE-Fix outperforms Chain on all datasets, but limited gains reflect its continued reliance on a holistic root embedding.The result motivates explicitly exploiting tree structure during visual reasoning.
- Ablative studies: Removing the in-node score makes RVG-TREE/Node significantly worse because that score corrects child confidence and provides a joint linguistic view.The ablation tests the complementary role of in-node scoring in bottom-up accumulation.
- Ablative studies: Without score accumulation, RVG-TREE/S performs worse, showing that compositional visual reasoning is crucial for referring-expression grounding.The comparison isolates recursive grounding-score accumulation.
- Ablative studies: Removing pairwise visual-feature scores makes RVG-TREE/F inferior, especially on longer RefCOCOg expressions, indicating that pairwise relationships help distinguish referents from context.The reported comparison concerns RVG-TREE and its nonpairwise counterpart.
- Comparison with state-of-the-art: RVG-TREE achieves state-of-the-art performance across the three datasets and outperforms both sentence-generation-comprehension and triplet compositional models.The authors attribute the improvement over triplet models to recursively applying triplet-like grounding scores along the tree.
- Explainability: Qualitative visualizations show reasonable intermediate maps: contextual subtrees can score multiple candidates while relational subtrees identify the final referent.In the “baseball player swinging bat at baseball” example, the relational subtree helps pinpoint the correct person.
- Results on detected regions: Detected bounding boxes reduce all methods’ performance, but RVG-TREE remains better than compositional baselines, demonstrating robustness to noisy visual regions.Methods without compositional reasoning experience the largest drops.
5 CONCLUSIONS
RVG-TREE localizes target regions by recursively accumulating vision-language grounding scores through a learned binary tree. Experiments on three referring-expression grounding benchmarks evaluate its reasoning, while the authors identify incomplete use of linguistic prior knowledge as its key limitation.
- RVG-TREE localizes target regions by recursively accumulating vision-language grounding scores.
- RVG-TREE learns a binary tree whose root grounding score accumulates confidence from two child sub-trees through a fully differentiable process.
- Experiments include ablative, quantitative, and qualitative evaluations on three benchmark datasets for referring expression grounding.
- The key limitation is that RVG-TREE does not fully exploit linguistic prior knowledge, although constituency-parser priors can improve performance.