Source-linked AI summary
Language-Conditioned Graph Networks for Relational Reasoning
Ronghang Hu, Anna Rohrbach, Trevor Darrell, Kate Saenko
TL;DR
Grounded language tasks need relational reasoning, while prior systems often rely on local appearance features. The paper proposes LCGN, which builds language-conditioned contextual representations through iterative message passing, and reports consistent improvements across tasks and datasets, including state-of-the-art results on GQA and CLEVR-Ref+.
Problem
Grounded language reasoning requires modeling task-relevant relationships between objects, but prior methods often use scene representations based mainly on local appearance features.
Method
LCGN augments each scene entity with language-conditioned relational context through iterative message passing and dynamically weighted graph edges.
Results
LCGN consistently improves over local appearance representations across tasks and datasets, achieving state-of-the-art results on GQA and CLEVR-Ref+.
Takeaways & Limitations
Context-aware representations can support complex relational reasoning through simple task-specific approaches across VQA and REF.
Takeaways & Limitations
GQA test-dev and test splits differ from train and val, creating a noticeable domain shift from validation to test evaluation.
Abstract
from arXiv · showhide
Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets. Our code is available at http://ronghanghu.com/lcgn.
1. Introduction
Grounded language tasks require locating relevant objects and reasoning over their relationships, but prior methods often pair complex inference with weak local scene representations. LCGN instead builds language-conditioned contextual representations and improves results across tasks and datasets.
- Grounded VQA and REF require identifying relevant objects and modeling relationships specified by language.
- Prior approaches support complex relational inference but rely on local appearance features with limited contextual information.
- LCGN augments each entity’s local appearance with relational context through iterative, language-conditioned message passing and dynamic edge weighting.
- The resulting representation can replace local features and support simple task-specific prediction, such as a single attention hop over the relevant object.
- LCGN focuses representation building on relationships relevant to the input task rather than encoding every possible first- and higher-order relation.
- LCGN improves over local appearance features across VQA and REF tasks and datasets, achieving state-of-the-art results on GQA and CLEVR-Ref+.
2. Related work
The paper situates LCGN among visual reasoning and graph-based representation methods for VQA and REF. Its distinguishing goal is a general representation approach that dynamically conditions message passing on language and applies across both tasks.
- VQA answers image questions, whereas REF localizes an image region matching a referring expression; both require reasoning about image content.
- LCGN is applied uniformly to VQA and REF across three datasets, demonstrating the intended generality of its contextualized object and region representations.
- Graph networks support relational inference by passing messages between image regions, with applications including detection, classification, interaction, and activity recognition.
- Unlike fixed question-conditioned graph connections, LCGN predicts dynamic edge weights at each message-passing iteration.
- Compared with approaches tailored to one task or lacking iterative higher-order relation gathering, LCGN offers a general scene representation for both VQA and REF.
- LCGN complements rather than replaces existing reasoning models by supplying contextualized visual representations in place of local features.
3. Language-Conditioned Graph Networks
LCGN encodes language-conditioned relational context into scene entities through iterative graph message passing. These contextualized representations then feed simple task-specific modules for VQA and REF.
- Context-aware scene representation: LCGN constructs contextualized representations containing relational information required by the reasoning procedure specified in the language input.
- Context-aware scene representation: Each scene entity begins with a local visual feature, and the model produces a context-aware output representation for later VQA or REF modules.
- Language-conditioned message passing: The model performs iterative message passing over T iterations, conditioning each entity’s relational context on the textual input.
- Language-conditioned message passing: LCGN uses a fully connected directed graph in which every entity can send information to every other entity.
- Textual command extraction: At each iteration, textual attention extracts a command vector from the input sequence to condition message passing.
- Language-conditioned message passing: For each receiver, softmax-normalized edge weights select and weight sender contributions conditioned on the iteration’s textual command.
- Language-conditioned message passing: Nodes send command-conditioned messages, aggregate incoming messages, and update their contextual representations before forming final outputs.
- Efficient implementation: The implementation avoids explicit N^2 pairwise computation through matrix multiplication and supports training with N as large as 196.
4. Experiments
Experiments evaluate LCGN on VQA and REF tasks requiring complex relational reasoning. Across feature types, datasets, and tasks, contextualized representations consistently improve simple task-specific models, with state-of-the-art results reported on GQA and CLEVR-Ref+.
- Experimental setup: LCGN is evaluated on GQA and CLEVR for VQA, and CLEVR-Ref+ for REF, including questions and expressions with higher-order relations.The evaluated tasks use language inputs and require resolving relations between objects.
- GQA VQA: Around 2% absolute accuracy improvement is obtained on GQA when single-hop prediction uses LCGN contextualized features instead of local features.The comparison uses released object-detection features and contrasts “single-hop + LCGN” with “single-hop.”
- GQA VQA: LCGN delivers consistent improvements across convolutional grid, object detection, and symbolic ground-truth local feature representations on GQA.The ablation compares three types of local feature representations, with symbolic representations based on ground-truth objects and attributes.
- Ablations: Multiple message-passing rounds, text-conditioned updates, and dynamic connection weights each support performance in the reported ablations.Using static connection weights causes a performance drop, with a similar but larger drop observed for REF.
- REF: Approximately 13% absolute improvement in REF accuracy is obtained on CLEVR-Ref+ with LCGN, achieving state-of-the-art performance.The REF model localizes a target object from candidate entities using a referring expression.
5. Conclusion
LCGN constructs language-conditioned, context-aware scene representations through iterative message passing for VQA and REF. Examples show relational information propagating across objects before simple task-specific outputs select answers or regions.
- LCGN is a generic approach for language-based reasoning tasks such as VQA and REF.
- The model builds rich context-aware scene representations through iterative message passing instead of task-specific inference procedures.
- On GQA, message-passing edges connect relevant objects across iterations before a single-hop classifier selects the attended object.
- On CLEVR, iterative propagation supports the question asking for the matte ball matching the gray metal ball's size, yielding the yellow prediction.
- On CLEVR-Ref+, the model propagates messages across four iterations and produces a bounding-box output for the selected target location.
A. Implementation details
The implementation uses 512-dimensional textual and contextual representations, dataset-specific visual preprocessing, Adam optimization, and spatial encodings for relational reasoning.
- Textual vectors and context features use dimensionality 512, while GQA local features are reduced to 512 dimensions with a linear layer.
- GQA training uses sigmoid cross entropy, Adam, batch size 128, and learning rate 3 × 10^-4.
- CLEVR and CLEVR-Ref+ use a two-layer convolutional network producing 14 × 14 × 512 feature maps and Adam with batch size 64.
- CLEVR uses softmax answer classification, while CLEVR-Ref+ uses softmax training for target selection and bounding-box offsets.
- Spatial information is added through object bounding-box coordinates or sinusoidal positional encodings for convolutional grid locations.
- All model parameters are trained on a single Titan Xp GPU, with parameter shapes listed in Table A.1.
B. Quantitative analysis on edge weights
The learned edge weights vary substantially with the question, indicating that LCGN selects different message-passing connections for different reasoning contexts.
- Across ten questions for the same CLEVR image, each receiver node connects to an average of 6.396 unique max-connected sender nodes.
- The learned edge weights are largely dependent on the input questions rather than fixed solely by the image.
C. Additional visualization examples
Additional visualizations illustrate LCGN message passing and task-specific outputs across GQA, CLEVR, and CLEVR-Ref+. They also expose ambiguity-related failures in GQA examples.
- The appendix presents additional LCGN visualizations for VQA on GQA and CLEVR and REF on CLEVR-Ref+.
- GQA VQA: In GQA examples, red lines mark message-passing edges whose connection weights exceed a threshold, with blue stars identifying sender nodes.
- Failure examples: Two GQA examples are failures, including cases attributed to answer ambiguity such as “woman” versus “lady.”
- CLEVR VQA: CLEVR visualizations show connection edge weights across message-passing steps and the highest-attention feature-grid location selected by the single-hop VQA classifier.
- CLEVR-Ref+ REF: CLEVR-Ref+ visualizations show message-passing edge weights alongside the selected target grid location, ground-truth box, and predicted bounding box.