PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Conditioned Graph Networks for Relational Reasoning

Ronghang Hu, Anna Rohrbach, Trevor Darrell, Kate Saenko

arXiv:1905.04405v2cs.CV

TL;DR

Grounded language tasks require identifying relevant objects and their relationships, yet prior methods often represent scenes with context-poor local features. LCGN builds language-conditioned contextual object representations through iterative graph message passing. It consistently improves performance across tasks and datasets, reaches state-of-the-art on GQA and CLEVR-Ref+, and yields around 2% absolute accuracy improvement in the reported single-hop comparison.

  • Problem

    Grounded language tasks require locating relevant scene objects and reasoning about their relationships, while prior methods often use local appearance features with little context.

    Source: §1 · p. 1

  • Method

    LCGN builds task-conditioned contextual representations for scene entities through iterative language-conditioned message passing, dynamically weighting which objects exchange relational information.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    LCGN consistently improves over local appearance features across tasks and datasets and reaches state-of-the-art on GQA and CLEVR-Ref+; in the reported single-hop comparison, it adds around 2% (absolute) improvement in accuracy.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    The authors conclude that message passing needs to depend on language inputs and that contextualized representations improve over local appearance features across multiple datasets.

    Source: §5 · p. 7

    The GQA test-dev and test splits were collected differently from train and val and have a noticeable domain shift.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets. Our code is available at http://ronghanghu.com/lcgn.

1. Introduction

Grounded language tasks require locating relevant objects and reasoning about their relationships. LCGN builds language-conditioned contextual object representations through iterative graph message passing, supporting relational reasoning across tasks.

1. Introduction

  • Grounded language comprehension requires locating relevant scene objects and reasoning about specified relationships between them.The passage illustrates this with a question about whether a person is left of a woman holding a blue umbrella.Source: §1 · p. 1
  • Prior models can perform complex relational inference, but often rely on local appearance features and manually designed inference structures or modules, making them task-specific.The paper identifies this limitation as motivation for a context-aware scene representation suitable for multiple tasks.Source: §1 · p. 1
  • LCGN augments each scene entity’s local appearance with relational context gathered through multiple rounds of dynamically weighted message passing conditioned on the input text.The resulting contextualized features can replace local appearance features in task-specific models; VQA can use a single attention hop over the relevant object.Source: §1 · pp. 1–2
  • Conditioning representations on the input text focuses each object’s context on task-relevant relations instead of constructing potentially enormous, unnecessary relational representations.An object can participate in multiple relations in different contexts.Source: §1 · p. 2
  • LCGN achieves consistent improvement over local appearance features across VQA and REF tasks and datasets, with state-of-the-art results on GQA and CLEVR-Ref+.The paper applies LCGN to Visual Question Answering and Referring Expression Comprehension.Source: §1 · p. 2

2. Related work

The section surveys VQA and REF benchmarks, graph-based contextual representations, and other reasoning models. It presents LCGN as a shared approach for both tasks and distinguishes its iterative, dynamic message passing from prior methods.

Visual question answering (VQA) and referring expression comprehension (REF)

  • Real-world VQA emphasizes perception, CLEVR benchmarks relational reasoning, and GQA combines real images with relational questions; CLEVR-Ref+ targets complex REF queries.RefCOCO and RefCOCOg are standard REF benchmarks, but many expressions in them do not require resolving relations.Source: §2 · p. 2
  • LCGN uses the same approach to build contextualized object or image-region representations for VQA and REF across three datasets.The authors present this as evidence of the approach’s generality for visual reasoning tasks.Source: §2 · p. 2

Graph networks and contextualized representations

  • Graph networks use message passing to contextualize image regions, and prior VQA work includes question-conditioned Relation Networks, graph convolutions, and structured attention.For REF, Language-guided Graph Attention Networks guide node and edge attention using subject, intra-class, and inter-class relations [40].Source: §2 · p. 2
  • Unlike [30]’s fixed sparse graph, LCGN predicts dynamic edge weights at each iteration; unlike [5], it avoids max-pooling over explicit pairwise vectors.The normalized edge weights improve message-passing efficiency and visualization, while [40] does not gather higher-order relational information iteratively; LCGN is designed for both VQA and REF.Source: §2 · pp. 2–3

Reasoning models

  • Other reasoning approaches include neural modules, memory-based reasoning, feature modulation, cross-modal attention, neural-symbolic programs, and modular REF attention.LCGN is intended to complement, not replace, these models: its contextualized visual representation can replace local features, and the authors compare with Stack-NMN [15] in Section 4.Source: §2 · p. 3

3. Language-Conditioned Graph Networks

LCGN constructs language-conditioned, context-aware representations of scene entities through iterative message passing. These representations support relational reasoning with simple task-specific output modules.

3. Language-Conditioned Graph Networks

  • LCGN builds context-aware representations for scene entities by iteratively passing messages conditioned on language, then feeds them to simple task-specific output modules.Figure 2 depicts each object sending messages to other objects to update context features before they are combined with local features for VQA or REF modules.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Context-aware scene representation

  • The scene is represented as a fully connected directed graph whose entity nodes receive context-aware representations through iterative message passing.Entities may be detected objects or spatial locations on the convolutional feature map.Source: §3.1 · p. 3, Figure 2 · p. 4
  • At each iteration, a BiLSTM-based textual attention mechanism extracts a command that conditions directed sender-to-receiver weights normalized over senders with softmax.The model uses a separate learned weight matrix for each iteration when computing textual commands.Source: §3.1 · pp. 3–4

3.2. Application to VQA and REF

  • For VQA, a single-hop attention classifier pools entities to classify answers; for REF, GroundeR-style retrieval scores candidates and, with grid locations, predicts box offsets; modules are trained end-to-end.REF training uses a softmax loss over candidate scores and an L2 loss for the box offset.Source: §3.2 · pp. 4–5

4. Experiments

The experiments evaluate LCGN for VQA on GQA and CLEVR and for referring expression comprehension on CLEVR-Ref+, datasets involving object relationships and relational expressions.

4.1. Visual Question Answering (VQA)

  • Around 2% absolute accuracy improvement over single-hop and performance exceeding MAC yield state-of-the-art results on GQA.The comparison uses released object detection features as local features.Source: Table 1 · p. 5, §4.1 · pp. 5–6
  • LCGN consistently improves performance over convolutional grid, object detection, and ground-truth object-and-attribute local features on GQA.The ground-truth symbolic-feature setting is evaluated only on the validation split with public scene-graph annotations.Source: §4.1 · p. 6, Table 2 · p. 6
  • 97.9% accuracy with single-hop + LCGN compares with 72.6% for single-hop on CLEVR, and is comparable to several state-of-the-art approaches.The authors attribute the gain to LCGN embedding relational context in its output scene representation.Source: Table 3 · p. 6, §4.1 · p. 6
  • On CLEVR, multiple message-passing rounds and text-conditioned messages improve performance, while static connection weights underperform dynamic weights.The authors report that T > 1 helps and that conditioning message passing on the input text is crucial; static weights predicted in the first round cause a performance drop.Source: §4.1 · pp. 6–7, Table 4 · p. 7

4.2. Referring Expression Comprehension (REF)

  • On CLEVR-Ref+, LCGN’s context-aware scene representation yields approximately 13% absolute improvement in REF accuracy, and the model achieves state-of-the-art REF performance on that dataset.Evaluation uses bounding-box localization of the single target object; a prediction is correct at 50% IoU or higher with the ground-truth box.Source: §4.2 · p. 7

5. Conclusion

LCGN supports language-based reasoning by building context-aware scene representations through iterative message passing. The paper reports that these representations improve over local appearance-based features across multiple feature types and datasets, and that message passing must depend on language inputs.

5. Conclusion

  • LCGN’s context-aware representations improve over local appearance-based representations across feature types and datasets.The authors also report that message passing must depend on the language inputs.Source: §5 · p. 7
  • LCGN builds rich, context-aware scene representations through iterative message passing conditioned on language inputs.The authors present it as a generic approach to language-based reasoning tasks such as VQA and REF, rather than relying on task-specific inference procedures.Source: §5 · p. 7

A. Implementation details

The implementation maps textual and contextual representations to 512 dimensions and adapts local visual features to 512 dimensions, with dataset-specific training and spatial-feature choices.

A. Implementation details

  • On GQA, textual, contextual, and reduced local features use 512 dimensions; training uses sigmoid cross entropy, Adam, batch size 128, and learning rate 3 × 10−4.Local features are reduced with a single fully connected layer without non-linearity.Source: A. Implementation details · p. 11
  • On CLEVR and CLEVR-Ref+, a two-layer convolutional network maps ResNet-101-C4 features to 14 × 14 × 512; Adam uses batch size 64 and learning rate 10−4.CLEVR uses softmax answer-classification loss; CLEVR-Ref+ uses softmax target-location selection and L2 bounding-box-offset loss.Source: A. Implementation details · p. 11
  • Spatial cues are added by concatenating object bounding-box coordinates for GQA detection or ground-truth features, and sinusoidal grid-position encodings for convolutional features.The grid encodings are used on GQA, CLEVR, and CLEVR-Ref+; the object coordinates are used on GQA.Source: A. Implementation details · p. 11

B. Quantitative analysis on edge weights

The analysis finds that learned edge weights vary with the input question, measured by changes in each node’s max-connected sender across questions about the same CLEVR image.

B. Quantitative analysis on edge weights

  • 6.396 unique max-connected sender nodes per node on average across the 10 questions for each CLEVR image, indicating that learned edge weights largely depend on the input question.For each node, the analysis counts the distinct senders with the highest edge weight across the image’s 10 questions.Source: B. Quantitative analysis on edge weights · p. 11

C. Additional visualization examples

Additional visualizations show LCGN message-passing paths and predictions for VQA on GQA and CLEVR, and REF on CLEVR-Ref+.

C. Additional visualization examples

  • GQA VQA examples show objects receiving messages along high-weight edges, classifier attention over detected objects, and failures including ambiguous answers such as “woman” versus “lady.”The examples are from the validation split; the last two rows show failure cases.Source: C. Additional visualization examples · p. 11, Figure C.1 · p. 12
  • CLEVR VQA examples show relevant objects first propagating messages within convolutional grids, then collecting context from other objects, alongside two failure examples.The classifier attention is shown over a 14×14 feature grid.Source: C. Additional visualization examples · p. 11, Figure C.2 · p. 13
  • CLEVR-Ref+ REF examples show target objects collecting messages from context objects, with selected grid locations, ground-truth boxes, and predicted boxes visualized.The examples are from the validation split; the last two rows show failure cases.Source: C. Additional visualization examples · p. 11, Figure C.3 · p. 14

Related papers

Loading 1905.04405v2…