Source-linked AI summary
Modeling Relationships in Referential Expressions with Compositional Modular Networks
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, Kate Saenko
TL;DR
The paper addresses grounding referential expressions that describe relationships among multiple entities, beyond holistic localization or fixed relationship categories. It introduces CMNs, which jointly parse expressions and align their components with individual regions and region pairs. The model outperforms baselines and state-of-the-art methods across multiple datasets, including under weak supervision.
Problem
Grounding expressions that describe relationships requires identifying multiple entities and their correspondence to language, but prior methods either treat expressions holistically or use fixed categories.
Method
CMNs use end-to-end learned soft attention to parse subject, relationship, and object components, then apply localization and relationship modules to regions and region pairs.
Results
CMNs outperform natural baselines and state-of-the-art methods on multiple referential-expression datasets, including relationship localization under weak supervision.
Takeaways & Limitations
Explicit compositional parsing combined with local and pairwise visual modules supports grounding entities and relationships expressed in arbitrary language.
Takeaways & Limitations
The approach focuses on referential expressions involving inter-object relationships representable as a subject, relationship, and object, and its parsing can diverge from syntactic analyses.
Abstract
from arXiv · showhide
People often refer to entities in an image in terms of their relationships with other entities. For example, "the black cat sitting under the table" refers to both a "black cat" entity and its relationship with another "table" entity. Understanding these relationships is essential for interpreting and grounding such natural language expressions. Most prior work focuses on either grounding entire referential expressions holistically to one region, or localizing relationships based on a fixed set of categories. In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene. We call this approach Compositional Modular Networks (CMNs): a novel architecture that learns linguistic analysis and visual inference end-to-end. Our approach is built around two types of neural modules that inspect local regions and pairwise interactions between regions. We evaluate CMNs on multiple referential expression datasets, outperforming state-of-the-art approaches on all tasks.
1. Introduction
Referential expressions can identify entities through relationships, so grounding requires linking language components to multiple regions rather than treating the expression as a single description.
- Grounding arbitrary natural-language expressions remains more challenging than detecting entities from predefined categories.
- The localization module scores individual regions, while the relationship module scores region pairs whose integrated scores determine the grounding result.
- Expressions such as “the woman holding a grey umbrella” require locating a subject and verifying its relationship to another object.
- Prior approaches either ground expressions holistically or rely on fixed entity and relationship categories.
- CMNs jointly parse arbitrary expressions into subject, relationship, and object components while grounding them with unary and pairwise region modules.
2. Related work
Earlier grounding methods scored candidate regions from visual and contextual features, while other approaches used fixed relationship categories or external parsing; CMNs instead learn explicit language-to-region correspondences end-to-end.
- Candidate-region grounding methods score each proposed region against the query and return the highest-scoring candidate.
- Vectorizing an entire expression makes explicit correspondences between textual components and image entities difficult to learn.
- CMNs parse expressions into components and align them with image regions end-to-end instead of treating the expression as a whole.
- Fixed-category relationship detectors represent subjects, relations, and objects as predefined discrete classes rather than arbitrary language phrases.
- Neural Module Networks decompose questions into components but depend on an external language parser and are not directly designed for referential-expression grounding.
3. Our model
CMNs decompose referential expressions into subject, relationship, and object representations, then ground them through unary region matching and pairwise relationship reasoning. The model combines these modules with strong or weak supervision for end-to-end learning.
- 3. Our model: CMNs localize a referential expression compositionally by grounding its subject, relationship, and object components and exploiting their interactions.The model learns these components jointly rather than treating the expression as a single vector.
- 3. Our model: The model scores region pairs by combining subject, object, and relationship components for expressions such as “the red apple on top of the bookshelf.”The pair must match both entity descriptions and the spatial configuration described by the relationship.
- 3. Our model: For inference, the subject score is the best pairwise score over possible object regions, and the highest-scoring region is selected as the subject grounding.This lets the model use an unobserved interacting object when determining the subject region.
- 3.1. Expression parsing with attention: The expression parser uses soft attention over the word sequence to generate qsubj, qrel, and qobj, avoiding reliance on an external syntactic parser.A two-layer bidirectional LSTM supplies contextual word states, and three attention predictions produce the component representations.
- 3.2. Localization module: A localization module assigns unary scores to regions matching either the subject or object textual representation.It uses the subject representation qsubj or object representation qobj as the language input to the module.
- 3.3. Relationship module: A relationship module assigns pairwise scores to region pairs according to the relationship representation.The implementation combines the spatial features of the two regions and does not use their visual features because they provided no noticeable performance boost.
- 3.4. End-to-end learning: When both subject and object regions are annotated, training uses strong supervision on the pairwise score.The strong loss directly optimizes the ground-truth subject-object region pair.
4. Experiments
Experiments test CMNs on synthetic and real-world grounding tasks, including relationship localization, referential expression grounding, and visual question answering. Across these settings, the model benefits from inter-object relationships and learned expression parsing, even under weak supervision.
- Experimental settings: CMNs are evaluated on synthetic shapes, Visual Genome relationship expressions, Google-Ref, and Visual-7W pointing questions.The experiments cover both controlled spatial reasoning and real-image grounding tasks.
- 4.1. Analysis on a synthetic dataset: On the synthetic dataset, weakly supervised CMNs achieve nearly perfect subject-localization accuracy and outperform a localization-only baseline.The baseline identifies matching green squares but fails to select the square specified by its relation to a red circle.
- 4.2. Localizing relationships in Visual Genome: On Visual Genome, the full model outperforms localization-only baselines for both subject retrieval and subject-object pair retrieval.The weakly supervised model also outperforms a strongly supervised baseline when retrieving subject-object pairs.
- 4.2. Localizing relationships in Visual Genome: Weakly supervised Visual Genome models produce reasonable attention weights over subject, relationship, and object words.Figure 4 visualizes these attention weights alongside ground-truth and predicted region pairs.
- 4.3. Grounding referential expressions in images: On Google-Ref, CMNs outperform the localization-only baseline and previous state-of-the-art methods under the same weak supervision.Replacing learned parsing and language representation with an external parser causes a significant performance drop.
- 4.4. Answering pointing questions in Visual-7W: On Visual-7W pointing questions, the full model outperforms the baseline, the external-parser variant, and previous work.The task asks the model to select the region corresponding to relational questions such as which tomato slice is under the knife.
5. Conclusion
The paper proposes Compositional Modular Networks, an end-to-end model for referential expressions involving relationships. CMNs jointly parse expressions and ground their entities and relationships using local and pairwise region modules.
- CMNs are an end-to-end trainable model for handling relationships in referential expressions.
- The model parses input expressions with soft attention and jointly learns linguistic and visual analyses from weak supervision.
- CMNs use modules for both local region features and pairwise interactions between regions.
- Experimental results show that CMNs outperform natural baselines and state-of-the-art methods on multiple datasets.