Source-linked AI summary
Modeling Context in Referring Expressions
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, Tamara L. Berg
TL;DR
Referring-expression models must generate and comprehend language that identifies particular objects in images. The paper adds explicit visual comparisons among same-category objects and ties generation across those objects. Across RefCOCO, RefCOCO+, and RefCOCOg, these methods improve referring-expression generation and comprehension over previous methods.
Problem
The paper addresses generating and comprehending natural-language expressions that identify particular objects within images, especially when similar objects share a scene.
Method
The model explicitly encodes visual differences among same-category objects and ties language generation across all depicted objects of the same type.
Results
Visual comparison and joint language generation improve referring-expression generation and comprehension across RefCOCO, RefCOCO+, and RefCOCOg.
Takeaways & Limitations
Modeling detailed visual context and jointly generating expressions helps produce descriptions that discriminate target objects while remaining complementary across objects.
Takeaways & Limitations
The fully automatic system depends strongly on detector performance, especially for general objects in testB.
Abstract
from arXiv · showhide
Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets - RefCOCO, RefCOCO+, and RefCOCOg, shows the advantages of our methods for both referring expression generation and comprehension.
1 Introduction
The paper studies generating and comprehending referring expressions for objects in images, where useful expressions must be informative, succinct, and unambiguous. It improves prior models by explicitly comparing similar objects and jointly tying generation across objects of the same type.
- Referring expressions support dialogue by identifying particular objects for a co-observer, with applications to human-computer interaction and robots.
- Well-formed expressions should be informative, succinct, and unambiguous because scenes often contain multiple objects of the same category.
- Prior deep models used target-object, whole-image, and location features for referring expression generation and comprehension.
- The paper explicitly encodes visual comparisons among relevant objects to select distinctive attributes for unambiguous descriptions.
- Tied language generation across same-type objects produces differentiated but complementary expressions rather than identical descriptions.
- Evaluations on three datasets show that visual and linguistic comparisons improve performance over previous state of the art.
2 Related Work
Prior work established the roles of attributes, spatial relations, and large-scale datasets in referring expression modeling. This paper builds on that foundation with image features and comparisons among objects in the same perceptual group.
- Earlier referring-expression research was constrained by data collection and weaker computer-vision algorithms.
- The framework extracts VGG-fc7 and location features for same-type objects, computes visual differences, and feeds these signals into LSTMs.
- Prior studies motivate modeling attributes, relative relations, and spatial relationships among objects in the same perceptual group.
- Large-scale natural-image referring-expression datasets enable training deep-learning models.
3 Models
The models extend deep-learning baselines with explicit comparisons among objects and joint generation of expressions for same-category objects. These changes aim to produce expressions that distinguish targets while coordinating language across an image.
- 3.1 Baselines: The baseline models combine CNN features for the target object and image context with target location and size, then use an LSTM to generate expressions.The visual inputs include target-object features, global image features, and a 5-dimensional location/size representation.
- 3.1 Baselines: MMI training penalizes expressions that could also be generated for another object, encouraging unambiguous references.The paper implements both maximum-likelihood Baseline and maximum-mutual-information MMI models.
- 3.2 Visual Comparison: Explicit visual differences between same-category objects help the model select attributes that discriminate the target from surrounding objects.For example, comparing three cars can support describing the uniquely red car when the other two are blue.
- 3.2 Visual Comparison: Appearance differences are computed from CNN representations and pooled across selected comparison objects, while location and size differences use up to five nearby same-category objects.The experiments vary whether comparisons use same-category objects, different-category objects, or all other depicted objects.
- 3.3 Joint Language Generation: Joint generation produces expressions for all same-category objects together rather than generating each expression independently.The LSTM adds connections between expressions, and hidden-state differences are combined with each target representation before word prediction.
- 3.3 Joint Language Generation: The joint model forwards the target hidden state and aggregated hidden differences to a softmax layer for predicting each word.The hidden differences are intended to push ambiguous information away across expressions.
4 Data
The study evaluates three MSCOCO-based referring-expression datasets that differ in collection procedure, expression style, and the number of same-category objects per image. These differences motivate visual comparison methods and determine the dataset splits used for evaluation.
- 4 Data: The experiments use RefCOCOg, collected non-interactively, and RefCOCO and RefCOCO+, collected interactively through a two-player game.All three datasets are built on the Microsoft COCO image collection.
- 4 Data: Images contain multiple objects of the same category, requiring expressions that uniquely identify one object among relevant alternatives.The object categories cover the 80 common MSCOCO objects with ground-truth segmentation.
- 4.1 Datasets & Analysis: RefCOCO and RefCOCO+ expressions average 3.61 and 3.53 words, whereas RefCOCOg expressions average 8.43 words.The interactive game encourages concise expressions, while non-interactive collection permits more complex descriptions.
- 4.1 Datasets & Analysis: RefCOCO and RefCOCO+ average 3.9 same-category objects per image, compared with 1.63 for RefCOCOg.The larger number of same-type objects in RefCOCO and RefCOCO+ provides cases where same-category visual comparisons are useful.
- 4.2 Dataset Splits: RefCOCOg uses a per-object train/test split, while RefCOCO and RefCOCO+ use people-versus-other-objects testA and testB splits.The latter split keeps all objects from an image in either training or testing and separates images with multiple people from images with other objects.
5 Experiments
Experiments evaluate context representations and the proposed visual-comparison and joint-language-generation methods for referring-expression comprehension and generation. Across three datasets, visual comparison improves comprehension and generation, while language tying further improves generation, with automatic detection introducing a clear performance constraint.
- Experimental setup: Experiments evaluate context and model innovations on referring-expression comprehension and generation across RefCOCO, RefCOCO+, and RefCOCOg.Comprehension uses an IoU threshold of 0.5, while generation is evaluated with automatic and human measures.
- 5.1 Analysis Experiments: Global image context does not improve comprehension, whereas a target-centered scaled window improves performance.The authors suggest identical global context across objects may introduce ambiguity.
- Context Representation: Same-category visual comparisons are most useful for referring-expression comprehension among the tested reference-object sets.The experiment compares same-category, different-category, and all-object comparisons using the visdif model.
- 5.2 Referring Expression Comprehension: Visual comparison features improve the Baseline and MMI comprehension models across all datasets and splits.The improvement is also observed when the system uses automatically detected objects, although all accuracies decrease because of imperfect detections.
- 5.3 Referring Expression Generation: Visual comparison improves generation quality, and tying language generation across objects consistently improves performance, with visdif+tie achieving the highest score under almost every measurement.Language tying is not used on RefCOCOg because objects from one image may be split between training and testing.
- 5.3 Referring Expression Generation: Human evaluations find that MMI and tied-generation methods perform better, with tied methods producing less ambiguous referring expressions.The evaluation counts an expression as correct when more than two of three Turkers select the true target.
- 5.3 Referring Expression Generation: The duplicate-expression metric is lower for MMI models, while language tying generates even more different expressions for multiple objects in an image.The metric measures the fraction of images containing the same generated expression for multiple objects; smaller values are better.
6 Conclusion
The paper introduces detailed visual context and joint generation methods for referring expressions, improving generation and comprehension performance over prior state of the art. Joint generation over relevant objects reduces ambiguity during generation.
- The proposed visual-comparison context improves referring expression generation and comprehension over previous state of the art.
- Joint generation produces expressions for all relevant objects together during referring expression generation.
- Joint generation improves results over previous approaches designed to reduce ambiguity during generation.
- Figure 4 presents referring expression comprehension results on RefCOCO and RefCOCO+ using visual comparison based on detections.
- Figures 5 and 6 present referring expression generation results, including joint generation with the full visdif+MMI+tie model.