Source-linked AI summary
Generation and Comprehension of Unambiguous Object Descriptions
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy
TL;DR
Image captioning is difficult to evaluate because many descriptions can be valid, motivating unambiguous referring expressions for localized objects. The paper jointly models their generation and comprehension with deep learning, releases an MS-COCO-based dataset, and reports that listener-aware modeling outperforms simpler generation methods.
Problem
Image captioning lacks an objective evaluation because many valid descriptions can describe the same image, whereas referring expressions can be evaluated by unique object identification.
Method
The paper jointly models referring-expression generation and comprehension for real images and text using CNN-RNN methods, with semi-supervised bootstrap training.
Results
Listener-aware generation consistently outperforms models that emit descriptions from region features without modeling the listener.
Takeaways & Limitations
Referring expressions provide an objectively measurable test bed for vision-and-language understanding, and successful generation requires considering the listener.
Takeaways & Limitations
The ReferIt comparison dataset can permit short descriptions when images contain only one object of a class and focuses more on context than objects.
Abstract
from arXiv · showhide
We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/mjhucla/Google_Refexp_toolbox
1. Introduction
The paper reframes image description as generating and comprehending referring expressions that uniquely identify objects or regions. It introduces a joint deep-learning model and a large-scale MS-COCO-based dataset, addressing image captioning’s evaluation difficulty.
- Image captioning is subjective and ill-posed because many valid descriptions can describe the same image, making automatic evaluation difficult.
- Referring expressions uniquely identify exactly one object or region, enabling a well-defined performance metric based on successful comprehension.
- The task includes generating an expression for a highlighted object and comprehending an expression by selecting its corresponding object.
- The authors jointly model generation and comprehension with CNN-RNN methods for real images and text.
- The model outperforms methods that generate referring expressions without considering the listener who must comprehend them.
- The paper releases a large-scale referring-expression dataset based on MS-COCO and a visualization and evaluation toolbox.
2. Related Work
Prior work studied referring-expression attributes, context, and comprehension, but often used simple or non-real-world settings. This paper extends the area with real-image datasets and joint generation–comprehension models that learn expressions directly from images.
- Referring expressions: Earlier referring-expression research examined attributes, spatial relationships, overspecification, underspecification, and speaker variation.
- Referring expressions: Context requires distinguishing the target from alternatives and considering how a listener perceives the description.
- Real-image referring expressions: Previous work often used small datasets of computer-generated or simple objects rather than complex real-world images.
- Real-image referring expressions: The paper builds on large-scale real-image data while jointly learning generation and comprehension models.
- Real-image referring expressions: Unlike approaches requiring explicit attribute categories or manually listed visual phrases, the deep models generate surface expressions directly from raw images.
- Image captioning and related tasks: Unlike standard image captioning, referring-expression systems must identify spatially localized objects and produce unambiguous descriptions.
3. Dataset Construction
The authors construct G-Ref from MS-COCO images containing multiple same-category instances, collect and verify descriptions through Mechanical Turk, and compare it with UNC-Ref.
- ReferIt contains 130,525 expressions for 96,654 objects in 19,894 natural-scene photographs, but often permits short descriptions and emphasizes context over objects.
- The new dataset uses MS-COCO, which provides more than 300,000 images and instance-level segmentation for 80 object categories.
- Selection required images with 2–4 instances of the same object type and target boxes occupying at least 5% of the image area.
- Iterative Mechanical Turk collection and verification produced 104,560 valid expressions, averaging 1.91 per object and 3.91 per image.
- G-Ref descriptions average 8.43 words versus 3.61 in UNC-Ref, whose descriptions are more concise and less flowery.
4. Tasks
The paper addresses generation of referring expressions for target regions and comprehension of those expressions to localize the referred object. Generation uses region-conditioned language modeling, while comprehension ranks candidate regions under a uniform-prior assumption.
- Description generation: Generation computes the most probable sentence for a target region and image, using beam search to approximate the globally most probable expression.The RNN generates the sentence one word at a time until an end-of-sentence symbol, with beam size 3.
- Description generation: The generated expression is trained to distinguish the target region from other candidate regions rather than merely describe the target in isolation.
- Description comprehension: Comprehension ranks a set of region proposals by their probability given the referring expression and image, then returns the highest-ranked region.
- Description comprehension: Assuming a uniform prior over regions, comprehension selects the proposal maximizing p(S|R, I).The paper notes that real applications should account for region saliency p(R|I).
- Description comprehension: At test time, proposals come from classifying multibox-generated bounding boxes, while ground-truth boxes provide an upper-bound evaluation condition.
5. The Baseline Method
The baseline combines whole-image, target-region, and location features with a CNN-LSTM language model. It is trained by maximizing the likelihood of referring expressions associated with observed image-region-sentence triplets.
- Architecture: The baseline architecture is illustrated with beginning-of-sentence and end-of-sentence symbols marking the generated sequence boundaries.
- Architecture: The baseline augments a CNN representation of the whole image with CNN features for the target region and its location information before LSTM generation.
- Architecture: VGGNet supplies the visual representations, with only its final layer fine-tuned while earlier layers remain fixed.
- Architecture: The concatenated region, image, and location-size features form a 2005-dimensional input to an LSTM parameterizing p(S|R, I).The LSTM uses 1024-dimensional word embeddings and hidden states and receives the visual representation at every time step.
- Training: Training data consists of image-region-expression triplets, and the baseline minimizes the negative log probability of each expression conditioned on its image and region.
6. The Full Method
The full method modifies maximum-likelihood generation to account for listener ambiguity by contrasting the target region with alternative regions. It uses mutual-information objectives and a computationally cheaper max-margin variant.
- Motivation: The baseline may generate nondiscriminative expressions because it maximizes p(S|R, I) without considering other objects in the image.The paper illustrates that attributes such as “pink” can help distinguish the target from a similar object.
- Discriminative training: The softmax loss is equivalent to maximizing mutual information between the sentence S and region R under a uniform prior over regions.The approach is therefore also called Maximum Mutual Information training.
- Discriminative training: MMI training raises p(S|R, I) while lowering p(S|R′, I) for alternative regions R′ in the same image.This directly incorporates whether a listener could interpret the expression unambiguously.
- Negative-region selection: Negative regions can be selected as easy ground-truth negatives, same-class hard ground-truth negatives, or hard multibox negatives, with five random negatives used per step.
- Implementation: The MMI model replicates a tied-weight network across candidate regions, retaining the same parameter count and optimization strategy as maximum-likelihood training.Thus the objective function is the only model difference described here.
- Max-margin MMI: MMI-MM compares the target with one random negative, requiring two network replicas and reducing memory use while stabilizing gradients through larger minibatches.The paper reports similar practical results to the softmax MMI objective.
7. Semi-supervised Training
Semi-supervised training expands a small set of image regions with human descriptions by generating and verifying descriptions for a larger set of unlabeled regions. An ensemble filters unreliable generated sentences before retraining.
- Motivation: Bounding boxes without descriptions can still support MMI training because generated sentences can be penalized when the model cannot decode them back to the correct region.MMI encourages higher probability for the target region than for alternative regions.
- Data and bootstrapping: The semi-supervised setting combines a small image-box-description dataset with a larger image-box dataset lacking descriptions.
- Data and bootstrapping: A generator trained on described regions produces automatic descriptions for the larger box-only dataset, after which the model is retrained on the union of human and generated data.The procedure is described as bootstrap learning.
- Verification: An ensemble of comprehension models keeps an automatically generated sentence only when every model maps it to the same correct object.The retained sentences form the verified dataset Dfiltered.
8. Experiments
The experiments evaluate comprehension, generation, end-to-end decoding, supervised training, semi-supervised training, and qualitative behavior. Across settings, listener-aware max-margin training outperforms the maximum-likelihood baseline, while failures remain for absent or very small objects.
- Evaluation Metrics: Comprehension uses precision@1: a prediction is correct when bounding-box IoU exceeds 0.5.Generation is assessed with human judgments and an automatic end-to-end test that decodes generated descriptions back to the target object.
- Comparing different training methods: All MMI variants outperform maximum-likelihood training, with hard negatives benefiting more than easy negatives.For multibox proposals, multibox negatives are preferable to ground-truth negatives; the selected Full Model uses max-margin training with hard multibox negatives.
- Fully-supervised Training: MMI training raises human-rated descriptions judged better than or equal to human captions from 15.9% to 20.4%, a 4.5% absolute improvement.The reported relative improvement is 28.5% over maximum-likelihood training.
- Semi-supervised Training: Adding automatically labeled bounding-box data to strongly labeled data improves semi-supervised validation performance on G-Ref and UNC-Ref.The experiment compares training on Dbb+txt alone with training on Dbb+txt ∪Dbb.
- Fully-supervised Training: On an image-based G-Ref split with multibox proposals and ground-truth descriptions, Precision@1 increases from 0.407 to 0.451 on the test set.The corresponding validation values are 0.404 for the baseline and 0.444 for the Full Model.
- Qualitative Results: Qualitatively, the Full Model generates longer, more discriminative descriptions and often handles short phrases, word changes, and relational references correctly.Failures include absent objects and tiny objects with insufficient detectability or training data.
9. Conclusions
The paper argues that referring expressions provide an objectively evaluable vision-language task and that successful generation requires modeling the listener. Its experiments support listener-aware joint generation and comprehension as a stronger approach than region-only captioning.
- 9. Conclusions: Referring expressions combine language and vision understanding with an objective performance metric, unlike standard image captioning.This makes the contributed datasets useful for evaluating vision-language systems.
- 9. Conclusions: Modeling a listener that decodes generated descriptions consistently outperforms emitting captions from region features alone.The conclusion connects this result to joint models of vision and language.