Source-linked AI summary
Comprehension-guided referring expressions
Ruotian Luo, Gregory Shakhnarovich
TL;DR
Referring-expression generation needs evaluation tied to whether a listener can identify the intended object, unlike generic image captioning. The paper uses comprehension models as differentiable training critics or test-time rerankers, and reports promising improvements across benchmark datasets, especially from generate-and-rerank. It also identifies future scope around alternative training regimes, contextual objects, and shared weights.
Problem
Generic image-captioning metrics do not directly capture whether an expression lets a listener identify the intended object, motivating comprehension-based evaluation.
Method
The paper uses a comprehension model to guide generation through discriminative loss during training or to rerank candidate expressions at test time.
Results
Both comprehension-guided approaches are promising across datasets, with generate-and-rerank obtaining particularly good results.
Takeaways & Limitations
Comprehension guidance provides a practical surrogate for human understanding when generating referring expressions.
Takeaways & Limitations
The paper leaves alternative training regimes, context objects, and generation-comprehension weight sharing for future work.
Abstract
from arXiv · showhide
We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic "image captioning" which lacks natural standard evaluation criteria, quality of a referring expression may be measured by the receiver's ability to correctly infer which object is being described. Following this intuition, we propose two approaches to utilize models trained for comprehension task to generate better expressions. First, we use a comprehension module trained on human-generated expressions, as a "critic" of referring expression generator. The comprehension module serves as a differentiable proxy of human evaluation, providing training signal to the generation module. Second, we use the comprehension module in a generate-and-rerank pipeline, which chooses from candidate expressions generated by a model according to their performance on the comprehension task. We show that both approaches lead to improved referring expression generation on multiple benchmark datasets.
1. Introduction
The paper treats referring expressions as captions whose success depends on whether a listener can identify the intended object. It uses comprehension models to guide generation during training or to rerank generated candidates at test time.
- Referring expression quality is evaluated by whether a human can locate the described object or region.
- The paper studies comprehension, which localizes an object from an expression, and generation, which produces a discriminative expression for an object.
- The collaborative architecture combines a generator with a comprehension model that evaluates whether generated expressions can be correctly dereferenced.
- The comprehension model is adapted into a differentiable proxy for human understanding and supplies training signal to the generator.
- Generate-and-rerank selects the candidate expression with the highest generation-comprehension score.
- Training by proxy adds discriminative comprehension loss to cross-entropy training and avoids additional region proposals at test time.
2. Related work
Prior work adapted captioning, retrieval, and multimodal embedding methods to referring expressions, while this paper replaces objectives less aligned with human comprehension with a comprehension-based surrogate.
- Image captioning systems commonly encode images with CNNs and generate word sequences with RNNs.
- Region-aware captioning methods use attention, word-region alignment, or per-region captions, but dense captions were not required to be unambiguous referring expressions.
- Earlier referring-expression systems scored expressions with P(S|I) or jointly trained comprehension and generation using max-margin MMI.
- The paper combines simple models while replacing max-margin loss with a surrogate closer to the eventual goal of human comprehension.
- The comprehension model belongs to the multimodal-embedding category and uses a similarity-based region-selection formulation.
- The generate-and-rerank method translates the speaker-listener idea to referring-expression generation by reranking sampled expressions with a listener model.
3. Generation and comprehension models
The paper defines separate generation and comprehension modules for image regions and referring expressions. Generation predicts an expression from image-region features, while comprehension selects the region matching a query expression.
- Expression generation: The generation module takes an image I and region r and outputs a referring expression w.
- Expression generation: Generation is an encoder-decoder model using target-object, global-context, and location features derived from the image and region.
- Expression generation: A unidirectional LSTM predicts each next word from visual features and the previous word embedding, trained with cross-entropy loss.
- Expression generation: At inference, beam search, greedy search, or sampling approximates the most probable expression instead of exactly computing argmax_w PG(w|I, r).
- Comprehension: The comprehension task selects a region from R given an image I and query expression q, using the posterior PC(r|I, q, R).
- Comprehension: The comprehension model encodes query expressions with a bidirectional LSTM and represents each region using the generation model’s visual encoder.
- Comprehension: Per-region logistic loss treats each region as a binary match decision, whereas softmax loss classifies which region should match the expression.
4. Comprehension-guided generation
The paper uses a trained comprehension model as a proxy for human understanding to guide referring-expression generation during training or test-time selection. Training combines generation and comprehension signals through differentiable approximations, while generate-and-rerank scores diverse candidates using both fluency and discriminativity.
- A comprehension model is used as a proxy for human comprehension to guide the expression generator in two approaches: training-time guidance and test-time reranking.The framework treats comprehension as an additional signal beyond standard generation training.
- 4.1. Training by proxy: The training-by-proxy method replaces nondifferentiable generated words with differentiable word distributions, enabling back-propagation from comprehension loss into the generator.The distribution P preserves the query representation's dimensions, sums to one per column, and is differentiable with respect to generator parameters.
- 4.1.1 Compound loss: The compound loss combines generation and comprehension losses, with λ controlling the relative importance of fluency and discriminativity.Both losses use the generator's word distribution, while replacing the one-hot sequence with P permits standard back-propagation.
- 4.1.2 Modified scheduled sampling: Modified scheduled sampling alternates between ground-truth inputs with generation loss and sampled inputs with comprehension loss, while decaying the ground-truth probability to a minimum.The minimum prevents the generator from losing all ground-truth information and being guided purely by comprehension loss.
- 4.1.3 Stochastic mixed sampling: SMIXEC samples a random continuation point from a geometric distribution, controlling how much ground-truth supervision remains during training.When p=0, the method becomes cross-entropy training; when p=1, it matches the MIXER annealing schedule.
- 4.2. Generate-and-rerank: Generate-and-rerank produces candidate expressions, scores them with a weighted fluency-and-comprehension function, and selects the highest-scoring candidate.Candidates are generated from the region and image, and beam search is avoided to increase candidate diversity.
5. Experiments
Experiments evaluate comprehension and generation across RefClef, RefCOCO, RefCOCO+, and RefCOCOg using automatic and human measures. Comprehension-guided generation improves several metrics and human judgments, although training schedules show dataset-dependent variation.
- Datasets: The experiments use RefClef, RefCOCO, RefCOCO+, and RefCOCOg, with dataset-specific splits and region-evaluation settings.RefCOCO and RefCOCO+ use testA for people and testB for other objects; RefCOCOg uses validation and RefClef uses test data.
- Comprehension evaluation: Comprehension is evaluated by selecting the referred region, with detector-proposal hits requiring IoU of 0.5 or higher.The experiments compare ground-truth-region and proposal-region settings, using precomputed proposals for all four datasets.
- Comprehension results: The comprehension model obtains state-of-the-art accuracy on RefClef and remains competitive with recent, more complex models on RefCOCO, RefCOCO+, and RefCOCOg.With the same visual feature encoder, it outperforms MMI on the reported comparison.
- Automatic generation evaluation: Generate-and-rerank consistently improves automatic comprehension accuracy and BLEU, and beats MLE+sample, isolating a benefit from comprehension-guided reranking.Both methods select from 100 sampled expressions, but MLE+sample chooses by lowest perplexity while reranking uses comprehension.
- Training schedules: No training-by-proxy schedule consistently wins: SMIXEC is strongest on RefCOCO, whereas compound loss, SMIXEC, and MSS differ across RefCOCO+ splits.The authors report no concrete explanation for these dataset-dependent differences.
- Human evaluation: Human evaluations find both comprehension-guided methods more informative on RefCOCO, while training by proxy performs less well on RefCOCO+ TestB.Results on RefCOCO+ TestA are similar to RefCOCO, but performance varies across training schedules and splits.
6. Conclusion
The paper uses learned comprehension models to improve referring-expression generation through training-time guidance and test-time generate-and-rerank. Both approaches are promising, while generate-and-rerank performs particularly well across datasets; future work targets broader training regimes, contextual objects, and shared representations.
- Contributions: Comprehension guidance improves referring-expression generation through training by proxy and generate-and-rerank methods.The first incorporates comprehension loss during generator training; the second selects among candidate expressions using comprehension scores.
- Results: Generate-and-rerank obtains particularly good results across datasets.
- Future work: Future work considers alternative training regimes, contextual objects, and weight sharing between generation and comprehension models.The current generation and comprehension models are completely separate.