Source-linked AI summary
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Licheng Yu, Hao Tan, Mohit Bansal, Tamara L. Berg
TL;DR
Referring-expression research requires systems to generate expressions for target objects and comprehend expressions by locating described regions. This paper jointly trains speaker and listener modules and adds a discriminative reinforcer, achieving state-of-the-art performance for both tasks on three datasets. The framework therefore unifies generation and comprehension while using reward-guided sampling to produce less ambiguous expressions.
Problem
Referring-expression generation and comprehension require coordinated speaker and listener behaviors for identifying objects in visual scenes.
Method
A CNN-LSTM speaker and embedding-based listener are jointly trained end to end, while a discriminative reward-based reinforcer guides sampling of more discriminative expressions.
Results
The model outperforms state of the art on both comprehension and generation across three datasets and multiple evaluation metrics.
Takeaways & Limitations
Joint interaction among speaker, listener, and reinforcer provides a unified framework for referring-expression generation and comprehension.
Takeaways & Limitations
The speaker and listener components are not restricted to the particular CNN-LSTM and triplet-based listener implementations described.
Abstract
from arXiv · showhide
Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generation. Our model is composed of three modules: speaker, listener, and reinforcer. The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions. The listener-speaker modules are trained jointly in an end-to-end learning framework, allowing the modules to be aware of one another during learning while also benefiting from the discriminative reinforcer's feedback. We demonstrate that this unified framework and training achieves state-of-the-art results for both comprehension and generation on three referring expression datasets. Project and demo page: https://vision.cs.unc.edu/refer
1. Introduction
Referring expression generation and comprehension model speaker-listener interactions for identifying objects in visual scenes. The proposed unified listener-speaker-reinforcer framework jointly trains both tasks and uses discriminative rewards to reduce ambiguity, outperforming prior methods on three datasets.
- Unified framework: The reinforcer learns a discriminative reward model that guides sampling toward expressions applying to the target but not other image objects.This targets clear, brief expressions while avoiding obscurity and ambiguity.
- Task setting: Referring expression generation produces an expression for a target object, whereas comprehension localizes the region described by an expression.These tasks computationally model the speaker and listener roles, respectively.
- Evaluation: The model is evaluated on three recent datasets collected from MSCOCO images.The paper focuses on three large-scale datasets used for referring-expression evaluation.
- Unified framework: The framework jointly trains CNN-LSTM speaker and embedding-based listener modules for both generation and comprehension.Joint learning lets the two modules interact rather than operate independently.
- Evaluation: The unified listener-speaker-reinforcer model significantly outperforms state of the art on comprehension and generation.Generation is evaluated with human judgments and automatic metrics, while comprehension results appear in Tables 1 and 2.
2. Related work
Related work frames referring expressions within broader vision-language research and distinguishes speaker-based generation from listener-based comprehension. Prior neural approaches use generative or joint-embedding models, while earlier speaker-listener combinations used offline reranking.
- Vision-language context: Vision-language research includes referring expression generation and comprehension alongside image captioning and visual question answering.These tasks emerged as evaluation applications of multimodal research.
- Vision-language context: Image captioning describes an image generally, whereas referring-expression tasks focus on particular regions or objects.The related-work discussion motivates more focused tasks because image-caption outputs are task dependent.
- Referring-expression models: One neural approach uses a CNN-LSTM encoder-decoder to model P(r|o), supporting generation and comprehension through posterior object selection.Comprehension selects the object with the largest posterior probability after applying Bayes’ rule.
- Referring-expression models: Earlier work combined a speaker for generation with an offline listener that reranked generated expressions, including work on abstract images.The paper’s approach extends speaker-listener interaction beyond this offline arrangement.
3. Model
The model jointly integrates CNN-LSTM speaker, embedding-based listener, and discriminative reinforcer modules for referring-expression generation and comprehension. Joint losses and reinforcement learning connect these modules during training to improve both tasks.
- 3.1. Speaker: The speaker encodes target-object appearance, global context, location/size, and comparison features before LSTM expression generation.The fused visual representation is fed into the LSTM, which generates the most likely referring expression.
- 3. Model: The framework combines a CNN-LSTM speaker, joint-embedding listener, and reinforcer that samples less ambiguous expressions for training.The speaker generates expressions for target objects, while the listener maps expressions and objects into a shared embedding space.
- 3.2. Listener: The listener learns object-expression compatibility with triplet hinge losses using mismatched expressions and objects from the same image.These negative matches support ranking the correct object-expression pair above alternatives.
- 3.3. Reinforcer: The reinforcer samples words from the speaker’s categorical outputs and uses a discriminative reward with policy gradients to favor less ambiguous expressions.Because sampling is non-differentiable, the reward updates the speaker through reinforcement learning.
- 3.4. Joint Model: The joint objective combines speaker losses, listener loss, and reward optimization, while listener embeddings are supplied as additional speaker input.At test time, speaker and listener predictions can be ensembled for comprehension.
- 3.5. Comprehension and Generation: Adding the listener improves comprehension performance to more than 4% over previous state-of-the-art results, while the jointly trained speaker alone already reaches state of the art.The speaker-only result is obtained with the listener contribution weight set to zero.
4. Experiments
Experiments evaluate the unified model on three referring-expression datasets, using ablations for comprehension and generation with both automatic and human evaluation. Jointly training speaker and listener modules, adding the reinforcer, and reranking improve performance across the reported settings.
- Experimental setup: Experiments use RefCOCO, RefCOCO+ and RefCOCO, with ground-truth or automatically detected candidate regions for comprehension.RefCOCO and RefCOCO+ use interactive collection, whereas RefCOCOg uses longer expressions from a non-interactive setting.
- Comprehension task: Comprehension ablations add listener and reinforcer modules to the speaker and evaluate both oracle-region and detector-based settings.The speaker selects the most probable candidate region, while the listener selects the candidate with the highest similarity score.
- Comprehension task: MMI-trained speakers outperform speakers trained without MMI, while joint training with the listener and reinforcer further improves the speaker.With MMI ranking, the jointly trained speaker outperforms the pure listener by around 2% on all three datasets.
- Comprehension task: The listener generally outperforms the speaker for comprehension, joint training improves the listener, and ensembling speaker and listener achieves the best overall results.Table 2 compares listener variations and ensembled listener+speaker modules under the same comprehension settings.
- Generation task: Jointly learned speakers achieve higher METEOR and CIDEr scores than single speakers across datasets, with improvements both with and without reranking.Human evaluation also finds that the full joint model with reranking achieves the best performance.
- Generation task: The full model’s reranked generation considers expressions for all target objects together, seeking relevance to each target and irrelevance to the others.This global comparison is illustrated in the joint expression-generation examples.
5. Conclusion
The paper presents a unified framework for referring-expression generation and comprehension, jointly training speaker and listener modules with a reinforcer for less ambiguous expressions. Experiments report state-of-the-art performance for both tasks across multiple datasets and evaluation metrics.
- The framework jointly trains speaker and listener modules and adds a reinforcer to help produce less ambiguous referring expressions.
- Experiments report that the model outperforms state of the art for both comprehension and generation on multiple datasets and evaluation metrics.