Source-linked AI summary
Actor-Critic Sequence Training for Image Captioning
Li Zhang, Flood Sung, Feng Liu, Tao Xiang, Shaogang Gong, Yongxin Yang, Timothy M. Hospedales
TL;DR
Image captioning is commonly trained by ground-truth likelihood, which does not directly optimize non-differentiable language metrics and creates exposure bias. The paper uses actor-critic reinforcement learning with per-token advantage and value computation to optimize caption rewards, achieving state-of-the-art performance on MSCOCO. Its λ = 1 setting uses Monte Carlo estimates, which are unbiased while short captioning episodes limit estimation variance.
Problem
Ground-truth likelihood training does not directly optimize evaluation metrics such as CIDEr and creates exposure bias between training and testing.
Method
The paper formulates image captioning as reinforcement learning with an actor policy network, a critic value network, and per-token advantage and value computation.
Results
The model achieves state-of-the-art performance on the MSCOCO benchmark.
Takeaways & Limitations
Reinforcement-learning training can directly optimize non-differentiable caption-quality metrics such as CIDEr while addressing the teacher-forcing mismatch.
Takeaways & Limitations
The method sets λ = 1, relying on Monte Carlo estimation whose variance is kept limited by image captioning’s relatively short episodes.
Abstract
from arXiv · showhide
Generating natural language descriptions of images is an important capability for a robot or other visual-intelligence driven AI agent that may need to communicate with human users about what it is seeing. Such image captioning methods are typically trained by maximising the likelihood of ground-truth annotated caption given the image. While simple and easy to implement, this approach does not directly maximise the language quality metrics we care about such as CIDEr. In this paper we investigate training image captioning methods based on actor-critic reinforcement learning in order to directly optimise non-differentiable quality metrics of interest. By formulating a per-token advantage and value computation strategy in this novel reinforcement learning based captioning model, we show that it is possible to achieve the state of the art performance on the widely used MSCOCO benchmark.
1 Introduction
Image captioning needs language-generation methods that avoid teacher-forcing exposure bias and directly optimize non-differentiable evaluation metrics. The paper proposes actor-critic reinforcement learning with per-token value and advantage estimates to address these issues.
- 1 Introduction: The paper targets richer natural-language image descriptions for communication with human users, motivated by limited recent gains on public benchmarks such as MSCOCO.It attributes this plateau hypothesis to greater research emphasis on image understanding than language generation.
- 1 Introduction: Teacher forcing creates exposure bias because training conditions on ground-truth prefixes, whereas testing conditions on the model’s own previous predictions.This mismatch can cause error accumulation during test-time generation.
- 1 Introduction: Actor-critic reinforcement learning trains captioning policies on sampled model outputs, avoiding train-test mismatch while directly optimizing metrics such as CIDEr.The approach treats caption generation as sequential decision-making, with language metrics used as reinforcement-learning rewards.
- 1 Introduction: The proposed model uses an actor policy network and a critic that estimates each state’s expected task-specific reward for continued token generation.Per-token value estimates provide advantages for training the actor; under exact critic values, the policy-gradient estimate is unbiased.
- 1 Introduction: Image captioning combines a very large token action space with relatively short episodes, a setting the proposed actor-critic approach is designed to exploit.The paper contrasts this structure with reinforcement-learning applications that typically involve smaller action spaces or longer episodes.
2 Related Work
Prior image-captioning work largely uses CNN encoders and RNN decoders trained with teacher forcing, while reinforcement-learning approaches address the mismatch between likelihood training and caption-quality metrics. Existing methods differ in how they assign sequence-level or action-level credit.
- Image captioning: Image captioning commonly combines a CNN image encoder with an RNN sentence decoder and trains caption likelihood through teacher forcing.Related work also explores dynamic attention and improved visual feature representations.
- Image captioning with reinforcement learning: Reinforcement-learning captioning methods seek to reduce the discrepancy between likelihood-based teacher forcing and evaluation metrics such as CIDEr.The cited methods include REINFORCE and other reinforcement-learning approaches for caption generation.
- Image captioning with reinforcement learning: The cited REINFORCE approach uses the reward from the current inference algorithm as a baseline, giving every token in a sampled caption the same sentence-level advantage.This treats token contributions uniformly within the sampled sentence.
- Actor-Critic: Actor-critic methods train an actor with policy gradients using advantages baselined by a critic, a pattern used by several state-of-the-art reinforcement-learning algorithms.The related background includes applications such as AlphaGo, where rewards may arrive only at the end of long episodes.
- Sequence generation: Sequence-generation work applies actor-critic methods to large action spaces, with critics estimating action values and using variance-control techniques.The paper situates image captioning within broader sequence-generation research while emphasizing its distinct action-space and episode structure.
3 Methodology
The method casts image captioning as a reinforcement-learning sequence-generation problem, using actor and critic networks to optimize task-specific language rewards. It combines CNN–RNN captioning with policy-gradient training, value-based advantage estimation, and staged supervised pretraining.
- 3.1 Problem formulation: The model represents caption generation as a finite MDP whose state contains CNN image features and the tokens generated so far.The conventional architecture uses a CNN encoder and RNN decoder; each token is treated as an action.
- 3.2 Model: The policy network maps the image and generated-token state to a categorical distribution over the dictionary's next-token actions.The image feature and start token form the initial state, and an LSTM produces the next-token distribution.
- 3.2 Model: The critic estimates each state’s expected future return, while the actor uses the resulting advantage to update token-generation probabilities.The value network is a separate LSTM with a shared CNN, and the advantage compares action value with state value.
- 3.3 Advantage function estimation: Advantage estimation uses forward-view TD(lambda), with n-step returns combining observed rewards and a bootstrapped value estimate.The method notes that this formulation matches GAE in a forward view and sets lambda = 1 for captioning because episodes are relatively short.
- 3.4 Value function estimation: The value network is trained by nonlinear regression toward the estimated action value, providing the critic needed for actor-critic updates.The value-function objective minimizes the discrepancy between estimated Q values and the critic's state-value prediction.
- 3.3 Advantage function estimation: The training objective uses completed-caption evaluation scores such as CIDEr as rewards rather than teacher-provided actions.The policy-gradient formulation maximizes expected cumulative reward, while standard cross-entropy pretraining initializes the actor before reinforcement-learning training.
4 Experiments
The experiments evaluate staged actor-critic training for image captioning on MSCOCO, comparing single-model performance, training reward, and computational cost against supervised and reinforcement-learning baselines. The proposed method improves CIDEr over the log-likelihood baseline, ranks third on the official challenge leaderboard, and is reported as the most efficient training method.
- Training Setup: Staged pretraining avoids early policy determinization and vanishing gradients that can occur when actor-critic training starts from scratch.The actor is first pretrained with cross-entropy, then the critic is pretrained from sampled actions of the fixed actor before joint training.
- Experimental Setup: The evaluation uses MSCOCO with 82,783 training images, 40,504 validation images, and a 4,051-image development split.The official test comparison uses 40,775 images evaluated through the COCO server.
- Results: 1.162 CIDEr versus 1.007 for the log-likelihood baseline on the MSCOCO development set, using single-model greedy decoding.The method also outperforms attention and memory-cell variants added to the LSTM.
- Results: Training reward is higher than Self-critical [17], which the authors attribute to per-token advantage and value computation.Average rewards were recorded over 1 million iterations at 10,000-iteration intervals.
- Results: Ranked 3rd on the MSCOCO image-captioning challenge leaderboard, outperforming supervised methods on all reported metrics with a single model.It also exceeds the other compared reinforcement-learning methods except on ROUGE-L C40.
- Computational Cost: The method has the lowest reported training cost because it omits an attention cell, while Self-critical [17] samples twice per iteration.Training-time comparisons use TensorFlow, an NVIDIA P100, and minibatch size 16.
5 Conclusion
The paper applies reinforcement learning to image captioning to optimize non-differentiable language metrics directly. Its actor-critic strategy enables per-token advantage and value computation, achieving state-of-the-art MSCOCO performance with a computationally efficient model.
- Reinforcement learning directly optimizes non-differentiable language metrics such as CIDEr for automated image captioning.
- The proposed actor-critic strategy enables per-token advantage and value computation for improved caption-model training.
- The computationally efficient model achieves state-of-the-art performance on the MSCOCO benchmark.