Source-linked AI summary
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, Kevin Murphy
TL;DR
Image captioning models and standard metrics do not reliably reflect human judgments, while SPICE and CIDEr are difficult to optimize directly. The paper uses policy gradients with Monte Carlo rollouts to optimize SPIDEr, reporting state-of-the-art COCO results and human preference for the resulting captions.
Problem
Existing MLE objectives and common captioning metrics correlate poorly with human quality judgments, while directly optimizing SPICE or CIDEr is difficult.
Method
The paper uses policy gradients with Monte Carlo rollouts to optimize captioning metrics, including the SPIDEr combination of SPICE and CIDEr.
Results
The method achieves state-of-the-art COCO leaderboard results and produces SPIDEr-optimized captions judged qualitatively superior by human raters.
Takeaways & Limitations
SPIDEr captures both fluency and semantic properties, supporting captions that human raters prefer over those optimized for MLE or COCO metrics.
Takeaways & Limitations
Optimizing SPICE alone produced long repetitive sentences judged poor by COCO metrics and humans.
Abstract
from arXiv · showhide
Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.
1 Introduction
Image captioning metrics often diverge from human judgments, motivating SPIDEr and a policy-gradient method that directly optimizes caption quality. The paper reports state-of-the-art COCO results and stronger human preferences for SPIDEr-optimized captions.
- Image captioning supports applications including text-based image retrieval, accessibility for blind users, and human-robot interaction.
- MLE-trained captioning models are evaluated with BLEU, METEOR, ROUGE, and CIDEr, but these metrics correlate weakly with human quality judgments.
- SPICE correlates strongly with human ratings but ignores syntactic quality, so optimizing it alone can produce poor captions.
- The proposed policy-gradient method improves on MIXER and enables optimization of arbitrary captioning metrics, including SPIDEr.
- SPIDEr linearly combines SPICE and CIDEr to address semantic fidelity and syntactic quality while satisfying both proposed metric criteria.
- SPIDEr optimization produces captions that human raters strongly prefer over captions from the same model trained with MLE or COCO metrics.
2 Related work
Prior captioning work commonly uses encoder-decoder models and MLE, but exposure bias and weak metric alignment motivate sequence-level policy-gradient optimization. The paper replaces MIXER’s mixed objective with Monte Carlo rollout estimates for more robust training.
- Most captioning systems use CNN-RNN encoder-decoder architectures, including the Show and Tell model used in this work.
- MLE training creates exposure bias because training conditions on ground-truth words while inference conditions on the model’s own predictions.
- Policy gradient treats caption scores as rewards and the RNN decoder as a stochastic policy that generates words sequentially.
- 2.3 MIXER: MIXER combines MLE for an initial prefix with REINFORCE for later words, gradually reducing the MLE prefix length.
- 2.3 MIXER: MIXER is sensitive to annealing schedules and hyperparameters, whereas Monte Carlo rollouts estimate future rewards and improve convergence speed and training stability.
3 Methods
The method uses policy gradients with Monte Carlo rollouts to optimize sequence-level caption rewards, including BCMR metrics, SPICE, and SPIDEr, in a CNN-RNN captioning system.
- Policy gradient formulation: Policy gradients treat next-word generation as stochastic actions and optimize arbitrary caption-level reward functions.The decoder policy generates words from the vocabulary, while the terminal reward can be based on BCMR or SPICE.
- Policy gradient formulation: Monte Carlo rollouts estimate each word choice’s value by averaging rewards from complete continuations conditioned on the current partial sequence.At terminal states, the action value equals the sequence reward.
- Policy gradient formulation: The method optimizes expected reward from the initial empty state using the policy gradient theorem and sampled trajectories.The objective averages value from the initial state across training examples; sampled paths approximate the gradient.
- Variance reduction: A learned baseline reduces gradient variance without changing the validity of the estimated gradient.The baseline is trained separately from Monte Carlo estimates, using an MLP over the RNN hidden state without back-propagating through that state.
- Training procedure: MLE pre-training warm-starts the stochastic policy before policy-gradient training, avoiding random exploration across the vocabulary’s exponentially many paths.Rollouts require only RNN forward passes and can be parallelized, making policy-gradient training about twice as slow as MLE in wall time.
- Reward functions: The reward design combines standard BCMR metrics or SPICE, while SPIDEr equally weights SPICE and CIDEr to balance semantic similarity and syntactic quality.Optimizing SPICE alone produced detailed captions with repeated phrases; the paper therefore combines SPICE with CIDEr.
- Model architecture: Experiments use a Show-and-Tell-style CNN-RNN with an ImageNet-pretrained Inception-V3 encoder and a one-layer 512-unit LSTM decoder.Image features are projected from 2,048 to 512 dimensions; training uses teacher forcing and inference uses greedy decoding.
4 Results
On COCO, the proposed policy-gradient methods improve automatic and human-evaluated caption quality, while SPIDEr produces more reasonable captions than BCMR despite lower COCO-metric scores. The method also converges faster and more stably than MIXER, although pure SPICE generates repetitive, ungrammatical captions.
- Automatic evaluation using BCMR metrics: PG-BCMR significantly outperforms MLE and the top five COCO methods on the official test evaluation.The comparison includes systems using attention, complex decoders, and high-level visual attributes.
- Comparison with MIXER: PG-BCMR also outperforms MIXER, while MIXER-BCMR-A approaches PG-BCMR after switching from vanilla SGD to Adam.The Adam-based variant slightly exceeds PG-BCMR on Meteor, but requires well-tuned parameters.
- Human evaluation: SPICE alone produces ungrammatical, repetitive captions, whereas combining SPICE with CIDEr gives much better results.SPICE captures scene-graph semantic similarity but is relatively insensitive to syntactic quality.
- Human evaluation: PG-SPIDEr generates more reasonable captions than PG-BCMR despite performing worse on the COCO metrics, and both appear better than MLE.The examples include captions from MLE, PG-SPICE, MIXER variants, PG-BCMR, and PG-SPIDEr.
5 Conclusion
The paper presents a robust policy-gradient method for optimizing captioning metrics, achieving state-of-the-art COCO leaderboard results and qualitatively superior captions when optimizing SPIDEr.
- The proposed policy-gradient method is robust and efficient across a variety of captioning metrics.
- Optimizing standard COCO metrics achieves state-of-the-art leaderboard results, despite their weak correlation with human judgment.
- Optimizing the proposed SPIDEr metric produces qualitatively superior captions according to human raters.
A Full derivation of policy gradient
The derivation formulates caption generation as a policy-gradient objective over sampled word sequences, then rewrites its gradient using incremental rewards and sequence-level Monte Carlo expectations.
- Policy-gradient objective: A caption is generated by sampling one vocabulary word per time step from the policy πθ(¯g|xn), parameterized by CNN-RNN weights θ.
- Policy-gradient objective: The objective is to maximize the expected reward of captions generated by following the policy over the training images.
- Gradient derivation: The gradient is computed by differentiating the policy objective and applying the chain rule to the joint conditional probability of the generated sequence.
- Monte Carlo estimation: Equations (15) and (17) yield a sequence-level Monte Carlo sampling algorithm, which the paper then further improves.
- Reward decomposition: The sequence reward is partitioned into prior-prefix and current-suffix contributions, allowing the prefix contribution to be shown as zero in the relevant summation.
B Variance reduction
The variance-reduction discussion addresses the high variance of Monte Carlo policy-gradient estimates caused by the enormous caption-sampling space and motivates subtracting a baseline.
- Variance problem: Monte Carlo estimation suffers from high variance because caption generation has a very high-dimensional sample space.
- Variance problem: With a vocabulary of about 9,000 words and caption length T = 30, the discrete sample space contains 9000^30 possible sequences.
- Baseline subtraction: Subtracting a constant from Qθ(g1:(t−1), gt) does not theoretically change the policy gradient.
- Baseline subtraction: Prior work indicates that this subtraction is effective at reducing variance when the sample size is small.