Source-linked AI summary
Self-critical Sequence Training for Image Captioning
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel
TL;DR
Image captioning must cope with exposure bias from teacher-forced training and evaluation with non-differentiable sequence metrics. The paper introduces SCST, which uses test-time inference to normalize REINFORCE rewards, and reports a CIDEr improvement from 104.9 to 114.7 on MSCOCO.
Problem
Image captioning seeks training that avoids exposure bias while directly optimizing the non-differentiable metrics used for evaluation.
Method
SCST applies REINFORCE while using the model’s own test-time inference output to normalize sampled-caption rewards.
Results
104.9 to 114.7 CIDEr: SCST improves the best MSCOCO result and establishes a new state-of-the-art.
Takeaways & Limitations
Directly optimizing CIDEr with SCST and greedy test-time decoding is highly effective for image captioning.
Takeaways & Limitations
The baseline must not depend on the sampled action, although it may depend on model parameters or time.
Abstract
from arXiv · showhide
Recently it has been shown that policy-gradient methods for reinforcement learning can be utilized to train deep end-to-end systems directly on non-differentiable metrics for the task at hand. In this paper we consider the problem of optimizing image captioning systems using reinforcement learning, and show that by carefully optimizing our systems using the test metrics of the MSCOCO task, significant gains in performance can be realized. Our systems are built using a new optimization approach that we call self-critical sequence training (SCST). SCST is a form of the popular REINFORCE algorithm that, rather than estimating a "baseline" to normalize the rewards and reduce variance, utilizes the output of its own test-time inference algorithm to normalize the rewards it experiences. Using this approach, estimating the reward signal (as actor-critic methods must do) and estimating normalization (as REINFORCE algorithms typically do) is avoided, while at the same time harmonizing the model with respect to its test-time inference procedure. Empirically we find that directly optimizing the CIDEr metric with SCST and greedy decoding at test-time is highly effective. Our results on the MSCOCO evaluation sever establish a new state-of-the-art on the task, improving the best result in terms of CIDEr from 104.9 to 114.7.
1. Introduction
Image captioning requires generating descriptions that capture fine-grained visual entities, attributes, and relationships. Existing training with teacher forcing mismatches test-time generation, while evaluation uses non-differentiable task metrics, motivating reinforcement-learning-based optimization and SCST.
- Image captioning generates natural-language descriptions of images and requires understanding global and local entities, attributes, and relationships.
- Teacher forcing creates exposure bias because training conditions on ground-truth words whereas testing conditions on the model’s previous predictions.This mismatch can cause error accumulation during test-time generation.
- Prior approaches address exposure bias by feeding back model predictions during training or matching training and free-running recurrent dynamics adversarially.
- Captioning models are trained with cross entropy but evaluated using discrete, non-differentiable metrics such as BLEU, ROUGE, METEOR, and CIDEr.
- SCST uses the model’s test-time inference output to normalize REINFORCE rewards, directly optimizes CIDEr with greedy decoding, and improves MSCOCO CIDEr from 104.9 to 114.7.It avoids separately estimating the reward signal and reward normalization.
2. Captioning Models
The paper uses CNN- and LSTM-based encoder-decoder captioning models, including static image-feature and spatial-attention variants. These models represent images and words in 512-dimensional spaces and are traditionally trained by maximizing sequence likelihood with cross entropy.
- Captioning Models: The captioning models encode images with a deep CNN and generate words with an LSTM decoder, using BOS and EOS sentence tokens.The image representation is treated as the first input word, and generated words are fed back into the LSTM.
- Captioning Models: The LSTM outputs a distribution over the next word using softmax, with maxout and sigmoid functions used in the recurrent updates.
- Captioning Models: The models use 512-dimensional hidden states, word embeddings, and image embeddings, with parameters traditionally learned by maximizing observed-sequence likelihood.
- Training Objective: Traditional training minimizes cross entropy for the target ground-truth sequence using the model’s conditional next-word probabilities.
- Attention Model: Attention models dynamically re-weight spatial CNN features to focus on specific image regions at each generation step.
- Attention Model: Feeding the attention feature to both LSTM gates and outputs produced significant gains with ADAM, whereas feeding it only to all gates did not boost performance.
3. Reinforcement Learning
The paper casts sequence generation as reinforcement learning so captioning models can optimize non-differentiable sequence metrics while addressing exposure bias. REINFORCE estimates policy gradients from sampled captions and uses action-independent baselines to reduce variance without changing the expected gradient.
- Sequence Generation as an RL Problem: An LSTM captioning model can be viewed as an agent whose policy predicts words, updates its state, and receives a metric reward after generating EOS.CIDEr is given as an example reward computed by comparing the generated sentence with ground-truth captions.
- Sequence Generation as an RL Problem: The training objective is to minimize the negative expected reward under the model policy.
- Policy Gradient with REINFORCE: REINFORCE computes gradients for non-differentiable rewards by sampling sequences from the model and weighting their log-probability gradients by reward.
- REINFORCE with a Baseline: A baseline can be subtracted from the reward when it does not depend on the sampled action, preserving the expected gradient while reducing estimator variance.
- REINFORCE with a Baseline: A baseline may depend on model parameters or time while remaining valid under the stated condition.
4. Self-critical sequence training (SCST)
SCST uses the model’s own test-time inference reward as the REINFORCE baseline, weighting sampled sequences by whether they outperform that baseline. The approach is extended with greedy decoding and time-dependent or partial-sequence baselines, though the tested generalizations added no significant MSCOCO gain.
- SCST baselines each sampled sequence with the reward from the model’s current test-time inference algorithm.This avoids learning a separate context-dependent expected-future-reward baseline.
- Samples outperforming the test-time baseline are increased in probability, while lower-reward samples are suppressed.The baseline therefore creates a reward difference that determines each sample’s optimization weight.
- Greedy decoding requires one additional forward pass and optimizes the system for fast greedy inference at test time.The paper focuses on the greedy-decoding setting for SCST.
- Time-dependent SCST conditions the baseline on the sampled prefix, using future rewards to reduce reward-signal variance.The baseline completes the sequence with the inference algorithm after the sampled prefix.
- True SCST uses the inference algorithm as a critic after sampling only n future tokens, but neither True SCST nor TD-SCST produced significant additional MSCOCO gains.The authors also found no benefit from learning a control variate for the SCST baseline on MSCOCO.
5. Experiments
The experiments evaluate SCST on MSCOCO using FC and attention-based captioning models, comparing training metrics, baselines, decoding strategies, and model architectures. SCST consistently improves captioning performance, with CIDEr optimization and attention models producing the strongest reported results.
- Experimental setup: 113,287 training images and 5,000-image validation and test sets support evaluation on BLEU-4, ROUGEL, METEOR, and CIDEr.The MSCOCO experiments use five captions per training image and report results on Karpathy splits.
- Experimental setup: SCST models use ResNet-101 image features, including pooled global representations for FC models and spatial attention over 14 × 14 features.Attention masks reweight 196 spatial locations to form attended image representations.
- SCST evaluation: SCST weights sampled captions by their reward difference from the greedy test-time inference sentence, aligning training with inference and reducing gradient variance.The method requires one additional forward pass and trains for greedy decoding at test time.
- SCST evaluation: SCST outperforms MIXER on FC-2K models trained to optimize CIDEr, while curriculum learning is unnecessary and disabling it speeds training without reducing performance.The comparison includes MIXER and MIXER without a baseline; the reported SCST gain over learned baselines is consistent across curriculum schedules and seeds.
- Metric optimization: Optimizing CIDEr produces the strongest overall performance across the MSCOCO evaluation metrics, so subsequent experiments focus primarily on CIDEr.Training on a given metric generally yields the best test performance on that same metric, but CIDEr optimization lifts the other metrics considerably.
- Model comparison: Attention models outperform FC models, and four-model ensembles improve performance further; SCST attention ensembles outperform the previous best system on all official metrics.The attention models establish state-of-the-art single-model and ensemble results on the Karpathy splits, without fine-tuning ResNet.
6. Example of Generated Captions
On an image of a boat in an unusual context, attention models—especially those trained with SCST—produced more accurate and descriptive captions than the FC models.
- The SCST-trained attention ensemble was the only system that correctly described the boat in an unusual context.The FC models failed completely on this image.
- Attention models handled an image composition unseen during training better than FC models.The example comes from the objects out-of-context dataset.
- SCST-trained attention models generated more accurate, descriptive captions with higher confidence on unusual-context images.Caption confidence was reflected by the average log-likelihoods of the generated words.
7. Discussion and Future Work
The paper presents SCST as an efficient REINFORCE baseline strategy for optimizing non-differentiable metrics, reporting improved MSCOCO captioning performance and a new state of the art.
- The method enables more effective training on non-differentiable metrics and significantly improves MSCOCO captioning performance.The paper describes SCST as a simple and efficient approach to baselining REINFORCE.
- Ensembled attention models trained with SCST established a new state-of-the-art result on the MSCOCO evaluation server.Table 6 reports performance for four ensembled attention models on the official server.
- SCST normalizes sampled-sentence rewards using the reward from the model’s test-time inference algorithm.This avoids estimating both action-dependent and action-independent reward functions.
Supplementary Material
Beam search generates captions by retaining multiple high-probability partial sequences instead of selecting only the most probable word at each step.
- Beam search maintains the N most probable subsequences generated so far during image-caption decoding.It then computes next-word posterior probabilities for each retained subsequence.
- Greedy decoding is the special case N = 1, whereas beam search explores multiple candidate continuations.
B. Performance of XE versus SCST trained models
The experiments compare cross-entropy and SCST-trained models under greedy and beam decoding, finding that directly optimizing CIDEr substantially improves MSCOCO evaluation metrics.
- SCST models were trained to directly optimize CIDEr, while XE models used scheduled sampling with an increasing feed-forward probability.The comparison uses corresponding bootstrap models.
- Greedy decoding and optimized beam search were both evaluated for all models.
- Optimizing CIDEr clearly improved all MSCOCO evaluation metrics substantially.The result was observed for the best Att2in model across training epochs on the validation split.
C. Examples of Generated Captions
Qualitative examples show similar performance on standard MSCOCO images, but attention models generalize better to objects in uncommon contexts, especially when trained with SCST.
- On MSCOCO images, all captioning systems perform qualitatively similarly.
- On images containing objects in uncommon contexts, attention models perform much better than the other systems.
- SCST-trained attention models produce more accurate and descriptive captions for uncommon-context images.
D. Further details and analysis of SCST training
Further analysis identifies a crucial EOS-token detail for stable CIDEr optimization and reports broad metric gains, while qualitative examples clarify where attention and SCST help most.
- Including EOS as a word prevents incomplete fragments from dominating CIDEr gains and substantially lowers their reward.Without EOS, fragments such as “with a” and “and a” dominated metric gains; including EOS completely resolved the issue.
- Optimizing CIDEr substantially improves all MSCOCO evaluation metrics.
- Performance on other metrics has not substantially improved without more substantially degrading CIDEr.
- All models perform well on standard MSCOCO test images, with qualitatively comparable captions.
- For atypical object contexts, FC models fail to describe images accurately, whereas attention models handle unseen compositions well.
- SCST-trained attention models return more accurate and detailed summaries for atypical-context images.