PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Video Captioning with Entailment Rewards

Ramakanth Pasunuru, Mohit Bansal

arXiv:1708.02300v1cs.CLcs.AIcs.CVcs.LG

TL;DR

Video-captioning models trained with word-level cross-entropy do not directly optimize sentence-level evaluation metrics, and phrase-matching rewards can score captions containing critical errors highly. The paper combines policy-gradient training with mixed loss and introduces CIDEnt, an entailment-corrected reward. Its CIDEnt model improves over the CIDEr-reward model and achieves state of the art on MSR-VTT.

  • Problem

    Word-level cross-entropy does not directly optimize sentence-level caption metrics, while undirected phrase matching can reward captions with critical errors.

    Source: §1 · p. 1, §4 · p. 3

  • Method

    The paper uses mixed-loss policy-gradient training and introduces CIDEnt, which adjusts CIDEr-based rewards using entailment to favor logically implied captions.

    Source: §3 · p. 3, §4 · p. 3

  • Results

    CIDEr-reward training improves over the cross-entropy baseline, and CIDEnt-RL significantly improves over CIDEr-RL on all automatic metrics and human evaluation, achieving state of the art on MSR-VTT.

    Source: §1 · p. 2, §6.1 · p. 4, §7 · p. 5

  • Takeaways & Limitations

    The authors plan to apply entailment-corrected rewards to image captioning and document summarization.

    Source: §7 · p. 5

    The dataset's very small dev-set causes tuning issues, and the authors plan a better train/dev re-split.

    Source: §6.3 · p. 5

Abstract

from arXiv · show

Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.

1 Introduction

Video-captioning models trained with word-level cross-entropy do not directly optimize the sentence-level metrics used for evaluation. The paper introduces mixed-loss policy-gradient training and CIDEnt, reporting improvements over prior reward baselines and state of the art on MSR-VTT.

1 Introduction

  • Word-level cross-entropy does not directly optimize sentence-level evaluation metrics such as CIDEr and BLEU, and video captioning also faces exposure bias.Exposure bias occurs when models see training-distribution inputs rather than their own predictions.Source: §1 · p. 1
  • Mixed-loss policy-gradient training combines metric-based rewards with cross-entropy to optimize sentence-level metrics while maintaining output fluency.The approach addresses exposure bias by training on model-generated predictions.Source: §1 · p. 1
  • CIDEnt penalizes CIDEr-based rewards when entailment is low, favoring captions logically implied by the ground truth and avoiding contradictory or unrelated information.It checks logically directed partial matches rather than relying only on undirected phrase matching.Source: §1 · pp. 1–2
  • CIDEnt-RL significantly improves over CIDEr-RL on all automatic metrics and human evaluation, while the CIDEr-reward model improves over the cross-entropy baseline across multiple datasets.The paper reports a new state of the art on MSR-VTT.Source: §1 · p. 2

2 Related Work

Prior video-captioning work explored diverse sequence-to-sequence architectures but generally trained with word-level cross-entropy. Related work also applied policy-gradient and mixed-loss methods to image captioning and summarization, while textual entailment research supplied relevant models and tasks.

2 Related Work

  • Video-captioning studies explored attention, hierarchical RNNs, 3D-CNN features, joint embedding spaces, and language fusion, but used word-level cross-entropy training.Examples include Venugopalan et al. (2015a), Yao et al. (2015), Pan et al. (2016a,b), and Venugopalan et al. (2016).Source: §2 · p. 2
  • Policy-gradient image-captioning work used non-differentiable evaluation metrics as rewards, with later work adding Monte Carlo roll-outs or a test-inference baseline.Paulus et al. (2017) also applied ROUGE rewards in a mixed-loss summarization setup.Source: §2 · p. 2
  • Recognizing Textual Entailment has a large SNLI dataset and leaderboard models, and prior work combined video captioning with entailment and video generation.The paper focuses on the decomposable intra-sentence attention model of Parikh et al. (2016).Source: §2 · p. 2

3 Models

The models use an attention-based sequence-to-sequence baseline, then train with policy-gradient rewards and a mixed loss that balances reward optimization with cross-entropy. The mixed-loss schedule starts from a cross-entropy-trained baseline.

3 Models

  • The attention baseline encodes video-frame features with a bidirectional LSTM and generates captions with an attention-based LSTM decoder.It is trained using cross-entropy against the ground-truth caption.Source: §3 · p. 2
  • Policy-gradient training treats generated words as actions and rewards a sampled caption sequence when it emits the end-of-sequence token.The objective minimizes negative expected reward, with gradients approximated using a sampled word sequence and a variance-reducing baseline estimator.Source: §3 · pp. 2–3
  • Mixed loss is a weighted combination of cross-entropy and reinforcement-learning losses, balancing reward optimization with readability and fluency.The cross-entropy component trains a conditioned language model, while the reinforcement component optimizes the reward metric.Source: §3 · p. 3
  • Training starts from the optimized cross-entropy baseline before switching to the mixed-loss objective.The schedule is described as supporting annealing and faster convergence.Source: §3 · p. 3

4 Reward Functions

The section presents CIDEr as an effective captioning reward, then introduces CIDEnt to correct phrase-matching rewards with entailment so logically inconsistent captions are penalized.

4 Reward Functions

  • The authors also tried MIXER’s time-step-based XE+RL annealing, but report that mixed-loss training maintained caption fluency better.Source: §4 · p. 3
  • CIDEr measures consensus across human references, correlates better with human evaluation than METEOR, ROUGE, and BLEU, and improves with more references.The video-captioning datasets provide 20–40 human references per video.Source: §4 · p. 3
  • CIDEr-RL achieves the best metric improvements and human-evaluation improvements among the tested reward metrics.The paper reports that it outperforms other metric-based rewards in video captioning.Source: §4 · p. 3
  • Undirected n-gram metrics can give high scores to captions with a critical wrong action, object, or negation despite the logical mismatch.For example, a football caption can score highly against a basketball reference.Source: §4 · p. 3
  • CIDEnt corrects a phrase-matching reward with a decomposable-attention entailment score, measuring whether a generated caption is implied by a reference.The entailment model is trained on SNLI in the image-caption domain, and the overall entailment score takes the maximum across references.Source: §4 · pp. 3–4
  • When entailment is very low, CIDEnt reduces the metric reward by λ; λ and β are tuned on development data.For CIDEr on MSR-VTT, the reported λ is roughly 0.45 and β is 0.33.Source: §4 · p. 4

5 Experimental Setup

The experiments use two video-captioning datasets, automatic caption metrics, and human judgments of relevance and coherence; hyperparameters are tuned on validation data.

5 Experimental Setup

  • MSR-VTT contains 10,000 videos with 20 references per video, while YouTube2Text/MSVD contains 1,970 videos with 40 references per video.Source: §5 · p. 4
  • Automatic evaluation uses METEOR, BLEU-4, CIDEr-D, and ROUGE-L, alongside human evaluation of caption relevance and coherence.Relevance concerns relation to video content; coherence concerns readability.Source: §5 · p. 4
  • The entailment classifier is reported as 92% accurate on entailment in the caption domain.The authors identify new summarization domains as future settings for which a multi-domain dataset may be used.Source: §5 · p. 4
  • Hyperparameters are tuned on the validation set, and results, including the baseline, use a five-model average ensemble.Source: §5 · p. 4

6 Results

CIDEnt-RL improves over CIDEr-RL on MSR-VTT and shows some gains on YouTube2Text; other reward choices reveal metric–human evaluation trade-offs and repetitive output.

6.1 Primary Results

  • On MSR-VTT, CIDEr-RL significantly improves over Baseline-XE on all metrics, and CIDEnt-RL significantly improves over CIDEr-RL on all automatic metrics and human relevance, reaching Rank1.The models are statistically equal on coherence in both human-evaluation comparisons.Source: §6.1 · pp. 4–5, Table 2 · p. 5

6.2 Other Datasets

  • On YouTube2Text, CIDEr-RL strongly improves over the cross-entropy baseline, while CIDEnt-RL shows some gains over CIDEr-RL on BLEU, CIDEnt score, and human relevance.The human relevance improvement is statistically significant.Source: §6.2 · p. 5

6.3 Other Metrics as Reward

  • BLEU-RL improves BLEU but performs worse than the cross-entropy baseline in human evaluation; BLEUEnt-RL improves BLEU and BLEUEnt scores but is also worse in human evaluation.SPICE reward produced long repetitive phrases.Source: §6.3 · p. 5
  • A small dev set caused tuning issues, prompting plans for a better train/dev re-split.Source: §6.3 · p. 5

6.4 Analysis

  • Figure 1 contrasts CIDEnt-RL’s ground-truth-style caption with a non-entailed caption from CIDEr-RL that still receives a high phrase-matching score.Source: §6.4 · p. 5

7 Conclusion

The paper combines mixed-loss policy-gradient training with an entailment-corrected CIDEnt reward for video captioning, reporting improved results and a new MSR-VTT state of the art.

7 Conclusion

  • CIDEnt further improves results and achieves the new state-of-the-art on MSR-VTT.The conclusion describes it as an entailment-corrected reward that builds on the mixed-loss policy-gradient approach.Source: §7 · p. 5

Supplementary Material

The supplement describes an attention-based LSTM captioning baseline and a policy-gradient objective for optimizing non-differentiable sequence rewards.

A Attention-based Baseline Model (Cross-Entropy)

  • The attention baseline encodes video frames with a bidirectional LSTM and generates captions with a single-layer LSTM using encoder-state attention.The context vector is a weighted combination of encoder hidden states, with weights serving as attention.Source: A Attention-based Baseline Model… · p. 6

B Reinforcement Learning (Policy Gradient)

  • Policy gradients optimize non-differentiable sequence rewards, while a decoder-state baseline reduces single-sample variance without changing the expected gradient.The baseline estimator is a linear regressor that takes the decoder hidden state as input.Source: B Reinforcement Learning (Policy… · p. 6

C.1 MSR-VTT Dataset

The experiments describe the video datasets, automatic metrics, and human-evaluation criteria used to assess the captioning models.

C.1 MSR-VTT Dataset

  • MSR-VTT contains 10,000 clips totaling 41.2 hours, with 20 human-annotated reference captions per video.The authors sample frames at 3 fps and extract Inception-v4 features.Source: §C.1 · p. 7

C.2 YouTube2Text Dataset

  • YouTube2Text contains 1,970 clips with an average of 40 human captions each; the standard split uses 1,200 training, 100 validation, and 670 test clips.Source: §C.2 · p. 7

C.3 Automatic Evaluation Metrics

  • Automatic evaluation uses METEOR, BLEU-4, CIDEr-D, and ROUGE-L with the Microsoft-COCO evaluation server.Source: §C.3 · p. 7

C.4 Human Evaluation

  • Human evaluation compares CIDEnt- and CIDEr-reward models on relevance to video content and caption coherence, including logic, fluency, and readability.Source: §C.4 · p. 7

D Training Details

Training details specify the model ensemble and architecture, mixed-loss optimization settings, and CIDEnt reward scaling.

D Training Details

  • The five-run ensemble averages decoder probabilities; models use a 50-step encoder, 16-step decoder, 1024-unit LSTMs, and projected Inception-v4 features.Word embeddings are 512-dimensional, and the 1536-dimensional image features are projected to 512 dimensions.Source: D Training Details · p. 7
  • Reward models combine cross-entropy and reinforcement losses; mixed-loss learning rates are 1 × 10^-5 on MSR-VTT and 1 × 10^-6 on YouTube2Text/MSVD.Source: D Training Details · p. 7
  • CIDEnt uses λ = 0.45 on MSR-VTT and λ = 0.75 on YouTube2Text/MSVD, roughly matching the baseline cross-entropy model's score on that metric.Source: D Training Details · p. 7

E Analysis

Figure 3 illustrates a caption-quality progression from the cross-entropy baseline to CIDEr-RL to CIDEnt-RL. The analysis attributes CIDEnt-RL’s advantage to checking entailment in addition to phrase overlap.

  • Figure 3 examples show CIDEnt-RL captions outperforming CIDEr-RL captions, which in turn outperform the cross-entropy baseline.Source: E Analysis · pp. 7–8, Figure 3 · p. 8
  • CIDEnt scores highly only when both CIDEr and the entailment classifier score highly.Source: E Analysis · p. 8
  • CIDEr-style metrics can score captions highly for phrase overlap even when they do not entail the ground truth, including captions with a wrong word.Source: E Analysis · p. 8
Loading 1708.02300v1…