Source-linked AI summary
Video Captioning via Hierarchical Reinforcement Learning
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang Wang
TL;DR
Fine-grained video captioning remains difficult because existing approaches often produce coarse descriptions of short videos rather than detailed accounts of multiple actions. The paper introduces hierarchical reinforcement learning with a Manager for sub-goals and a Worker for primitive word-level actions, achieving state-of-the-art performance on Charades Captions and MSR-VTT. The authors also identify future gains from multiple feature modalities.
Problem
Existing video-captioning methods can describe short videos coarsely, but detailed captioning of videos containing multiple fine-grained actions remains challenging.
Method
The paper uses hierarchical reinforcement learning with a Manager that designs segment goals, a Worker that generates words, hierarchical attention, and an internal critic.
Results
The HRL model achieves state-of-the-art performance on both the widely used MSR-VTT dataset and the newly introduced Charades Captions dataset.
Takeaways & Limitations
Hierarchical reinforcement learning provides a compositional framework for reinforcing video captioning at different levels and generating fine-grained descriptions.
Takeaways & Limitations
The authors identify reliance on current feature choices as a scope boundary and plan to explore multiple modalities, including C3D features and optical flow.
Abstract
from arXiv · showhide
Video captioning is the task of automatically generating a textual description of the actions in a video. Although previous work (e.g. sequence-to-sequence model) has shown promising results in abstracting a coarse description of a short video, it is still very challenging to caption a video containing multiple fine-grained actions with a detailed description. This paper aims to address the challenge by proposing a novel hierarchical reinforcement learning framework for video captioning, where a high-level Manager module learns to design sub-goals and a low-level Worker module recognizes the primitive actions to fulfill the sub-goal. With this compositional framework to reinforce video captioning at different levels, our approach significantly outperforms all the baseline methods on a newly introduced large-scale dataset for fine-grained video captioning. Furthermore, our non-ensemble model has already achieved the state-of-the-art results on the widely-used MSR-VTT dataset.
1. Introduction
Video captioning must move beyond coarse short-video descriptions to represent multiple dependent, fine-grained actions in detailed language. The paper proposes hierarchical reinforcement learning that divides caption generation into guided segments and reports strong results on Charades Captions and MSR-VTT.
- Motivation: Video captioning requires coherent understanding of multiple scenes to generate several description segments, unlike image captioning’s static-scene setting.
- Motivation: Fine-grained activities can overlap and occur sequentially, making short sentences and заранее predicted temporal intervals inadequate for detailed descriptions.
- Approach: The proposed “divide and conquer” solution splits a long caption into segments, then generates each segment word by word under higher-level guidance.
- Approach: The HRL framework combines a Manager that sets lower-resolution goals, a Worker that selects words, and an internal critic that evaluates segment completion.
- Approach: Hierarchical attention lets the Manager capture wider temporal dynamics while the Worker focuses on local dynamics conditioned on goals.
- Contributions and results: The paper introduces Charades Captions for fine-grained captioning and reports state-of-the-art results on both Charades Captions and MSR-VTT.
2. Related Work
Prior video-captioning work includes sequence-to-sequence, attention, hierarchical RNN, and reinforcement-learning methods, but exposure bias and policy-gradient variance remain challenges. The paper positions its approach as an early application of hierarchical reinforcement learning to vision-language video captioning.
- Video Captioning: Video-captioning research spans sequence-to-sequence generation and improvements including attention, hierarchical RNNs, C3D features, joint embeddings, language fusion, and multitask learning.
- Video Captioning: Maximum-likelihood training creates exposure bias because ground-truth previous words are generally unavailable during testing.
- Reinforcement Learning: REINFORCE directly optimizes nondifferentiable metrics, but policy-gradient estimates can have high variance and instability without suitable normalization.
- Reinforcement Learning: Baselines, actor-critic methods, and related techniques reduce policy-gradient variance by estimating or subtracting reference values.
- Related Captioning Tasks: Dense video captioning detects multiple events and describes each, whereas this method targets one or multiple sentences for continuous action sequences.
- Hierarchical Reinforcement Learning: Hierarchical reinforcement learning uses a high-level agent to set sub-goals at lower temporal resolution and a low-level agent to select primitive actions.
- Hierarchical Reinforcement Learning: The paper identifies its work as the first to apply hierarchical reinforcement learning at the intersection of vision and language.
3. Our Approach
The approach uses an encoder-decoder architecture whose decoder is a hierarchical reinforcement-learning agent. A Manager produces latent goals at lower temporal resolution, while a Worker generates words and an internal critic signals goal completion.
- Overview: Video frame features pass through successive Worker and Manager encoders before the HRL decoder generates a vocabulary sequence.The Worker encoder is a low-level Bi-LSTM and the Manager encoder is a high-level LSTM.
- Overview: The Manager emits goals when needed, and the Worker generates words over subsequent time steps to fulfill each semantic segment.The internal critic sends a binary segment signal to help the Manager update goals, and decoding ends at <EOS>.
- Attention: Both Manager and Worker use attention over video features, with broader temporal focus for the Manager and locally conditioned focus for the Worker.Attention contexts are computed as weighted sums of encoder hidden states, with weights acting as an alignment mechanism.
- Policy Network: The Worker receives the attention context, Manager goal, and previous action, then outputs probabilities over vocabulary actions.The Worker LSTM processes the concatenated inputs before projecting to the softmax layer.
- Internal Critic: The internal critic uses an RNN over the generated word sequence to estimate whether the Worker has reached a goal.Its signal is modeled from the critic hidden state and trained using ground-truth completion signals.
3.3. Learning
Learning alternates between stochastic Worker optimization and deterministic Manager optimization. The framework uses policy gradients, baselines, segment-level rewards, exploration for latent goals, and cross-entropy warm-starting.
- Policy Learning: The Manager uses a deterministic policy for latent goals, whereas the Worker uses a stochastic policy to select words from the vocabulary.This distinction allows simultaneous cross-entropy warm-starting of the composite Manager-Worker agent.
- Worker Learning: The Worker maximizes discounted reward through REINFORCE, with sampled policy-gradient estimates and a baseline to reduce variance.The baseline is estimated by a linear regressor from the Worker hidden state, with gradient flow cut between the Worker and estimator.
- Manager Learning: The Manager is trained through segments generated by a fixed Oracle Worker, so its latent goals are optimized according to segment-level rewards.A Manager goal is followed for c steps, producing a segment and a resulting state and scalar reward.
- Manager Learning: Manager exploration adds Gaussian perturbations to its deterministic policy, while its gradient also uses a baseline estimator based on the Manager hidden state.The noise variance can be selected to suit the environment.
- Reward Definition: Rewards use delta CIDEr for each generated word rather than assigning the final caption CIDEr score to every word.The framework can also accommodate other reasonable rewards, such as deltaBLEU.
- Training Algorithm: Manager and Worker policies are trained alternately, with goal exploration disabled during Worker training and beam search used at test time.Testing requires only one forward pass.
4. Experimental Results
Experiments evaluate the HRL captioning framework on MSR-VTT and the newly introduced Charades Captions dataset using standard captioning metrics and qualitative analyses. HRL outperforms the baselines, converges faster, and shows stronger gains for longer, fine-grained descriptions.
- Evaluation: The experiments use BLEU, METEOR, ROUGE-L, and CIDEr-D with standard MS-COCO evaluation code.
- MSR-VTT Results: HRL outperformed the XE-baseline, RL-baseline, and other listed state-of-the-art methods on MSR-VTT.The RL-baseline improved over the XE-baseline on all metrics, while HRL achieved the strongest overall comparison.
- Charades Captions Results: All HRL variants significantly improved over the baselines on Charades Captions, with larger gains than on MSR-VTT for longer descriptions.Average caption lengths were 24.13 words for Charades Captions and 9.28 words for MSR-VTT.
- Goal Dimension: HRL-16 performed best on almost all metrics, although HRL-64 was slightly better on CIDEr and BLEU@4.The reported ordering was HRL-16 > HRL-32 ≥ HRL-64, indicating that higher goal dimensionality did not guarantee better performance.
- Qualitative Analysis: HRL generated semantically meaningful phases that matched ground-truth captions better than the baselines in qualitative comparisons.The phases were produced through segment-by-segment generation.
- Learning Curves: HRL models converged faster and reached higher CIDEr validation peaks than the baselines, with HRL-16 reaching the highest point.
5. Conclusion
The paper proposes hierarchical reinforcement learning to improve fine-grained video description generation. It reports state-of-the-art performance on both MSR-VTT and Charades Captions, while identifying multimodal features and attention exploration as future directions.
- The paper proposes hierarchical reinforcement learning for generating fine-grained video descriptions containing rich activities.
- The HRL model achieves state-of-the-art performance on MSR-VTT and Charades Captions.
- Future work includes exploring attention space and using multiple feature modalities such as C3D features and optical flow.
A. Attention Visualization
Attention visualization shows that different learned text segments attend to different temporal regions of the video. For the segment “is cooking on the stove,” frames containing cooking receive greater attention.
- Different learned text segments attend to different temporal frames, linking generated segments with video dynamics.
- For “is cooking on the stove,” the first half of the video receives larger attention values because it contains the cooking action.
B. Qualitative Examples on MSR-VTT
Additional MSR-VTT examples indicate that HRL captures more detailed and fine-grained video content than the baseline methods. In one example, it identifies both an event and its scene.
- HRL captures more details and generates more fine-grained descriptions in the additional MSR-VTT examples.
- In one example, HRL identifies both a group of people dancing and the beach setting, whereas the baselines omit the scene.
C. Network Architecture
The experimental architecture combines frame and sequence encoders with a manager, worker, and internal critic. A visualization also shows segment-specific attentions and selected keyframes.
- Encoders: The architecture uses a Bi-LSTM low-level encoder and an LSTM high-level encoder for projected frame features.Frames are sampled at 3fps, extracted with ResNet-152 as 2048-dimensional features, and projected to 512 dimensions.
- Worker: The worker combines an LSTM, attention, word embeddings, and a projection module to produce vocabulary-token probabilities.Its projection sequence is Linear → Tanh → Linear → SoftMax.
- Attention Visualization: Figure 6 visualizes colored text segments, their associated attentions, and a keyframe selected from the most noticeable area for each segment.
- Manager: The manager uses an LSTM, attention, and a linear projection into latent goal space.
- Internal Critic: The internal critic is an RNN with a GRU, word embedding, linear layer, and Sigmoid function.Its hidden and embedding sizes are 128 for MSR-VTT and 64 for Charades Captions.
D. Training Details
Training uses validation-tuned regularization and optimization settings, with separate procedures for cross-entropy models and reinforcement-learning models. XE models use Adadelta and schedule sampling, while RL and HRL models warm-start from pretrained XE models.
- Regularization and Initialization: Hyperparameters are tuned on the validation set, with dropout 0.5, gradient clipping to [-10, 10], and uniform initialization in [-0.1, 0.1].
- Dataset Settings: MSR-VTT uses encoder step size 50 and maximum caption length 30, while Charades Captions uses 150 and 60, respectively.
- Optimization: XE training uses Adadelta with batch size 64, learning-rate reductions based on CIDEr stagnation, and schedule sampling; RL and HRL training warm-start from pretrained XE models.The RL and HRL models continue training with learning rate 0.1.