Source-linked AI summary
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu
TL;DR
Video captioning seeks to generate one or more sentences describing realistic videos, while prior deep-learning work largely focused on single short clips. The paper uses hierarchical RNNs with attention-based sentence generation and paragraph-level inter-sentence modeling, achieving reported state-of-the-art results on two large-scale datasets.
Problem
Deep-learning video captioning had mainly addressed single short clips, leaving paragraph generation for long, semantically rich videos insufficiently addressed.
Method
A hierarchical RNN stacks a temporal- and spatial-attention sentence generator beneath a paragraph generator that uses sentential embeddings and paragraph history to initialize successive sentences.
Results
The approach achieves state-of-the-art results on YouTubeClips and TACoS-MultiLevel, with reported BLEU@4 scores of 0.499 and 0.305, respectively.
Takeaways & Limitations
Modeling inter-sentence dependency enables the approach to generate paragraphs for long videos rather than treating sentences independently.
Takeaways & Limitations
The method struggles with very small or occluded objects and passes sentential information through the paragraph recurrent layer only unidirectionally.
Abstract
from arXiv · showhide
We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.
1. Introduction
Video captioning aims to generate one or more sentences describing unconstrained videos, but earlier deep-learning methods primarily handled single short clips. The paper proposes hierarchical RNNs to generate coherent paragraphs for longer videos.
- Motivation: Video captioning generates one or multiple sentences describing video content, including general YouTube videos and fine-grained cooking activities.The authors motivate applications including video retrieval, automatic subtitling, and blind navigation.
- Limitations of prior work: Large-scale datasets such as YouTubeClips and TACoS-MultiLevel make video captioning more challenging, and existing methods generally perform poorly on them.These datasets contain thousands of lexical entries and dozens of hours of videos.
- Limitations of prior work: Earlier RNN methods focused on a single sentence for a short video clip, while paragraph generation for long videos had not been attempted with deep learning.Graphical-model approaches could generate multiple sentences, but their results remained far from perfect.
- Approach: The proposed hierarchical RNN models temporal dependencies among paragraph sentences so later descriptions can reflect earlier semantic context.The paper illustrates this with sequential cooking activities, where one action makes particular subsequent actions more likely.
- Approach: The framework combines a sentence generator with a paragraph generator, using sentential embeddings and paragraph state to produce successive sentence initial states.The sentence generator uses temporal and spatial attention, while the paragraph generator models inter-sentence dependency.
- Evaluation: The approach is evaluated on YouTubeClips and TACoS-MultiLevel and is reported to significantly outperform other state-of-the-art methods.The authors describe it as the first application of hierarchical RNNs to video captioning.
2. Related Work
Video captioning research progressed from rule-based systems to statistical models and neural sequence-to-sequence methods. Neural approaches use visual feature sequences and RNN decoders to generate variable-length descriptions.
- Neural methods: Neural video captioning applies RNNs to translate visual sequences into natural language, following the encoder-decoder paradigm of neural machine translation.An encoder summarizes deep convolutional features, and an RNN decoder generates a variable-length sentence.
- Neural methods: Compared with image captioning, video captioning must model action and motion features and process substantially more data in a sequence-to-sequence structure.Image captioning primarily requires object appearance features and usually has a simpler instance-to-sequence structure.
3. Hierarchical RNN for Video Captioning
The hierarchical RNN combines a sentence generator with a paragraph generator to produce context-dependent video descriptions. Attention mechanisms select visual features, while gated recurrent components model language and inter-sentence context.
- Hierarchical framework: The framework stacks a sentence generator for word-level captioning beneath a paragraph generator that models inter-sentence dependency.The paragraph generator combines sentence embeddings with paragraph history and produces the next initial state for the sentence generator.
- Gated recurrent units: The gated recurrent components adaptively retain or discard information through reset and update gates, addressing simple RNN limitations on long-term dependencies.The reset gate drops potentially irrelevant information, while the update gate controls preservation of the previous hidden state.
- Sentence generator: A recurrent language layer updates at each word and encodes the sentence semantics into a compact representation.The implementation uses a 512-dimensional gated recurrent layer with ReLU activation for language modeling.
- Sentence generator: The sentence generator uses temporal and spatial attention to focus selectively on video features before multimodal fusion.Spatial attention uses features from multiple image patches, while appearance and action channels pass through similar attention processes.
- Sentence generator: Attention produces a weighted feature vector that lets the sentence generator focus on a subset of video features during generation.The model uses separate feature channels for object appearance and action/motion, each with its own learned attention parameters.
- Paragraph generator: After each sentence, the paragraph state reinitializes the next sentence generator state so subsequent sentences incorporate preceding semantic context.The paragraph state combines the second recurrent layer's hidden state with the sentence embedding.
4. Training and Generation
The framework trains paragraph generation with likelihood-based costs and generates paragraphs through beam search, selecting sentences iteratively and using paragraph context to reinitialize the sentence generator.
- Training: The model is trained jointly from scratch with randomly initialized parameters.
- Training: Each training word is scored by its likelihood conditioned on preceding sentences, prior words in the current sentence, and the corresponding video.
- Training: The paragraph cost sums word-level costs across sentences and corresponds to paragraph perplexity given the video.
- Training: Training uses L1 and L2 regularization, BPTT, SGD, RMSPROP, and a learning rate of 10^-4 to reduce overfitting and avoid gradient explosion.
- Generation: Beam search expands candidate word sequences, retains the L lowest-cost sequences, and stores up to J complete sentence candidates.
- Generation: The lowest-cost candidate sentence is passed to the paragraph generator, which reinitializes sentence generation until an end-of-paragraph sequence is produced.
5. Experiments
The experiments evaluate h-RNN on YouTubeClips and TACoS-MultiLevel against state-of-the-art and feature-configuration baselines. Results show strong performance, while human evaluation and generated examples support the value of hierarchical paragraph modeling.
- Datasets: YouTubeClips contains 1,967 open-domain clips averaging 9 seconds, with 80,839 sentences from multiple parallel annotations.Because annotations describe the same interval independently, the dataset evaluates the model with paragraph length N = 1.
- Datasets: TACoS-MultiLevel contains 185 indoor cooking videos averaging 6 minutes, with 16,145 intervals and 52,478 sentences.Annotations pair temporal intervals with short sentences describing fine-grained activities and small interacting objects.
- Evaluation: The evaluation uses BLEU, METEOR, and CIDEr, with higher scores indicating stronger correlation between generated sentences and human judgment.Prior work reported all three metrics for YouTubeClips but only BLEU for TACoS-MultiLevel, so the authors compute additional comparison metrics there.
- YouTubeClips results: 0.499−0.453 in BLEU@4 improves h-RNN over LSTM-E on YouTubeClips, while h-RNN also performs better across all three reported metrics.The comparison includes feature-only h-RNN-Vgg and h-RNN-C3D baselines.
- TACoS-MultiLevel results: 0.305−0.292 in BLEU@4 improves h-RNN over LRCN on TACoS-MultiLevel and exceeds the other state-of-the-art and baseline methods.The results also indicate that both video features and the hierarchical structure are important for this task.
- Qualitative analysis: Generated examples show h-RNN avoids repeated events and captures event ordering that independently generated RNN-sent sentences miss.The figure contrasts errors in red with h-RNN outputs in green across corresponding time intervals.
- Human evaluation: 8.50% improvement over RNN-cat is obtained in human preference selections for h-RNN after excluding intervals with identical generated sentences.The evaluation used 3,148 intervals and recorded 773 selections for h-RNN versus 472 for RNN-cat.
- Limitations: The method remains limited by small-object detection errors, unidirectional paragraph information flow, and discrepancy between training and evaluation objectives.On TACoS-MultiLevel, object confusion includes cucumber versus carrot and mango versus orange; early incorrect sentences can pass misleading information forward.
6. Conclusion
The paper proposes a hierarchical-RNN framework that models inter-sentence dependency to generate paragraph descriptions for long videos, achieving state-of-the-art results on two large-scale datasets.
- The hierarchical-RNN framework models inter-sentence dependency while generating a sequence of sentences from video data.
- The approach generates paragraphs for long videos and achieves state-of-the-art results on two large-scale datasets.