Source-linked AI summary
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
Jingkuan Song, Zhao Guo, Lianli Gao, Wu Liu, Dongxiang Zhang, Heng Tao Shen
TL;DR
Video captioning systems often apply visual attention to every generated word, including non-visual words that can rely on language context. hLSTMat addresses this with hierarchical LSTMs and adjusted temporal attention, and achieves state-of-the-art performance on MSVD and MSR-VTT.
Problem
Existing video-captioning attention models use visual information at every decoding step without distinguishing visual words from non-visual words.
Method
hLSTMat combines hierarchical LSTMs, temporal attention, and adjusted temporal attention to select relevant frames and decide whether to use visual or language information.
Results
hLSTMat achieves state-of-the-art performance on both the MSVD and MSR-VTT datasets.
Takeaways & Limitations
The framework automatically chooses when and where to use video visual information and when to rely on sentence context during caption generation.
Abstract
from arXiv · showhide
Recent progress has been made in using attention based encoder-decoder framework for video captioning. However, most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g. "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of video captioning. To address this issue, we propose a hierarchical LSTM with adjusted temporal attention (hLSTMat) approach for video captioning. Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict the related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the video caption generation. To demonstrate the effectiveness of our proposed framework, we test our method on two prevalent datasets: MSVD and MSR-VTT, and experimental results show that our approach outperforms the state-of-the-art methods on both two datasets.
1 Introduction
Video captioning must connect visual content understanding with natural language, but existing attention models use video frames at every decoding step without distinguishing visual from non-visual words. The proposed hLSTMat framework combines hierarchical LSTMs with temporal and adjusted temporal attention to select relevant visual information or rely on language context.
- Motivation: Video captioning is challenging because it must integrate diverse temporal visual content with natural-language generation.Compared with image captioning, videos contain diverse objects, scenes, actions, attributes, and salient contents.
- Problem: Existing visual-attention models use video frames at every time step without considering whether the predicted word is visual or non-visual.Words such as “man,” “shooting,” and “gun” have visual signals, whereas “the,” “a,” and “is” can rely on language context.
- Approach: The paper proposes hLSTMat, a unified encoder-decoder framework combining hierarchical LSTMs, temporal attention, and adjusted temporal attention.The framework uses deep neural networks to extract 2D CNN features for each frame before decoding.
- Approach: Temporal attention selects where to attend in the video, while adjusted temporal attention determines when to use visual information or language context.The hierarchical LSTMs are designed to obtain low-level visual information and high-level language context information.
2 The Proposed Approach
The proposed hLSTMat decoder combines hierarchical LSTMs with temporal and adjusted temporal attention to generate video captions from frame-level visual features and language context. Temporal attention selects relevant frames, while adjusted attention controls whether word generation relies on visual information or sentence context.
- Framework: The hLSTMat framework uses a CNN encoder and an attention-based hierarchical LSTM decoder for video captioning.The decoder integrates two LSTM layers and attention mechanisms to model visual and language information.
- Hierarchical LSTM: The bottom LSTM decodes visual features, while the top LSTM mines higher-level language context for caption generation.The two layers are arranged hierarchically so the top layer receives the bottom layer's output.
- Word Prediction: The decoder predicts each vocabulary word from the top hidden state and the adjusted temporal attention vector.The resulting word probabilities are optimized by minimizing the negative log-likelihood of the target sentence.
- Temporal Attention: Temporal attention dynamically weights frame-level features instead of collapsing all frames into a single average vector.The attention weights quantify feature relevance at each time step and produce a context vector from the video frames.
- Temporal Attention: The model uses the current hidden state from the first LSTM layer to compute temporal context focused on salient video features.This differs from using the previous hidden state for context computation.
- Adjusted Temporal Attention: Adjusted temporal attention determines when to use visual information and when to rely on language context for predicting the next word.It aims to use nearly no visual information for non-visual words and the most relevant visual information for visual words.
3 Experiments
The experiments evaluate the video-captioning algorithm through CNN-encoder comparisons, component analyses, and comparisons with state-of-the-art methods.
- The evaluation studies CNN encoder influence, proposed-component effectiveness, and performance against state-of-the-art methods.
3.1 Datasets
The study uses two publicly available video-captioning datasets: MSVD and MSR-VTT. They differ in scale, annotations, vocabulary coverage, and video-content diversity.
- The experiments use two publicly available datasets that were widely used in previous work.
- MSVD: MSVD contains 1,970 short clips, approximately 80,000 description pairs, and about 16,000 vocabulary words.The split includes 1,200 training, 100 validation, and 670 testing videos.
- MSR-VTT: MSR-VTT contains 10,000 web clips, approximately 20 sentences per clip, 20 categories, and 200,000 clip-sentence pairs.The dataset covers a wide variety of visual content.
3.2 Implementation Details
Implementation uses dataset-specific text preprocessing, special sentence-boundary tokens, standard captioning metrics, and regularized mini-batch optimization.
- Text and video preprocessing: MSVD descriptions are lowercased and tokenized with NLTK wordpunct, while MSR-VTT descriptions are split on spaces.The resulting training vocabularies contain 13,010 and 23,662 words, respectively.
- Sentence generation: Training adds <BOS> and <EOS> tokens, while testing begins with <BOS> and selects the maximum-probability word until <EOS>.
- Optimization: LSTM units and word embeddings are both size 512, with dropout 0.5, element-wise gradient clipping at 10, and early stopping based on validation performance.Optimization uses Adadelta with batch sizes 64 for MSVD and 256 for MSR-VTT.
- Evaluation: Performance is evaluated using the standard BLUE and METEOR metrics.
3.3 The Effect of Different CNN Encoders
The CNN-encoder experiment on MSVD finds ResNet-152 performs best among the tested encoders, although its advantage over Inception-v3 is small.
- Inception-v3 is a strong competitor, reaching 82.7% B@1, 72.0% B@2, 62.5% B@3, 51.9% B@4, and 33.5% METEOR.The reported gap between Inception-v3 and ResNet-152 is very small.
3.4 Architecture Exploration and Comparison
The architecture study compares basic LSTM, hLSTMt, hLSTMat, and prior methods on MSVD, finding hLSTMat strongest across every reported metric. Adjusted attention improves performance beyond temporal attention alone.
- 82.9% B@1, 72.2% B@2, 63.0% B@3, 53.0% B@4, and 33.6% METEOR are achieved by hLSTMat on MSVD.
- hLSTMat outperforms basic LSTM, hLSTMt, MP-LSTM, and SA across all reported metrics.
- Adjusted attention improves video-captioning performance compared with hLSTMt without the adjusted mechanism.
3.5 Compare with the-state-of-the-art Methods
Across MSVD and MSR-VTT comparisons, the proposed approach achieves strong results against methods using static, combined, or temporal features. The reported gains include higher MSVD scores than key baselines and best performance on all MSR-VTT metrics.
- Static frame-level features: 8.7% B@4 and 2.5% METEOR improvements over p-RNN are reported when using static frame-level features on MSVD.
- Static frame-level features: 53.0% B@4 and 33.6% METEOR outperform HRNE and HRNE-SA on MSVD.
- Combined spatial and temporal features: p-RNN(V+C) reaches 49.9% B@4 and 32.6% METEOR, exceeding p-RNN(V) and p-RNN(C).V denotes VGGNet, C denotes C3D, and the table compares methods using different feature combinations.
- Comparison with multiple deep features: 53.0% B@4 and 33.6% METEOR are achieved using only ResNet-152, exceeding approaches combining multiple deep features.
- Human evaluation: Questionnaire results favor the proposed method in Overall Quality and Caption Accuracy, but p-RNN scores higher in Information coverage.The questionnaire was collected from ten users with different academic backgrounds.
- MSR-VTT comparison: 38.3% @B4 and 26.3% METEOR are the best reported MSR-VTT results, improving over temporal attention alone by 1.1% @B4 and 0.2% METEOR.
4 Conclusion and Future Work
The paper concludes that hLSTMat combines hierarchical LSTMs with temporal and adjusted temporal attention to select between visual information and sentence context. It reports state-of-the-art performance on MSVD and MSR-VTT, while proposing future integration of temporal and visual features.
- hLSTMat integrates hierarchical LSTMs, temporal attention, and adjusted temporal attention to choose between visual information and sentence context.
- The framework considers low-level video visual features and language context information simultaneously.
- State-of-the-art performance is reported on both the MSVD and MSR-VTT datasets.
- Future work will incorporate both temporal and visual features to test performance.