Source-linked AI summary
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, Yueting Zhuang
TL;DR
Video representation must capture temporal structure across both local actions and longer-range dependencies, while added recurrent non-linearity can increase computation. HRNE uses a hierarchical recurrent encoder to model video temporal information, and video-captioning experiments on MSVD and M-VAD demonstrate its effectiveness, including the best METEOR result against systems using multiple features.
Problem
Video temporal structure is layered across actions and within actions, but straightforward LSTM implementation cannot model these multiple granularities effectively.
Method
HRNE hierarchically encodes temporal information by processing short LSTM chains and composing consecutive temporal chunks at higher levels.
Results
HRNE demonstrates effectiveness on MSVD and M-VAD video captioning, achieving the best METEOR result on MSVD despite using only one ConvNet feature while compared systems combine multiple features.
Takeaways & Limitations
The generic HRNE video representation is presented as applicable to a wide range of video-analysis applications beyond the captioning setting tested here.
Takeaways & Limitations
The paper leaves video classification as future work for validating the encoder's generalization capability.
Abstract
from arXiv · showhide
Recently, deep learning approach, especially deep Convolutional Neural Networks (ConvNets), have achieved overwhelming accuracy with fast processing speed for image classification. Incorporating temporal structure with deep ConvNets for video representation becomes a fundamental problem for video content analysis. In this paper, we propose a new approach, namely Hierarchical Recurrent Neural Encoder (HRNE), to exploit temporal information of videos. Compared to recent video representation inference approaches, this paper makes the following three contributions. First, our HRNE is able to efficiently exploit video temporal structure in a longer range by reducing the length of input information flow, and compositing multiple consecutive inputs at a higher level. Second, computation operations are significantly lessened while attaining more non-linearity. Third, HRNE is able to uncover temporal transitions between frame chunks with different granularities, i.e., it can model the temporal transitions between frames as well as the transitions between segments. We apply the new method to video captioning where temporal information plays a crucial role. Experiments demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks. Notably, even using a single network with only RGB stream as input, HRNE beats all the recent systems which combine multiple inputs, such as RGB ConvNet plus 3D ConvNet.
1. Introduction
Video representation must capture temporal structure efficiently, but existing approaches struggle with long-range dependencies, computation, and multiple temporal granularities. HRNE addresses these challenges with hierarchical recurrent encoding and demonstrates effectiveness on video captioning.
- Temporal modeling remains a fundamental problem in video representation for computer vision.
- Optical-flow-based methods and two-stream ConvNets require expensive optical-flow inputs, while aggregation methods can miss temporal structure.
- 3D ConvNets avoid optical flow but generally handle only short clips of about 16 frames, while LSTM performance favors clips of 30 to 80 frames.
- HRNE divides long videos into short frame chunks, encodes them with LSTM, and combines chunk outputs for higher-level recurrent processing.
- HRNE shortens information paths while adding non-linearity, reducing computational cost compared with straightforward LSTM stacking.
- HRNE models temporal transitions at multiple granularities, including both within-action and between-action dependencies.
- Applied to video captioning on MSVD and M-VAD, HRNE experiments demonstrate the method’s effectiveness.
2. Related Works
Video representation research progressed from hand-crafted and pooled ConvNet features toward recurrent models that preserve temporal order. Related hierarchical recurrent approaches also include language-side modeling for query suggestion and video caption generation.
- Dense Trajectories represented videos using tracked interest points, local descriptors, and BoW or Fisher-vector aggregation.
- Early ConvNet video methods explored temporal fusion, while two-stream models used stacked optical-flow frames with RGB inputs.
- Average pooling, Fisher vectors, and VLAD treat local features as sets, losing the order-dependent information in video sequences.
- Video captioning systems commonly use LSTMs as language models conditioned on pooled visual features or attention-based context vectors.
- A hierarchical recurrent model for query suggestion first forms query-level states, then models session-level dependencies for generating suggestions.
- Yu et al. use hierarchical GRU generators to model sentence and paragraph dependencies on the language-processing side of video captioning.
3. The Proposed Approach
HRNE hierarchically models video sequences by applying recurrent temporal filters to short chunks and then modeling dependencies among chunk representations. This reduces long-range information paths while supporting multiple temporal granularities and attention-based temporal selection.
- Hierarchical Recurrent Neural Encoder: The model adds non-linearity through hierarchical recurrent layers while reducing the computational burden associated with stacking recurrent layers.The approach is motivated by the usefulness of additional non-linearity and the long paths introduced by straightforward stacking.
- Hierarchical Recurrent Neural Encoder: Temporal filters use short LSTM chains to explore local temporal structure, replacing a single matrix filter because consecutive temporal items have dependencies.The filtering result is the mean of the LSTM chains’ hidden states.
- Hierarchical Recurrent Neural Encoder: HRNE divides long videos into short frame chunks, processes them with LSTM filters, and feeds chunk representations to a higher-level recurrent layer.The first layer captures local temporal structure, while the second models dependencies among subsequences.
- Hierarchical Recurrent Neural Encoder: HRNE shortens input-to-output paths compared with stacked LSTM, using n + ⌈T/n⌉ steps instead of T + 1 steps for a filter length n and sequence length T.For T = 1,000 and n = 30, the path is 64 steps rather than 1,001 steps.
- Attention mechanism: Attention dynamically weights temporal locations and is inserted between visual inputs and recurrent filters, between hierarchy levels, and before the caption decoder.This allows the LSTM to focus on different temporal locations when input and output sequences are not strictly aligned.
- Video captioning: HRNE is demonstrated on video captioning, a task in which temporal information is important.The encoder produces video representations for downstream video processing tasks.
4. Experimental Setup
The experiments evaluate HRNE for video captioning on MSVD and M-VAD using GoogLeNet frame features and comparisons with established captioning baselines.
- Datasets: Experiments use the MSVD and M-VAD video captioning benchmarks to validate HRNE.MSVD contains short clips with multiple descriptions, while M-VAD contains clips from DVD movie narrations.
- Visual features: GoogLeNet extracts frame-level visual features, with videos truncated or zero-padded to 160 frames before a learned linear embedding.The embedded features are then provided as input to HRNE.
- Evaluation: Evaluation mainly uses METEOR because prior analysis found it more consistent with human judgment than BLEU and ROUGE.BLEU, ROUGE-L, and CIDEr are also standard captioning metrics used in the evaluation framework.
- Baselines: The comparison includes factor-graph, mean-pooling, encoder-decoder, temporal-attention, embedding, and paragraph-RNN captioning systems.These baselines use varied strategies for visual aggregation, recurrent encoding, attention, feature-text embedding, or hierarchical language decoding.
- Optimization: Training uses dataset-specific LSTM widths, ADAM optimization, a batch size of 128, and early stopping or a 200-epoch limit.The reported learning rate is η = 2 × 10^-4, with additional regularization applied because of overfitting on M-VAD.
5. Experimental Results
HRNE is evaluated on MSVD and M-VAD against systems using static or fused visual features. It achieves strong METEOR results with a single RGB feature stream, including further gains from attention.
- MSVD static features: On MSVD with static frame-level features, HRNE outperforms mean pooling, temporal attention, paragraph RNN, and S2VT, while attention further improves it.The paper attributes the advantage over S2VT to shorter input flow and multi-granularity temporal encoding.
- MSVD fusion comparison: With one GoogLeNet feature stream, HRNE achieves the best MSVD METEOR result against systems combining multiple ConvNet features.Compared systems include fusion of RGB, optical flow, VGGNet, AlexNet, C3D, or GoogLeNet features.
- MSVD fusion comparison: 29.8% METEOR is achieved by S2VT with RGB frames on VGGNet and optical flows on AlexNet, while HRNE achieves the best METEOR in the fusion comparison.SA reaches 29.6% METEOR and 41.9% BLEU-4; p-RNN exceeds HRNE on BLEU but not in almost all other reported cases.
- Qualitative results: Generated-caption examples indicate that HRNE can describe difficult videos accurately, with attention generally improving results.The attention trend is consistent with the reported MSVD results.
- M-VAD: On M-VAD, HRNE reaches 5.8% METEOR, improving to 6.8% with attention and outperforming S2VT and SA3.The attention-enhanced result also exceeds S2VT trained with both M-VAD and MPII-MD.
6. Conclusions and Future Work
HRNE generates video representations focused on temporal modeling and addresses longer-range, nonlinear, multigranular temporal structure. Extensive video-captioning experiments demonstrate its efficacy, while video-classification validation remains future work.
- HRNE models video temporal information with a hierarchical recurrent encoder focused on temporal modeling.
- HRNE reduces input-flow length, supports longer-range temporal structure, adds nonlinearity and flexibility, and models transitions at multiple granularities.
- Extensive video-captioning experiments demonstrate the efficacy of HRNE.
- The encoder’s generalization to video classification is identified as future work.