Source-linked AI summary
Hierarchical Boundary-Aware Neural Encoder for Video Captioning
Lorenzo Baraldi, Costantino Grana, Rita Cucchiara
TL;DR
Video-captioning encoders must represent long, structured videos without relying on continuous recurrence that mixes distinct temporal segments. This paper introduces a boundary-aware LSTM encoder that detects discontinuities, adapts its temporal connections, and forms hierarchical segment representations. Across three datasets, it discovers suitable video decompositions and improves movie-description state-of-the-art results while remaining competitive on short user-generated videos.
Problem
Plain recurrent video encoders struggle with long sequences and layered video structure, while distinct shots or actions can require separating temporal segments.
Method
A boundary-aware LSTM detects appearance or action discontinuities, resets its state and memory, outputs segment summaries, and passes them to another recurrent layer for hierarchical encoding.
Results
The method improves state-of-the-art movie-description results and is competitive on short user-generated video across M-VAD, MPII-MD, and MSVD.
Takeaways & Limitations
Learned temporal boundaries provide an adaptive hierarchical video representation for captioning.
Takeaways & Limitations
MSVD mainly contains short clips with a single action and is therefore less appropriate for evaluating the method’s ability to identify video structure.
Abstract
from arXiv · showhide
The use of Recurrent Neural Networks for video captioning has recently gained a lot of attention, since they can be used both to encode the input video and to generate the corresponding description. In this paper, we present a recurrent video encoding scheme which can discover and leverage the hierarchical structure of the video. Unlike the classical encoder-decoder approach, in which a video is encoded continuously by a recurrent layer, we propose a novel LSTM cell, which can identify discontinuity points between frames or segments and modify the temporal connections of the encoding layer accordingly. We evaluate our approach on three large-scale datasets: the Montreal Video Annotation dataset, the MPII Movie Description dataset and the Microsoft Video Description Corpus. Experiments show that our approach can discover appropriate hierarchical representations of input videos and improve the state of the art results on movie description datasets.
1. Introduction
Video captioning requires encoders that handle long, hierarchically structured videos rather than treating all frames as one continuous sequence. The paper proposes a boundary-aware recurrent encoder that detects temporal discontinuities, adapts its connectivity, and improves representation and captioning performance.
- Motivation: Video captioning is important for video retrieval, content search, automatic subtitles, and accessibility for visually impaired people.The task combines computer vision and language to describe video content in natural language.
- Problem: Plain recurrent encoders struggle with long captioning sequences and cannot adequately represent the layered structure of videos.LSTMs show good learning capabilities on sequences of 30–80 frames, shorter than many video-captioning inputs.
- Problem: Edited videos contain shots with different appearances, while user-generated videos can combine multiple actions within one shot.The encoder must model temporal dependencies both within actions and across interactions without mixing distinct segments.
- Approach: The proposed time boundary-aware LSTM detects action or appearance discontinuities and adaptively modifies temporal connectivity to produce variable-length video encodings.Its encoding granularity and length depend on the input video itself.
- Approach: The boundary-aware LSTM resets its state and memory when required, while encoding content and temporal structure in a trainable end-to-end layer.This cell includes a boundary detection module that discovers discontinuities in the input video.
- Evaluation: The hierarchical encoder is evaluated on M-VAD, MPII-MD, and MSVD, improving state-of-the-art movie-description results while remaining competitive on short user-generated videos.The experiments also investigate whether the encoder discovers appropriate input-video decompositions.
2. Related Works
Video-captioning research moved from template-based and image-like methods toward recurrent sequence models, but standard encoders do not explicitly adapt to video structure. The paper instead learns temporal segments and builds hierarchical representations from them.
- Prior captioning approaches: Early video-captioning methods used visual subject–verb–object triplets and templates, limiting natural-language richness and generalization to unseen data.Recurrent networks subsequently enabled sequence-based sentence generation from visual representations.
- Recurrent approaches: Mean-pooling frame features reduced early video captioning to an image-captioning-like task by ignoring the video’s sequential nature.Later recurrent approaches addressed both video encoding and sentence decoding with sequence-to-sequence architectures.
- Proposed distinction: When a boundary is detected, the boundary-aware LSTM reinitializes its state and outputs a representation of the completed segment.Figure 2 contrasts these reset or modified connections with those of a standard LSTM encoder.
- Hierarchical encoders: Hierarchical recurrent encoders abstracted visual features at multiple timescales by processing overlapping video chunks with stacked recurrent layers.This prior strategy used a hand-crafted sliding-window organization of temporal processing.
- Proposed distinction: The proposed encoder learns its temporal structure from input data rather than imposing a hand-crafted sliding-window variation of LSTM processing.Its strategy is intended to keep each chunk’s cell memory focused on homogeneous information.
3. Method
The method uses a boundary-aware LSTM encoder that detects input discontinuities, adaptively changes temporal connectivity, and builds hierarchical video representations for caption generation.
- Time boundary-aware encoder: The boundary-aware recurrent cell detects appearance or action changes and reinitializes hidden state and memory to separate video segments.Its learnable boundary function depends on the current input and hidden state rather than fixed segment boundaries.
- Hierarchical representation: At each segment end, the encoder outputs a hidden state summarizing that segment, producing variable-length segment representations.A higher-level LSTM receives these segment outputs to form a hierarchical representation of the whole video.
- Training the boundary detector: The boundary detector is trained with a stochastic binary neuron and a differentiable backward estimator, then used deterministically at test time.During training, the step function is approximated by the identity function for backpropagation.
- Sentence generation: The encoder-decoder generates captions by conditioning a recurrent decoder on the video descriptor and preceding caption words.The decoder predicts each next word and models word probabilities with a softmax layer.
- Sentence generation: The decoder uses a GRU with reset and update gates to control how previous hidden-state information contributes to each output.Word vectors are projected into a lower-dimensional embedding space before decoding.
4. Experimental setup
The experiments evaluate video captioning on two movie-description datasets and one short user-generated-video dataset using standard caption metrics and fixed preprocessing and training procedures.
- Datasets: The evaluation covers M-VAD, MPII-MD, and MSVD, with the first two designed for movie description and MSVD containing short YouTube clips.MSVD is considered less appropriate for evaluating video-structure discovery because it mainly contains short clips with a single action.
- Datasets: M-VAD contains 84.6 hours from 92 Hollywood movies and 46,523 video clips aligned with single descriptions.Its descriptions come from Descriptive Video Service audio tracks.
- Datasets: MPII-MD contains over 68K aligned sentences and video snippets from 94 HD movies, with 56,861 training, 4,930 validation, and 6,584 test samples.The video-description alignments were manually corrected.
- Datasets: MSVD contains 2,089 YouTube clips and 85K English descriptions, split into 1,200 training, 100 validation, and 670 test videos.Only English captions are used in the experiments.
- Evaluation: Performance is measured with BLEU-4, ROUGEL, METEOR, and CIDEr using the Microsoft CoCo evaluation toolkit.These metrics compare generated captions with reference descriptions using n-gram, sequence, alignment, and similarity-based criteria.
- Preprocessing: Visual inputs combine ResNet50 appearance features computed every 5 frames with C3D motion features computed every 16 frames.Captions are lowercased, tokenized, and restricted to words appearing at least five times.
- Training details: Training uses Adadelta, minibatches of 128, dropout with retain probability 0.5, 512-dimensional embeddings, and 1024-dimensional recurrent states.Models are trained for up to 100 epochs or until validation loss stops improving.
5. Results and Discussion
The boundary-aware encoder improves video-captioning performance, especially on movie-description datasets, while learning nonuniform temporal boundaries that produce more homogeneous segments. Its learned boundaries outperform equally spaced or externally detected shot boundaries, and remain effective on MSVD despite its lack of hierarchical structure.
- M-VAD: 7.3% METEOR improves over the 6.7% single- and two-layer LSTM baselines on M-VAD.The gain is 0.6 percentage points.
- MPII-MD: On MPII-MD, the approach exceeds state of the art on CIDEr and ROUGEL and nearly matches the semantic method on BLEU-4 and METEOR.It achieves this without exploiting caption semantics or building concept classifiers.
- Qualitative results: The boundary-aware encoder generally generates better descriptions than the plain LSTM encoder on M-VAD and MPII-MD examples.This agrees with the quantitative results reported for both datasets.
- MSVD: On MSVD, where hierarchical structure is absent, the method improves over plain techniques and remains capable of accurate captioning in difficult cases.The experiment tests whether the strategy harms performance when no hierarchy is present.
- Learned boundaries: Less than three boundaries are detected in the vast majority of M-VAD and MPII-MD videos, with cut probability generally increasing between 20% and 80% of video duration.Additional peaks occur near video beginnings and endings, partly because clips are not precisely aligned with captions.
- Learned boundaries: Forcing equally spaced segments reduces METEOR by 0.2% on M-VAD and 0.5% on MPII-MD.The comparison preserves the number of segments detected by the original boundary-aware encoder.
- Learned boundaries: Only 33.7% of learned boundaries lie within 15 frames of shot boundaries, indicating that the model detects camera changes, appearance variations, and softer boundaries.Using external shot boundaries yields 7.1% METEOR on M-VAD and 6.6% on MPII-MD, below the complete model.
6. Conclusion
The paper proposes a boundary-aware video encoder that discovers hierarchical structure and adapts recurrent temporal connections accordingly. It achieves competitive results across popular benchmarks and is presented as potentially applicable beyond video captioning.
- Conclusion: The boundary-aware encoder discovers hierarchical video structure and modifies recurrent-layer temporal connections accordingly.The proposed architecture is described as generic enough for applications such as video classification and action detection.