PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequence to Sequence -- Video to Text

Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko

arXiv:1505.00487v3cs.CV

TL;DR

Open-domain video description requires modeling complex temporal dynamics with variable-length frame inputs and word outputs. The paper proposes S2VT, an end-to-end sequence-to-sequence LSTM trained on paired videos and sentences, using RGB and optical-flow features. It achieves state-of-the-art performance on MSVD and outperforms related work on two challenging movie-description datasets.

  • Problem

    Video description must capture temporal structure while handling variable-length sequences of frames and words in open-domain content.

  • Method

    S2VT uses an end-to-end sequence-to-sequence LSTM that reads CNN-based frame features sequentially and generates caption words sequentially from paired video-sentence data.

  • Results

    S2VT achieves state-of-the-art performance on MSVD and outperforms related work on the M-VAD and MPII-MD movie-description datasets.

  • Takeaways & Limitations

    The model jointly handles variable-length inputs and outputs while modeling temporal structure in videos and language.

Abstract

from arXiv · show

Real-world videos often have complex dynamics; and methods for generating open-domain video descriptions should be sensitive to temporal structure and allow both input (sequence of frames) and output (sequence of words) of variable length. To approach this problem, we propose a novel end-to-end sequence-to-sequence model to generate captions for videos. For this we exploit recurrent neural networks, specifically LSTMs, which have demonstrated state-of-the-art performance in image caption generation. Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip. Our model naturally is able to learn the temporal structure of the sequence of frames as well as the sequence model of the generated sentences, i.e. a language model. We evaluate several variants of our model that exploit different visual features on a standard set of YouTube videos and two movie description datasets (M-VAD and MPII-MD).

1. Introduction

Video description must model both variable-length frame inputs and variable-length word outputs while capturing salient events and temporal context. S2VT addresses this with an end-to-end sequence-to-sequence LSTM trained on paired video clips and sentences.

  • Motivation: Unlike image description, video description must handle a variable-length sequence of input frames as well as a variable-length output sentence.Earlier approaches addressed variable-length inputs through holistic representations, frame pooling, or fixed-frame subsampling.
  • Motivation: Video description is challenging because videos contain diverse content and require selecting salient events in context.The model learns from video clips paired with natural-language sentences describing depicted events.
  • S2VT Approach: S2VT directly maps frames to words with an end-to-end sequence-to-sequence model that processes both inputs and outputs sequentially.The encoding and decoding representations are learned jointly from a parallel video-sentence corpus.
  • S2VT Approach: A stacked LSTM reads CNN features from frames one by one, then generates the caption word by word after all frames have been encoded.The visual inputs can include RGB CNN outputs and optical-flow CNN outputs computed between consecutive frames.
  • Contribution: The sequence-to-sequence design handles variable frame counts, exploits temporal structure, and learns a language model for natural, grammatical sentences without explicit attention.The authors report state-of-the-art performance on MSVD, M-VAD, and MPII Movie Description datasets.

2. Related Work

Prior video-description systems commonly compressed or semantically staged video information before sentence generation, limiting their use of frame ordering or requiring fixed representations. S2VT instead uses recurrent sequence modeling to encode visual sequences and generate captions.

  • Earlier Video Description: Earlier video-captioning work used metadata, clustering, retrieval, or two-stage semantic pipelines that identified objects, actions, and scenes before sentence generation.These approaches often combined individual classifiers with probabilistic graphical models and language models.
  • Image Captioning: Image-captioning systems typically encode an image into a fixed-length CNN representation and decode it into a word sequence, with LSTMs helping model long-range dependencies.The paper adapts this sequence-decoding perspective to videos, whose visual input is itself sequential.
  • Video Captioning: Mean-pooling video methods reduce frame features to one vector before decoding, thereby ignoring frame ordering and temporal information.This limitation motivates modeling the frame sequence directly.
  • Temporal Modeling: Other contemporaneous methods addressed temporal information using fixed-volume 3-D convnets, dense trajectories, or attention over frame features.These approaches differ from S2VT in their fixed-volume assumptions or use of explicit attention.
  • S2VT Compared with Related Work: Unlike encoder-decoder language translation models using separate LSTMs, S2VT uses one LSTM for both encoding and decoding, allowing shared weights.The model also differs from activity-classification systems by generating captions after encoding the complete optical-flow sequence.

3. Approach

S2VT maps variable-length sequences of video frames to variable-length word sequences using LSTMs, jointly encoding visual content and decoding language. The architecture shares parameters across encoding and decoding while incorporating RGB and optical-flow representations.

  • Sequence-to-sequence formulation: S2VT treats video description as sequence translation from frames (x1, . . . , xn) to words (y1, . . . , ym), with potentially different input and output lengths.The model typically receives many more frames than words.
  • Sequence-to-sequence formulation: The model first encodes frames one at a time into a latent representation, then decodes that representation into a sentence one word at a time.This encoding-decoding design addresses variable-length inputs and outputs.
  • Architecture: S2VT uses a single LSTM for both encoding and decoding, allowing parameter sharing between the two stages.This differs from approaches that use separate LSTMs for encoding and output generation.
  • Architecture: A two-layer LSTM stack models visual frames in its top layer and output words in its lower layer, with each layer using 1000 hidden units.The first layer’s hidden representation is provided as input to the second layer.
  • Training and inference: During training, frames are encoded without loss, decoding begins with <BOS>, and the loss is computed only while generating the sentence.The model optimizes the decoding log-likelihood over the training dataset using stochastic gradient descent.
  • Training and inference: At inference, zero padding supplies the visual input during decoding, and words are selected by maximum softmax probability until <EOS>.The explicit end-of-sentence token permits sequences of varying lengths.
  • Input representations: RGB frames and optical-flow inputs are processed through CNNs, while embedded word vectors are combined with visual representations for language modeling.The combined model rescored candidate words using a weighted sum of RGB and flow predictions.

4. Experimental Setup

The evaluation tests S2VT across YouTube and movie-description corpora using METEOR, compares visual feature variants and baselines, and examines temporal ordering. The datasets differ in annotation sources and the number of descriptions per video.

  • Datasets: The experiments use MSVD, MPII-MD, and M-VAD, covering YouTube clips and Hollywood movie snippets with natural-language descriptions.Together, the corpora provide video-sentence pairs from web videos and script- or audio-description data.
  • Datasets: MSVD contains short YouTube clips selected to depict a single activity and paired with English descriptions elicited from annotators.The study uses provided data splits and samples every tenth frame.
  • Datasets: MPII-MD has around 68,000 clips from 94 Hollywood movies, whereas M-VAD has about 49,000 clips from 92 movies with automatically aligned audio descriptions.MPII-MD descriptions come from movie scripts and audio-description data; M-VAD contains only audio-description data.
  • Evaluation: Evaluation uses METEOR, which scores hypothesis-reference alignment through exact, stemmed, paraphrase, and WordNet-synonym matches.The metric’s semantic matching distinguishes it from BLEU, ROUGE-L, and CIDEr.
  • Implementation: Training unrolls the LSTM for 80 time steps, a trade-off between memory use and the number of frames supplied to the model.With frames sampled one in ten, 94% of YouTube training videos fit this limit.
  • Model variants: The model variants compare RGB features from AlexNet and VGG, optical flow, randomly reordered frames, and an RGB–flow ensemble.The ensemble combines RGB (VGG) and Flow (AlexNet) predictions with weighted averaging.
  • Baselines: The study compares S2VT with factor-graph, mean-pooled, Soft-Attention, and Temporal-Attention approaches.The comparisons include mean-pooled baselines using different CNN features and attention-based video models.

5. Results and Discussion

S2VT performs strongly across YouTube and movie-description datasets, with results showing benefits from temporal modeling and combining RGB with optical flow. Generated descriptions are often relevant, although many closely resemble training sentences.

  • MSVD dataset: 27.9% METEOR improves over mean-pooled baselines at 26.9% and 27.7% on MSVD.The basic S2VT AlexNet model uses RGB video frames.
  • MSVD dataset: Randomly ordering frames substantially lowers METEOR, showing that S2VT benefits from exploiting temporal structure.
  • MSVD dataset: 42.9% of MSVD predictions exactly match a training sentence, while another 38.3% are within one edit of one.Many generated descriptions are reported to be relevant.
  • Movie description datasets: 7.1% METEOR on MPII-MD improves over SMT at 5.6%, while performance is similar to Visual-Labels.The model used for these movie datasets is S2VT trained on RGB frames and VGG features.
  • Movie description datasets: 6.7% METEOR on M-VAD outperforms temporal attention at 4.3%, mean pooling at 6.1%, and Visual-Labels at 6.3%.
  • Movie description datasets: 7.0% METEOR is achieved on the public LSMDC test set, which combines M-VAD and MPII-MD.

6. Conclusion

The paper presents S2VT as a sequence-to-sequence approach that sequentially encodes frames and generates words, handling variable-length sequences while modeling temporal structure. It achieves state-of-the-art performance on MSVD and outperforms related methods on two challenging movie-description datasets.

  • S2VT sequentially reads frames and generates words, handling variable-length input and output while modeling temporal structure.
  • S2VT achieves state-of-the-art performance on MSVD and outperforms related work on two large, challenging movie-description datasets.
  • Additional data significantly benefits the model, suggesting high capacity to learn complex temporal structure in challenging movie-description datasets.
Loading 1505.00487v3…