PaperlayerView exact revision on arXiv

Source-linked AI summary

From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning

Jingkuan Song, Yuyu Guo, Lianli Gao, Xuelong Li, Alan Hanjalic, Heng Tao Shen

arXiv:1708.02478v2cs.CV

TL;DR

Video captioning must account for subjective and model uncertainty that deterministic decoders ignore. The paper proposes MS-RNN, using multimodal and stochastic LSTM components with latent variables, and reports state-of-the-art or comparable benchmark performance while generating multiple descriptions.

  • Problem

    Existing video-captioning approaches propagate deterministic hidden states and ignore subjective judgment and model uncertainty, although the same video can receive different descriptions.

  • Method

    MS-RNN uses latent stochastic variables within a multimodal recurrent encoder-decoder framework to model uncertainty and generate multiple video descriptions.

  • Results

    MS-RNN outperforms state-of-the-art benchmarks on MSVD and achieves state-of-the-art performance with appearance features while remaining comparable to models using appearance and motion features.

  • Takeaways & Limitations

    Modeling uncertainty enables MS-RNN to produce duplicate and comprehensive descriptions consistent with the variable nature of video captioning.

Abstract

from arXiv · show

Video captioning in essential is a complex natural process, which is affected by various uncertainties stemming from video content, subjective judgment, etc. In this paper we build on the recent progress in using encoder-decoder framework for video captioning and address what we find to be a critical deficiency of the existing methods, that most of the decoders propagate deterministic hidden states. Such complex uncertainty cannot be modeled efficiently by the deterministic models. In this paper, we propose a generative approach, referred to as multi-modal stochastic RNNs networks (MS-RNN), which models the uncertainty observed in the data using latent stochastic variables. Therefore, MS-RNN can improve the performance of video captioning, and generate multiple sentences to describe a video considering different random factors. Specifically, a multi-modal LSTM (M-LSTM) is first proposed to interact with both visual and textual features to capture a high-level representation. Then, a backward stochastic LSTM (S-LSTM) is proposed to support uncertainty propagation by introducing latent variables. Experimental results on the challenging datasets MSVD and MSR-VTT show that our proposed MS-RNN approach outperforms the state-of-the-art video captioning benchmarks.

I. INTRODUCTION

Video captioning commonly uses encoder-decoder models with CNN encoders and RNN decoders, but deterministic formulations ignore subjective and model uncertainty. MS-RNN addresses this gap by modeling uncertainty with latent stochastic variables so one video can receive multiple descriptions.

  • Video captioning translates visual content into meaningful textual sentences for applications including video indexing, search, and human-robot interaction.
  • The encoder-decoder scheme typically combines a CNN encoder with an RNN decoder and outperforms traditional methods using hand-crafted features.
  • Existing encoder-decoder approaches focus on bridging video and language, semantic information, attention, and sentence attributes or key concepts.
  • Prior approaches are deterministic and omit subjective judgment and model uncertainty from video-captioning calculations.Different people can provide different descriptions for the same video, while abstraction of the captioning process also creates model uncertainty.
  • MS-RNN introduces latent stochastic variables to model uncertainty and generate multiple sentences describing each video from different aspects.The paper also proposes an end-to-end framework and a multimodal LSTM that combines visual and word features into higher-level representations.
  • RNNs process sequential streams by updating hidden states with a deterministic nonlinear function, while most existing RNNs propagate deterministic hidden states.

B. Visual Captioning

Earlier visual-captioning systems used visual information, domain knowledge, CNN features, and RNN language generation. This paper frames captioning uncertainty using latent variables and integrates the resulting decoder into an encoder-decoder approach.

  • B. Visual Captioning: Early video-captioning systems detected visual information, selected predicates and objects using domain knowledge, and generated sentences with syntactic rules.
  • B. Visual Captioning: Later systems extracted video appearance features with CNNs and used RNNs with previous words to predict the next word through a softmax.
  • C. What is Uncertainty: Uncertainty includes measurement and systematic errors, natural variation, inherent randomness, model uncertainty, and subjective judgment.
  • D. Variational auto-encoder: A VAE captures variation by introducing latent random variables and rewriting the objective around observed variables and their latent representation.
  • D. Variational auto-encoder: For video captioning, MS-RNN adapts this latent-variable idea to conditional sentence generation using a stochastic layer to capture uncertainty information.
  • III. THE PROPOSED APPROACH: The proposed decoder follows the conventional encoder-decoder framework and combines an encoder with the MS-RNN decoder as its major contribution.

A. Problem Formulation

The model formulates captioning as sequential word prediction from video features and introduces an encoder-decoder architecture whose MS-RNN decoder combines multimodal and stochastic recurrent components.

  • A. Problem Formulation: A video is represented by N frame-level features, while its description contains T one-hot word vectors.
  • A. Problem Formulation: At each time step, the model uses the video and preceding words to predict the next word until the end-of-sentence marker.
  • B. Encoder: The encoder computes compact, representative continuous visual features for the decoder from CNN-derived video representations.
  • B. Encoder: Sentence words are mapped from sparse one-hot vectors into embeddings using a learned parameter matrix.
  • C. Decoder with MS-RNN: The S-LSTM propagates uncertainty through latent variables, aligning prior and posterior distributions during training so the learned prior supports testing.
  • C. Decoder with MS-RNN: The listed architecture is presented as an end-to-end multi-modal stochastic RNN for video captioning.
  • C. Decoder with MS-RNN: The architecture includes a basic LSTM for word-level features, an M-LSTM for multimodal encoding, and an S-LSTM for latent-variable introduction.

1) LSTM for Word Features:

The word-feature and multimodal stages use recurrent processing to encode sentence structure and fuse textual features with a pooled video representation into higher-level features.

  • 1) LSTM for Word Features:: A basic LSTM transforms the input word sequence into word features containing encoded temporal information.
  • 1) LSTM for Word Features:: The standard LSTM uses forget, input, and output gates to update its cell state and output activation.
  • 2) Multimodal LSTM Layer:: The M-LSTM takes word features and a video-level feature as inputs to fuse them into a common feature space.
  • 2) Multimodal LSTM Layer:: The video-level feature is obtained with basic mean pooling rather than an advanced temporal or spatial attention mechanism.
  • 2) Multimodal LSTM Layer:: The authors state that attention mechanisms can further boost video-captioning performance, although this model uses basic visual-feature utilization.
  • 2) Multimodal LSTM Layer:: M-LSTM is described as a novel LSTM variant that generates plausible features from multiple views while retaining numerical stability.
  • 1) LSTM for Word Features:: Compared with basic RNN units, LSTM is presented as capable of handling long-term dependencies.
  • 2) Multimodal LSTM Layer:: The M-LSTM combines word features and video input with initialized recurrent vectors to extract higher-level features.

3) Backward Stochastic LSTM:

The backward stochastic LSTM uses future sequence information and reparameterized latent variables to model posterior and prior distributions during caption generation.

  • 3) Backward Stochastic LSTM:: The S-LSTM approximates posterior distributions over sequential latent variables using a backward LSTM and a stochastic unit.
  • 3) Backward Stochastic LSTM:: The backward LSTM computes its output from the next embedded word, the present multimodal feature, and the future recurrent output.
  • 3) Backward Stochastic LSTM:: The stochastic cell samples Gaussian noise and uses the reparameterization trick to construct latent variables from learned means and variances.
  • 3) Backward Stochastic LSTM:: This future dependence supports posterior estimation because the posterior uses present and future information rather than past outputs and deterministic states.
  • 3) Backward Stochastic LSTM:: Feed-forward networks estimate the prior mean and variance from the current multimodal feature and the preceding latent variable.
  • 3) Backward Stochastic LSTM:: The posterior mean and variance are estimated from the backward output and preceding latent variable, while training and testing use different latent sampling expressions.
  • 3) Backward Stochastic LSTM:: The concatenated latent and multimodal features are transformed into a word probability distribution by a softmax layer.

D. Loss Function

The loss function follows variational inference: it combines expected caption log likelihood with a KL divergence between approximate posterior and prior distributions, then optimizes all parameters.

  • D. Loss Function: The objective is based on variational inference and a conditional variational autoencoder loss.
  • D. Loss Function: The ELBO includes expected log likelihood under the approximate posterior distribution Q(z|a, l).
  • D. Loss Function: The expected likelihood is computed from softmax probabilities conditioned on the latent variable and multimodal feature.
  • D. Loss Function: The KL term measures the non-symmetric difference between the approximate posterior Q(z|a, l) and prior P(z|l).
  • D. Loss Function: Posterior and prior distributions are factorized over latent variables, with the backward LSTM helping approximate their time-step parameters.
  • D. Loss Function: The global KL divergence is calculated across the whole generated sentence from the time-step divergences.
  • D. Loss Function: All parameters are learned by maximizing the loss, using backpropagation through time and Adadelta optimization.

IV. EXPERIMENT

The model is evaluated on MSVD and MSR-VTT, two standard video-captioning benchmarks with multiple human descriptions and diverse visual content.

  • MSVD contains 1,970 YouTube video clips averaging about 9 seconds, with approximately 80,000 clip-description pairs from Amazon Mechanical Turkers.
  • MSR-VTT is a large-scale dataset collected from a commercial video search engine, covering comprehensive categories and diverse visual content.
  • MSR-VTT is divided into 65% training, 5% validation, and 30% testing subsets.

A. Evaluation Metrics

The experiments use mean-pooled video appearance features, standardized sentence preprocessing, beam-search decoding, and repeated testing to evaluate stochastic caption generation.

  • Performance is evaluated with BLUE, METEOR, CIDEr, and ROUGE-L using the Microsoft COCO evaluation server.
  • Mean pooling aggregates frame-level appearance features into a representative video-level feature, avoiding the extra operations of soft attention.
  • MSVD preprocessing lowercases and tokenizes words, producing a 13,010-word vocabulary; MSR-VTT produces a 23,662-word vocabulary.
  • During testing, the model takes a video and <bos> tag as inputs, then generates captions with beam search.
  • Each test video is input five times to obtain repeated performances and an average performance, while different latent variables can yield different captions.

C. Results on MSVD Dataset

On MSVD, integrating stochastic S-LSTM with multimodal M-LSTM improves evaluation metrics over M-LSTM alone and enables varied, more comprehensive captions.

  • Generated Captions: Repeated testing shows stable and reasonable performance, while generated sentences describe videos from various aspects.
  • Component Analysis: 1.3% M, 3.3% C and 1% RL performance increases result when MS-RNN integrates M-LSTM and S-LSTM instead of using M-LSTM alone.
  • Generated Captions: M-LSTM is deterministic and generates one sentence, whereas M-LSTM+S-LSTM can produce different sentences for a video.
  • Generated Captions: M-LSTM+S-LSTM generally provides more specific, comprehensive, and accurate descriptions than M-LSTM.
  • Generated Captions: The model may produce duplicate and comprehensive results, consistent with the nature of video captioning.
  • Limitations: Wrong captions arise from cooking-video overfitting, omitted motion features, and occasional failure to identify object counts.

E. Comparison Results on MSVD Dataset

On MSVD, MS-RNN with ResNet-based appearance features outperforms or matches existing methods across reported metrics, including methods using attention or motion features.

  • Spatial Representations: MS-RNN (R) achieves the best performance on all evaluation metrics using only appearance features.
  • Spatial Representations: Compared with MFA-LSTM (R), MS-RNN (R) improves B@1, B@2, B@3, B@4, M, and C by 1.6%, 2.8%, 3%, 2.9%, 1.6%, and 5%, respectively.
  • Spatial Representations: ResNet-based video representations perform better than VggNet- and GoogleNet-based features across the compared approaches.
  • Comparison with Existing Methods: Against temporal-attention methods, MS-RNN (R) reports increases of 3.8%, 9.4%, 12.3%, 12.7%, and 4.8% on B@1, B@2, B@3, B@4, and M.
  • Comparison with Motion-Feature Methods: Using only appearance features, MS-RNN achieves B@2 72.6%, B@3 63.5%, B@4 53.3%, and M 33.8%, with comparable B@1 82.9% and C 74.8%.

F. Comparison Results on MSR-VTT Dataset

On MSR-VTT, MS-RNN performs strongly with mean-pooled visual features, surpassing comparable baselines and approaching models using richer visual inputs.

  • MS-RNN achieves 39.8% B@4, 26.1% M, 40.9% C, and 59.3% RL on MSR-VTT.
  • 4% B@4 and 0.8% M improvements over MP-LSTM (VGG-19+C3D) are achieved by MS-RNN (R).
  • MS-RNN (R) exceeds SA-LSTM (VGG-19+C3D) by 3.2% B@4 despite SA-LSTM using soft attention.
  • MS-RNN (R) obtains results comparable to MS-RNN (R+C) on B@4, M, and RL using only a single appearance feature.
  • The framework reports state-of-the-art performance with mean video appearance features and comparable results to methods using appearance and motion features.
  • Future work will integrate attention mechanisms and consider motion features to further improve video captioning performance.
Loading 1708.02478v2…