PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, Kate Saenko

arXiv:1604.01729v2cs.CLcs.CV

TL;DR

Video-captioning models rely on scarce paired video-sentence data, while large text corpora offer linguistic knowledge that may aid description. This paper integrates text-trained language models and distributional representations into an LSTM video-captioning model, finding significantly better grammaticality and modestly better descriptive quality across YouTube and movie datasets.

  • Problem

    Video-captioning models trained on limited caption text can show linguistic irregularities, while paired video-sentence data are scarce despite the availability of large raw-text corpora.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    The authors integrate text-trained language models and distributional word representations into an LSTM-based video-captioning model.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Across YouTube and two movie-description datasets, linguistic knowledge significantly improved grammaticality while modestly improving descriptive quality.

    Source: §1 · p. 1, §6 · p. 5

  • Takeaways & Limitations

    The evaluated techniques are described as generic and applicable to other video and image captioning models.

    Source: §6 · p. 5

Abstract

from arXiv · show

This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.

1 Introduction

Video captioning models need large paired video-sentence corpora, but such data are scarce while raw text is abundant. The paper investigates ways to use linguistic knowledge from text corpora to improve video descriptions.

1 Introduction

  • Automatic natural-language video descriptions can support content-based video retrieval and description for visually impaired users.The motivation contrasts scarce paired video-sentence data with widely available raw text corpora.Source: §1 · p. 1
  • The paper explores using language-corpus knowledge to capture general linguistic regularities for video description.Its approaches include pre-training on plain text, integrating an LSTM language model, and using distributional word vectors.Source: §1 · p. 1
  • Across three video-caption datasets, external linguistic knowledge significantly improved grammaticality and modestly improved descriptive quality.Grammar improvements were established through crowdsourced human evaluations; descriptive quality was assessed by human judgments and automated comparisons.Source: §1 · p. 1
  • The study compares approaches on a standard YouTube corpus and two large movie-description datasets.Human judgments show that external linguistic knowledge significantly affects grammar.Source: §1 · p. 1

2 LSTM-based Video Description

The paper uses S2VT, a sequence-to-sequence model that encodes video frames and decodes them into a sentence. Its two LSTM layers separately encode visual features and generate language conditioned on the visual representation.

2 LSTM-based Video Description

  • S2VT maps a sequence of video-frame features to a fixed-dimensional vector, then decodes that representation into output words.Its two LSTM layers encode the video and decode language conditioned on the visual representation.Source: §2 · p. 2
  • The first LSTM encodes frame features extracted from a CNN's penultimate layer, while the second decodes the resulting state into words.The frame features are taken after the ReLU operation.Source: §2 · p. 2
  • The authors modify S2VT to incorporate linguistic knowledge at different stages of training and generation.They describe the methods as sufficiently general to be incorporated into other CNN-RNN captioning models.Source: §2 · p. 2

3 Approach

The paper explores several ways to add prior linguistic knowledge to an S2VT-based video-captioning model. The approaches use text-trained language models and distributional word representations at different points in training and generation.

3 Approach

  • Existing captioning models trained only on caption-dataset text can exhibit linguistic irregularities associated with restricted language models and small vocabularies.The paper evaluates techniques for integrating prior linguistic knowledge into a CNN/LSTM video-to-text network.Source: §3 · p. 2
  • Early fusion pre-trains language-modeling network portions on raw text, then fine-tunes them on paired video-text data.The pretrained weights initialize the S2VT embedding and LSTM layers.Source: §3 · p. 2
  • Late fusion re-scores each video model's word distribution using a weighted average of its scores and the language model's scores.The weighting parameter α is tuned on the validation set.Source: §3 · p. 2
  • Deep fusion concatenates the language model and video model hidden states to predict the output word.The method fixes the language-model weights while training the full video-captioning network.Source: §3 · pp. 2–3
  • The authors replace one-hot word inputs with 300-dimensional GloVe vectors pretrained on six billion tokens.They also explore predicting both one-hot words and GloVe vectors; the vector-prediction loss influences network weights, but its predicted vector is not used.Source: §3 · p. 3

4 Experiments

Across the YouTube and movie datasets, adding linguistic knowledge significantly improved grammaticality, while descriptive-quality gains were modest. Experiments used automated text-comparison metrics and crowdsourced human ratings.

4 Experiments

  • Evaluation used METEOR and BLEU against human descriptions, plus crowdsourced relevance and grammar ratings on a random sample of 200 clips per dataset.Movie datasets were evaluated using METEOR and grammar ratings only.Source: §4 · p. 3

4.1 Youtube Video Dataset Results

  • On YouTube, Deep Fusion performed well on METEOR and BLEU, GloVe embeddings substantially increased METEOR, and their combination performed best.The reported scores use GloVe embeddings only as input; fine-tuning the embedding matrix reduced validation results by 0.4 METEOR.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Movie Description Results

  • Grammar improved significantly on YouTube and movie datasets, while descriptive-quality gains were modest and movie METEOR gains were small but consistent.YouTube human ratings correlated with METEOR; movie grammar ratings improved with p < 0.0001.Source: §4.2 · p. 4, §4.1 · p. 4, Table 2 · p. 4
  • The movie corpora each provide one ground-truth description per video, making learning and evaluation challenging.METEOR scores are fairly low when generated descriptions are compared with a single reference.Source: §4.2 · p. 4, Table 2 · p. 4

5 Related Work

Prior video-description models decode video representations with LSTMs, while this work investigates linguistic knowledge from external text. Its experiments indicate that language-side contributions on YouTube are considerable relative to visual improvements.

5 Related Work

  • The work investigates distributional semantic embeddings and LSTM language models trained on external text to aid existing CNN-RNN video-description models.Related video-description work also used external image-caption data, whereas this paper focuses on linguistic knowledge.Source: §5 · pp. 4–5
  • Unlike the cited MT approach, the monolingual language-model output is used as input when training the full video-description network.The paper applies late and deep fusion approaches using large monolingual-English corpora.Source: §5 · p. 5
  • On YouTube, the experiments show language's contribution is considerable and only slightly less than the visual contribution.The related models discussed focus primarily on improving video representation through hierarchical visual pipelines and attention.Source: §5 · p. 5

6 Conclusion

The proposed techniques improve grammar significantly and descriptive quality modestly across the evaluated datasets. Although evaluated on one video-captioning network, the techniques can also be applied to other video and image captioning models.

6 Conclusion

  • The techniques significantly improve grammaticality while modestly improving descriptive quality across all evaluated datasets.The evaluations use human judgments for grammar and descriptive quality, alongside automated comparisons for descriptive quality.Source: §6 · p. 5
  • Although evaluated on a specific video-captioning network, the techniques are generic and can be applied to other video and image captioning models.Source: §6 · p. 5
  • Figure 4 shows representative movie-description clips for S2VT, Glove, and Glove+Deep, alongside the ground-truth sentence.Source: Figure 4 · p. 5, §6 · p. 5

Related papers

Loading 1604.01729v2…