Source-linked AI summary
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko
TL;DR
Video description remains difficult because open-domain videos have scarce paired descriptive data and large, complex language spaces. The paper uses a unified convolutional-recurrent network, initialized with image classification and captioning knowledge, to translate video pixels directly into sentences. Its fully deep models outperform previous work on sentence generation, while image-description pretraining improves performance; temporal information remains underused.
Problem
Open-domain video description is constrained by scarce paired video-sentence data, large vocabularies, and complex frame-word dependencies.
Method
A unified convolutional-recurrent network translates video pixels directly to sentences, transferring knowledge from image classification and captioning datasets.
Results
Fully deep video-to-text models outperform previous work, and image-description data improves performance across evaluation criteria.
Takeaways & Limitations
End-to-end sentence generation and auxiliary image data provide a viable way to improve open-domain video descriptions despite limited video-description data.
Takeaways & Limitations
The approach falls short in better utilizing temporal information in videos.
Abstract
from arXiv · showhide
Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.
1 Introduction
The paper targets open-domain video description, where large vocabularies, scarce paired video-text data, and complex frame-word dependencies challenge existing methods. It proposes an end-to-end convolutional-recurrent model that transfers knowledge from image classification and captioning data.
- Open-domain videos remain difficult to describe because existing solutions often restrict actions and objects to narrow domains with small vocabularies.
- Limited paired video-sentence data and complex dependencies between frame sequences and word sequences further constrain video description.
- The system translates video pixels directly into natural-language sentences with a single deep neural network.
- A convolutional network processes video frames, while a recurrent network models the meaning state and word sequence.
- The approach connects an LSTM directly to a convolutional network, avoiding supervised intermediate role representations and rigid sentence templates.
- The model transfers still-image classification and captioning knowledge to video, and the paper reports evaluation on the YouTube corpus with improvement over prior work.
2 Related Work
Related work largely addressed video description through narrow-domain systems, retrieval or tagging, and two-step role prediction followed by template generation. Image-captioning advances and recurrent neural networks motivate this paper’s direct end-to-end video-to-text approach.
- Most prior video-description research focused on narrow domains with limited vocabularies of objects and activities.
- In-the-wild online-video research commonly emphasized retrieval and event-tag prediction rather than generating descriptive sentences.
- Previous YouTube-corpus methods detected fixed subject-verb-object-style role tuples before generating grammatical sentences with templates.
- Deep-learning breakthroughs in still-image captioning established a closely related image-to-text research direction.
- Recurrent neural networks with LSTM units offered a natural way to combine vector representations of visual inputs with sequence generation.
- Unlike prior end-to-end alternatives that used CRF-based intermediate roles, this approach decodes sentences directly from convolutional features and avoids semantic-role labeling.
3 Approach
The approach translates videos into sentences by combining convolutional visual features with recurrent sequence generation. A CNN summarizes sampled video frames, and stacked LSTMs decode that representation word by word, using image-caption data to supplement limited video-description training data.
- The model applies image-to-text translation to videos by transforming visual content into a fixed-dimensional vector and decoding it with a recurrent neural network.The visual vector represents the video, while the sequence model generates an English sentence.
- The network uses two stacked LSTM layers to decode the visual feature vector into textual output while modeling long-range temporal dependencies with gated memory.The LSTM memory cell is controlled by input, output, and forget gates.
- Each video is represented by mean-pooled fc7 features extracted from sampled frames, producing a single 4,096-dimensional vector for the LSTM.Frames are sampled at one in every ten frames before mean pooling.
- During training, the model predicts each next word from the visual features and preceding words while optimizing sentence log-likelihood with stochastic gradient descent.Teacher-forced ground-truth words are used during training; inference proceeds sequentially until the end-of-sentence token.
4 Experiments
Experiments use the Microsoft Research Video Description Corpus and image-description data to train and compare LSTM video-to-text models against role-based baselines. Evaluation combines SVO accuracy, automated sentence-generation metrics, human judgments, and individual-frame tests.
- Datasets: The Microsoft Research Video Description Corpus contains 1,970 YouTube snippets, with roughly 40 English descriptions per video; experiments use 1,200 for training, 100 for validation, and 670 for testing.Clips last 10–25 seconds and typically depict a single activity or short sequence.
- Datasets: Image-caption data supplement scarce video data: Flickr30k contains about 30,000 images with five or more descriptions, while models are also trained on COCO2014 before video fine-tuning.The model variants include YouTube-only, Flickr-fine-tuned, COCO-fine-tuned, and combined Flickr–COCO models.
- Models: Baseline systems HVC and FGM predict subject, verb, object, and scene information, with FGM combining visual confidences and text-derived knowledge before template-based sentence generation.HVC predicts confidence over 45 subjects, 218 verbs, and 241 objects.
- Evaluation Metrics and Results: For the LSTM models, subject, verb, and object predictions are extracted from dependency parses of generated sentences and evaluated against valid or most frequent ground-truth triples.The valid-triple and most-frequent-triple criteria are reported separately in Tables 1 and 2.
- Evaluation Metrics and Results: Generated sentences are evaluated with BLEU-4 and METEOR, while human studies assess video relevance through rankings and grammatical correctness through ratings.The human evaluations use 1–5 scales, and higher values indicate better performance.
5 Discussion
The experiments show that image-description pretraining and full-sentence training improve video description, while temporal representation and grammatical correctness remain imperfect.
- Base LSTM model: Full-sentence training enables the LSTM to outperform template-based generation on BLEU and METEOR despite slightly lower SVO binary accuracy than prior work.The model learns fluency and vocabulary in context rather than explicitly optimizing subject, verb, and object identification.
- Transferring helps: Image-description pretraining improves performance across all evaluation criteria.COCO pretraining produces larger gains in object and verb prediction, while combined Flickr30k and COCO pretraining yields only marginal improvement, possibly from overfitting.
- Automated evaluation: Fully deep video-to-text models outperform previous work in automated evaluation.The discussion identifies full-sentence training as probably the main reason for these improvements.
- Mean pooling is significant: Mean pooling frame features gives significantly better results than training and testing on individual video frames.This suggests mean pooling is a reasonable representation for short clips when generating simple sentential descriptions.
- Human evaluation: Generated sentences are judged more relevant to video content than previous models, but remain far behind human ground-truth descriptions.The template-based FGM receives the highest sentence-only grammaticality ratings, while the LSTM sometimes repeats prepositions and articles.
6 Conclusion
The paper presents an end-to-end neural video-description pipeline from pixels to sentences and reports better sentences than related approaches. Image-description data improves performance, but the approach does not yet fully exploit video temporal information.
- 6 Conclusion: The model uses neural networks throughout the pipeline from video pixels to sentences, enabling potential end-to-end training and tuning.The conclusion frames this as the paper’s proposed video-description approach.
- 6 Conclusion: The approach generates better sentences than related approaches in extensive experimental evaluation.
- 6 Conclusion: Image-description data improves performance compared with relying only on video-description data.
- 6 Conclusion: The approach falls short in utilizing temporal information in videos, leaving improved temporal modeling as future work.