Source-linked AI summary
Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
Atousa Torabi, Christopher Pal, Hugo Larochelle, Aaron Courville
TL;DR
Video-description research lacked large open-domain paired video–caption datasets, while DVS offered professionally written, temporally aligned movie descriptions. The paper presents an automated extraction and alignment pipeline and uses it to build a large DVS-derived collection. The resulting dataset contains 84.6 hours of paired clips and sentences from 92 DVDs.
Problem
Video-description research lacks large open-domain paired video–caption datasets, while script-based sources can require substantial manual cleaning for accurate alignment.
Method
The paper isolates DVS narration, separates it from dialogue, aligns it with movie audio, segments it using narration pauses, and transcribes the resulting clips.
Results
84.6 hours of paired video clips and sentences were collected from 92 DVDs, yielding 48,986 filtered clips and 55,904 sentences.
Takeaways & Limitations
The M-VAD dataset provides a publicly available, professionally captioned resource for video-description research and defines an official balanced data split.
Takeaways & Limitations
The corpus uses separate dataset splits because vocabulary can differ substantially across movie genres.
Abstract
from arXiv · showhide
In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.
1. Introduction
The paper addresses the lack of large open-domain paired video–caption datasets by using professionally written DVS narrations and developing largely automatic extraction and alignment methods. It introduces the M-VAD dataset, collected from 92 DVDs and intended for video-description research.
- Video annotation requires substantial paired data for high-quality natural-language generation, but large video-description datasets were unavailable.
- DVS provides professionally written descriptions of visual elements, gestures, actions, and interactions, positioned within movies with maximum misalignment limited to 2 seconds.
- Compared with movie scripts, DVS is more closely tied to the released visual content, while script-based alignment can be imprecise and require substantial manual cleaning.
- The authors develop methods for isolating DVS narration and automatically aligning descriptions with movie scenes by detecting short narration pauses.
- The resulting M-VAD dataset was collected from 92 DVDs and is presented as useful for high-capacity deep-learning models for in-the-wild video description.
2. Related Work
Prior work established strong progress in image annotation but left open-domain video annotation constrained by limited paired datasets and narrow domains. The paper positions DVS-derived movie descriptions as a richer alternative.
- Deep-learning approaches have advanced image description generation, supported by large datasets containing paired images and natural-language captions.
- Video annotation has progressed less extensively because open-domain paired video-caption datasets remain scarce.
- Existing video datasets such as TaCos, YouCook, and MSVD are described as small or focused on toy domains with limited vocabularies.
- The paper contrasts its DVS-derived collection with prior DVD-based work that used descriptions from both DVS and movie scripts.
3. DVS-Derived Dataset Collection
The collection pipeline exploits DVS audio’s relationship to movie dialogue: it isolates vocals, separates narration, segments pauses, aligns tracks, and transcribes the resulting clips. This supports scalable dataset construction with limited manual effort.
- DVS audio provides professionally written natural sentences and is increasingly available on DVDs, making it a source for large paired video–sentence datasets.
- DVS segmentation relies on narration occurring during natural dialogue pauses, where vocal-isolated movie audio becomes nearly flat relative to the DVS signal.
- The pipeline isolates vocal content, separates DVS narration from dialogue, and extracts narration segments with a fixed threshold.
- The method aligns movie and DVS audio using FFT-based cross-correlation, then applies LMS noise cancellation and signal subtraction to suppress dialogue.
- The system adds two seconds to each clip end to compensate for potential 1–2-second narration–scene misalignment without human intervention.
- Professional transcription services produce descriptions with more than 98 percent transcription accuracy, while short segments reduce transcription effort and cost.
4. DVS-Derived Dataset Statistics and Comparison With Other Datasets
The dataset contains 48,986 filtered video clips totaling 84.6 hours from 92 DVDs, with 55,904 sentences. Compared with existing resources, it is the largest DVS-derived dataset described and uses professionally written captions across varied movie content.
- 48,986 filtered video clips average 6.2 seconds, totaling 84.6 hours, and are paired with 55,904 sentences from 92 DVDs.Some clips contain more than one sentence.
- The dataset is described as the biggest available DVS-derived dataset, containing twice as many DVDs as the 46-DVD DVS portion of the most similar prior resource.Other existing datasets have fewer videos and narrower domains, with crowdsourced descriptions.
- The corpus has more adjectives than verbs, indicating detailed descriptions of visual-element characteristics, while five of its ten most frequent verbs are seeing-related synonyms.The vocabulary statistics were obtained after tagging the corpus with a POS tagger.
5. DVS-Derived Dataset Corpus Preparation
The corpus preparation includes name normalization, a genre-balanced official split, qualitative LSTM examples, and corpus-level POS statistics.
- Corpus preparation: People’s names are replaced with the token “SOME-ONE,” because removing proper names was beneficial in the authors’ LSTM-based training work.The dataset contains over 500 proper names before replacement.
- Corpus preparation: The official training, validation, and test split contains 38,949, 4,888, and 5,149 video clips, respectively, with DVD genres balanced across sets.Genre balancing reflects vocabulary differences among movie genres.
- Qualitative results: Figure 3 compares four LSTM-generated sentences with their original DVS sentences for films including Charile st. cloud and The big year.The examples are preliminary qualitative results based on the official dataset split.
- Qualitative results: The qualitative samples are meaningful and closely related to the visual context even when they do not match the DVS reference exactly.
6. Conclusion
The paper introduces a large public DVS-derived video dataset and an automatic collection process requiring minimal human intervention. It also provides a balanced split for an official task.
- The authors introduce a large DVS-derived video dataset that is publicly available to the research community.
- Automatic DVS segmentation and alignment enabled dataset collection with minimal human intervention.
- A balanced data split is provided to define an official task.