Source-linked AI summary
A Dataset for Movie Description
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, Bernt Schiele
TL;DR
Movie-description research lacks large, clean, transcribed DVS aligned to full-length videos, despite DVS being visually grounded and costly to produce. The paper constructs such a dataset with aligned scripts and benchmarks retrieval and semantic-parsing-based generation. DVS is reported to be more correct and relevant to the movie than script sentences, while the adapted approach shows competitive performance on TACoS Multi-Level and promising results on the new data.
Problem
Transcribed DVS suitable for computer vision is scarce, while scripts often mismatch finished movies and require cleaning.
Method
The paper builds a dataset of transcribed, action-aligned DVS and scripts, then benchmarks retrieval and semantic-parsing-based video-description approaches.
Results
DVS tends to be more correct and relevant to the movie than scripts, while the adapted approach achieves competitive TACoS Multi-Level performance and promising results on the new dataset.
Takeaways & Limitations
The dataset supports research on video description, activity recognition, visual grounding, and understanding plots across multiple sentences.
Takeaways & Limitations
The automatically extracted labels remain imperfect because different labels may represent similar concepts and annotations may omit verbs used in collected sentences.
Abstract
from arXiv · showhide
Descriptive video service (DVS) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the Movie Description dataset contains a parallel corpus of over 54,000 sentences and video snippets from 72 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing DVS to scripts, we find that DVS is far more visual and describes precisely what is shown rather than what should happen according to the scripts created prior to movie production.
1. Introduction
DVS offers visually grounded movie descriptions but is costly to produce, motivating automated video-description research. The paper introduces an aligned DVS-and-script dataset, benchmarks generation methods, and finds DVS more faithful to what appears onscreen than scripts.
- Motivation: DVS narrates important visual information, including actions, gestures, scenes, and character appearance, for blind or visually impaired audiences.Descriptions are prepared by trained describers and read by professional narrators.
- Motivation: Up to 60 person-hours may be required to describe a 2-hour movie, leaving only a small subset of movies and television programs available.
- Motivation: Video description requires recognizing scenes, activities, and objects, then deciding which visual information is salient enough to verbalize.
- Motivation: Scripts often mismatch the finished movie, making automatic alignment unreliable and sometimes mixing correct and irrelevant information.Scripts are typically produced before filming, whereas DVS describes what is shown more precisely.
- Contribution: The proposed dataset provides transcribed DVS aligned to full-length HD movies, together with aligned movie scripts, sentences, video snippets, and computed features.The authors manually align each DVS sentence to movie actions and benchmark retrieval and semantic-parsing-based generation.
- Contribution: The first experiments benchmark several description approaches, with semantic parsing enabling a translation model to outperform retrieval on the new dataset.The adapted approach achieves competitive performance on TACoS Multi-Level without using its annotations.
2. Related Work
Prior work uses scripts for movie understanding and DVS mainly for linguistic analysis or limited automation. The paper addresses the lack of clean, transcribed, action-aligned DVS suitable for computer vision.
- Video description: Video-description research commonly uses retrieval or templates, while image-description work more often generates novel sentences from visual-linguistic relations.
- Movie scripts: Movie scripts have supported scene discovery, action annotation, weak supervision, and localization of short actions in longer clips.
- DVS resources: Earlier DVS research analyzed language in a nonpublic corpus containing artifacts such as dialogue and production notes.The paper distinguishes its cleaner corpus, which contains only actual DVS.
- DVS resources: The only cited computer-vision use of DVS segments events and infers character interactions, but relies on an unevaluated heuristic because its DVS is not aligned.
3. The Movie Description dataset
The Movie Description dataset combines transcribed DVS and scripts with temporal alignment to full movies, producing a large open-domain corpus of professionally described video.
- DVS collection: The collection begins with 46 diverse-genre blu-ray movies containing DVS, whose audio is extracted and semi-automatically segmented.
- DVS collection: DVS segments are crowd-transcribed with sentence timestamps, then manually aligned because speech timing can differ slightly from corresponding visual content.
- Script collection: Scripts are mined from web resources, selected for suitable draft labels and DVS overlap, and aligned to movies through subtitles and dynamic programming.
- Filtering: Manual filtering removes introductions and endings, on-screen text, irrelevant descriptions, and sentences about audio, sounds, or music.
- Filtering: 8.9% of script words and 2.7% of DVS words are removed during alignment and filtering.
- Statistics: Over 50K video-sentence pairs spanning more than 56 hours form a large open-domain corpus with professional sentences and multi-sentence description potential.
- Visual features: The released data includes video snippets and uniformly sampled frames, alongside visual features from trajectories, object detectors, and scene classifiers.
4. Approaches to video description
The benchmark compares nearest-neighbor sentence retrieval with translation-based systems that map visual input to sentences through semantic or visual representations.
- Nearest neighbor: Nearest-neighbor retrieval selects the closest training sentence using L1-normalized visual features and intersection distance.
- SMT: The SMT approach first predicts an intermediate semantic representation from visual input, then translates that representation into a natural-language sentence.
- SMT: Because annotated semantic representations are unavailable, the method automatically mines them from sentences using semantic parsing.
- SMT Visual words: A visual-words alternative directly translates activity, object, and scene representations derived from dense trajectories and pretrained recognition models.
5. Semantic parsing
The paper introduces semantic parsing to extract structured annotations from natural-language descriptions, reducing reliance on manual labels. It applies these representations to video-description generation and finds competitive results, while noting imperfections in automatically extracted labels.
- 5. Semantic parsing: Semantic parsing extracts annotations from natural sentences so video-description models can be trained without tedious manual annotation.The approach is evaluated on a corpus with available annotations before being applied to the movie-description dataset.
- 5. Semantic parsing: SRL and WSD map sentence words into semantic roles and WordNet senses, exemplified by Agent, Action, Patient, and Location assignments.For “He shot a video in the moving bus,” the parse identifies the man, shooting action, video, and bus location.
- 5. Semantic parsing: The parser decomposes clauses, chunks phrases, and disambiguates noun-phrase and verb-phrase head senses using ClausIE, OpenNLP, and IMS.Phrase-level disambiguation links expressions such as “moving bus” to bus#1 and “begin to shoot” to shoot#2.
- 5. Semantic parsing: VerbNet supplies role labels through syntactic-frame matching followed by semantic restrictions on argument types.Only syntactic candidates satisfying VerbNet’s semantic restrictions are accepted.
- 5. Semantic parsing: The extracted roles are grouped into SUBJECT, VERB, OBJECT, and LOCATION, with labels represented either as text chunks or WordNet senses.Sense-labels group multiple text labels by shared senses, while the resulting representation is a four-role tuple.
- 5. Semantic parsing: Automatically extracted labels remain imperfect, although their quantity is close to that of manual labels.Some verbs in human sentences may be absent because the annotations were created before sentence collection.
- 5. Semantic parsing: The authors conclude that automatic parsing can replace manual annotations while achieving competitive results.They subsequently apply the parsing approach to the movie-description dataset.
6. Evaluation
The evaluation compares DVS and script sentences with automatic video-description systems, finding that semantic parsing with SMT outperforms retrieval baselines while human-authored sentences remain strongest.
- DVS–script comparison: 126 sentence pairs from five movies were evaluated by humans for correctness and relevance against the corresponding video intervals.The study used overlapping intervals with at least 75% intersection-over-union and conducted 252 AMT tasks.
- Video description benchmark: 250 randomly selected test snippets were rated by human judges for correctness, grammar, and relevance, using mean rank where lower is better.The test set used five movies, with the remaining 67 movies for training.
- Video description benchmark: Pre-trained visual features—LSDA, PLACES, and HYBRID—outperformed the DT feature among nearest-neighbor retrieval systems, with HYBRID performing best.The comparison is based on mean rankings from the human evaluation.
- Video description benchmark: The actual movie script and DVS sentences ranked significantly better on average than every automatic approach.The “Movie script/DVS” entries are the corpus’s human-authored test sentences rather than generated outputs.
- Video description benchmark: Semantic parsing with SMT outperformed nearest-neighbor baselines and SMT using visual words, while sense labels performed slightly worse than text labels.The authors attribute the sense-label disadvantage to word-sense-disambiguation errors.
7. Conclusions
The paper introduces a movie dataset combining aligned DVS and script descriptions, then evaluates visual-feature retrieval and annotation-free semantic-parsing methods. DVS is more correct and relevant to the movie than script sentences, and the dataset supports larger-scale study of stories and plots.
- Conclusions: The dataset combines movie scripts with transcribed DVS aligned to movies and includes first experiments using visual features and a recent video-description method.The method is adapted to work without manual annotations by relying on semantic parsing of labels.
- Conclusions: The adapted approach achieves competitive performance on TACoS Multi-Level and promising results on the movie-description dataset.
- Conclusions: DVS tends to be more correct and relevant to the movie than script sentences.
- Conclusions: The dataset enables future study of stories and plots across multiple sentences in an open-domain, large-scale setting.