PaperlayerView exact revision on arXiv

Source-linked AI summary

Movie Description

Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, Bernt Schiele

arXiv:1605.03705v1cs.CVcs.CL

TL;DR

The paper addresses limited access to large, aligned, visually grounded movie-description data and the costly creation of audio descriptions. It constructs LSMDC from transcribed ADs and scripts aligned to full-length movies, then benchmarks and challenges description systems. ADs are more correct and relevant to movies than scripts, while Frame-Video-Concept Fusion receives the strongest human evaluations; long-tail novelty remains difficult.

  • Problem

    Creating ADs can take up to 60 person-hours for a 2-hour movie, leaving only a small subset of movies and TV programs available to blind people.

  • Method

    The paper builds LSMDC from transcribed ADs and movie scripts aligned to full-length movies, and evaluates retrieval, semantic-parsing, LSTM, and challenge approaches.

  • Results

    AD tends to be more correct and relevant to the movie than scripts, while Frame-Video-Concept Fusion receives the best human ratings for correctness, grammar, and relevance.

  • Takeaways & Limitations

    Movie-focused AD data supports video-description research, but human preferences favor medium-length, moderately diverse descriptions over longer and more novel outputs.

  • Takeaways & Limitations

    All approaches face the long-tail distributions of movie data, and additional training data cannot fully resolve novel content in new movies.

Abstract

from arXiv · show

Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. In total the Large Scale Movie Description Challenge (LSMDC) contains a parallel corpus of 118,114 sentences and video clips from 202 movies. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are indeed more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in a challenge organized in the context of the workshop "Describing and Understanding Video & The Large Scale Movie Description Challenge (LSMDC)", at ICCV 2015.

1 Introduction

Audio descriptions make movies accessible to blind and visually impaired people while providing visually grounded data for computer vision and computational linguistics. The paper introduces LSMDC, a large aligned corpus of movie ADs and scripts, and benchmarks description approaches.

  • AD provides audio narratives of important visual information, including actions, gestures, scenes, and character appearance.
  • Describing video requires recognizing scenes, activities, and objects, then deciding which visual information is salient enough to verbalize.
  • Scripts can diverge from filmed content because they are produced before shooting, whereas AD narrations describe key visual elements actually present in the video.
  • The benchmark includes retrieval, semantic-parsing, and Visual-Labels approaches for generating movie descriptions.
  • LSMDC provides transcribed AD and script sentences aligned to full-length movies, with an evaluation challenge and public and blind test sets.
  • Visual-Labels outperforms prior work on both M-VAD and MPII-MD under automatic and human evaluation.

2 Related work

Prior video-description research spans controlled, small-scale, and domain-specific settings, while movie scripts have mainly supported recognition and alignment tasks. LSMDC extends this landscape with movie-focused descriptions and supports subsequent multimodal and temporal-context studies.

  • Earlier video-description work often used controlled settings, small datasets, or single domains such as cooking.
  • Movie scripts have supported scene and action annotation, activity knowledge bases, and script-to-subtitle alignment.
  • Weakly supervised movie-action studies using scripts have used semantic parsing or ordering constraints, but some limited recognition to two frames.
  • Prior AD research examined character interactions, supported AD production with scripts, detected scene boundaries, and analyzed a non-public corpus from 91 movies.
  • LSMDC has enabled visual-semantic movie-to-book embeddings, MovieQA questions combining dialogue and AD, and temporal-context fill-in-the-blank tasks.

3 Datasets for movie description

The dataset combines AD and script collection with audio isolation, transcription, alignment, and visual-feature extraction. Its construction addresses the difficulty of separating AD narration from mixed movie sound and accurately locating descriptions in video.

  • MPII-MD combines AD and script data from Blu-ray movies with sentence-level manual alignment of transcribed audio to video actions.
  • AD collection retrieves Blu-ray audio streams, segments mixed AD tracks, and obtains crowd-sourced transcripts with timestamps for spoken sentences.
  • The corpus includes 39 selected scripts, with only 11 movies overlapping scripts and ADs for direct comparison, yielding 50 scripts total.
  • AD and script sentences are manually aligned to video, while irrelevant, on-screen, and introduction or ending descriptions are filtered out.
  • Video clips are extracted from aligned intervals and represented with improved dense trajectories, object-detection CNNs, and scene-classification CNNs.
  • Vocal isolation exploits the fact that AD narration is inserted during natural dialogue pauses, making the movie signal relatively flat while the narrator speaks.
  • The automatic segmentation pipeline isolates vocals, separates AD narration from dialogue, and thresholds the result to extract AD segments.

3.3 The Large Scale Movie Description Challenge (LSMDC)

LSMDC joins M-VAD and MPII-MD into a movie-focused challenge corpus while preventing movie overlap across splits. The resulting datasets provide large-scale sentence–clip resources and broader coverage than several earlier corpora.

  • LSMDC combines M-VAD and MPII-MD after identifying overlap and excluding script-based alignments from MPII-MD validation and test sets.
  • The combined dataset contains over 118K sentence–clip pairs and 158 hours of video across training, validation, public-test, and blind-test splits.
  • The splits contain 91,908 training, 6,542 validation, 10,053 public-test, and 9,578 blind-test clips and balance movie genres.
  • Vocabulary and part-of-speech statistics are computed after tagging and stemming corpus words, counting each stemmed word and its variants once.
  • Compared with prior corpora, LSMDC focuses on open-domain movies rather than web videos or restricted domains such as cooking.

4 Approaches for movie description

The paper benchmarks SMT- and LSTM-based approaches for generating descriptions from aligned videos and sentences. The LSTM-based Visual-Labels approach uses semantic label groups and group-specific visual features before generating text.

  • Approaches: The two benchmarked approaches are an SMT-based method adapted from prior work and a proposed LSTM-based method.Both approaches use visual classifiers trained on labels extracted from natural-language descriptions.
  • SMT-based approach: The SMT pipeline predicts a Subject-Verb-Object-Location tuple with a CRF, then translates that representation into a sentence.Its intermediate semantic representation is mined automatically from sentences through semantic parsing.
  • LSTM-based approach: The Visual-Labels LSTM uses separate semantic groups and selects visual features most relevant to verbs, objects, and places.Improved Dense Trajectories support verbs, LSDA scores support objects, and PLACES-CNN scores support places.
  • Semantic parsing: Semantic parsing uses semantic role labeling and word-sense disambiguation to map words into roles and WordNet senses.For example, a sentence is converted into Agent, Action, Patient, and Location fillers with disambiguated role fillers.
  • LSTM-based approach: The LSTM receives classifier scores and previously generated words at each time step to generate a description.The study compares one- and two-layer architectures, a factored two-layer variant, and several dropout placements.

5 Evaluation

The evaluation section compares audio descriptions with movie scripts and benchmarks the proposed and related video-description approaches. The supplied passages identify these evaluation components but do not report their outcomes.

  • Evaluation scope: The evaluation compares audio descriptions with movie scripts.The comparison is presented as the first dataset analysis.
  • Evaluation scope: The evaluation benchmarks the video-description approaches introduced earlier and other related work.This benchmarking follows the comparison of AD and script data.
  • Evaluation scope: The supplied evaluation materials include a human-evaluation table for script and AD sentence correctness and relevance to video.The table uses forced choice and majority votes from five judges.

5.1 Comparison of AD vs script data

The study compares ADs and scripts on overlapping movie intervals using human judgments of correctness and relevance to the video. The evaluation retains 279 sentence pairs after overlap filtering and removes two identical pairs.

  • Data selection: The comparison uses 11 MPII-MD movies for which both AD and script data are available.Only overlapping time intervals are selected for comparison.
  • Data selection: 279 sentence pairs remain after selecting intervals with at least 75% intersection-over-union overlap.Two pairs with identical sentences are removed.
  • Human evaluation: Five Amazon Mechanical Turk judges compare each sentence pair for correctness and relevance to the video.Both video intervals are used as references during the forced-choice evaluation.

5.2 Semantic parser evaluation

The semantic parser is evaluated component by component using human judgments. Clause splitting, NLP processing, semantic role labeling, and word-sense disambiguation are assessed, with NLP and WSD identified as the weakest components.

  • Parser evaluation: The evaluation measures clause splitting, NLP tagging and chunking, semantic role labeling, and word-sense disambiguation.Correctness is manually assessed on a randomly sampled set of sentences by human judges.
  • Parser evaluation: NLP and WSD are the poorest-performing components of the semantic parsing pipeline.Some NLP errors arise from incorrect part-of-speech tagging, while WSD is described as difficult.
  • Parser evaluation: The component evaluation focuses on correctness rather than a downstream video-description metric.The reported assessment uses human judgments of sampled parsed sentences.

5.3 Evaluation metrics for description

The evaluation combines automatic caption metrics with human rankings of generated descriptions. Human judges assess correctness, relevance, grammar, and usefulness for blind viewers.

  • Automatic evaluation: Automatic evaluation uses BLEU-1 through BLEU-4, METEOR, ROUGE-L, and CIDEr.Preliminary discussions focus on METEOR, while the final LSMDC evaluation reports all measures.
  • Human evaluation: Human judges rank descriptions from different systems for correctness, relevance, grammar, and helpfulness for blind viewers.The ranking setup compares multiple system outputs using randomized sentences.
  • Human evaluation: Grammar measures fluency and readability independently of whether the sentence correctly describes the video.
  • Human evaluation: Correctness measures whether a sentence's content matches the video, independently of completeness and grammatical quality.
  • Human evaluation: Relevance measures whether a sentence contains salient or important events and objects from the video.
  • Human evaluation: Helpfulness measures how useful a description would be to a blind person trying to understand or follow the movie without seeing it.

5.4 Movie description evaluation

The study evaluates semantic-parsing, retrieval, and Visual-Labels approaches for movie description on MPII-MD and M-VAD. Visual-Labels performs strongly across feature, classifier, architecture, and dataset comparisons, outperforming prior systems while trading grammar for longer descriptions.

  • Experimental setup: The MPII-MD split uses 83 movies for training and 11 movies with 22 alignments for validation and testing; M-VAD uses 72, 10, and 10 movies.
  • Semantic parsing + SMT: SMT-Best selects the IDT 30 variant, which achieves a highest score of 5.59 among the tested SMT versions.Text-label variants perform slightly better than sense-label variants, attributed to semantic-parser disambiguation errors.
  • Visual labels + LSTM: 7.10 METEOR is achieved by PLACES features, while stacking IDT + LSDA + PLACES improves performance to 7.24 METEOR.
  • Visual labels + LSTM: Feature selection by label semantics, label-specific one-vs-all SVMs, and more visual labels improve Visual-Labels performance while reducing LSTM input dimensionality to 263.
  • LSTM architectures and training: Dropout, METEOR-based iteration selection, and ensembles of differently initialized LSTMs are identified as important for reducing over-fitting and improving sentence generation.The ensemble averages word predictions from three LSTMs trained with different data orderings.
  • Comparison to related work: 1.44 METEOR points separate Visual-Labels from the compared MPII-MD result, and human evaluation finds it better in correctness and relevance but worse in grammar than S2VT.S2VT produces shorter sentences, averaging 7.4 words versus 8.7 for Visual-Labels.
  • Comparison to related work: On M-VAD, Visual-Labels outperforms S2VT and Temporal attention in METEOR, although scores are consistently lower than on MPII-MD.The authors suggest that M-VAD is more challenging because MPII-MD has more precise manual alignments.

5.5 Movie description analysis

The analysis examines how sentence difficulty and verb semantics affect movie-description performance. Scores decline for longer, rarer, and structurally atypical sentences, while frequent and visual verbs are easier for the systems.

  • Sentence difficulty: All three methods obtain lower METEOR scores as sentence length increases or average word frequency decreases.The correlation is stronger for word frequency than for sentence length.
  • WordNet verb topics: The cognition topic performs best but is highly biased toward “look at,” whereas motion and contact remain challenging because of their diverse verbs.Topics with abstract verbs such as “be,” “have,” and “start” receive lower scores.
  • Analysis summary: Sentences with frequent or visual verbs score higher, while verb-less sentences and sentences lacking ordinary subjects score lower.The summary attributes these patterns to dataset biases.

6 The Large Scale Movie Description Challenge

The LSMDC challenge compared four movie-description systems using automatic and human evaluations. Visual-Labels led automatic metrics, while Frame-Video-Concept Fusion received the strongest overall human ratings, revealing a trade-off between descriptive diversity and correctness.

  • 6.1 LSMDC quantitative results: Visual-Labels achieved the highest scores on all automatic evaluation measures except BLEU-1 and BLEU-2, where S2VT led.Visual-Labels scored 7.1 METEOR, compared with 7.0 for S2VT, 6.1 for Frame-Video-Concept Fusion, and 5.2 for Temporal Attention.
  • 6.1 LSMDC quantitative results: Shorter sentences from Frame-Video-Concept Fusion and Temporal Attention contributed to their lower automatic scores relative to the references.Their averages were 5.16 and 3.63 words, respectively, versus 8.74 words for the references.
  • 6.1 LSMDC quantitative results: Human judges preferred Frame-Video-Concept Fusion for correctness, grammar, and relevance, whereas Visual-Labels was judged most helpful for blind users.The authors suggest that informativeness was weighted more heavily in the helpfulness evaluation than description errors.
  • 6.2 LSMDC qualitative results: Qualitative examples show that systems sometimes captured visually relevant actions absent from the reference, but all approaches also missed actions such as running away or phone interaction.Visual-Labels occasionally captured details that other methods missed, while Temporal Attention often generated generally applicable sentences.
  • 6.1 LSMDC quantitative results: Visual-Labels and S2VT generated longer, more diverse, and more novel descriptions, while the other systems produced shorter, simpler, and more frequently reused descriptions.All methods used a relatively small set of visual and language concepts compared with the reference descriptions, reflecting the long-tail challenge.

7 Conclusion

The paper introduces LSMDC, a large movie-description dataset pairing aligned audio descriptions and scripts, and evaluates systems for generating descriptions. The results show a trade-off between descriptive diversity and human-rated correctness, while the dataset remains challenging because of long-tail content.

  • 7 Conclusion: LSMDC contains aligned audio descriptions and movie scripts from 202 movies, totaling 118,114 sentences with aligned clips.The dataset is designed for movie description and comparison of the two description sources.
  • 7 Conclusion: Visual-Labels trains visual classifiers and feeds their scores into an LSTM, using separate verb, object, and place labels and selecting reliable classifiers.The approach addresses weak sentence annotations by removing noisy negatives and separating semantic groups.
  • 7 Conclusion: Frame-Video-Concept Fusion received the best human ratings by producing medium-length sentences with intermediate diversity, whereas longer descriptions were more error-prone.Visual-Labels was especially well received for helpfulness to blind users because of important visual details.
  • 7 Conclusion: All approaches struggled with the long-tail distribution of movie content, and additional training data may not fully resolve novel content in new movies.The authors expect techniques using different modalities to help address this challenge.
Loading 1605.03705v1…