PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoAD: Movie Description in Context

Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, Andrew Zisserman

arXiv:2303.16899v1cs.CV

TL;DR

Movie AD generation must account for context and faces a shortage of suitable training data. AutoAD combines pretrained language and vision-language models with visual, prior-AD, and subtitle context, using partial-data pretraining and improved AD data. The paper reports promising movie-AD results and zero-shot LSMDC results comparable to the fine-tuned state of the art, while identifying character naming as a limitation.

  • Problem

    High-quality movie AD depends on context, while suitable paired training data is limited and general captioning data transfers poorly to movies.

    Source: §1 · pp. 1–2

  • Method

    AutoAD bridges pretrained language and vision-language models for visual text generation, incorporates prior AD and subtitle context, and uses partial-data pretraining.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    The authors report promising automatic-AD results and zero-shot LSMDC multi-description results comparable to the fine-tuned state of the art.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents partial-data pretraining as applicable when full data is difficult to obtain and contributes cleaned MAD data and a text-only movie-AD pretraining resource.

    Source: §1 · p. 2, §7 · p. 8

    Character naming remains a clear limitation of the AutoAD pipeline.

    Source: §7 · p. 8

Abstract

from arXiv · show

The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. Generating high-quality movie AD is challenging due to the dependency of the descriptions on context, and the limited amount of training data available. In this work, we leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation. In order to obtain high-quality AD, we make the following four contributions: (i) we incorporate context from the movie clip, AD from previous clips, as well as the subtitles; (ii) we address the lack of training data by pretraining on large-scale datasets, where visual or contextual information is unavailable, e.g. text-only AD without movies or visual captioning datasets without context; (iii) we improve on the currently available AD datasets, by removing label noise in the MAD dataset, and adding character naming information; and (iv) we obtain strong results on the movie AD task compared with previous methods.

1. Introduction

Movie audio description narrates visual events for visually impaired audiences, with storytelling shaped by the soundtrack and contextual information. AutoAD addresses the challenge of limited suitable training data by combining pretrained models with contextual inputs and partial-data pretraining.

1. Introduction

  • AD combines dense descriptions of important visual elements over time with storytelling, while avoiding information already conveyed by audio and fitting between dialogue.Descriptions typically include characters’ names, emotions, and actions.Source: §1 · p. 1
  • Movie AD generation takes continuous movie frames and produces text for an annotated time interval, evaluated against ground-truth AD.Unlike descriptions for ActivityNet videos, this task concerns long-form feature films; LSMDC treats descriptions and character names separately.Source: §1 · p. 1
  • Suitable training data is scarce: general image- and video-caption datasets transfer poorly to movies, while human annotation does not scale and scripts or plots are limited and weakly grounded in vision.The paper identifies paired visual, textual, and contextual data as a particular challenge for training AD models.Source: §1 · p. 2
  • AutoAD combines temporal and dialogue context with pretrained language and vision-language models, and uses large-scale partial-data pretraining to address limited AD training data.The paper also reports improving MAD annotations and collecting text-only movie AD for pretraining.Source: §1 · p. 2

2. Related Works

Related work spans image and video captioning, visual storytelling, and movie understanding. AutoAD extends pretrained image-captioning approaches to movie AD, a task with distinct storytelling and audio-complementarity requirements.

Image Captioning

  • Image-captioning work bridges CLIP visual representations and GPT-2 language representations [53] [58], an approach this paper extends to automatic video AD.The cited approaches report promising results that generalize across domains.Source: §2 · p. 2

Video Captioning

  • Video-captioning datasets rely on manual labels that do not scale, noisy ASR, caption transfer limited by seed data, or models trained with limited data.Other approaches use HowTo100M ASR or extend image-text representations to multiple frames.Source: §2 · p. 2

Visual Storytelling

  • Unlike dense video captioning, movie AD serves storytelling, targets movies, and complements the audio track; unlike LSMDC visual storytelling, it describes continuous video with story and character context.TPAM uses local visual context, whereas this model conditions recurrently on previous outputs and pretrains GPT on in-domain text-only AD.Source: §2 · p. 2

Movie Understanding

  • Movie-understanding studies align films with plots, books, scripts, and YouTube summaries, but these sources are limited and often do not closely relate to visual elements in frames.LSMDC, M-VAD [84], and MPII-MD provide captions for short movie clips; QuerYD provides textual descriptions for longer videos.Source: §2 · p. 2

3. Method

AutoAD adapts a frozen CLIP encoder and GPT language model for movie AD, adding visual and textual context. It also pretrains components with visual-only or AD-only data when complete paired data are unavailable.

3.1. Visual Captioning with Prompt Tuning

  • AutoAD adapts an image-captioning pipeline by mapping CLIP features into prompt vectors for a pretrained GPT model.The visual mapping network is trained while CLIP and GPT are used as pretrained components.Source: §3.1 · p. 3

3.2. Benefiting from Temporal Context

  • Movie AD generation uses multiple frames, previous AD descriptions, and subtitles to provide visual and story context.The multi-frame transformer models temporal relations; subtitles can supply character names, relationships, and emotions, and a variant encodes only recent character names.Source: §3.2 · pp. 3–4
  • The model is trained with ground-truth past AD, while inference tests oracle ground-truth context and recurrent predicted context.In the recurrent setting, predicted past descriptions provide context for subsequent clips.Source: §3.2 · p. 4

3.3. Pretraining with Partial Data

  • Partial-data pretraining trains available components when a modality is missing, using visual-only captioning data or text-only movie AD.Visual-only pretraining deactivates context modules and keeps the language model frozen; AD-only pretraining trains the contextual AD module and language model without visual data.Source: §3.3 · p. 4

4. Denoising MAD Dataset

The original MAD annotations rely on ASR and subtitle-based speaker separation, which introduces transcription errors and dialogue leakage. The authors replace this with audio-only collection and release denoised MAD-v2 in named and unnamed forms.

4. Denoising MAD Dataset

  • The original MAD annotations were derived from mixed movie audio and separated using ASR plus DVD subtitles, creating transcription errors and character-dialogue leakage.MAD provides AD caption-video clip pairs from 488 movies; its annotations were collected from audio tracks mixed with the original movie soundtrack.Source: §4 · p. 5
  • An audio-only collection pipeline replaces subtitle-based separation with improved ASR and audio-based speaker separation to address transcription errors and dialogue leakage.It uses WhisperX word-timestamped transcription, sentence segmentation, diarization, first-person-pronoun proportions to identify the AD speaker, then synchronizes segments with visual features.Source: §4 · p. 5
  • MAD-v2 is described as more robust, with fewer errors and less character-dialogue leakage; its named variant retains character names while its unnamed variant replaces them with “someone”.The original annotations are called MAD-v1; character names in MAD-v2-Unnamed are identified with NER and anonymized.Source: §4 · p. 5

5. Partial Pretraining with AudioVault Dataset

Because paired AD and visual data are difficult to obtain amid movie copyrights, the authors collect text-only AD and proxy subtitle data from AudioVault. The resulting dataset is an order of magnitude larger than prior AD datasets.

5. Partial Pretraining with AudioVault Dataset

  • AudioVault draws on mixed audio from over 7,000 movies absent from MAD-v1, using ASR from non-AD speakers as proxy subtitles and excluding movies in LSMDC or MAD to prevent test-time leakage.The AD narrations are collected with a denoising pipeline similar to the one described for MAD.Source: §5 · p. 5
  • 3.3 million AD utterances and 8.7 million subtitle utterances make AudioVault an order of magnitude larger than prior AD datasets.AudioVault-AD and AudioVault-Sub include their utterances’ timestamps within each movie.Source: §5 · p. 6

6. Experiments

The experiments evaluate AutoAD on movie audio description datasets, ablate context and partial-data pretraining, and compare it with prior methods. Context and pretraining improve results in several settings, while subtitle gains are limited and recurrent character naming remains difficult.

6.1.1 Datasets

  • MAD-eval contains 10 movies excluded from LSMDC training and test sets; the experiments use named and unnamed evaluation versions.The dataset description also reports transcribed subtitles and training with MAD-v1 or cleaned MAD-v2.Source: §6.1.1 · p. 6

6.1.2 Architecture

  • The visual mapping network projects CLIP frame features into GPT-2’s embedding space, while GPT-2 is frozen in most experiments.The model also takes up to 64 past AD tokens and 32 subtitle tokens; subtitle context uses the four most recent dialogues within one minute.Source: §6.1.2 · p. 6

6.1.3 Training and Inference Details

  • Experiments use batches of 8 sequences with 16 consecutive video-AD pairs each, train for 10 epochs by default, and report top-1 beam-search outputs.Beam search uses size 5; recurrent generation feeds past greedy outputs back as context.Source: §6.1.3 · p. 6

6.1.4 Evaluation Metrics

  • Evaluation reports ROUGE-L, CIDEr, SPICE, and BertScore, with higher values indicating better text generation against ground truth.BertScore measures word matching using pretrained BERT embeddings.Source: §6.1.4 · p. 6

6.2. Experiments on Movie Audio Descriptions

  • CIDEr rises from 6.7 to 17.8 with oracle AD context and to 12.6 with recurrent AD context; subtitle context provides no gain, while name-only subtitles yield a slight gain.Longer AD context generally improves performance, but recurrent gains flatten beyond three sentences and attention incurs quadratic GPT-2 cost.Source: §6.2 · pp. 7–8, Figure 4 · p. 7
  • MAD-v2 outperforms MAD-v1 across named and unnamed settings despite having fewer annotations, which the authors interpret as evidence of reduced label noise.Visual-only pretraining improves CIDEr to 8.4 with CC3M and 10.0 with WebVid, while AudioVault-AD pretraining raises recurrent CIDEr from 12.6 to 14.1.Source: §6.2 · p. 7

6.2.1 Qualitative Results

  • Oracle examples use identities from past ground-truth AD, whereas recurrent outputs often use pronouns but still identify actions or objects; AutoAD also achieves competitive zero-shot LSMDC results (CIDEr 16.7 vs 15.4).Further training on LSMDC data can improve its multi-sentence results.Source: §6.2.1 · p. 8, §6.3 · p. 8

6.3. Comparison with Other Works

  • AutoAD clearly outperforms ClipCap and CapDec on the movie AD comparison, though evaluation is limited to methods using frozen CLIP features.The authors say these results highlight the importance of context for movie AD.Source: Table 4 · p. 7, §6.3 · p. 8

7. Conclusion and Future Work

The conclusion presents AutoAD as a movie AD pipeline that incorporates context and benefits from partial-data pretraining, alongside cleaned MAD annotations and a text-only movie AD resource. Character naming remains a limitation, and selecting when to generate AD is left for future work.

7. Conclusion and Future Work

  • The pipeline’s clear limitation is character naming—identifying who is doing what—while choosing when to generate AD remains future work.The paper focuses on generating AD for a given time interval.Source: §7 · p. 8

Appendix

  • The appendix outlines the AD collection pipeline, gives qualitative examples and implementation details, and lists movie IDs used in the MAD-v2 split.Source: Appendix · p. 13

A. AD Collection Pipeline Additional Details

The MAD-v2 pipeline extracts AD text and timestamps from mixed movie audio, addressing speaker identification and synchronization challenges. The appendix also describes text-only AudioVault collection and contrasts the pipeline with MAD-v1.

A.1. AD Collection Pipeline for MAD-v2

  • MAD-v2 addresses mixed-track speaker identification and source-audio synchronization, which is essential because its visual features come from differently synchronized movie files.AudioVault text-only collection does not require the final synchronization step.Source: §A.1 · p. 13, §A.2 · p. 13
  • WhisperX transcribes mixed audio, sentence segments receive diarized speaker identities, and the AD speaker is selected by low first- and second-person pronoun frequency before timestamps are shifted using FFT cross-correlation.The pipeline outputs AD text with corresponding timestamps.Source: §A.1 · p. 13, Figure A.1 · p. 13

A.2. AD Collection Pipeline for AudioVault

  • AudioVault text-only AD uses the mixed-audio transcription and speaker-identification steps, then filters out movies with fewer than 100 AD narrations or over 5% first- and second-person pronouns.Synchronization is unnecessary for these text-only annotations.Source: §A.2 · pp. 13–14

A.3. Comparison with MAD-v1

  • Unlike MAD-v1’s subtitle-based speaker separation, the pipeline uses audio-based separation, needs only mixed audio, and applies Whisper for more accurate punctuation and identity spelling.MAD-v1 can mistake dialogue for AD when official subtitles omit dialogue.Source: §A.3 · p. 14

B. Qualitative Examples of MAD-v2 vs MAD-v1.

The qualitative examples indicate that MAD-v2 produces more accurate AD than MAD-v1, particularly in name spelling and excluding dialogue. The examples identify recurring transcription and formatting errors in MAD-v1.

  • MAD-v2 is described as more accurate than MAD-v1, particularly in spelling names correctly and excluding dialogue.The comparison examples mark erroneous transcriptions and include manual verification.Source: B. Qualitative Examples of MAD-v2 vs… · p. 14, Figure A.2 · p. 15, Figure A.3 · p. 16
  • The Pursuit of Happyness examples show MAD-v1 errors including dialogue leakage, incorrect ASR, missing words, and incorrect name spelling.One example also involves name spelling alongside incorrect ASR.Source: Figure A.2 · p. 15
  • The Sully examples show MAD-v1 errors including dialogue leakage, incorrect ASR, missing words, number spelling, and sentence partitioning.Source: Figure A.3 · p. 16

C. Quantitative Comparison between MAD-v2 vs MAD-v1 on Grounding

The paper uses frozen CLIP zero-shot video-language grounding as an indicator of dataset quality and reports that MAD-v2 annotations benefit grounding performance. The comparison varies AD text quality and timestamps between MAD-v1 and MAD-v2.

D. Additional Implementation Details

Implementation details specify frame and text-context settings, evaluation preprocessing, and a vision-language fusion comparison. Qualitative examples show reasonable action and expression descriptions while illustrating limits in character identification.

D. Additional Implementation Details

  • The model uses 8 frames per clip, while subtitle context consists of the four most recent dialogues within a one-minute window.The four dialogues can span from a few seconds to minutes before the AD timestamp; experiments also vary the number of prior AD sentences.Source: D. Additional Implementation Details · p. 17
  • ROUGE-L, CIDEr, SPICE, and METEOR preprocessing removes punctuation and lowercases text; BertScore inputs are likewise lowercased and stripped of punctuation.Source: D. Additional Implementation Details · p. 17
  • Raw text context inputs outperform feeding context AD sentences as language features, at 12.6 CIDEr versus about 8.0 CIDEr.Source: D. Additional Implementation Details · p. 17

E. Additional Qualitative Examples

  • Qualitative examples show action and facial-expression descriptions, but character identification remains limited and recurrent outputs can use pronouns instead of correct names.Oracle predictions can learn names from ground-truth context, yet one example confuses Gatsby with Daisy’s dance partner.Source: E. Additional Qualitative Examples · p. 17

F. Dataset Splits

The appendix specifies which movies are used for training, evaluation, or neither, and provides movie IDs for the splits. MAD-v2 uses 488 MAD training movies, while MAD-eval contains 10 movies selected from MAD and LSMDC splits.

F. Dataset Splits

  • The appendix lists movie IDs for the MAD and LSMDC splits used and not used in the paper.The splits are also available on the paper website.Source: F. Dataset Splits · p. 17
  • MAD-v2 uses 488 movies, all for training, with cleaner ADs produced by the described automated pipeline; these are the MAD training movies proposed in.Source: F. Dataset Splits · p. 18
  • MAD-eval contains 10 movies from MAD validation/test and LSMDC validation, excluding LSMDC training movies to ease future comparison.Its annotations come from LSMDC and include named and unnamed versions.Source: F. Dataset Splits · p. 19
  • Another 151 MAD validation/test movies are used for neither training nor testing because they overlap with LSMDC train/test movies.Source: F. Dataset Splits · p. 19

Related papers

Loading 2303.16899v1…