PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description

Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, Andrew Zisserman

arXiv:2407.15850v2cs.CV

TL;DR

Audio description requires concise accounts of visual events, especially character identities and actions, but fine-tuning data are limited and zero-shot methods lack character knowledge. AutoAD-Zero combines visual and text prompting with a VLM-to-LLM generation process, and reports competitive results on movie and TV AD, including state-of-the-art CRITIC scores.

  • Problem

    Audio description must convey visual content, particularly who is present and what they do, while fine-tuning data are insufficient and existing zero-shot techniques lack character knowledge.

    Source: §1 · pp. 1–2

  • Method

    AutoAD-Zero uses visual and text prompts to provide character information to a VLM, then an LLM summarises the VLM’s detailed video description into an AD.

    Source: §1 · p. 2

  • Results

    AutoAD-Zero significantly outperforms existing training-free AD methods, competes with fine-tuned models, and achieves state-of-the-art CRITIC scores.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The two-stage design offers extensibility to newer VLMs and LLMs and flexibility to tailor the second-stage LLM to specific styles.

    Source: §7 · p. 14

    If the first-stage VLM fails to describe an action, the second-stage LLM cannot summarise it.

    Source: §7 · p. 14

Abstract

from arXiv · show

Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs), and develop visual and text prompting strategies for this task. Our contributions are three-fold: (i) We demonstrate that a VLM can successfully name and refer to characters if directly prompted with character information through visual indications without requiring any fine-tuning; (ii) A two-stage process is developed to generate ADs, with the first stage asking the VLM to comprehensively describe the video, followed by a second stage utilising a LLM to summarise dense textual information into one succinct AD sentence; (iii) A new dataset for TV audio description is formulated. Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.

1 Introduction

Audio description must convey visual information, especially who is present and what they do, but fine-tuning data are insufficient and zero-shot systems lack character knowledge. AutoAD-Zero uses visual and text prompts with a VLM-to-LLM pipeline, and reports competitive performance, state-of-the-art CRITIC scores, and a new TV-AD dataset.

1 Introduction

  • The two-stage pipeline prompts a VLM for a detailed, character-aware video description, then uses an LLM to summarise it into an AD sentence.Visual circles and text prompts associate character names with people in the frames.Source: §1 · p. 2, Figure 1 · p. 2
  • The training-free approach can be used with newly released VLMs and proprietary models without retraining.The authors expect it to benefit from improvements in models’ descriptive power.Source: §1 · pp. 2–3
  • AutoAD-Zero introduces TV-AD, a dataset of TV-series audio descriptions with ground-truth aligned annotations.The work also investigates differences between TV and movie ADs.Source: §1 · p. 3
  • AutoAD-Zero significantly outperforms existing training-free methods and is competitive with fine-tuned models, achieving state-of-the-art CRITIC scores.The evaluation includes multiple movie and TV datasets.Source: §1 · p. 3

2 Related Work

Related work frames AD as distinct from video captioning because it requires dense, coherent storytelling with concise formatting and character information. Prior work also explores trained character recognition, zero-shot generation, and visual prompting.

Previous AD and Video Captioning Models

  • AD differs from video captioning by requiring dense storytelling over time, concise formatting, coherent context, and character information.Prior work trained a separate character-recognition module and found it critical relative to GPT-4-based AD generation.Source: §2 · p. 3

Data Acquisition

  • Movie and TV AD dataset acquisition has been constrained by copyright, leading early works to share short clips or precomputed features.Later work aligned YouTube videos with AD annotations, while another collected publicly shareable amateur short films [10].Source: §2 · pp. 3–4

Prompting for LLMs and VLMs

  • Prompting research includes learned vector prompts and text-only instructions compatible with black-box models.Examples include “let’s think step by step” and chain-of-thought examples.Source: §2 · p. 4

Visual Prompting

  • Visual prompting adds marks to images without fine-tuning; prior work found circles can direct attention, while other methods use arbitrary markers or segmentation overlays [44].These approaches support visual grounding or referring-expression and keypoint tasks.Source: §2 · p. 4

Training-Free VLM Adaptation

  • Training-free VLM adaptation has been explored across multimodal applications, including medical tasks, where models still lack medical expertise [45] [42] [43].Source: §2 · p. 4

Related AD Methods

  • AutoAD-Zero differs from MM-Narrator and LLM-AD [7] by combining character names in text prompts with color-coded visual indicators.MM-Narrator uses multimodal in-context learning and memory; LLM-AD overlays character names on frames.Source: §2 · p. 4

3 Prompting VLM with Character Information

The character-information pipeline matches detected video faces to cast portraits, then uses visual and text cues to help a VLM associate character identities with people in frames.

3 Prompting VLM with Character Information

  • The approach replaces explicit VLM tuning with off-the-shelf face detection and face matching, followed by visual character indications.Source: §3 · pp. 4–5

3.1 Character Recognition by Face Detection

  • A video-specific character bank pairs each cast member’s name with the corresponding actor’s profile portrait.The cast list is queried for the movie or TV episode.Source: §3.1 · p. 5
  • Face detection runs independently on each frame and returns face embeddings together with bounding-box coordinates.Source: §3.1 · p. 5
  • Portrait embeddings are extracted from the character-bank images, then compared with in-frame embeddings using cosine similarity to select the best-matched portrait.The matching score is the cosine similarity between the portrait and frame features.Source: §3.1 · p. 6
  • The matched face is represented by its character ID, bounding box, and matching score.Source: §3.1 · p. 6
  • Low-confidence matches are removed, and when multiple faces in one frame receive the same ID, only the highest-scoring match is retained.A threshold filters uncertain matches, which may result from poor video quality or motion blur.Source: §3.1 · p. 6

3.2 Visual Prompting VLM by Circling Character Faces

  • Each character receives a unique circle color, and the VLM receives the marked frames alongside prompts pairing character names with colors.The circles connect the visual face indications with the names in the text prompt.Source: §3.2 · p. 6

4 Two-Stage Training-Free AD Generation

AutoAD-Zero generates ADs in two training-free stages: a VLM builds a character-aware video description, then an LLM condenses it into one AD sentence. Character-first visual prompting and staged prompts guide description, while the summary stage applies rules for content and style.

4.1 Stage I: VLM-Based Video Description

  • The VLM identifies characters by name, links them to colored circles, and answers questions about actions, character interactions, and facial expressions.The prompt also asks about states, gestures, movements, and object interactions.Source: §4.1 · p. 7
  • Adding directions about the environment and character appearances produces no significant improvement.Source: §4.1 · p. 8
  • The multi-question prompt yields a dense description; directly asking the VLM for a one-sentence AD makes it difficult to enforce AD format while preserving description quality.The questions follow a fixed order, and the model may also be asked to summarize at the end.Source: §4.1 · pp. 7–8

4.2 Stage II: LLM-Based AD Summary

  • The second stage gives a text-only LLM the first-stage description and asks it to produce one AD sentence under content and style rules.The instructions specify character naming and action selection, with AD-window duration hints and examples.Source: §4.2 · p. 8
  • The summary rules favor the most important characters and actions, use a training-split list of 15 frequent verbs, and treat static-state and talking-related actions as non-informative.The prompt also adjusts detail to the AD window and provides examples of AD style.Source: §4.2 · p. 8

5 TV-AD Dataset

TV-AD is a dataset of ground-truth ADs aligned with TV episodes, with non-overlapping series in its training and evaluation splits. Its formulation aligns AudioVault and TV soundtracks, transcribes them, and filters transcripts to extract AD text.

5.1 Dataset Details

  • TV-AD separates its approximately 31k-AD training split from an approximately 3k-AD evaluation split, with no TV series shared between them.The evaluation split covers publicly available TV videos from TVQA and is slated for public release.Source: §5.1 · p. 9, Table 1 · p. 9
  • 2.07s is the mean duration of TV-AD descriptions, versus 2.51s for CMD-AD; TV descriptions more often fall within 1–2s.The authors suggest shorter episodes, fast-paced plots, and dialogue may contribute to more compressed TV descriptions.Source: §5.1 · p. 9, Figure 4 · p. 9

5.2 Dataset Formulation: Aligning ADs with TV Series

  • TV soundtrack alignment and extraction are challenging because soundtracks and recording speeds can differ, while short ADs often mix with sound effects.The pipeline aligns AudioVault and TV soundtracks, then extracts and transcribes human-annotated ADs.Source: §5.2 · p. 9
  • The preprocessing step transcribes both soundtracks with timestamps and diarizes AudioVault speakers, then aligns them using masked mel-spectrogram correlation and a RANSAC-fitted linear timestamp map.The method skips the text-text alignment step used in the original proposal because audio-audio alignment alone was found precise enough.Source: §5.2 · p. 10
  • AD filtering first removes AudioVault sentences matching TV dialogue by WER, then applies majority-vote speaker diarization and heuristic exclusions.Text matching removes most non-AD content, but transcription and alignment errors can leave some behind; diarization errors may also leave a few non-AD sentences.Source: §5.2 · pp. 10–11

6 Experiments

The experiments evaluate character recognition and AD generation on movie and TV datasets, using character-list overlap and three AD metrics. AutoAD-Zero is competitive on AD metrics, reaches state-of-the-art CRITIC performance, and improves character-recognition results over AutoAD-II.

6.1 Datasets

  • Evaluation covers a four-movie MovieNet character-recognition subset, CMD-AD and MAD-Eval movie AD sets, and a TV-AD evaluation split of about 3k ADs across 100 episodes.Source: §6.1 · p. 11

6.2 Evaluation Metrics

  • Character recognition is measured by IoU, precision, and recall against ground-truth character-name lists; AD quality is assessed with CIDEr, CRITIC, and LLM-AD-Eval.CIDEr uses TF-IDF-weighted word matching, CRITIC compares character-name sets, and LLM-AD-Eval scores action, object, and interaction matching from 1 to 5.Source: §6.2 · pp. 11–12

6.3 Implementation Details

  • The character bank uses the top 10 IMDb characters and actor profile images, with InsightFace-based face detection and matching at threshold ϵ = 0.2.VideoLLaMA2-7B processes eight video frames for stage I, and LLaMA3-8B generates the stage-II AD summary.Source: §6.3 · p. 12

6.5 Audio Description Predictions

  • AutoAD-Zero surpasses training-free MM-Narrator on MAD-Eval and is competitive with fine-tuned models on CIDEr and LLM-AD-eval, while achieving state-of-the-art CRITIC performance.CRITIC evaluates character identification; the comparison spans CMD-AD, MAD-Eval, and TV-AD.Source: Table 3 · p. 13, §6.5 · pp. 13–14

6.4 Character Recognition Results

  • On four MAD-Eval movies, AutoAD-Zero achieves higher character-recognition IoU, precision, and recall than AutoAD-II.The reported scores are measured on final Video-LLM outputs.Source: Table 2 · p. 12, §6.4 · p. 13

7 Discussion and Extensions

AutoAD-Zero combines visual character prompting with a two-stage description-and-summarisation process. The discussion highlights its extensibility and style flexibility while noting prompt-design, stage-error, and possible data-leakage concerns; the appendix outlines further implementation, ablation, dataset, and results material.

7 Discussion and Extensions

  • The two-stage design allows use of more advanced VLMs and LLMs and lets the second-stage LLM be tailored to specific styles.The appendix lists potential extensions involving expert knowledge and additional context such as previous ADs, dialogue, and plot summaries.Source: §7 · p. 14
  • Prompt design remains heuristic and could be optimised; errors also accumulate when the first-stage VLM misses an action, since the second-stage LLM cannot summarise it.The authors also note that prior exposure to popular characters and scenes may undermine the task's zero-shot nature, and suggest recent or amateur films for evaluation.Source: §7 · p. 14

Appendix

  • The appendix covers implementation details, additional ablations, TV-AD details, further quantitative and qualitative results, and the text prompts used by the VLM and LLM.It also includes the LLM-based AD evaluation prompt.Source: Appendix · p. 18

A Additional Implementation Details

The appendix describes a rule-based transcript filter and compares the two-stage method's inference time with a single-stage model.

Details of Rule-Based AD Filtering

  • The final filtering step treats transcript sentences containing listed conversational, credit, or title-related strings as non-AD sentences.The rule-based heuristic checks a string list that includes dialogue expressions and credits such as “Directed by” and “Music by.”Source: A Additional Implementation Details · p. 18

Inference Time Analysis

  • AutoAD-Zero takes 0.67s per AD (0.50s for stage I and 0.16s for stage II), versus 0.20s for single-stage AutoAD-III.The two-stage design allows more flexible outputs and can benefit from improving VLM/LLM descriptive power.Source: A Additional Implementation Details · p. 18

B Additional Ablation Studies

The ablations examine character-prompt design, description content, summary instructions, question order, model choices, and repeatability. They support a default character prompt using circles, face labels, and in-text names, and emphasize character-aware descriptions followed by LLM summarisation.

Character Prompting Strategies

  • Without character information scores fall; text-only names can mis-associate multiple people, while names overlaid in images do not outperform the default and may incur OCR errors.The strategies and their comparisons are reported in Table B.1.Source: Table B.1 · p. 19, B Additional Ablation Studies · p. 20
  • In-text names are preferable to in-image names; circles slightly outperform boxes; body labels perform similarly to face labels, while using both simultaneously degrades performance, possibly from confusion over duplicate identity labels.The default setting is “Circle + Face + In-text” (Exp. F).Source: B Additional Ablation Studies · p. 20

Factors for Video Description

  • Removing character actions or interactions significantly lowers CIDEr and CRITIC, while omitting facial expressions lowers CRITIC; adding environments, appearances, or object interactions adversely affects performance.The authors suggest excessive or partly redundant information may make it harder for the LLM to summarise the details most important for ADs; Table B.2 identifies the ablated factors.Source: B Additional Ablation Studies · p. 20, Table B.2 · p. 21

Factors for AD summary

Change of Question Order in Video Description

  • Changing stage-I question order shows the importance of asking the model to identify characters first in the chain-of-thought process.Table B.4 compares question order for characters, actions, interactions, and facial expressions.Source: B Additional Ablation Studies · p. 21, Table B.4 · p. 21

Different VLM and LLM Models

  • Directly producing a one-sentence AD without the second-stage summary degrades performance; VideoLLaMA2 outperforms Video-LLaVA, while LLaMA3 and Gemma2 perform similarly.Repeated experiments show relatively low variance across ablations and the final result; the model comparisons and repeats are reported in Tables B.5 and B.6.Source: B Additional Ablation Studies · p. 22, Table B.5 · p. 22, Table B.6 · p. 22

C Details of TV-AD

The TV-AD section lists the series, seasons, and episodes used and states that evaluation materials will be released. Some episodes are excluded because the AudioVault soundtracks and video sources are misaligned.

TV-AD Details

  • The evaluation split uses video sources sampled from TVQA, and ground-truth AD annotations for both training and evaluation splits will be released.The passage also states that the evaluation split will be publicly available.Source: C Details of TV-AD · p. 22
  • The TV-AD details list series alongside adopted seasons and episodes, with some episodes filtered out for soundtrack–video misalignment.Table C.7 provides the series and episode details.Source: Table C.7 · p. 23

D More Quantitative Results

Additional MAD-Eval results report that AutoAD-Zero outperforms other training-free methods on Rouge-L, METEOR, and BLEU-1.

D More Quantitative Results

E More Qualitative Results

The appendix illustrates qualitative outputs and failure cases, and provides detailed prompts for the two-stage description process and LLM evaluation. The examples show dense VLM descriptions being summarized into single AD sentences, while failures include missed minimal motion and misread action direction.

E More Qualitative Results

  • The examples include predicted and ground-truth ADs alongside dense VLM descriptions, showing how stage-I output is summarized into one AD sentence.Fig. E.2 shows examples from CMD-AD and TV-AD, with character names linked to colored visual indicators and text prompts.Source: E More Qualitative Results · p. 23, Figure E.2 · p. 24
  • The illustrated failures include a VLM missing minimal motion such as nodding and occasionally misreading an action’s direction.The latter is described as an “arrow of time” issue.Source: E More Qualitative Results · pp. 23–24

F Detailed Text Prompts

  • Stage I prompts the VLM to identify characters and describe their actions, interactions, and facial expressions, while avoiding hallucinated details.Optional prompts add environments, appearances, costumes, and object-character interactions.Source: F Detailed Text Prompts · p. 25
  • Stage II prompts the LLM to turn the dense description into one succinct AD sentence, prioritizing characters and actions and adjusting length to clip duration.Instructions also discourage mood, camera, talking, and positional details, and prohibit adding information absent from the input.Source: F Detailed Text Prompts · p. 26
  • The LLM-AD evaluation prompt requests an integer matching score from 0 to 5 for a predicted and correct AD pair.The prompt directs evaluation toward matching visual elements, especially actions, objects, and interactions.Source: F Detailed Text Prompts · p. 27

Related papers

Loading 2407.15850v2…