FocusedAD: Character-centric Movie Audio Description

1 Introduction

• Because BVI audiences process AD alongside the soundtrack, descriptions must stay concise while preserving narrative-critical details.Character descriptions and scene transitions are among the elements identified as important to the narrative.Source: §1 · p. 1
• Global-scene captioning can produce redundant object lists instead of event-driven storytelling and struggles to focus on narrative-salient regions.The introduction contrasts descriptions such as lists of objects with narration of a character’s actions.Source: §1 · p. 1
• Character naming requires tracking regions across frames and linking them to identities, while sparse subtitle names and appearance shifts complicate recognition.These challenges are especially relevant in multi-character scenes and when characters’ appearances or makeup vary.Source: §1 · pp. 1–2
• FocusedAD combines character-region propagation, a clustering-built query bank, dynamic soft prompts, and joint caption reasoning over scene, character, and text tokens.Its soft-prompt mechanism adapts to scenes with varying numbers of characters.Source: §1 · p. 2
• FocusedAD achieves state-of-the-art performance on MAD-eval-Named and Cinepile-AD, with zero-shot results on both.The introduction highlights these results as evidence of its performance on the two test datasets.Source: §1 · p. 2

2.1 Video Caption

• Dense video captioning pairs multiple captions with temporal localization, but describing event segments in isolation can lack the narrative coherence required for AD.Recent approaches jointly optimize captioning and localization, while the cited discussion distinguishes AD’s need for coherence across segments.Source: §2.1 · p. 2
• Recent MLLM video-captioning approaches still rely heavily on global scene features, risking fine-grained detail imbalance and semantically irrelevant redundancy.FocusedAD is presented as using character perception and focused captioning to generate character-centric captions.Source: §2.1 · p. 2

2.2 Audio Description

• Prior AD pipelines use global features rather than story-relevant details, while FocusedAD targets narrative-salient regions for storyline-coherent descriptions.The related work also notes dataset constraints such as keyframe-only content and frame-level feature limits.Source: §2.2 · p. 2
• AD needs explicit character names; FocusedAD uses automated clustering to build a character query bank intended to better handle visual variation across lighting and scene contexts.The overview describes CPM propagating character regions and DPM integrating visual and text priors before FCM generates character-centric AD.Source: §2.2 · p. 2, Figure 2 · p. 3

3 Method

• The method connects character-name-linked visual priors and prior AD and subtitle text to generate character-centric descriptions of movie clips.CPM associates character regions with names, while DPM connects those visual priors to textual context for FCM.Source: §3 · p. 3

3.1 Character Perception Module

• CPM identifies active main characters, calibrates character queries from portraits, and propagates detected regions across key frames.It uses clustering to obtain character representations, then detects characters and propagates their regions in both directions.Source: §3.1 · pp. 3–4

3.2 Dynamic Prior Module

• DPM uses previous ADs and subtitles as text context and soft prompts to adapt descriptions to scenes with varying numbers of characters.The prompts incorporate visual and text priors into the caption module and can weight character descriptions by narrative significance.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

3.3 Focused Caption Module

• FCM pools character-region features into character-level tokens and combines them with scene and text tokens for LLM-generated AD.Temporal token merging reduces redundant character tokens before they are sent to the LLM.Source: §3.3 · p. 5

4.1 Training Datasets

• Storyboard-v2 contains 11,250 triplets of no-dialogue movie clips, movie-specific character query banks, and AD ground truth.Its construction addresses dataset limitations including restricted visual features, missing event-level segmentation, absent character-action descriptions, or limited context.Source: §4.1 · p. 5

4.2 Test Datasets

• MAD-eval-Named retains 10 independent movies, while Cinepile-AD derives dialogue-free clips and visual-description ground truth from CinePile.MAD-eval-Named is drawn from a quality-filtered set of 162 LSMDC movies; Cinepile-AD also collects preceding ADs and subtitles as context.Source: §4.2 · p. 6

4.3 Architecture

• The architecture uses Hiera image features, CPM character outputs, FaceNet recognition with memory attention, and Qwen2-7B-Instruct as the captioning LLM.Source: §4.3 · p. 6

4.4 Training and Inference Details

• Training uses a global batch size of 128 for one epoch, and inference reports top-1 outputs from beam search with beam size 5.Training uses 4 NVIDIA H20 GPUs; generated sequences are limited to 60 tokens unless a full stop is predicted.Source: §4.4 · pp. 6–7

5.1 Evaluation Metrics

• SPICE, METEOR, and BertScore assess generated AD text against ground truth, while the redundancy-aware metric flags content-bearing words below a similarity threshold to all ground-truth vectors.The redundancy score is averaged over valid words, quantifying how concisely the model captures ground-truth semantics.Source: §5.1 · p. 7

5.2 Audio Description on GT Movie Clips

• K-means character query lists improve character-recognition accuracy and movie AD quality compared with the alternative configuration, supporting the Character Perception Module’s effectiveness.The ablation table reports SPICE, METEOR, and BertScore for AD-generation variants.Source: §5.2 · p. 7, Table 1 · p. 7
• Combining previous ADs and subtitles with soft prompts further improves AD generation by helping the model focus on content relevant to the current segment.The reported comparisons are C1 vs C2 for contextual priors and A1 vs A2 for soft prompts.Source: §5.2 · p. 7
• The Focused Caption Module combines character-level tokens with scene-level visual representations and language embeddings to direct attention toward main-character regions and narrative-relevant descriptions.Source: §5.2 · p. 8

5.3 Qualitative Results

• Qualitative examples show the Character Perception Module recognizing active characters and the Focused Caption Module associating them with generated ADs; faces appear in 31% of clips at frame 0 and 26% in middle frames.The authors describe identifying characters at any frame and propagating that information through clips as essential for high-quality AD generation.Source: §5.3 · p. 8, Figure 8 · p. 8

5.4 Comparison with SOTA Approaches

• FocusedAD achieves state-of-the-art performance on both MAD-eval-Named and Cinepile-AD, including zero-shot evaluation against fine-tuning-based AD models.The comparison also includes general video-understanding MLLMs that emphasize scene-level features, whereas FocusedAD emphasizes character-centric details.Source: §5.4 · p. 8, Table 2 · p. 8

• FocusedAD is a framework for automatic movie AD that targets character-centric narration and selection of narrative-relevant details.Source: §6 · p. 8
• The approach propagates character regions across video sequences and generates descriptions focused on story-critical elements.It also introduces an automated character query-bank construction pipeline to address character-identification challenges in existing AD datasets.Source: §6 · pp. 8–9
• FocusedAD achieves state-of-the-art performance across multiple benchmarks, including zero-shot results on MAD-eval-Named and Cinepile-AD.Source: §6 · p. 9