PaperlayerView exact revision on arXiv

Source-linked AI summary

From Visual Cues to Spoken Narration: Rethinking Audio Description

Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach

arXiv:2609.01725v1cs.CV

TL;DR

Audio Description requires jointly determining what visual event to narrate and when to insert the spoken description, a capability largely absent from prior pre-segmented-clip approaches. Cue2Narrate introduces LongLSMDC and a two-stage paired-window pipeline, improving localization and AD generation across evaluation settings.

  • Problem

    Prior AD work largely assumes pre-segmented visual clips and does not jointly address visual-event selection and narration timing in untrimmed, multi-segment video.

  • Method

    Cue2Narrate uses a dual-head audio-visual localizer to predict visual cue and spoken narration windows, then generates AD text from localized visual evidence.

  • Results

    Cue2Narrate outperforms video-only and audio-only localization baselines and improves AD generation over the corresponding fine-tuned base VLM under predicted and ground-truth windows.

  • Takeaways & Limitations

    LongLSMDC establishes a benchmark for multi-segment AD generation on long-form clips, while Cue2Narrate recovers the offset between visual events and spoken narration.

  • Takeaways & Limitations

    The pipeline separates localization and generation, and each AD is generated independently without cross-description context.

Abstract

from arXiv · show

Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visual event) and when (position for inserting the AD) to narrate, to achieve the best user experience. Prior work has largely reduced the problem to video captioning of pre-segmented video clips, i.e., what is largely predefined and when is ignored entirely. We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips. A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance: a visual cue window and a spoken narration window. A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence, trained with a Description Ranking Loss that ranks captions (negative samples) of the same frames lower than the GT AD. To benchmark this new problem statement, we introduce the LongLSMDC benchmark with up to 8-min movie clips (~6.5min on average). On LongLSMDC, Cue2Narrate outperforms video-only and audio-only localization baselines by 5--12 points in avg. mAP. Under both predicted- and GT-window evaluation, Cue2Narrate improves AD generation over the corresponding fine-tuned base VLM. These results establish the first benchmark for multi-segment AD generation on long-form clips. Data & Code: https://github.com/multimodal-ai-lab/Cue2Narrate

1 Introduction

Audio description requires deciding both which visual event to narrate and when to voice it, but prior work largely assumes pre-segmented clips and overlooks timing. Cue2Narrate addresses this gap with paired-window localization and localized AD generation for long-form video.

  • Audio description narrates visual content not conveyed through dialogue for blind and low-vision audiences.Descriptions are voiced over the soundtrack and should be concise.
  • Professional AD requires two decisions: which visual event matters and when its description should be voiced in a dialogue gap.The visual and spoken timing windows may occur during or slightly before or after the event.
  • Cue2Narrate predicts paired visual cue and spoken narration windows, then generates AD text from the localized visual window for insertion into the predicted spoken window.This pipeline targets untrimmed, long-form clips rather than pre-segmented video.
  • 11.8 points: Cue2Narrate’s localizer surpasses a video-only ActionFormer baseline in avg. mAP on LongLSMDC.The introduction also reports gains of 5.0 points over an audio-only variant and 3.6 points over a dual-head ActionFormer baseline.
  • LongLSMDC provides long-form clips with dense multi-segment annotations for jointly evaluating what and when in AD generation.The benchmark includes both window types per AD utterance.

2 Related Work

Prior AD systems generally generate descriptions from known or assumed temporal contexts, while timing prediction remains limited. LongLSMDC and Cue2Narrate instead support paired-window, multi-segment prediction on long-form clips.

  • Automated AD generation has mostly been treated as text generation from a known temporal window.AutoAD variants add character recognition or speech-gap processing, but timing remains incomplete or differently framed.
  • LongLSMDC is the first listed benchmark supporting multi-segment prediction of both visual and spoken windows on long-form clips.Compared datasets are either short-clip or provide spoken-only segments on longer clips.
  • Prior methods either use pre-extracted or assumed visual context, focus on QA-based AD evaluation, or predict only a single timing window.Cue2Narrate learns paired visual and spoken windows using video and audio supervision.
  • Temporal action localization supplies multi-segment boundary modeling for untrimmed video, while dense video captioning produces descriptions per segment.These related paradigms motivate the localization-and-generation formulation but do not themselves establish joint AD timing prediction.

3 LongLSMDC Dataset

LongLSMDC extends movie AD data to long, non-overlapping clips with paired visual and spoken annotations. Its annotations capture that the event’s on-screen timing and narration timing are often offset.

  • LongLSMDC segments full-length LSMDC movies into non-overlapping clips of up to 8 minutes, averaging approximately 6.5 minutes.Each AD annotation is assigned to the clip containing its temporal midpoint.
  • The dataset contains 1,995 training clips, 170 validation clips, and approximately 40 AD annotations per clip on average.Training supervision includes paired, visual-only, and spoken-only annotations.
  • 88% of paired annotations exhibit a temporal offset between visual cue and spoken narration windows.The windows differ in onset, duration, or both, making separate prediction relevant.
  • LongLSMDC provides 37,302 spoken-visual, 21,295 visual-only, and 20,197 spoken-only training annotations.The total is 78.8k training annotations and 83.8k including validation.
  • The benchmark evaluates predicted visual windows for generation and predicted spoken windows for temporal matching.The architecture description states that spoken-window matching follows the alignment used in human annotation practice.

4 Cue2Narrate

Cue2Narrate addresses long-form audio description by jointly localizing what visual event to describe and when to voice it, then generating concise narration from predicted visual evidence. Its two-stage design uses dual audio-visual temporal heads and a window-conditioned VLM generator with ranking-based training.

  • Pipeline: Cue2Narrate factorizes AD generation into Stage 1 paired-window prediction and Stage 2 sentence generation from the predicted visual cue window.The visual cue window determines what is described, while the spoken narration window determines where the description is inserted.
  • Stage 1: Audio-Visual Dual-Head Localizer: The localizer predicts independent visual cue and spoken narration windows over a shared audio-visual representation, allowing different durations and arbitrary temporal offsets.In LongLSMDC, 88% of paired annotations have a temporal offset between the two windows.
  • Stage 1: Audio-Visual Dual-Head Localizer: Audio and video provide complementary localization cues: audio supports spoken-window localization through silence patterns, whereas video supports visual-window localization through scene changes and actions.The model uses VideoMAE visual features and resampled log-mel audio features in a temporal multi-scale backbone.
  • Stage 1: Audio-Visual Dual-Head Localizer: LongLSMDC annotations supervise the visual and spoken heads asymmetrically, while paired annotations train both heads and prevent collapse to a shared window.The heads use sigmoid focal classification and 1D center-based DIoU regression losses.
  • Stage 2: Window-Conditioned AD Generator: Stage 2 adapts a Qwen VLM with LoRA to generate one-sentence ADs from uniformly sampled frames inside the visual cue window.The vision encoder remains frozen while a trainable projection aligns visual tokens with the language model.
  • Stage 2: Window-Conditioned AD Generator: Description Ranking Loss ranks the ground-truth concise AD above a frozen-VLM vanilla caption from the same frames, targeting residual captioning priors beyond cross-entropy.Training uses ground-truth visual windows, while inference uses Stage 1 predictions.

5 Experiments

Experiments evaluate Cue2Narrate’s paired-window localization and AD generation on long-form benchmarks, using modality, architecture, decoding, transfer, and qualitative analyses. The results show complementary audio-video localization, improved generation over fine-tuned baselines, and identifiable evaluation and generation failure modes.

  • 5.1 Localization Results: 42.2 visual and 42.0 spoken mAP on LongLSMDC exceed video-ActionFormer by 11.8 points and the audio-only variant by 5.0 points.Cue2Narrate also reaches 41.0 spoken mAP on CMD-AD after fine-tuning and 70.0 visual mAP on MAD-Eval, while achieving 90.0 F1 versus CA3D’s 65.3 F1 on the same split.
  • 5.1 Localization Results: The dual-head design preserves temporal offsets between visual cue and spoken narration windows rather than forcing a shared or averaged target.Same-head versus cross-head evaluation shows an approximately 4–5 point drop, indicating distinct learned temporal targets.
  • 5.1 Localization Results: Audio identifies dialogue gaps, while video distinguishes narratively salient events from arbitrary pauses, giving the modalities complementary localization roles.The spoken head is weak for 2–4 s events but stronger for 6 s+ events, whereas visual-head AP remains comparatively flat across durations.
  • 5.2 Generation Results: Under GT spoken windows, Qwen3 + Cue2Narrate reaches 37.3 CIDEr, compared with 32.5 for Qwen2.5 + Cue2Narrate and 17.3 for Gemini-2.5 Pro.Gemini scores higher on LLM-Eval, while Cue2Narrate’s Description Ranking Loss targets compact reference-AD style reflected by CIDEr.
  • 5.2 Generation Results: Cue2Narrate improves Qwen3-VL generation by 1.3 CIDEr under GT windows and 1.4 CIDEr under predicted windows on LongLSMDC.The gains persist across backbones, datasets, and evaluation settings, and transfer to CMD-AD with gains from 8.8 to 11.5 predicted and 23.8 to 25.1 GT.
  • 5.2 Generation Results: Matched-pair text metrics measure generation quality conditional on successful localization and do not penalize unmatched predictions or missed ground-truth segments.Localization coverage is reported separately, and CIDEr values from GT-window and predicted-window settings should not be treated as a direct localization penalty.

6 Conclusion

The paper reframes automatic Audio Description as jointly predicting what to narrate and when to voice it in long-form, untrimmed video. It introduces LongLSMDC and Cue2Narrate to benchmark and address this paired-window problem.

  • Automatic Audio Description is reframed as joint prediction of the visual event and its narration timing.The paired windows capture where an event is visible and where its description is voiced, including their temporal offset.
  • LongLSMDC benchmarks long-form movie clips with paired visual and spoken windows for multi-segment AD generation.The benchmark is built for untrimmed clips containing multiple AD events.
  • Cue2Narrate combines a dual-head audio-visual localizer with a window-conditioned generator.The localizer predicts the two windows separately before generation uses the visual cue window.

7 Limitations

The paper identifies future work around integrating localization and generation, modeling context across descriptions, evaluating with blind and low-vision users, and verifying character banks.

  • Jointly training localization and generation could let generation feedback shape window prediction.The current pipeline keeps the two stages separate, making joint training a proposed next step.
  • Generator context could be expanded to include spoken narration windows and adjacent Audio Descriptions.The goal is to improve coherence across consecutive descriptions by incorporating surrounding dialogue and previously voiced descriptions.
  • A study with blind and low-vision users is left for future work beyond the paper’s scope.The paper reports a 20-evaluator pairwise study but identifies user evaluation as a complementary direction.
  • Human verification of automatically constructed character banks remains a natural extension.Character identity is assigned automatically, while the appendix quantifies agreement rates.

8 Ethical Considerations

Cue2Narrate is framed as an assistive system that should complement human describers, with disclosure, review, and fairness auditing needed because generated descriptions can be wrong or biased.

  • Cue2Narrate is intended to expand Audio Description coverage while complementing rather than replacing human describers.The paper positions the system as an assistive tool for blind and low-vision audiences.
  • Generated descriptions can be factually wrong, making disclosure of machine generation and human review important for dependent users.The paper emphasizes the consequence for users who rely on generated narration as their primary access to visual content.
  • Training data and the underlying VLM may introduce demographic biases into descriptions.The paper identifies descriptive-fairness auditing as future work.
  • LongLSMDC materials inherit LSMDC’s restricted-access, research-only terms, while code and evaluation scripts are unrestricted.The underlying video content is not redistributed.

A Additional LongLSMDC and Character-bank details

Additional details describe LongLSMDC’s construction, character-bank processing, vocabulary, localization metrics, coverage measures, and reported stability and boundary-precision considerations.

  • Dataset construction: LongLSMDC segments full-length movies into randomized, non-overlapping 5–8-minute clips to make multi-event processing feasible.Random durations δ ∼U(300s, 480s) prevent duration-specific biases and produce variable clip complexity.
  • Character banks: The character bank assigns detected faces to named characters by comparing in-frame and portrait-level features with cosine similarity.A cast list supplies character names, and the resulting bank maps frame-level detections to identities.
  • Dataset construction: Midpoint assignment places each AD annotation in exactly one clip and converts its timestamps to clip-relative coordinates.Annotations on clip boundaries, representing less than 1%, are excluded.
  • Dataset construction: Over 78,000 training annotations are distributed across more than 2,100 clips, with 1–147 AD annotations per clip.The dataset contains 1,995 training clips and 170 validation clips, with mean density varying by annotation category.
  • Vocabulary: LongLSMDC has richer action-verb and character-descriptor diversity than CMD-AD, reflecting longer clips and broader movie coverage.Examples of varied verbs include “walks”, “turns”, “looks”, and “stands”.
  • Dataset statistics: LongLSMDC contains up to 8-minute clips with 29–40 ADs per clip, compared with CMD-AD’s 2–3-minute clips and 1–5 ADs.This higher density supports long-form, multi-segment AD evaluation.
  • Character banks: Character-bank agreement is 70.4% among 3,770 mentions for which portrait matching is possible.The bank identifies 2,655 mentions; many non-identifications involve faces that are off-screen, turned away, or viewed from behind.
  • Evaluation: Localization reports mAP across temporal IoU thresholds 0.2–0.6, using greedy matching between predicted and ground-truth segments.Average Precision is computed per video and mAP is averaged across test videos.

B Fine-Grained IoU Analysis for LongLSMDC

Fine-grained evaluation shows that Cue2Narrate localizes paired visual and spoken windows more precisely than competing approaches as temporal-overlap requirements become stricter. Transfer results and protocol analysis clarify both the role of fine-tuning and the conservatism of top-1 scoring.

  • LongLSMDC comparison: 30.4/28.6 mAP at tIoU 0.5 for Cue2Narrate exceeds video-only’s 22.8/17.2 and audio-only’s 21.8/25.3 on LongLSMDC.The widening margin at stricter thresholds is consistent with audio supporting spoken-slot timing and video disambiguating narratable visual events.
  • Transfer evaluation: Cue2Narrate retains its lead across all tIoU thresholds in zero-shot transfer to CMD-AD and MAD-Eval.The supplied passage reports the same qualitative pattern across both transfer benchmarks.
  • Transfer evaluation: 42.7 versus 12.8 mAP at tIoU 0.4 on CMD-AD shows that fine-tuning primarily improves boundary precision rather than event recall.At loose tIoU 0.2, fine-tuning has little effect, with approximately 56 versus 57 mAP.
  • Evaluation protocol: 90.0 versus 65.3 F1 under MAD-Eval’s top-1 protocol reflects single-best-detection boundary precision, not an advantage from producing more candidates.Each clip contributes only its highest-scoring predicted segment, so additional correct predictions on multi-event clips are discarded.

C Human Evaluation

The human evaluation compares Cue2Narrate with Shot-by-Shot through pairwise judgments, while the accompanying material defines the evaluation setup and AD-generation constraints. The figure reports a preference for Cue2Narrate across all evaluated dimensions.

  • Localization context: Figure 10 plots mAP against tIoU thresholds from 0.2 to 0.6, comparing five methods on visual cue and spoken narration windows.Hatched and solid bars distinguish the two window heads; bar height encodes mAP percentage.
  • Generation constraints: The generation prompt requires exactly one present-tense sentence describing only visible information and focusing on important people, actions, and interactions.It also prohibits hallucination and restricts character naming unless visually explicit or provided.
  • Metric reproducibility: The Action Score evaluation is reproducible without closed-source software after offline preprocessing of ground-truth annotations.Predictions are parsed with rule-based dependency parsing and compared using an open-source sentence-embedding model plus verb matching.
  • Human preference study: ∼65% versus 35% preference on Narrative Quality & Clarity favors Cue2Narrate over Shot-by-Shot.The study sampled 100 LongLSMDC clips, used 20 evaluators per clip, and compared anonymized AD tracks across three axes.
  • Training design: Description Ranking Loss uses frozen Qwen3-VL captions as long, descriptive negatives against concise ground-truth ADs.The same prompt format is used across clips, so negatives reflect the backbone’s default captioning prior rather than AD-style framing.

G.1 Systematic error categories

The paper organizes generation errors into five categories and uses qualitative examples to examine selective silence awareness and dense temporal segmentation. These analyses also clarify the limits of the human evaluation and dataset licensing.

  • Systematic error categories: Five error categories are identified: wrong-subject attribution, under-specified verbs, boundary drift, hallucinated detail, and wording differences from the reference.Wrong-subject attribution is illustrated when several plausible visual subjects share the frame.
  • Selective silence awareness: Four predicted AD segments align with ground truth despite multiple silence gaps, showing selective rather than silence-triggered narration on CMD-AD.Audio identifies silence periods, while visual input determines whether the corresponding content is salient enough to describe.
  • Dense continuous annotations: 4–5 adjacent ground-truth AD segments within a 20-second window are closely tracked by predictions in a dense LongLSMDC sequence.The model partitions extended low-activity periods into multiple segments based on visual changes.
  • Data-use boundaries: LSMDC is restricted to noncommercial academic research, while MAD-Eval is governed by an NDA limiting use to academic research.The paper states that all resources are used within their respective licensing terms.

I Model size and budget

The implementation uses a compact shared localizer and LoRA-adapted VLM, while qualitative examples illustrate errors and challenges in dense, multi-event movie sequences. The supplied passages do not report a complete computational budget.

  • Model size: 6.64M parameters comprise the Stage 1 audio-visual dual-head localizer across video-only, audio-only, and full variants.Mode flags disable computation paths without changing the total parameter count.
  • Model size: LoRA adapts Qwen2.5-VL-7B or Qwen3-VL-8B while freezing the vision encoder.The reported configuration uses rank 16, α=32, and dropout 0.05 on the language model and multimodal projection.
  • Qualitative errors: Predicted descriptions include wrong subjects, generic clothing statements, and mismatched actions relative to the ground-truth examples.Examples include “They look at each other” versus “Moody searches the shelves” and “They are all wearing costumes” versus snowflakes outside.
  • Dense scenarios: Dense LongLSMDC examples contain 7 or 9 ground-truth segments within 50-second windows, including rapid adjacent events.These sequences involve minimal gaps, complex scenes, and extended dialogue-free periods requiring multiple insertions.
  • Qualitative errors: Additional examples show mismatched characters, actions, and scene interpretations in both predicted and ground-truth descriptions.The supplied examples include incorrect identities, omitted actions, and descriptions of different visual events.
Loading 2609.01725v1…