Source-linked AI summary
More than a Moment: Towards Coherent Sequences of Audio Descriptions
Eshika Khandelwal, Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Andrew Zisserman, Gül Varol, Makarand Tapaswi
TL;DR
Automatic AD generation often describes video intervals independently, although ADs are intended to form a coherent sequence. CoherentAD generates multiple candidates per interval and selects across the sequence, while StoryRecall and repetition metrics evaluate sequence-level quality. The authors report improvements on CMD-AD and TV-AD, with user studies providing further validation.
Problem
Independent AD generation and per-interval evaluation do not capture the coherence, narrative coverage, or redundancy of a sequence of descriptions.
Source: §1 · pp. 1–2Method
CoherentAD generates multiple AD candidates per interval and auto-regressively selects among them using prior selections as context.
Source: §1 · p. 2, §3.3 · p. 3Results
Takeaways & Limitations
StoryRecall and repetition metrics assess whether AD sequences convey ground-truth narrative content while limiting redundancy.
Source: §4 · p. 3The authors suggest neighbouring context may fill gaps and improve coherence, and that editing or merging selected candidates could further improve sequence-level fluency.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Audio Descriptions (ADs) convey essential on-screen information, allowing visually impaired audiences to follow videos. To be effective, ADs must form a coherent sequence that helps listeners to visualise the unfolding scene, rather than describing isolated moments. However, most automatic methods generate each AD independently, often resulting in repetitive, incoherent descriptions. To address this, we propose a training-free method, CoherentAD, that first generates multiple candidate descriptions for each AD time interval, and then performs auto-regressive selection across the sequence to form a coherent and informative narrative. To evaluate AD sequences holistically, we introduce a sequence-level metric, StoryRecall, which measures how well the predicted ADs convey the ground truth narrative, alongside repetition metrics that capture the redundancy across consecutive AD outputs. Our method produces coherent AD sequences with enhanced narrative understanding, outperforming prior approaches that rely on independent generations.
1 Introduction
Automatic audio description has often treated each interval as a separate caption, despite ADs needing to carry a coherent visual story across time. CoherentAD selects among multiple candidates across a sequence, and the paper introduces sequence-level evaluation for narrative coverage and repetition.
1 Introduction
- Independent interval-level generation can repeat information and miss movie narrative structure, while per-AD metrics may reward correct names or objects without capturing sequence coherence or redundancy.Single-ground-truth matching also overlooks that multiple descriptions may be valid for an interval.Source: §1 · pp. 1–2
- CoherentAD is a training-free method that generates multiple AD-like candidates per clip and auto-regressively selects descriptions that advance the narrative and add visual details.The approach aims to encourage diverse visual descriptions across the video.Source: §1 · p. 2
- The paper introduces StoryRecall for conveying ground-truth visual details and narrative points, plus repetition metrics, and reports improvements on CMD-AD and TV-AD validated by user studies.The evaluation moves beyond comparing one interval at a time.Source: §1 · p. 2
2 Related Work
Prior AD generation spans fine-tuned end-to-end models and training-free multi-stage frameworks, with newer work also addressing coherence and redundancy across consecutive outputs. This paper situates its approach in the training-free setting and proposes multi-AD evaluation focused on repetition and storyline coherence.
2 Related Work
- End-to-end AD methods are fine-tuned on domain-specific datasets, whereas training-free frameworks build multi-stage systems from pretrained VLMs and LLMs.The passage names prior works in both categories, including Han et al. (2024) and Xie et al. (2024a).Source: §2 · p. 2
- Coherence-focused prior methods condition on previous ADs, predict temporal segments, or jointly process adjacent clips to reduce redundancy.AutoAD-I and UniAD use recursive generation; AutoAD-II predicts segments; DistinctAD uses a contextual attention mechanism.Source: §2 · p. 2
- Existing evaluation includes captioning, retrieval, and LLM-based metrics, while a smaller set of methods measures coherence or redundancy across ADs.The paper proposes a multi-AD metric suite focused on repetitions and overall storyline coherence.Source: §2 · p. 2
3 Sequence-Level AD Generation
CoherentAD generates multiple candidates for each predefined video interval and selects a coherent, non-redundant sequence in three stages. It first creates and condenses visual descriptions, then generates candidates, and finally selects among them using context-conditioned criteria.
3 Sequence-Level AD Generation
- CoherentAD has three stages: interval description, multiple AD generation per interval, and selection of the optimal sequence.It is a training-free method for generating a coherent, non-redundant sequence over predefined intervals.Source: §3 · p. 2
3.1 Video interval to summarised narrative
- A VLM extracts structured visual details and an LLM condenses them into a concise but complete paragraph for candidate generation.The paper notes that details missed in the initial extraction cannot be recovered later because subsequent stages rely on text.Source: §3.1 · pp. 2–3
3.2 Multiple AD candidates per interval
- An LLM generates up to m diverse, independent, concise candidates that group related events and collectively cover the preceding description.Allowing fewer than m candidates is intended to avoid redundancy while preventing fragmentation.Source: §3.2 · p. 3
3.3 Coherent sequence selection
- Four independent LLM scorers assess guideline adherence, redundancy, story advancement, and visual-element counts, with scores combined as a weighted average.The highest-scoring candidate is selected auto-regressively using the previous r selected descriptions as context.Source: §3.3 · p. 3, Figure 2 · p. 3
4 Sequence-Level AD Evaluation
The paper evaluates ADs as sequences because they should convey a story while avoiding redundancy. It introduces StoryRecall for narrative coverage and repetition measures for repeated content across outputs.
4 Sequence-Level AD Evaluation
- Sequence-level AD evaluation targets two goals that single-description comparisons can miss: conveying the story and remaining non-redundant.The proposed evaluation uses StoryRecall and repetition metrics.Source: §4 · p. 3
4.1 StoryRecall
- StoryRecall compares concatenated ground-truth and predicted sequences with an LLM, scoring conveyed ground-truth visual content from 1 to 5.Paraphrasing and reordering are allowed if the core narrative remains intact, and extra predicted information is not penalised.Source: §4.1 · p. 3
4.2 Repetition metrics
- Repetition evaluation measures exact matches and lexical overlap between each description and the next three descriptions, reporting a score for each offset.The table reports results on CMD-AD and TV-AD; its first row gives the inherent ground-truth repetition level as a lower bound.Source: §4.2 · p. 4, Table 1 · p. 4
5 Experiments
CoherentAD improves sequence-level narrative coverage on CMD-AD while producing repetition scores close to ground truth, and the multi-candidate setup improves both StoryRecall and repetition. Qualitative examples show distinct, visibly grounded details across intervals.
5 Experiments
- CoherentAD achieves 2.63 vs 2.43 StoryRecall on CMD-AD, with 5.21% repetition versus ground truth’s 3.71% and zero exact repeats; TV-AD performance is comparable.The reported results compare CoherentAD with prior state of the art on StoryRecall and report repetition scores close to ground truth.Source: §5 · p. 4
- The multi-candidate setup raises StoryRecall from 2.49 to 2.63 and lowers repetition from 8.17% to 5.21%; its wider VLM outputs yield diverse descriptions.Shot-by-Shot has 19.06% partial repetitions between consecutive predictions, compared with 8.17% in the ablation without multiple candidate generation.Source: §5 · p. 4
- Qualitative examples show baseline repetitions, while CoherentAD supplies distinct, visibly grounded details at each interval to form a coherent, non-redundant narrative.The paper also reports good results on the ADQA benchmark.Source: §5 · p. 4
6 Conclusion
The conclusion frames repetitive, incoherent sequential descriptions as a limitation of current methods. CoherentAD addresses this with a training-free approach that searches among candidates using multiple coherence criteria.
6 Conclusion
- Current methods produce repetitions and incoherence in sequential audio descriptions.Source: §6 · p. 4
- CoherentAD is training-free and uses multiple coherence criteria to address those problems.Source: §6 · p. 4
- The approach formulates generation as sequence search among multiple candidates for each AD interval.Source: §6 · p. 4
Limitations
The method assumes reference-provided temporal intervals and does not predict when descriptions should be placed. It also omits neighboring context and post-selection editing or merging.
Limitations
- CoherentAD takes reference-provided temporal intervals as input and does not address predicting when an AD should be placed.Source: Limitations · p. 4
- Generation and selection omit neighboring context, although ADs may refer to nearby events; adding context could help fill gaps and improve coherence.Source: Limitations · pp. 4–5
- The method selects one candidate per interval without post-processing; editing or merging candidates could further improve sequence-level fluency.Source: Limitations · p. 5
A Implementation Details
The implementation extracts structured visual details, condenses them into narrative paragraphs, generates multiple candidate ADs, and scores candidates across sequence-level criteria. It uses 7B/8B open-source models; processing is offline, with an average of 22.05 seconds per description.
A.1 Video interval to summarised narrative
- The VLM samples 16 frames per interval and extracts actions, interactions, environmental changes, characters, and objects before an LLM creates a concise, grounded narrative paragraph.The LLM is instructed to retain meaningful visual content while avoiding inference, dialogue, and speculation.Source: §A.1 · p. 7
A.2 Multiple candidate generation
- Candidate generation produces up to m diverse ADs per interval, treating lmax as a target length and grouping related observations into cohesive visual events.The target length is adapted from Xie et al. (2025).Source: §A.2 · p. 7
A.3 Criterion weights for sequence selection
- Selection scores are normalized to [0,1], weighting AD-guideline adherence and story advancement at 0.40 each, redundancy at 0.25, and counts at 0.29.The counts score combines participants (0.13), actions (0.11), and salient details (0.05); ten small weight variations retained StoryRecall at 2.60±0.02.Source: §A.3 · p. 7
A.4 Model sizes and hardware setup
- The pipeline uses Qwen2-VL-7B for structured visual descriptions and LLaMA-3.1-Instruct-8B for candidate generation and scoring, with experiments on NVIDIA A6000 GPUs.Each setting was run once, and reported values reflect those single runs.Source: §A.4 · p. 7
A.5 Computational cost
- The full pipeline combines VLM video description, LLM summarization, candidate generation, and scoring; its table caption reports the highest StoryRecall and lowest repetitions.The pipeline processes an average of 22.05 seconds per description, with time broken down by stage.Source: Table A.1 · p. 8, Table A.2 · p. 8
- AD generation is an offline process, so computational costs are incurred only once and are described as practical relative to movie-production timelines and budgets.The process can be parallelized across video segments.Source: §A.5 · pp. 7–8
A.6 Use of AI Assistants
- The authors used Cursor for code autocompletion and OpenAI to test zero-shot prompts during development.Source: §A.6 · p. 8
B.1 Ablation of pipeline stages
The full multi-stage pipeline outperforms the reported variants, while ablations show drops when direct single-AD generation or omission of summarization is used. Scoring with three prior descriptions gives the best reported balance of StoryRecall and repetition.
B.1 Ablation of pipeline stages
- Direct single-AD generation by the VLM yields StoryRecall 2.38 and substantially higher repetition; even a simplified prompt often fails to meet its constraints.The simplified prompt retained core constraints including visible action or interaction, provided character names, no inference or emotion, and a word limit.Source: §B.1 · p. 8
- Removing summarization lowers StoryRecall from 2.63 to 2.58 and increases partial repeats, with candidates often repeating the same visual event.Guideline-violating candidates such as “talking” or “conversing” are discarded during scoring, reducing viable options.Source: §B.1 · p. 8
- The proposed multi-stage pipeline outperforms the other variants, and its full configuration achieves the highest StoryRecall and lowest repetitions.The full configuration comprises video description, summarization, candidate generation, and scoring.Source: §B.1 · p. 8
B.2 Varying context during scoring
- Using r=3 prior descriptions gives the highest StoryRecall (2.63) with low overall repetition (5.21, 4.45, 4.18); r=5 reduces repetition further but does not improve StoryRecall.Exact repeats are zero in all tested cases.Source: §B.2 · p. 8
C ADQA Benchmark
On ADQA, CoherentAD produces concise descriptions that fit AD intervals while retaining strong visual-appreciation performance, and it leads prior methods on narrative understanding. Its results also show that conventional similarity metrics do not capture all aspects of AD usefulness.
C ADQA Benchmark
- CoherentAD scores 50.34 on Visual Appreciation, second only to Q2VL, and leads prior methods on Narrative Understanding with CC 64.0 and accuracy ratio 88.0.Q2VL’s densely summarized outputs are paragraph-length and unconstrained by AD timing, while CoherentAD produces single-sentence descriptions.Source: C ADQA Benchmark · p. 9
- Despite lower CIDEr (13.2) and LLM-AD-Eval (2.17), CoherentAD’s ADQA results establish it as the most effective method across Visual Appreciation and Narrative Understanding.The authors note that LLM difficulty with large information dumps may contribute to lower Narrative Understanding scores for summarized paragraphs.Source: C ADQA Benchmark · p. 9
D Additional Qualitative Results
Appendix D adds qualitative comparisons of CoherentAD with the ground truth and two baselines, highlighting repetitions, and documents prompts for drafting descriptions and assessing candidates.
D Additional Qualitative Results
- Figures A.1 and A.2 compare ground truth, CoherentAD, AutoAD-Zero, and Shot-by-Shot outputs, highlighting repeated content in red.The captions identify the comparisons but do not state a qualitative winner.Source: Figure A.1 · p. 10, Figure A.2 · p. 11
- The storyboard prompt orders clip events step by step and includes locations, objects, visible text, and setting changes, while the summary identifies the main event and participants.The prompts direct the model to flag uncertainty when something is unclear and not to guess beyond what is visible.Source: D Additional Qualitative Results · p. 12
- The narrative-summary prompt requires grounded, active-voice prose that retains meaningful visible actions and setting changes while excluding inference, dialogue, and emotion.Drafts are refined against checks for hallucination, forbidden content, state-to-verb rewrites, and coverage.Source: D Additional Qualitative Results · p. 13
- Candidate generation converts a visual paragraph into up to five concise, self-contained, present-tense ADs describing only physically visible content.Related observations are grouped into coherent visual moments, and the minimum number of candidates needed for coverage is requested.Source: D Additional Qualitative Results · p. 14
- Candidate checks separately assess AD-rule adherence, repeated events relative to prior descriptions, and how much a candidate advances the visual narrative.The scoring prompts use scales of 1–3 for adherence and redundancy, and 1–5 for narrative progression.Source: D Additional Qualitative Results · pp. 15–17
- The visual-content prompt counts salient participants and other details, requiring distinct, explicit, visually relevant elements rather than routine background items.It asks whether removing a participant would change the moment and whether omitting a detail would leave a blind viewer missing something essential.Source: D Additional Qualitative Results · p. 18