PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning

Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung-Ching Lin, Zicheng Liu, Lijuan Wang

arXiv:2311.17435v1cs.CVcs.AI

TL;DR

Long-form audio description requires coherent, character-centric narration across extended videos, while prior approaches often rely on fine-tuning. MM-Narrator combines multimodal perception, textual and visual memory, and complexity-selected few-shot demonstrations; it outperforms existing approaches in most scenarios, and its AD is reported comparable to human annotations across considered aspects.

  • Problem

    Long-form AD must track evolving character-centered storylines across extended videos, while prior approaches primarily rely on downstream fine-tuning.

    Source: §1 · p. 2, §5.2 · p. 6

  • Method

    MM-Narrator uses GPT-4 for recurrent multimodal AD generation with short-term textual and long-term visual memory, complemented by complexity-selected few-shot MM-ICL.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    MM-Narrator outperforms fine-tuning-based and LLM/LMM approaches in most scenarios, and its AD is reported comparable to human annotations across considered aspects.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported results place training-free, memory-augmented AD generation as a strong approach for long-form videos within the evaluated scenarios.

    Source: §6 · p. 9

    Character identities remain unknown until their names are first spoken in dialogue.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD). Unlike previous methods that primarily focused on downstream fine-tuning with short video clips, MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even beyond hours, in an autoregressive manner. This capability is made possible by the proposed memory-augmented generation process, which effectively utilizes both the short-term textual context and long-term visual memory through an efficient register-and-recall mechanism. These contextual memories compile pertinent past information, including storylines and character identities, ensuring an accurate tracking and depicting of story-coherent and character-centric audio descriptions. Maintaining the training-free design of MM-Narrator, we further propose a complexity-based demonstration selection strategy to largely enhance its multi-step reasoning capability via few-shot multimodal in-context learning (MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics. Additionally, we introduce the first segment-based evaluator for recurrent text generation. Empowered by GPT-4, this evaluator comprehensively reasons and marks AD generation performance in various extendable dimensions.

1. Introduction

Audio description must narrate visual events coherently while complementing existing audio, a demanding task over long videos and evolving storylines. MM-Narrator combines multimodal perception and memory with complexity-based few-shot demonstrations; the authors report strong performance and AD comparable to human annotations.

1. Introduction

  • AD must complement existing dialogue, clarify who is present and what happens, and capture when and how actions unfold alongside subtle visual cues.AD primarily supports visually impaired audiences by transforming visual content into spoken narratives.Source: §1 · p. 2
  • Human-annotated AD is costly and can be inconsistent, while AD also supports audiences and uses beyond visual impairment.The passage also frames AD as a testbed for long-form multimodal reasoning.Source: §1 · p. 2
  • Long-form AD requires tracking character-centered storylines over extended durations, unlike frame-by-frame dense video captioning.Short-term textual memory supports coherent narration, while long-term visual memory helps re-identify characters, especially in scenes without dialogue.Source: §1 · p. 2
  • Complexity-based MM-ICL selects multimodal demonstrations by reasoning complexity and is reported to surpass random sampling and similarity retrieval.The approach uses chain-of-thought complexity to learn from a smaller candidate pool of demonstrations.Source: §1 · p. 2
  • MM-Narrator outperforms fine-tuning-based and LLM/LMM baselines, including GPT-4V, on most classic captioning metrics, with AD comparable to human annotations across considered aspects.The system is training-free and generates story-coherent, character-centric AD for long-form videos.Source: §1 · p. 2

2. Related Work

Prior work distinguishes AD from clip captioning and has explored fine-tuned models, prompting, and ICL example selection. MM-Narrator instead combines memory-based long-form generation with training-free AD and complexity-selected multimodal demonstrations.

2. Related Work

  • MM-Narrator generates AD training-free, unlike prior methods that rely on downstream fine-tuning.AD differs from video captioning by narrating coherent storylines, characters, and actions while complementing regular audio.Source: §2 · pp. 2–3
  • Prior video-understanding approaches fine-tune models or prompt LLMs with textualized video, but lengthy or speech-dense inputs can strain token limits.MM-Narrator instead uses short-term textual and long-term visual memory with register-and-recall for long-form AD.Source: §2 · p. 3
  • Earlier ICL selection work uses similarity, mutual information, or perplexity, while complex multimodal generation had not been explored in that literature.MM-Narrator measures demonstration complexity by chain-of-thought reasoning steps and selects intuitive examples for AD generation.Source: §2 · p. 3

3. Method

MM-Narrator generates long-video AD autoregressively with GPT-4, using multimodal perception and recalled context. Its MM-ICL extension selects intuitive demonstrations by reasoning-step count, while SegEval supports context-sensitive recurrent text evaluation.

3. Method

  • The training-free framework generates AD sequences autoregressively and adds complexity-based few-shot MM-ICL to enhance multimodal reasoning.The method is organized around recurrent AD generation and MM-ICL.Source: §3 · p. 3

3.1. Recurrent AD Narrator

  • At each video iteration, multimodal experts provide perceptions, past memories are recalled, and an LLM is prompted to generate an AD.Visual inputs include per-frame features and text-formed outputs from vision experts.Source: §3.1 · p. 3
  • Spoken dialogue supplies complementary visual information and can identify characters by name when video metadata is unavailable.Subtitles may come from online sources or automated speech recognition.Source: §3.1 · p. 4
  • A short-term queue stores recent timestamped ADs, while long-term visual memory registers frame signatures and retrieves similar past character-related ADs.Retrieval is focused on named entities and, for simplicity, activated when a single individual is detected in a frame.Source: §3.1 · p. 4

3.2. Multimodal In-Context Learning

  • Similarity-based MM-ICL did not improve AD generation empirically, whereas the method selects examples with the fewest reasoning steps as intuitive demonstrations.Demonstrations expand from (Q, A) pairs to (Q, R, A) tuples, where R represents reasoning steps.Source: §3.2 · pp. 4–5
  • SegEval uses GPT-4 to assess recurrent text generation with oracle contexts, and its marking criteria can vary the context window to measure text- or sequence-level qualities.The illustrated example has a re-scaled r of 2.25.Source: Figure 3 · p. 5

4. Segment-based GPT-4 Evaluator

SegEval uses GPT-4 to assess recurrent audio descriptions across multiple qualities, addressing subjectivity and the limits of reference-based captioning metrics. It compares predicted and human-annotated segments under an oracle context, with context length controlling which qualities can be measured.

4. Segment-based GPT-4 Evaluator

  • The evaluator uses an oracle context window around each target segment, with its length W adjustable to vary the evaluation context.For each segment, it considers W - 1 adjacent segments.Source: §4 · p. 5
  • SegEval asks GPT-4 to score predicted and ground-truth AD segments independently, then uses their raw-mark ratio as the final score.The oracle context is formed from ground-truth annotations; a rescaled ratio above 1.0 indicates GPT-4 might favor the predicted AD for that aspect.Source: §4 · p. 5
  • Changing W lets SegEval assess text-level qualities such as originality and consistency at W = 1, or sequence-level qualities such as coherence, diversity, and specificity when W > 1.The authors motivate SegEval by noting that AD is subjective, with annotator disagreement challenging traditional reference-based captioning metrics.Source: §4 · p. 5

5. Experiments

Experiments compare MM-Narrator with fine-tuning and training-free baselines, then examine its components, demonstration selection, and recurrent-generation evaluation. Results favor MM-Narrator across reported comparisons, while SegEval assesses qualities that reference-based captioning scores may miss.

5.1. Evaluation Setup

  • Evaluation uses MAD-v2-Named for training and MAD-eval-Named for testing, reporting captioning metrics and Recall@k within Neighbours for sequence generation.MAD-v2-Named contains 334,296 ADs and 628,613 subtitles from 488 movies; MAD-eval-Named contains 6,520 ADs and 10,602 subtitles from 10 movies.Source: §5.1 · pp. 5–6

5.2. Comparison with State-of-the-Art Approaches

  • R-L 12.1 vs 11.9, S 4.5 vs 4.4, and R@k/N 48.0 vs 42.1 for MM-Narrator versus AutoAD-I; MM-ICL further outperforms VLog and MM-Vid by a large margin.With an external character bank, MM-Narrator generates outcomes comparable to fine-tuning-based AutoAD-II.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

5.3. Building MM-Narrator From Image Captioner

  • Adding multiple frames, subtitles, and a task-specific hint builds multimodal perception; subtitles alone may not improve performance, while the hint guides their use for character-centric AD.The recurrent system also adds a short-term memory queue, multimodal demonstrations, and long-term visual memory, optionally enhanced by an external character bank.Source: §5.3 · p. 7, Figure 5 · p. 7

5.4. Ablations on Multimodal In-Context Learning

  • Complexity-based MM-ICL with CoT performs more favorably than random and similarity-based sampling, while the hardest, longest-reasoning examples can reduce performance.The results support selecting more straightforward examples, measured by shorter reasoning-step counts; adding reasoning tuples also yields consistent gains in the reported comparisons.Source: Table 4 · p. 7, §5.4 · pp. 7–8

5.5. Evaluating AD Generation with GPT-4

  • SegEval uses GPT-4 to assess recurrent-generation qualities across adjustable context windows, and its rankings align with other experiments except for MM-Vid diversity.Reference-based captioning scores can drop when GPT-4V produces richer, more detailed ADs than the shorter human references; SegEval supports extendable marking criteria.Source: Table 5 · p. 8, §5.5 · p. 8

6. Conclusion

MM-Narrator combines short-term textual context with long-term visual memory to generate story-coherent, character-centric audio descriptions for long-form videos. Its training-free system outperforms fine-tuning-based and LLM-based approaches in most scenarios, while SegEval finds its descriptions comparable to human annotations across several aspects.

6. Conclusion

  • MM-Narrator outperforms existing fine-tuning-based and LLM-based approaches in most scenarios on traditional captioning metrics.The GPT-4-powered SegEval results also suggest its descriptions are comparable to human annotations across several considered aspects.Source: §6 · p. 9
  • Combining immediate textual context with long-term visual memory supports story-coherent and character-centric AD generation for long-form videos.The system is recurrent and training-free.Source: §6 · p. 9
  • MM-Narrator introduces a GPT-4-powered evaluator for more comprehensive measurement of recurrent text-generation qualities.Source: §6 · p. 9

Appendix

The appendix provides further details on AD generation, complexity-based MM-ICL, SegEval, implementation, and future work. It describes how MM-Narrator forms recurrent-generation prompts from multimodal clues and few-shot demonstrations.

Appendix

  • The appendix covers AD generation, complexity-based MM-ICL, SegEval, implementation details, and future work.Source: Appendix · p. 12

A. AD Generation

  • MM-Narrator's recurrent-generation prompts combine task instructions and hints, a main query, and few-shot multimodal demonstrations.The query builder formats main and demonstration queries using visual captions, recent context ADs, and character dialogues.Source: A. AD Generation · p. 12
  • MM-ICL demonstrations use question-answer pairs or, with chain-of-thought, question-reasoning-answer tuples.The appendix also presents qualitative examples on MAD-eval-Named and other long-form videos.Source: A. AD Generation · p. 12

B. Discussions of Complexity-based MM-ICL

The appendix measures demonstration reasoning complexity by counting atomic chain-of-thought steps, then ranks examples to form simple and hard pools. It also describes how constrained chain-of-thought generation supports reliable demonstrations.

B. Discussions of Complexity-based MM-ICL

  • A CoT-specific constraint prompts reasoning that derives the answer from the question, reducing the risk of GPT-4 adding its own AD prediction.The authors note that unconstrained output may produce a prediction different from the human-annotated answer.Source: B. Discussions of Complexity-based… · p. 12
  • The authors found raw reasoning-step counts generated by an LLM insufficiently consistent across examples, motivating their atomic-step measure.Source: B. Discussions of Complexity-based… · p. 12
  • Natomic measures reasoning complexity by counting newline-separated atomic steps in the chain of thought.This makes multi-step reasoning examples comparable when one example's reasoning over individual frame captions spans several steps.Source: B. Discussions of Complexity-based… · p. 12
  • The 10% shortest demonstrations form a simple pool with maximum Natomic 5, while the 10% longest form a hard pool with minimum Natomic 12.Figure 6 shows the distribution of demonstrations over reasoning complexity, quantified by Natomic.Source: B. Discussions of Complexity-based… · p. 12, Figure 6 · p. 12

C. AD Evaluation with GPT-4

SegEval rates groups of audio descriptions against an oracle context window, using GPT-4 to mark text-level and sequence-level qualities. Its criteria distinguish context-free textual qualities from qualities assessed in relation to context.

C. AD Evaluation with GPT-4

D. Implementation Details

The implementation combines multimodal expert services with recurrent video segmentation and speech generation. It also specifies how demonstrations and memory are configured for experiments.

D. Implementation Details

  • Azure Cognitive Services supplies frame captions and people detection, WhisperX handles speech recognition, and CLIP-ViT-L14 plus GPT-4 support character re-identification.GPT-4 is prompted to extract people names from text for long-term visual memory registration and recall.Source: D. Implementation Details · p. 13
  • MAD-v2-Named demonstrations use ClipCap captions from released CLIP-ViT features, human-annotated recent context ADs, and no long-term memory retrievals.For occasional content-filtering errors (<0.1%), ClipCap captions the mean-pooled clip feature as a fallback AD.Source: D. Implementation Details · p. 13
  • Experiments sample 8 frames per clip, use subtitles within a 0.25-minute window, retain 7 recent ADs, and sample 5 demonstrations.Long videos are segmented with PySceneDetect, and gTTS creates voice-over audio for each clip.Source: D. Implementation Details · p. 13

E. Future works

The future-work discussion identifies stronger audio-visual speaker and character identification, better-timed AD delivery, and factuality-focused evaluation as priorities. Appendix figures provide prompt, reasoning, evaluation, and qualitative examples.

E. Future works

  • Future AD generation work should integrate audio-visual speaker and character identification, strategically time descriptions, and build a more comprehensive external character bank for retrieval.The proposed directions aim to improve contextual relevance and timely delivery without interrupting critical dialogue or action.Source: E. Future works · p. 13
  • SegEval does not adequately assess factuality, and traditional reference-based scores cannot precisely measure AD factual accuracy.The text proposes GPT-4V as a promising direction for more nuanced multimedia factuality assessment.Source: E. Future works · p. 13
  • Figure 7 breaks down the AD-generation prompt, while Figures 8 and 9 illustrate chain-of-thought reasoning and GPT-4-based SegEval examples.Figure 7 labels the prompt overview, task introduction and hint, MM-ICL demonstrations, main query, and long-term visual memory.Source: Figure 7 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16
  • Figures 10 and 11 add qualitative comparisons on MAD-eval-Named and demonstrations on other long-form videos.Examples condition generation on the current clip and context extending from the video start to the relevant timestamp.Source: Figure 10 · p. 17, Figure 11 · p. 17

Related papers

Loading 2311.17435v1…