PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual AD Narration with Interleaved Multimodal Sequence

Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, Limin Wang

arXiv:2403.12922v3cs.CV

TL;DR

Uni-AD tackles AD generation, where narrations need to identify characters and fit the video storyline. It combines interleaved multimodal prompting, character refinement, and contextual information with contrastive loss; the authors report state-of-the-art performance on multiple AD benchmarks.

  • Problem

    AD generation must describe visual elements for video comprehension while naming relevant characters and maintaining contextual coherence within the storyline.

    Source: §1 · p. 2

  • Method

    Uni-AD prompts a frozen pre-trained LLM with interleaved multimodal inputs, mapping visual features into textual space and refining character information while incorporating context and contrastive loss.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    Uni-AD achieves state-of-the-art performance on multiple AD generation benchmarks, and the authors report that fine-grained alignment, precise character information, and contextual data benefit AD generation.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The reported results support interleaved multimodal prompting with refined character information and contextual data as a framework for AD generation.

    Source: §5 · p. 8

    With more accurate character predictions from AutoAD-Zero, the character-refinement module declines in performance; the authors attribute this to its shallow cross-attention design and limited CMDAD training data.

    Source: §C.5 · p. 11

Abstract

from arXiv · show

The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies. With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name and provide reasonable, contextual descriptions to help audience understand the storyline of movie. To achieve this goal, we propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs with interleaved multimodal sequence as input, termed as Uni-AD. To enhance the alignment of features across various modalities with finer granularity, we introduce a simple and lightweight module that maps video features into the textual feature space. Moreover, we also propose a character-refinement module to provide more precise information by identifying the main characters who play more significant roles in the video context. With these unique designs, we further incorporate contextual information and a contrastive loss into our architecture to generate smoother and more contextually appropriate ADs. Experiments on multiple AD datasets show that Uni-AD performs well on AD generation, which demonstrates the effectiveness of our approach. Our code is available at: https://github.com/ant-research/UniAD.

1. Introduction

Uni-AD generates audio descriptions by combining video, text, character, and contextual information in an interleaved multimodal sequence. It refines character information and uses context with a contrastive loss; the paper reports better performance than previous methods on multiple AD datasets.

1. Introduction

  • AD describes scenes, characters, actions, and storylines, supplementing dialogue to support video comprehension, but most videos lack AD because manual annotation is costly and annotators may differ.The paper notes potential benefits for visually impaired viewers and sighted viewers in activities such as language learning and sight-free video consumption while driving.Source: §1 · pp. 1–2
  • Unlike conventional video captioning, AD generation uses multiple modalities and contextual information, and aims to narrate character names and actions coherently within a plot.Inputs may include video, text, character portraits and names, context video, and past ADs.Source: §1 · p. 2
  • Uni-AD aligns inputs in an interleaved multimodal sequence to support open-source LLM use, while preserving relative order so semantically related embeddings and tokens can align at finer granularity.The paper presents this formulation as a simpler alternative to growing adapters and to training-free methods that require complex prompt engineering and may hallucinate.Source: §1 · p. 2
  • The character-refinement module identifies AD-related main characters rather than treating every character appearing in a clip as relevant narration information.The paper contrasts this goal with prior identification of all appearing characters and says the trained module can be applied to any videos.Source: §1 · p. 2
  • Uni-AD combines past visual content and ADs as context, then adds contrastive loss to discourage repeated descriptions when the current video resembles a past clip.The paper contrasts its use of past video and ADs with previous works that concatenate only past ADs.Source: §1 · p. 2
  • Uni-AD outperforms previous methods on multiple AD datasets while targeting finer-grained feature alignment and more accurate, coherent descriptions.Its design combines interleaved multimodal inputs, character refinement, contextual information, and contrastive loss.Source: §1 · p. 2

2. Related Work

Prior AD-generation work spans specialized tools, learned visual prompts, character banks, visual-language bridging, and training-free prompting. Uni-AD draws on interleaved multimodal sequences and visual-conditioned prompting to address modality alignment and scalability concerns.

2.1. Audio Descriptions Generation

  • AD generation advanced from annotation tools and evaluation mechanisms to deep-learning models: AutoAD-I [15] prompted GPT-2 with learnable visual vectors, [16] added a character bank, and [17] used Q-former to bridge visual and language spaces.Source: §2.1 · p. 2
  • Earlier AD-generation approaches are described as facing poor scalability, complex prompt engineering, or weak modality alignment, which Uni-AD addresses with interleaved input and larger LLMs.The related-work discussion links these drawbacks to the approaches reviewed in AD generation.Source: §2.1 · p. 3

2.2. Interleaved Sequence for Multimodal Learning

  • Interleaved image, video, and text sequences retain semantic relationships across modalities, motivating their use for finer-grained alignment in AD generation.Prior multimodal learning work used cohesive sequences and large interleaved datasets for pre-training.Source: §2.2 · p. 3

2.3. LLMs for Video Understanding

  • Video-LLM approaches include prompt tuning, adapter-based alignment, and training-free querying; adapters become harder to scale as LLM size grows, while Uni-AD follows visual-conditioned prompt tuning.The training-free category uses visual experts to transform video into text for LLM reasoning.Source: §2.3 · p. 3

3. Methodology

Uni-AD filters character information, maps visual inputs into embeddings, and combines them with text and context in a prompt for a frozen LLM. Its method also models contextual video and ADs, discourages repetition with contrastive loss, and trains with autoregressive supervision.

3.1. Method Overview

  • Uni-AD addresses the need for character names and contextual coherence by combining current video inputs and context in an interleaved multimodal sequence with character refinement.The framework has visual-modality alignment and multimodal prompt-generation stages; a contrastive loss can discourage duplicate ADs.Source: §3 · p. 3, §3.1 · p. 3

3.2. Uni-AD Pipeline

  • A multi-layer transformer encoder with fixed learnable vectors maps video frames and character portraits into embeddings, allowing visual tokens to interact and controlling representation length without adding LLM adapters.The authors motivate frame interaction for temporal relations and easier scaling to larger LLMs.Source: §3.2 · p. 3
  • The prompt combines mapped video and character embeddings with text while retaining the relative positions of multimodal information for finer-grained alignment.The architecture filters character information before mapping visual content and forms an interleaved sequence with text and contextual information.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Character-Refinement Module

  • The character-refinement module identifies AD-related characters from an external character bank because appearing in a clip does not mean a character belongs in its AD.It uses character exemplars as queries and video features as keys and values in three cross-attention and feed-forward layers, then predicts whether each character is AD-related.Source: §3.3 · p. 5

3.4. Contextual Information Modeling

  • Context modeling concatenates past ADs with the prompt and maps past video features together with the current clip, while contrastive loss addresses near-identical ADs for similar consecutive clips.The loss constrains the average likelihood score of the last AD to be lower than that of the current ground-truth AD.Source: §3.4 · p. 5

3.5. Objective Function

  • Training queries a frozen LLM with a visual-conditioned prompt using autoregressive supervision, with the complete objective adding contrastive loss to the autoregressive loss.The autoregressive loss is defined over target AD tokens conditioned on the prompt and preceding tokens.Source: §3.5 · p. 6

4. Experiments

Experiments evaluate Uni-AD on multiple AD benchmarks, compare it with prior approaches, and test character refinement, visual mapping, contextual inputs, and qualitative outputs.

4.1. Implementation Details

  • Experiments use MAD-v2-Named for training and MAD-eval-Named for evaluation, and train on CMDAD for CMDAD and TVAD evaluation.They report ROUGE-L, CIDEr, R@k/N, CRITIC, and LLM-AD-eval; training uses 10 epochs on 8 A100 GPUs.Source: §4.1 · p. 6

4.2. Comparison with state-of-the-art approaches

  • With context, Uni-AD reaches 17.1 Rouge-L, 28.2 Cider, and 54.2 Recall@5/16; it also outperforms prior methods on MAD-eval-Named and is competitive on CMDAD and TVAD.Extending Uni-AD to LLaMA2-7B adds 1.6 growth points on Cider, 0.4 on Rouge-L, and 1.8 on Recall@5/16.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6

4.3. Ablation Study

  • Character refinement improves AD generation across settings, but its MAD-eval-Named performance remains limited at 0.41 Precision and 0.77 Recall.The ablation evaluates the module without pre-training or contextual information; the gap to ground-truth character inputs indicates room for improvement.Source: §4.3 · p. 7, Table 3 · p. 7
  • Increasing visual-mapping latent vectors improves Uni-AD under both GPT-CLIP-B32 and LLaMA-CLIP-L14 settings, while AutoAD-II remains basically unchanged.The authors attribute this to Uni-AD retaining more visual details with more vectors, enabling finer-grained alignment and better AD generation.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Integrating Contextual Information

  • Adding context video clips and contrastive loss both enhances performance; the most recent context video with contrastive loss performs best.The authors link this to current AD being most relevant to preceding content.Source: §4.4 · p. 7, Table 5 · p. 7
  • Using predicted past ADs in the recurrent setting decreases performance, whereas the oracle setting improves as more ground-truth context ADs are added.The authors attribute recurrent degradation to predicted-versus-ground-truth discrepancies that provide inaccurate inference-time context and widen the training-testing gap.Source: §4.4 · p. 7, Table 6 · p. 8

4.5. Qualitative Results

  • Qualitative examples show character refinement improves character identification, while context and contrastive loss reduce repeated ADs and support more coherent descriptions.More visual tokens yield more detailed descriptions; Uni-AD is described as more video-consistent than AutoAD-II and ClipCap, while MM-Narrator includes unsupported details.Source: §4.5 · p. 8, Figure 4 · p. 8

5. Conclusion

Uni-AD is a unified framework for generating audio descriptions with interleaved multimodal inputs, character refinement, and contextual information. The paper reports state-of-the-art performance on multiple AD benchmarks and describes supplementary implementation, ablation, and qualitative materials.

5. Conclusion

  • Uni-AD achieves state-of-the-art performance on multiple AD generation benchmarks.Source: §5 · p. 8
  • The framework prompts pre-trained LLMs with interleaved multimodal sequences and uses character refinement to provide more precise character information.Source: §5 · p. 8
  • The authors report that fine-grained feature alignment, precise character information, and contextual data can benefit AD generation.Source: §5 · p. 8

A. Overview

  • The supplementary material contains model architecture and experiment settings, further ablations, and additional qualitative results.Source: A. Overview · p. 9

B.1. Architecture details

The appendix documents the Uni-AD visual mapping network and character-refinement module, including architecture details and the setup used for CMDAD and TVAD.

B.1. Architecture details

  • For CMDAD and TVAD, the character-refinement module is trained on CMDAD’s training set and evaluated on both datasets.Source: §B.1 · p. 9
  • InternVideo2.5 is used to extract video-frame features for CMDAD and TVAD.Source: §B.1 · p. 9
  • Tables S1 and S2 provide architecture details for Uni-AD’s visual mapping network and character-refinement module.Source: Table S1 · p. 9, Table S2 · p. 9, §B.1 · p. 9

B.2. Datasets

The appendix describes the AD datasets, comparison methods, and evaluation metrics used in the experiments. The materials cover movie and TV-series data, several model families, and both captioning and AD-specific evaluation measures.

B.2. Datasets

  • MAD-v2-Named contains 334,296 ADs and 628,613 subtitles from 488 movies; MAD-eval-Named has 6,520 ADs and 10,602 subtitles from 10 movies.CMDAD contains 101k ADs across more than 1432 movies, with 100 evaluation movies; TVAD has 31k training ADs and 3k evaluation ADs; AudioVault-AD contains 3.3 million utterances from 7,057 movies.Source: §B.2 · p. 9

B.3. Baselines

  • The comparison methods range from CLIP-feature captioning and GPT-2-based AD generation to Flamingo-style and Q-former models and prompt-based LLM pipelines.The listed methods include ClipCap, AutoAD-I, AutoAD-II, AutoAD-III, AutoAD-Zero, and MM-Narrater.Source: §B.3 · pp. 9–10

B.4. Metircs

  • Evaluation uses ROUGE-L and CIDEr alongside retrieval R@k/N, character-precision CRITIC, and LLM-AD-eval scores from 1 to 5.Source: §B.4 · p. 10

C. More Ablation Studies

The supplementary ablations examine visual-mapping design choices under a fixed setting that uses character refinement and no pre-training.

C. More Ablation Studies

  • All supplementary ablation experiments use the character-refinement module and omit pre-training.Source: C. More Ablation Studies · p. 10
  • Table S3 ablates visual-mapping-network structure by varying the number of learnable latent vectors in Uni-AD (GPT-2-B32).Source: Table S3 · p. 10
  • Table S4 tests whether video and image features share a single visual mapping network.Source: Table S4 · p. 10

C.1. Ablation on visual mapping network

The visual mapping ablations favor a multi-layer transformer encoder with 30 learnable vectors, and show that separate video and image mapping networks matter for Uni-AD(GPT-2-B32) but not Uni-AD(LLaMA-L14).

C.1. Ablation on visual mapping network

  • The visual mapping network with 30 learnable vectors performs best; frame interaction improves over an MLP, while the MLP performs worst.Without learnable vectors, the transformer encoder's visual embedding length is limited to the eight frames used in the experiments.Source: §C.1 · p. 10

C.2. Ablation on sharing visual mapping network

  • Separate video and image mapping networks are important for Uni-AD(GPT-2-B32), but not necessary for Uni-AD(LLaMA-L14).The authors report that video and image can be mixed for training the mapping network when visual features and the LLM are sufficiently strong.Source: §C.2 · p. 10

C.3. Ablation on image-video interaction

The ablation tests whether combining image and video inputs in the visual mapping networks improves their representations; the reported results do not show a benefit for Uni-AD.

C.3. Ablation on image-video interaction

  • More image-video interaction in visual mapping does not further benefit Uni-AD.The ablation concatenates the current character's image with video for image mapping, and all recognized characters' images with video for video mapping.Source: §C.3 · pp. 10–11, Table S5 · p. 11

C.4. Ablation on the Threshold in Character-Refinement Module

The threshold used by the Character-Refinement Module affects AD generation; a high threshold can remove too much character information and yield poorer results.

C.4. Ablation on the Threshold in Character-Refinement Module

  • The Character-Refinement Module threshold has a considerable impact on AD generation, and a high threshold can cause poor results by removing too much character information.The threshold ablation results are reported in Table S6.Source: §C.4 · p. 11, Table S6 · p. 11

C.5. Limitation of contextual information and character-refinement module

The paper reports that Uni-AD uses interleaved multimodal sequences for finer-grained alignment and outperforms prior methods on multiple AD datasets, while its module benefits are not uniform across settings. Additional qualitative examples illustrate character refinement and the effect of learnable-vector count.

C.5. Limitation of contextual information and character-refinement module

  • Uni-AD uses interleaved multimodal sequences for finer-grained alignment, and outperforms previous methods on multiple AD datasets.The authors also describe contextual information modeling and character refinement as submodules; their effectiveness on CMDAD and TVAD is tested under a new setup using VideoLLaMA features and AutoAD-Zero character predictions.Source: §C.5 · p. 11
  • With AutoAD-Zero character predictions on CMDAD, the character-refinement module reduces performance, which the authors attribute to its shallow cross-attention design and limited training data.They suggest future work could use more powerful multimodal large models to recognize main roles.Source: §C.5 · p. 11
  • On CMDAD with AutoAD-Zero character predictions, context yields a better LLM AD eval result, while omitting context yields a better CIDEr result; overall differences are not substantial.The authors suggest that stronger VideoLLaMA visual representations and limited CMDAD training data may make context modeling less noticeable.Source: §C.5 · pp. 11–12, Table S8 · p. 12

D. Additional Qualitative Analyses

  • In qualitative examples, character refinement removes irrelevant bank entries and helps identify a female police officer as Caroline; more learnable vectors also include a briefly appearing female character in the AD.These examples are from MAD-eval; Figure S1 and Figure S2 provide additional qualitative analyses.Source: D. Additional Qualitative Analyses · p. 12, Figure S1 · p. 12, Figure S2 · p. 12

Related papers

Loading 2403.12922v3…