PaperlayerView exact revision on arXiv

Source-linked AI summary

TAVGBench: Benchmarking Text to Audible-Video Generation

Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, Yuchao Dai

arXiv:2404.14381v1cs.CVcs.MM

TL;DR

Text-to-video generation leaves synchronized audio generation from text comparatively unexplored, and the task needs a benchmark with aligned audio-video data. The paper introduces TAVGBench, an alignment metric, and a latent-diffusion baseline; TAVDiffusion is reported to outperform comparison models on video and audio quality metrics.

  • Problem

    Generating synchronized audio alongside text-generated video remains largely unexplored, while TAVG requires both modalities to be generated together from text.

    Source: §1 · p. 1

  • Method

    The paper introduces TAVGBench, an automatic annotation pipeline and AVHScore, and proposes TAVDiffusion with cross-attention and contrastive learning for audio-video alignment.

    Source: §1 · p. 1, §4.4 · p. 6, §6 · p. 8

  • Results

    TAVDiffusion is reported to outperform comparison models on video and audio quality metrics, and to achieve an AVHScore of 29.06.

    Source: §5.2 · pp. 6, 8

  • Takeaways & Limitations

    TAVGBench provides a large-scale dataset with audio and video descriptions to support supervised training and evaluation of TAVG models.

    Source: §1 · p. 1

Abstract

from arXiv · show

The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics.

1 INTRODUCTION

Text-to-video methods generate visual content, but generating synchronized audio from text alongside video remains largely unexplored. The paper introduces TAVG and proposes a benchmark, alignment metric, and latent-diffusion baseline for the task.

1 INTRODUCTION

  • TAVG requires generating audio and video together from written descriptions, extending text-to-video generation beyond visual-only output.The paper frames the task as addressing the gap in generating synchronized audio alongside videos from text.Source: §1 · p. 1, Figure 1 · p. 2
  • TAVGBench provides over 1.7 million audio-visual pairs and an automatic pipeline that combines BLIP2 video descriptions and WavCaps audio descriptions with ChatGPT refinement.The benchmark is intended to support supervised training and comprehensive evaluation and comparison.Source: §1 · p. 1
  • The paper introduces AVHScore to quantify audio-video alignment in a multimodal semantic space.This addresses the limitation of existing audio and video generation metrics, which focus on each modality separately only where stated in the supplied passages?Source: §1 · p. 1
  • TAVDiffusion is presented as a latent-diffusion baseline that uses feature interaction and feature constraints to align audio and video latent variables.The paper evaluates it on TAVGBench with conventional and proposed metrics.Source: §1 · p. 1
  • The paper lists TAVG, TAVGBench with its annotation pipeline and AVHScore, and TAVDiffusion as its main contributions.Source: §1 · p. 1

2 RELATED WORK

Prior work develops text-guided video and audio generation and explores joint audio-video generation, often conditioning one modality on the other. The paper identifies the lack of a comprehensive TAVG benchmark and large-scale dataset.

Text to video generation; Text to audio generation

  • Text-to-video and text-to-audio research has progressed from GAN and autoregressive approaches toward diffusion models, with methods addressing temporal coherence and text conditioning.The cited approaches include latent-diffusion video methods and audio methods using CLAP or T5 text embeddings.Source: §2 · p. 2

Audio video mutual/joint generation

  • Prior joint audio-video work includes mutual generation and methods that generate both modalities, retrieve audio, or optimize generation for alignment.Examples include MM-diffusion for joint generation and methods that generate video before retrieving audio.Source: §2 · p. 2

Uniqueness of our benchmark

  • The authors identify the absence of a comprehensive benchmark and large-scale dataset specifically for text-to-audible-video generation.Their proposed solution combines training and evaluation data, alignment metrics, and a baseline.Source: §2 · p. 2

3 THE TAVGBENCH

TAVGBench combines a large collection of audio-video clips with textual descriptions for both modalities. Its annotation pipeline refines separate audio and video descriptions, while AVHScore measures their alignment.

3.1 Dataset statistics

  • TAVGBench contains 1.7 million valid 10-second clips from AudioSet, totaling 11.8K hours, after invalid videos were excluded.The source collection contained 2 million aligned audio-video pairs from YouTube.Source: §3.1 · p. 3
  • Unlike datasets describing only one modality or the limited-scale FAVDBench, TAVGBench describes both audio and video and averages 2.32 sentences and 49.98 words per annotation.The supplied comparison also notes that WebVid videos have watermarks that restrict their application in actual scenarios.Source: §3.1 · p. 3

3.2 Annotation details

  • BLIP2 and WavCaps generate initial video and audio descriptions, respectively, and ChatGPT refines and combines them into coherent annotations.The refinement step is motivated by initial annotations that lacked coherence and context; the authors report enhanced contextual relevance and linguistic refinement.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Evaluation metric

  • AVHScore addresses the need to assess audio-video alignment in addition to separate audio and video quality, using ImageBind features in a shared space.It averages cosine similarity between each video frame and its corresponding audio input; N denotes the number of video frames.Source: §3.3 · p. 3

4 A BASELINE

TAVDiffusion is a latent-diffusion baseline for generating audio and video together from text. Its two-stream design combines cross-attention for feature interaction with contrastive learning for explicit audio-visual alignment.

4 A BASELINE

  • TAVDiffusion establishes a latent-diffusion baseline for text-to-audible-video generation.The model uses a two-stream architecture to generate audio and video jointly.Source: §4 · p. 4, Figure 4 · p. 5

4.1 Preliminary: Latent diffusion model

  • Latent diffusion encodes data into a latent space, adds noise through a forward process, and learns reverse denoising to recover the sample.The denoised latent is decoded back into the input space.Source: §4.1 · p. 4

4.2 TAVDiffusion

  • Separate audio and video latent streams undergo forward and reverse diffusion, using a shared schedule of diffusion hyperparameters.Parallel UNet branches denoise the two modalities; the text embedding conditions the process.Source: §4.2 · p. 5

4.3 Multimodal interaction

  • Cross-attention lets audio and video features interact, and the resulting features feed the diffusion UNet decoder.The two streams cross-attend to one another during the diffusion process.Source: §4.3 · pp. 5–6

4.4 Multimodal alignment

  • Because feature interaction alone does not explicitly enforce alignment, EAS uses contrastive learning to align audio and video representations.Paired examples from the two modalities are positives; examples with different indices are negatives.Source: §4.4 · p. 6

4.5 Objective Function

  • Training combines the diffusion task loss with the feature-alignment loss using a weighted sum.The loss weight λ is empirically set to 0.1.Source: §4.5 · p. 6

5 EXPERIMENTAL RESULTS

The experiments evaluate TAVDiffusion on TAVGBench and FAVDBench, compare it with two-stage baselines, and test its alignment strategies. The reported results include stronger quality metrics than the comparisons, qualitative audio-visual alignment, and improvements from both ablated strategies.

5.1 Implementation details

  • Evaluation uses 3,000 samples from the TAVGBench evaluation subset and 1,000 from the FAVDBench test subset, comparing TAVDiffusion with designed two-stage methods.The stated baselines include AnimateDiff paired with AudioLDM or Diff-Foley. Video is assessed with FVD, KVD, and CLIPSIM; audio with FAD; and alignment with AVHScore.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 Main results

  • TAVDiffusion outperforms all comparison methods on video and audio quality metrics, with FVD 776.25, KVD 65.53, FAD 1.46, CLIPSIM 24.18, and AVHScore 29.06.The evaluations include TAVGBench and FAVDBench; neither the model nor comparison methods trained on FAVDBench data.Source: §5.2 · pp. 6, 8, Table 2 · p. 7
  • Qualitative examples show TAVDiffusion producing realistic visuals and prompt-aligned audio across noisy and quiet scenes.The examples include a performer with guitar and singing, a car scene, background music and cheers, and birds chirping.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3 Ablation studies

  • Multimodal cross-attention and explicit audio-visual alignment each enhance video and audio quality and the alignment score.The ablation evidence includes quantitative results and qualitative visualizations, including a wolf whose mouth movements match the barking prompt and audio.Source: §5.3 · p. 8, Table 3 · p. 8, Figure 6 · p. 8

5.4 Potential applications

  • TAVGBench's video, audio, and text descriptions also support multimodal tasks such as generating videos from text and audio prompts.The dataset may also support training audible-video captioning models.Source: §5.4 · p. 8

6 CONCLUSION

The paper introduces TAVGBench, an annotation method, AVHScore, and TAVDiffusion for text-to-audible-video generation. It reports experimental effectiveness and identifies a multimodal diffusion transformer as future work.

6 CONCLUSION

  • The paper introduces TAVGBench, automatic audio-visual annotation, AVHScore, and TAVDiffusion, which uses latent diffusion with cross-attention and contrastive learning for audio-visual alignment.TAVGBench contains over 1.7 million video clips.Source: §6 · p. 8

Related papers

Loading 2404.14381v1…