PaperlayerView exact revision on arXiv

Source-linked AI summary

Sound-Guided Semantic Video Generation

Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, Sangpil Kim

arXiv:2204.09273v4cs.CVcs.AI

TL;DR

Existing video-generation methods can produce motion without semantic guidance, while non-verbal sound provides cues about scene events and context. The paper maps sound into StyleGAN’s latent space, using CLIP-based multimodal guidance to generate a sound-consistent video trajectory. The authors report better video quality than state-of-the-art methods and introduce a high-resolution landscape audio-video dataset.

  • Problem

    Motion generated from noise may lack semantic meaning, and generating realistic video from non-verbal sound requires semantic consistency, plausible motion, and temporal coherence.

    Source: §1 · p. 1

  • Method

    The framework maps audio into StyleGAN’s latent space and uses CLIP-based multimodal guidance and a frame generator to predict a sound-consistent latent trajectory.

    Source: §1 · p. 2

  • Results

    The model outperforms state-of-the-art methods in video quality and generates video that better reflects varied sound sources.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The new high-resolution landscape audio-video dataset is effective for experiments on high-resolution audio-visual generation models.

    Source: §1 · p. 3

    The first frame is sampled from StyleGAN’s latent space, which covers only the training domain; color change is rarely observed when the pretrained StyleGAN and video datasets have a domain gap.

    Source: §5 · p. 13

Abstract

from arXiv · show

The recent success in StyleGAN demonstrates that pre-trained StyleGAN latent space is useful for realistic video generation. However, the generated motion in the video is usually not semantically meaningful due to the difficulty of determining the direction and magnitude in the StyleGAN latent space. In this paper, we propose a framework to generate realistic videos by leveraging multimodal (sound-image-text) embedding space. As sound provides the temporal contexts of the scene, our framework learns to generate a video that is semantically consistent with sound. First, our sound inversion module maps the audio directly into the StyleGAN latent space. We then incorporate the CLIP-based multimodal embedding space to further provide the audio-visual relationships. Finally, the proposed frame generator learns to find the trajectory in the latent space which is coherent with the corresponding sound and generates a video in a hierarchical manner. We provide the new high-resolution landscape video dataset (audio-visual pair) for the sound-guided video generation task. The experiments show that our model outperforms the state-of-the-art methods in terms of video quality. We further show several applications including image and video editing to verify the effectiveness of our method.

1 Introduction

The paper addresses the lack of semantically meaningful motion in sound-guided video generation and the challenge of generating realistic video from non-verbal sound. It maps sound into StyleGAN’s latent space with multimodal guidance, generates a sound-consistent latent trajectory, and introduces a high-resolution landscape audio-video dataset.

1 Introduction

  • Existing video generation often creates motion from noise without guidance, while realistic non-verbal-sound generation also requires semantic consistency, plausible motion, and temporal coherence.Prior sound-based approaches often focus on talking faces, and existing methods can struggle to generate diverse visuals for sounds such as waves.Source: §1 · pp. 1–2
  • The framework semantically maps sound into StyleGAN’s W+ space and uses a CLIP-based multimodal embedding space to guide the latent trajectory.A frame generator predicts subsequent latent codes from audio; its three layers predict coarse, mid, and fine styles, providing nonlinear guidance at each time step.Source: §1 · pp. 2–3
  • The authors introduce a high-resolution landscape video dataset with audio-visual pairs for evaluating high-resolution sound-guided generation.They describe existing audio-visual benchmarks as lacking high-fidelity video or sound-video pairs.Source: §1 · pp. 3–4
  • The method outperforms state-of-the-art methods in video quality and generates videos that better reflect varied sound sources.It also determines latent movement direction and size and generates rich-style video from different sound attributes.Source: §1 · pp. 3–4

2 Related Work

Related work spans StyleGAN latent-space traversal, multimodal guidance, and sound-conditioned video generation; the paper positions its approach as generating high-resolution video from non-verbal sound.

2 Related Work

  • Prior StyleGAN work analyzes or edits latent codes, while this study embeds audio into StyleGAN space and uses audio-visual multimodal embeddings to guide traversal [21].The passage contrasts text-driven latent mapping with the goal of directing temporal style variation through CLIP guidance.Source: §2 · p. 4
  • The figure contrasts conventional latent interpolation in the top grid with audio-guided latent-code movement in the bottom grid.The paper also describes sound semantics generating images at each time step using latent vectors optimized with CLIP guidance.Source: Figure 2 · p. 4, §2 · p. 5
  • Sound-conditioned video work includes VAE-based next-frame prediction and methods that use non-verbal sound to represent scene context.Other cited work generates facial expressions from verbal sound.Source: §2 · p. 5
  • Earlier StyleGAN video methods synthesize continuous videos, while Jeong et al. [12] explore latent space for music-guided video with guidance not apparent to the user.The paper describes its own target as high-resolution video guided by sounds such as wind and rain.Source: §2 · p. 5

3 Sound-guided Video Generation

The framework maps sound into StyleGAN’s latent space, then generates a temporally consistent video sequence conditioned on sound. CLIP-based multimodal embeddings guide sound-image alignment, while recurrent, multi-scale latent prediction and adversarial discriminators support video generation.

3.1 Inverting Sound Into the StyleGAN Latent Space

  • Sound Inversion Encoder maps audio features into StyleGAN’s W+ space, and CLIP image-audio and image-text similarities guide sound-semantic alignment.An L2 constraint on latent codes encourages generated images across time to share content and semantics.Source: §3.1 · pp. 6–7

3.2 Sound-guided Semantic Video Generation

  • A recurrent network predicts sound-conditioned latent codes over time, while separate coarse, mid, and fine recurrent blocks model different levels of style detail.The latent groups correspond to StyleGAN layers 0–2, 3–5, and 6–15.Source: §3.2 · p. 7
  • Image and video discriminators are trained adversarially to distinguish generated outputs from real examples.The video discriminator classifies 3D patches and averages their responses; the image discriminator evaluates images along the time axis.Source: §3.2 · p. 8

4 Experiments

The experiments evaluate the method on audio-video datasets and report strong video quality and audio-video semantic consistency. They also examine CLIP-based inversion, recurrent style blocks, user judgments, and text-constrained generation.

4.1 Datasets

  • The Landscape dataset contains 9,280 ten-second clips from 928 high-resolution videos, totaling approximately 26 hours across nine scenes.It was created because Sub-URMP has limited clips focused on orchestra scenes, while other available paired datasets lack high fidelity or sound-video pairs.Source: §4.1 · pp. 9–10

4.2 Comparison to Existing Sound-guided Video Generation

  • Our approach produces the best-quality results, more visually correlated videos from sound, and high-fidelity videos compared with the reported methods.The qualitative comparison covers the landscape and Sub-URMP datasets; Sound2Sight and CCVS use an initial frame, while TräumerAI and our method use audio only.Source: §4.2 · pp. 10–11, Figure 6 · p. 11

4.3 Ablation Studies

  • CLIP-based inversion improves landscape-image reconstruction, while coarse and mid recurrent blocks control viewpoint and meaningful motion and the fine block handles texture.Table 3b compares MSE and LPIPS for inversion; Figure 7 contrasts individual recurrent blocks with the full block.Source: Table 3 · p. 10, §4.3 · pp. 11–12, Figure 7 · p. 12

4.4 User Study

  • 100 AMT participants rated our method significantly above other state-of-the-art approaches in realness, naturalness, and semantic consistency.Participants assessed generated videos using five-point Likert scales for realness and semantic consistency and selected the result that best expressed the target attribute for naturalness.Source: §4.4 · pp. 12–13, Figure 8 · p. 13

4.5 Sound-guided Video Editing with Text Constraints

  • Combining sound with text-query constraints yields more versatile video styles, with style mixing enabling styles at different levels.For editing, the initial latent code is selected from sampled codes by closest CLIP embedding cosine distance to the text prompt.Source: §4.5 · p. 13

5 Discussion

The discussion notes limits from StyleGAN’s train-domain coverage and the need for pretrained image-generator weights. It also flags a potential ethical concern when generated video uses art or ethically problematic source material.

5 Discussion

  • StyleGAN covers only its train domain, so color change is rarely observed when the pretrained model and video datasets have a domain gap.The authors nevertheless report applicability to cases such as sound-guided face video generation.Source: §5 · p. 13
  • Training requires pretrained image-generator weights, increasing training time, and latent movement size trades off against identity preservation.The movement-size and identity tradeoff is described with further detail in the supplement.Source: §5 · p. 13
  • Using art or ethically incorrect source video may imply intentions contrary to the original creator’s or raise ethical concerns.The authors also describe user-desired video generation as a potential use of the method.Source: §5 · pp. 13–14

6 Conclusion

The paper combines multimodal embeddings with StyleGAN to generate sound-guided video, and reports qualitative and quantitative improvements over state-of-the-art methods. It also presents a high-fidelity landscape audio-video dataset and text-constrained video generation.

6 Conclusion

  • The method uses CLIP space to encode sound information into StyleGAN’s latent space, with recurrent blocks representing rich style information across frames.The paper also curates a high-fidelity landscape audio-video dataset and demonstrates video generation with text constraints.Source: §6 · p. 14

Related papers

Loading 2204.09273v4…