Source-linked AI summary
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen
TL;DR
Neural Foley aims to generate sound effects that are simultaneously high-quality, semantically relevant, and temporally synchronized with silent videos, a combination existing approaches struggle to achieve. FoleyCrafter integrates trainable semantic and temporal modules with a pretrained text-to-audio model. Experiments report state-of-the-art benchmark performance, while text prompts provide controllable Foley generation.
Problem
Existing Neural Foley methods struggle to synthesize sounds that are simultaneously high-quality, semantically relevant, and temporally synchronized with videos.
Method
FoleyCrafter integrates a semantic adapter and temporal controller with a pretrained text-to-audio model for semantic and temporal video-audio alignment.
Results
FoleyCrafter achieves state-of-the-art performance on commonly used benchmarks across audio quality and semantic and temporal video alignment evaluations.
Takeaways & Limitations
FoleyCrafter supports flexible, fine-grained video-to-audio generation controlled by video content and text prompts.
Takeaways & Limitations
Temporal synchronization is limited by the detector's capabilities and by the availability of strong, relevant training data, especially for complex visual scenes.
Abstract
from arXiv · showhide
We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: the semantic adapter for semantic alignment and the temporal controller for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal controller incorporates an onset detector and a timestampbased adapter to achieve precise audio-video alignment. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes are available at https://github.com/open-mmlab/FoleyCrafter.
1 Introduction
Neural Foley seeks realistic sound effects that are both semantically related and temporally synchronized with video, but existing approaches trade off audio quality and alignment. FoleyCrafter combines a pretrained text-to-audio model with semantic and temporal modules, and experiments report state-of-the-art performance with text-prompt control.
- Motivation: Neural Foley must synthesize high-quality, semantically relevant, and temporally synchronized sounds for silent media, but this remains challenging.Traditional Foley is labor-intensive and time-consuming, while Neural Foley targets automatic generation for immersive audiovisual experiences.
- Limitations of Existing Methods: Video-to-audio methods trained on large paired datasets can suffer from noisy, complex real-world recordings, limiting generated audio quality.These datasets are often crawled from the Internet and may contain environmental sounds recorded in the wild [28] [30].
- Limitations of Existing Methods: Video-to-text followed by text-to-audio can improve sound quality but may produce unsynchronized sounds because translated text conditions lose visual details.The method group is described as achieving impressive sound quality while struggling to bridge video and text [p05].
- FoleyCrafter: FoleyCrafter integrates a learnable module into a pretrained text-to-audio model, using a semantic adapter and temporal controller for video-aligned generation.The semantic adapter uses parallel cross-attention on video features, while the temporal controller combines onset detection with timestamp-based alignment.
- Results: FoleyCrafter achieves state-of-the-art results in quantitative, qualitative, and user-study evaluations while supporting fine-grained control through text prompts.The reported evaluations assess audio quality and semantic and temporal video alignment.
2 Related Work
Prior work spans diffusion-based text-to-audio generation and video-to-audio synthesis, but open-domain systems still struggle to produce sounds that are both high-quality and visually synchronized. Video-to-audio methods differ in whether they train directly from paired data or use specialized conditioning strategies.
- Diffusion-based Audio Generation: Latent diffusion models advanced open-domain audio generation, with systems such as AudioLDM, Tango, Make-an-Audio, and Auffusion targeting text-conditioned synthesis.These approaches use different conditioning, representation, and architectural strategies [18] [19] [12] [16] [32].
- Video-to-Audio Generation: Open-domain video-to-audio generation remains challenging because systems must jointly achieve high audio quality and visual synchronization.Earlier Neural Foley models often targeted specific genres or narrow visual cues [6] [22] [37].
- Video-to-Audio Generation: Video-to-audio approaches include models trained from scratch, such as SpecVQGAN, Im2Wav, and Diff-Foley, using cross-modal or contrastive conditioning strategies.These methods respectively use video tokens, CLIP features, or aligned video-audio pretraining [17] [20] [25].
3 Approach
FoleyCrafter integrates trainable semantic and temporal modules with a fixed pre-trained text-to-audio generator to synthesize audio that aligns with video content and timing.
- 3.2 FoleyCrafter: FoleyCrafter combines a semantic adapter for video-semantic alignment with a temporal controller for video-audio synchronization around a fixed pre-trained text-to-audio generator.Only the FoleyCrafter modules are trained with ground-truth audio supervision, preserving the generator’s established synthesis capability.
- 3.2.1 Semantic Adapter: The semantic adapter extracts frame-level visual features with a CLIP encoder and learnable projections, then integrates them through visual cross-attention alongside text conditioning.Separate visual and text cross-attention outputs are combined with a weight parameter, retaining text-to-audio functionality while adding video conditioning.
- 3.2.1 Semantic Adapter: Text and visual embeddings are processed by parallel cross-attention layers, with pre-trained text projections fixed and newly introduced visual projections learned for alignment with the audio generator.The visual projections map video embeddings into the condition space used by the pre-trained text-to-audio model.
- 3.2.1 Semantic Adapter: Randomly dropping the text condition in approximately 90% of semantic-adapter training encourages video-driven audio generation while retaining controllability through text prompts.The design is intended to capture visual cues effectively without removing the ability to combine video and text conditions.
- 3.2.2 Temporal Controller: The temporal controller predicts binary sound-versus-silence timestamps from video and feeds the resulting mask into a temporal adapter that controls audio generation.The temporal adapter uses replicated UNet encoder blocks and injects timestamp-conditioned residual information into the original generator.
4 Experiments
FoleyCrafter is evaluated against state-of-the-art baselines for semantic alignment, audio quality, and temporal synchronization, with qualitative, prompt-control, and ablation studies. Across these evaluations, it achieves state-of-the-art results, aligns sound events more accurately with visual cues, supports controllable generation, and benefits from its temporal controller.
- Quantitative Comparison: FoleyCrafter achieves state-of-the-art semantic-alignment, audio-quality, and temporal-synchronization performance on VGGSound and AVSync15.The evaluation uses MKL, CLIP, and FID for semantic alignment and audio quality, plus onset detection metrics for synchronization.
- Qualitative Comparison: FoleyCrafter generates sound at the most accurate time relative to visual cues, closely matching the ground-truth audio pattern.In the gunshot example, SpecVQGAN and Diff-Foley miss the onset, whereas FoleyCrafter synchronizes it with the video.
- Text-Prompt Control: Text prompts provide fine-grained control: “high pitch” increases high-frequency content, while negative prompts remove unwanted sounds such as wind noise.For a horse-running video, the negative prompt produces clearer hoof sounds by suppressing wind and environmental noise.
- Ablation Study: Removing the temporal controller noticeably reduces onset precision because the semantic adapter captures video-level semantics but not accurate synchronization features.Without the temporal controller, generated sounds remain relevant but can begin at random timestamps.
5 Limitations and Broader Impatct
FoleyCrafter’s synchronization depends on the temporal detector and strong, relevant training data, with complex scenes posing a particular challenge. The technology also carries a risk of misuse for generating fake content on video and social platforms.
- Limitations: Synchronization performance is limited by the temporal detector and by the availability of strong, relevant training data.Complex visual scenes are especially challenging because onset-event training data are scarce in those contexts.
- Broader Impact: FoleyCrafter could be misused to generate fake content on video or social platforms, motivating cautious and responsible application.The paper advises users and researchers to screen uses of the technology carefully.
6 Conclusion
FoleyCrafter adds synchronized sound effects to silent videos through a pluggable module integrated with a text-to-audio generator.
- 6 Conclusion: FoleyCrafter integrates into a text-to-audio generator instead of training a video-to-audio model from scratch or using video-to-text translation.This integration targets high-quality audio while synchronizing generation with video content.
- 6 Conclusion: The framework uses a semantic adapter for semantic alignment and a temporal controller for temporal synchronization.
- 6 Conclusion: Extensive experiments on standard benchmarks demonstrate FoleyCrafter’s effectiveness.
A Appendix
The appendix provides supplementary material covering training datasets, user-study details, qualitative results, and separate video results.
- A.1 Details of Training Dataset: Appendix A.1 describes the training datasets used by FoleyCrafter.
- A.2 Details of user study: Appendix A.2 documents the user study methodology.
- A.3 More qualitative results: Appendix A.3 presents additional qualitative results.
- Supplementary video results: Video results are provided in a separate supplementary file sourced from Sora.
A.1 Details of Training Dataset
FoleyCrafter trains its semantic and temporal components separately using datasets selected for visual labels, timestamp detection, and temporal adaptation.
- A.1 Details of Training Dataset: The semantic adapter is trained on VGGSound, which contains approximately 199,176 YouTube videos with annotated video-content labels.Labels are prefixed with “The sound of” to form generation prompts.
- A.1 Details of Training Dataset: The timestamp detector is trained on AVSync15, containing 1,500 strongly correlated audio-visual pairs curated from VGGSound Sync.
- A.1 Details of Training Dataset: The temporal adapter is trained on AudioSet Strong, which contains 103,463 videos with sound-event and timestamp labels.
A.2 Details of user study
The user study compares FoleyCrafter with baseline methods through pairwise audio judgments from 20 participants across semantic, temporal, and overall quality criteria.
- A.2 Details of user study: The study randomly samples VGGSound test results generated by FoleyCrafter and baseline methods for questionnaire comparisons.Baselines include SpecVQGAN, Diff-Foley, and V2A-Mapper.
- A.2 Details of user study: Participants compare each method pair for semantic alignment, temporal alignment, and overall audio quality.
- A.2 Details of user study: FoleyCrafter is preferred by users on all three evaluation metrics.The preference score is computed as selections divided by appearances.
A.3 More qualitative results
FoleyCrafter demonstrates qualitative video-to-audio generation across diverse videos, while supporting text control and improved temporal synchronization. Supplementary examples include generated-video applications, text-conditioned results, temporal comparisons, and varied genres.
- Foley Generation for Generated Videos: FoleyCrafter is also applied to generated videos, where visual information helps produce sound effects for shown subjects and environments.A Sora video is used as an example of Foley generation for generated videos.
- Text-based video to audio generation: The semantic adapter uses visual information together with text prompts to generate controllable audio for video.Parallel cross-attention enables text-based video-to-audio generation, while visual information supports appropriate sounds for subjects and environments.
- Temporal Synchronization Comparison: The temporal controller enhances temporal alignment between generated audio and visual cues, with additional comparisons provided against other methods.Figure 9 presents more intuitive temporal synchronization comparisons, with corresponding video results in supplementary material.
- Video to Audio generation on various genres: FoleyCrafter generates audio for realistic videos, games, and animation featuring people, animals, and musical instruments.The supplementary examples use VGGSound test cases spanning these video types and visual subjects.