Source-linked AI summary
Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
Ciara Rowles, Varun Jampani, Simon Donné, Shimon Vainer, Julian Parker, Zach Evans
TL;DR
Video-to-audio systems can require heavy training pipelines or control stacks, and end-to-end models rely on large aligned datasets. Foley Control adds pooled video cross-attention to a frozen text-to-audio model, delivering competitive alignment with few trainable parameters and retaining modularity.
Problem
Video-to-audio systems have advanced fidelity and semantic coverage, but many rely on heavy training pipelines or control stacks, while end-to-end approaches require large, tightly aligned video–audio datasets.
Source: §1 · p. 1Method
Foley Control connects V-JEPA2 video embeddings to a frozen Stable Audio Open DiT using pooled video cross-attention after text cross-attention.
Source: §1 · p. 1, §3.3 · p. 5Results
Foley Control delivers competitive semantic and temporal alignment while training only a small fraction of the parameters used by fully multimodal systems; pooled tokens match denser grid features without degrading synchronization.
Source: §5 · p. 8Takeaways & Limitations
The framework preserves prompt control and allows encoders or the text-to-audio backbone to be swapped or upgraded without end-to-end retraining.
Source: §5 · p. 8The setup caps video duration and relies on pooled tokens that may miss rare fine-grained spatial cues; it assumes clean inputs and does not model spatial acoustics or streaming alignment.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Foley Control is a lightweight approach to video-guided Foley that keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them. We connect V-JEPA2 video embeddings to a frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact video cross-attention after the model's existing text cross-attention, so prompts set global semantics while video refines timing and local dynamics. The frozen backbones retain strong marginals (video; audio given text) and the bridge learns the audio-video dependency needed for synchronization -- without retraining the audio prior. To cut memory and stabilize training, we pool video tokens before conditioning. On curated video-audio benchmarks, Foley Control delivers competitive temporal and semantic alignment with far fewer trainable parameters than recent multi-modal systems, while preserving prompt-driven controllability and production-friendly modularity (swap/upgrade encoders or the T2A backbone without end-to-end retraining). Although we focus on Video-to-Foley, the same bridge design can potentially extend to other audio modalities (e.g., speech).
1 Introduction
Video-to-audio systems offer strong fidelity and semantic coverage, but many rely on heavy training pipelines or control stacks. Foley Control instead freezes a text-to-audio backbone and adds a small video–audio bridge for timing and localized dynamics.
1 Introduction
- Many video-to-audio systems require heavy training pipelines or control stacks, while end-to-end models demand large, tightly aligned video–audio datasets.Real-world noise and imprecise timestamps further degrade supervision and underrepresent long-tail events.Source: §1 · p. 1
- Foley Control connects V-JEPA2 video embeddings to a frozen Stable Audio Open DiT with compact video-conditioned cross-attention layers.The video cross-attention follows the existing text cross-attention, so prompts establish high-level semantics before video refines timing and localized dynamics.Source: §1 · pp. 1–2
- Pooling video embeddings and adding RoPE keep the architecture compact while preserving prompt-driven controllability and temporal grounding.The framework reuses the frozen text-to-audio prior and adds trainable capacity for video alignment rather than relearning audio generation.Source: §1 · p. 2
2 Related Work
Prior video-to-audio work ranges from lightweight controllers on pretrained generators to fully multimodal models trained end to end. Foley Control places video conditioning within a frozen diffusion transformer’s existing attention layers.
2 Related Work
- FoleyCrafter [37] adds semantic and timing controllers to a pretrained U-Net generator, using a separate timestamp detector to sharpen synchronization around transient events.Its frozen backbone retains audio fidelity while the controllers improve video–audio alignment under modest compute.Source: §2 · p. 2
- MMAudio [5] and HunyuanVideo-Foley [26] jointly train multimodal systems, with HunyuanVideo-Foley using a massive curated dataset and representation-alignment loss.MMAudio combines multimodal and audio-only transformer blocks with a synchronization module; HunyuanVideo-Foley uses dual-stream diffusion and a DAC-style autoencoder.Source: §2 · p. 2
- Stable Audio Open [7] provides a high-fidelity latent-diffusion text-to-audio backbone, while V-JEPA2 [1] supplies video representations focused on predictable dynamics.V-JEPA2 is pretrained with masked feature prediction in representation space and supports motion understanding and temporal reasoning.Source: §2 · pp. 2–3
- Other approaches include few-step rectified-flow synthesis, expert stitching, and reasoning-based Foley pipelines [33] [31] [20] [18] [3].These works may retrain large joint models or use reasoning-based pipelines, whereas Foley Control freezes a text–audio backbone and learns a lightweight bridge.Source: §2 · p. 3
2.1 Architectural Positioning of our approach
- Foley Control integrates video conditioning into a frozen transformer’s existing attention layers, unlike adapter frameworks that partition control pathways [37, 25].Fully multimodal diffusion transformers such as MMAudio [5] and HunyuanVideo-Foley [26] require much more paired data and compute, while Foley Control freezes the text–audio prior.Source: §2.1 · p. 3
3 Method
The method uses pretrained video and audio components, curated video–audio pairs, and a frozen audio diffusion backbone. It adds pooled video conditioning inside the DiT and uses RoPE for temporal correspondence.
3.1 Preliminaries
- The setup maps video frames to V-JEPA2 spatiotemporal tokens and synthesizes aligned audio by denoising audio-autoencoder latents with a frozen Stable Audio DiT.The video tokens provide keys for cross-modal alignment with the audio latent sequence.Source: §3.1 · pp. 3–4
3.2 Dataset Curation
- The Kinetics-700-derived corpus is filtered to remove silent, low-quality, or conceptually distinct samples.ImageBind and Meta Audiobox Aesthetics scores are used to select high-fidelity, semantically consistent audio–video pairs.Source: §3.2 · p. 4
3.3 Framework Overview
- The Stable Audio Open backbone, VAE, and text pathway remain frozen while video cross-attention is inserted after text cross-attention in each DiT block.Audio latents serve as queries and video tokens as keys and values; only the introduced collaboration sublayer parameters are trainable.Source: §3.3 · pp. 4–5
- RoPE is applied independently to audio queries and video keys to preserve relative temporal phase information for aligning motion and audio onsets.The video path also uses a lightweight two-layer MLP adapter before forming keys and values.Source: §3.3 · p. 4
- Standard RoPE was sufficient for stable temporal correspondence without an additional alignment network in the adapter setting.Video cross-attention follows text cross-attention so prompts establish global semantics before video-specific timing and dynamics are injected.Source: §3.3 · p. 5
- Pooling each effective frame to one token yields 32 video tokens per four-second segment, with inputs capped at 12 seconds.This reduces token count and computational cost while retaining salient spatial context and stabilizing optimization; the figure depicts video, text, and noise entering the DiT.Source: §3.3 · p. 5, Figure 1 · p. 5
4 Experiments
The experiments evaluate Foley Control against recent video-to-audio systems on MovieGenBench, test pooling strategies, and report training and evaluation protocols. The reported comparisons are competitive, while the pooling ablation finds that lower-resolution video embeddings can preserve alignment with reduced compute and memory.
4.1 Experimental Setup
- The experiments train collaboration layers using V-JEPA2 embeddings over a frozen StableAudioDiT backbone, with curated Kinetics-700 training data and Meta Movie Audio Bench evaluation.The Stable Audio DiT, VAE, and CLAP text encoder remain frozen.Source: §4.1 · p. 5
4.2 Ablation Studies
- Across 400k steps, Single and grid8 KL-PANNs differ by less than 0.03 on Kinetics-700 validation without text guidance; Single is on par or slightly better.The runs use a fixed 30% subset of curated training data.Source: Table 1 · p. 6, Figure 2 · p. 6, §4.2 · p. 6
4.3 Comparison to Existing Work
- Comparisons use the same MovieGenBench test split, frame-rate and duration limits, and one unprocessed audio sample per clip, with perceptual, statistical, semantic, and synchronization metrics.The metric suite includes Fréchet distance and KL divergence, ImageBind consistency, and Synchformer synchronization.Source: §4.3 · p. 7
- Kling-Foley AudioEval results are omitted because training-corpus overlap could produce inflated or noncomparable scores.Source: §4.3 · p. 8
- On MovieGenBench, Foley Control performs competitively under demanding ambient conditions against systems including MMAudio, HunyuanVideo-Foley, ThinkSound, and FRIEREN.MovieGenBench emphasizes cinematic sound design and long-range temporal dependencies.Source: §4.3 · pp. 6, 8, Table 2 · p. 8
- Nearly two orders of magnitude less paired data and compute than HunyuanVideo-Foley still yields competitive synchronization and semantic alignment for Foley Control.Foley Control trains for 400k steps with an effective batch size of 384 and requires no auxiliary audio-only corpus.Source: §4.3 · p. 8
5 Conclusion
Foley Control adds a compact video bridge to a frozen text-to-audio generator, preserving the audio prior and prompt control while delivering competitive alignment. The design remains modular, though pooled tokens and capped video duration constrain its current scope.
5 Conclusion
- Foley Control inserts compact trainable collaboration layers after text cross-attention to add video guidance to a frozen text-to-audio generator.V-JEPA2 embeddings, token pooling, and RoPE ordering cues support temporal control while preserving the audio prior and prompt controllability.Source: §5 · p. 8
- On curated data and MovieGenBench, Foley Control delivers competitive semantic and temporal alignment while training only a small fraction of parameters compared with fully multimodal systems.Aggressively pooled video tokens match denser grid features while reducing compute and memory without degrading synchronization.Source: §5 · p. 8
- Encoders or the text-to-audio backbone can be swapped or upgraded without end-to-end retraining, supporting production settings with evolving models and latency or VRAM budgets.Source: §5 · p. 8
- The current setup caps video duration and uses pooled tokens that may miss rare fine-grained spatial cues; it does not explicitly model spatial acoustics or streaming alignment.The stated future-work directions include longer-context conditioning and spatial audio generation.Source: §5 · p. 9