HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation

Motivation

• Synchronized audio is essential for immersive generated video, but traditional Foley production is costly and time-intensive.Recent video generation advances produce photorealistic dynamic sequences, while manual frame-by-frame Foley creation is incompatible with their efficiency.Source: §1 · p. 1

Limitations of Existing Methods

• Text-to-audio methods generate high-fidelity audio from descriptions but cannot inherently align it with video content.Source: §1 · p. 1
• Existing video-to-audio methods face multimodal data scarcity, modality imbalance, and limited audio quality.Source: §1 · p. 1

Contributions

• HunyuanVideo-Foley builds a ∼100k-hour text-video-audio dataset through automated labeling and filtering.Source: §1 · pp. 1–2
• Its dual-stream MMDiT and single-stream audio DiT architecture uses joint audio-video attention, textual cross-attention, and REPA-based audio guidance.REPA aligns internal audio representations with pretrained audio features to improve generation quality and stability.Source: §1 · p. 2

Text-to-Audio Generation

• Text-to-audio research advanced from diffusion and autoregressive synthesis to latent diffusion with pretrained text encoders for semantic alignment.Source: §2.1 · p. 2

Video-to-Audio Generation

• Video-to-audio methods inject visual features into pretrained text-to-audio models or train multimodal models from scratch.Source: §2.2 · p. 2

Open Challenges

• Existing video-to-audio systems still exhibit suboptimal audio quality, imprecise temporal alignment, and insufficient visual-semantic correspondence.Source: §2.2 · p. 2
• HunyuanVideo-Foley uses distinct attention mechanisms for video-audio and text-audio relationships to improve visual-semantic alignment and synthesized audio quality.Source: §2.2 · p. 2

Representation Alignment

• REPA has been adopted across generative modeling tasks to align internal features with representations from pretrained encoders.Source: §2.3 · p. 2

3.1 TV2A Data Pipeline

• The data pipeline filters video-audio samples, balances annotations, and constructs a ∼100k-hour text-video-audio training dataset.Filtering removes unsuitable clips, while speech-music detection, audio classification, and generated captions annotate retained segments.Source: §3.1 · p. 3

Multimodal Architecture

• HunyuanVideo-Foley uses N1 multimodal transformer blocks followed by N2 unimodal audio transformer blocks.The framework processes visual, textual, audio, and synchronization representations through specialized encoders.Source: §3.2 · p. 3, Multimodal Representations · p. 3, Figure 2 · p. 4
• Dual-phase attention concatenates audio and visual tokens for joint self-attention while using cross-attention to inject textual guidance.Source: Multimodal Alignment with Dual-Phase… · p. 4

Interleaved RoPE

• Interleaved RoPE aligns audio and visual sequences before position encoding so temporally adjacent tokens receive consecutive embeddings.The sequences are length-aligned, interleaved, position-encoded, and then decoupled for subsequent attention.Source: Interleaved RoPE · p. 4

Modulation with Synchronization Features

• Synchronization features and flow-timestep embeddings condition transformer blocks through adaptive modulation and gated residual integration.The conditioning signal is formed by summing synchronization and timestep embeddings, then produces modulation and gate parameters.Source: Modulation with Synchronization Features · p. 5

REPA Training Strategy

• REPA aligns intermediate diffusion representations with pretrained ATST-Frame audio features by maximizing their cosine similarity.An MLP maps internal DiT states into the representation space, providing semantic and acoustic guidance during generation.Source: §3.4 · p. 5

Kling-Audio-Eval

• On Kling-Audio-Eval, FD improves from 9.01 to 6.07, KL from 2.17 to 1.89, and IB from 0.30 to 0.38 versus MMAudio, although IS, CE, and CLAP are slightly lower.The comparison spans distribution matching, audio quality, and visual-semantic alignment metrics.Source: Text-Video-to-Audio Generation · p. 6, Table 1 · p. 6

VGGSound-Test

• On VGGSound-Test, HunyuanVideo-Foley leads in IS and PQ, maintains state-of-the-art IB, and achieves comparable DeSync and CLAP despite weaker FD and KL than some baselines.The reported distribution gap is attributed to generally poor-quality, non-professional recordings in VGGSound.Source: Text-Video-to-Audio Generation · p. 6, Table 2 · p. 6

MovieGen-Audio-Bench

• On MovieGen-Audio-Bench, HunyuanVideo-Foley outperforms baselines in nearly all objective metrics and all subjective evaluations, improving PQ, DeSync, and IB over MMAudio while matching its CLAP performance.Table 3 reports both objective and subjective evaluation results.Source: Text-Video-to-Audio Generation · p. 6, Table 3 · p. 6

Overall Results

• Across three datasets, HunyuanVideo-Foley establishes state-of-the-art TV2A performance, substantially improving visual-semantic alignment while leading or remaining competitive on audio quality, temporal alignment, text-semantic alignment, and distribution matching.It achieves optimal distribution matching on Kling-Audio-Eval and substantial IB gains over all baselines.Source: Text-Video-to-Audio Generation · p. 6, Table 3 · p. 6

Architecture Ablations

• Ablations show that joint audio-video attention followed by text cross-attention outperforms alternative multimodal designs, while interleaved RoPE and unimodal audio DiT improve alignment and overall performance.The proposed attention ordering particularly improves DeSync, and replacing the audio-only transformer with an audio-video dual-stream DiT degrades results.Source: Model Architecture · p. 7, Table 5 · p. 7

Representation Alignment

• REPA performs best when applied in the unimodal audio DiT, with ATST providing the strongest representation-alignment results; combining EAT and ATST degrades most metrics.ATST improves audio quality, temporal alignment, and text-semantic alignment, while shallower unimodal layers tend to work better for REPA.Source: Model Architecture · p. 7, Table 6 · p. 7, Table 7 · p. 7

• HunyuanVideo-Foley achieves new state-of-the-art text-video-to-audio performance, particularly in video-semantic alignment, temporal synchronization, and audio quality.Source: §5 · p. 8
• The framework uses a REPA strategy to produce high-fidelity audio with balanced alignment to visual dynamics and text context.Source: §5 · p. 8
• An efficient data pipeline provides support for scaling text-video-audio training data.Source: §5 · p. 8

• Bandwidth tagging assigns “high-quality” captions to audio above 16 kHz and appends the tag to inference captions.The strategy addresses varying sampling rates in the training data.Source: §A.1 · p. 10
• The method associates the “high-quality” tag with higher sampling rates, producing outputs with enhanced high-frequency detail preservation.Source: §A.1 · p. 10

• The metrics cover text-semantic consistency, acoustic quality, visual-semantic alignment, temporal synchronization, and generated-versus-real audio distribution similarity.Source: §A.2 · p. 10
• Fréchet Distance, Kullback-Leibler divergence, and Inception Score assess distribution similarity, probability divergence, and audio quality and diversity.Source: §A.2 · p. 10
• AudioBox-Aesthetics reports production quality and production complexity, with complexity scores potentially increasing for noisy or artifact-laden audio.Source: §A.2 · p. 10
• ImageBind cosine similarity evaluates video-audio cross-modal alignment, and DeSync estimates audio-visual synchronization errors.Source: §A.2 · p. 10
• LAION-CLAP measures text-audio semantic similarity, while MOS-Q, MOS-S, and MOS-T rate acoustic naturalness, semantic matching, and audio-visual synchronization.Source: §A.2 · p. 10

• Audio Reconstruction

• PESQ, STOI, SI-SDR, and Mel Distance evaluate perceptual speech quality, intelligibility, signal distortion, and spectrogram distance.Source: • Audio Reconstruction · p. 10
• The study compares FoleyCrafter, V-AURA, Frieren, MMAudio, and ThinkSound using officially released pretrained versions.Source: §A.3 · p. 11
• All baselines are evaluated on identical hardware with their original inference scripts, selecting the best available pretrained variant when multiple versions exist.Source: §A.3 · p. 11
• FoleyCrafter uses visual relevance and temporal control, V-AURA uses high-frame-rate visual features, and Frieren uses rectified-flow spectrogram generation.Source: §A.3 · p. 11

B.1 Radar Chart.

• HunyuanVideo-Foley consistently outperforms existing methods across almost all metric dimensions on all three evaluation benchmarks.The comparison is presented through three radar-chart subplots covering Kling-Audio-Eval, VGGSound-Test, and MovieGen-Audio-Bench.Source: §B.1 · p. 11, Figure 3 · p. 11

B.2 Spectrogram.

• The method maintains precise temporal alignment between audio events and actions in walking, skateboarding, and other motion scenarios.The visualizations cover step timing and duration, as well as synchronization with temporal characteristics of human movement.Source: §B.2 · p. 11, Figure 4 · p. 12
• HunyuanVideo-Foley preserves high-frequency spectral components without spectral leakage, including in complex skiing scenarios.The ice hockey example also shows robust detection of subtle motion variations synchronized with sound patterns.Source: §B.2 · p. 11, Figure 5 · p. 12