Source-linked AI summary
Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, Ran Yi
TL;DR
Open-source audio-video generation still lacks robust fine-grained synchronization because joint diffusion faces correspondence drift, inefficient temporal interaction, and guidance that does not explicitly reinforce cross-modal alignment. Harmony addresses these issues with Cross-Task Synergy, a Global-Local Decoupled Interaction Module, and SyncCFG, and achieves state-of-the-art synchronization performance.
Problem
Open-source audio-video generation struggles with robust alignment because joint diffusion involves shifting noisy correspondences, limited temporal interaction, and guidance that is agnostic to audio-video consistency.
Method
Harmony combines auxiliary audio-driven video and video-driven audio training with decoupled global-local interaction and synchronization-enhanced classifier-free guidance.
Results
Harmony achieves the best audio-video alignment on Harmony-Bench and establishes a new state-of-the-art in audio-video synchronization.
Takeaways & Limitations
Cross-task supervision, separate style-timing interaction, and explicit synchronization guidance provide a framework for fine-grained audio-video alignment.
Takeaways & Limitations
Standard CFG remains limited because its text-oriented guidance provides no mechanism to isolate or amplify synchronization between audio and video.
Abstract
from arXiv · showhide
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
1. Introduction
Open-source joint audio-video generation still struggles with precise synchronization, especially across speech, ambient sound, and fine-grained temporal correspondence. Harmony addresses these gaps through cross-task training, decoupled interaction, and synchronization-focused guidance, achieving strong alignment results on Harmony-Bench.
- 1. Introduction: Existing open-source systems remain behind closed-source models in audio-video alignment, with some supporting only ambient sounds and others only speech.The supplied comparison caption evaluates sound-type coverage and temporal alignment across joint generation models.
- 1. Introduction: Joint diffusion can suffer Correspondence Drift because two highly noisy latents evolve concurrently, causing the optimal cross-modal mapping to shift during early denoising.This shifting correspondence impedes stable learning of audio-video alignment.
- 1. Introduction: Conventional CFG strengthens modality-specific conditionality but does not inherently enhance correspondence between generated audio and video.Harmony’s SyncCFG addresses this limitation by using cross-task information to isolate and amplify synchronization guidance.
- 1. Introduction: Harmony combines Cross-Task Synergy, Global-Local Decoupled Interaction, and SyncCFG to target drift, style-timing conflicts, and weak cross-modal guidance.Cross-task training uses auxiliary audio-driven video and video-driven audio tasks; interaction separates global style from local timing, while SyncCFG amplifies alignment during inference.
- 1. Introduction: Harmony achieves the best audio-video alignment on Harmony-Bench, maintaining fine-grained temporal synchronization in complex acoustic scenes.The benchmark evaluates joint generation of human speech and ambient sounds.
2. Related Work
Video generation has progressed from GANs to diffusion-based temporal models and increasingly capable Diffusion Transformers. Joint audio-video generation is emerging, but early open-source systems were often limited to coarse environmental sounds and lacked meaningful speech generation.
- 2. Related Work: Video generation evolved from GANs to diffusion models, with temporal architectures such as AnimateDiff and SVD extending diffusion beyond images.Architectures also progressed from UNets toward Diffusion Transformers.
- 2. Related Work: Recent research increasingly targets unified audio-video generation within a single framework.The passage identifies a growing body of work pursuing simultaneous synthesis of both modalities.
- 2. Related Work: Early open-source joint models mainly synthesized coarse environmental sounds and could not generate meaningful human speech, while later systems began incorporating speech-video generation.JAM-Flow and UniAVGen are cited as subsequent efforts addressing speech-video generation.
3. Method
Harmony addresses cross-modal alignment in joint audio-video diffusion through Cross-Task Synergy, global-local interaction, and synchronization-focused guidance. Together, these designs target Correspondence Drift, temporal-style alignment, and the inability of standard CFG to amplify synchronization.
- 3. Method: Harmony combines Cross-Task Synergy, a Global-Local Decoupled Interaction Module, and synchronization-enhanced CFG to address cross-modal alignment in joint audio-video diffusion.The framework targets Correspondence Drift, temporal-style alignment, and synchronization-specific inference guidance.
- 3.2.1. The Challenge of Correspondence Drift: Audio-driven training converges faster to high alignment than joint generation under the same architecture, supporting clean modality signals as useful alignment supervision.The comparison uses lip-sync alignment scores and attributes the disparity to anchoring one modality with a deterministic, noise-free signal.
- 3.2.2. Cross-Task Synergy: Cross-Task Synergy jointly trains the primary task with audio-driven video and video-driven audio tasks, supplying stable uni-directional alignment signals against Correspondence Drift.The driven tasks use clean modality latents, and their losses are combined with the joint-generation loss.
- 3.3. Global-Local Decoupled Interaction Module: The bidirectional global-local module separates precise temporal correspondence from holistic style consistency, preventing interference between the two alignment objectives.RoPE-Aligned Frame-wise Attention handles temporal correspondence, while Global Style Alignment handles holistic consistency.
- 3.4. Synchronization-Enhanced CFG: Standard CFG strengthens text adherence without isolating audio-video correspondence, whereas SyncCFG repurposes guidance to amplify synchronization-specific differences.Its subtraction term captures visual changes correlated with audio, such as mouth movements or object impacts.
4. Experiments
Harmony is evaluated on joint generation of speech, ambient sounds, and complex scenes using automated quality, fidelity, and synchronization metrics. It achieves strong quantitative and qualitative alignment, including precise localization of active sound sources.
- The evaluation covers ambient sound-video, speech-video, and complex-scene generation across video quality, audio fidelity, and audio-visual synchronization.The training corpus includes over 4 million clips spanning human speech and environmental sounds.
- Harmony achieves a C score of 5.61 and the lowest (best) Sync-D score of 7.53, demonstrating strong temporal alignment.These results are reported as validating the cross-task synergy mechanism for cross-modal coherence.
- Compared with UniVerse-1 and Ovi, Harmony produces synchronized lip movements and dynamically aligned mandolin-playing video with richer corresponding audio.The competing systems show lip-sync failures, irrelevant noise, less dynamic audio, or minimal visual motion in the cited examples.
- Audio-to-video attention localizes the active speaker’s mouth and distinguishes among multiple people, while also identifying primary and ambient environmental sound sources.Examples include attention to an animal alongside rain or birdsong in the surrounding scene.
- The ablation study progressively evaluates the Global-Local Decoupled Interaction module, RoPE alignment, Cross-Task Synergy, and SyncCFG against a baseline.All ablated models are trained and evaluated on the human-speech dataset.
5. Conclusion
The conclusion attributes audio-visual synchronization failures to correspondence drift, global-versus-local alignment conflict, and standard CFG limitations. Harmony addresses these issues with three corresponding components and reports state-of-the-art synchronization.
- Harmony combines Cross-Task Synergy, Global-Local Decoupled Interaction, and Synchronization-Enhanced CFG to address three identified synchronization problems.The components target drift, separate style from timing, and amplify alignment during inference, respectively.
- The experiments report that Harmony establishes state-of-the-art audio-video synchronization and outperforms simply scaling up models.The supplied conclusion frames this as a foundation for accessible, well-aligned audio-visual models.
- The supplementary material includes implementation details, benchmark settings, quantitative and qualitative comparisons, voice-clone details, and audio-driven results.It also provides a demo video and project page.
B. Implementation Details
Harmony is trained on diverse speech and environmental-audio data through a three-stage curriculum. The curriculum progresses from audio pre-training and timbre disentanglement to final cross-task audio-visual training.
- The corpus combines public and newly collected sources covering human speech and environmental sounds, with automated annotation and consistency filtering.The human-speech collection includes a high-quality subset of 2 million video clips annotated with transcripts, video captions, and background-sound captions; environmental data includes AudioCaps, Clotho, and WavCaps.
- Stage 1 pre-trains the audio model on a balanced 1:1 mixture of speech and environmental sounds for 100,000 iterations.The reference audio is a randomly selected 1–3 second segment, supporting learning of timbre and content replication.
- Stage 2 uses mismatched reference and target content to disentangle invariant speaker or environmental acoustic characteristics from specific content.Speech uses cross-utterance samples from the same speaker, while environmental sounds use non-overlapping clips from the same recording.
- The final cross-task stage trains the full audio-visual model for 10,000 iterations with batch size 128 using balanced speech and environmental-sound data.A constant learning rate of 1e-5 is used across training stages.
C.1. The Harmony-Bench Dataset
Harmony-Bench is designed to evaluate audio-visual generation across ambient sounds, speech, and complex scenes. Its subsets test fidelity, temporal synchronization, semantic consistency, and multimodal alignment with automated metrics.
- Harmony-Bench contains 150 test cases divided into three progressively challenging subsets of 50 items each.The benchmark addresses limitations in prior benchmarks, including missing human-speech evaluation and weak synchronization coverage.
- The ambient sound-video subset evaluates synchronization of non-speech acoustic events with visual dynamics using separately conditioned audio and video captions.Its evaluation centers on audio fidelity, temporal synchrony, and semantic consistency between generated audio and visual events.
- The speech-video subset evaluates speech fidelity, lip synchronization, and alignment between ambient sounds and corresponding visual actions.It uses a mix of real-world and synthetic multilingual data, as described in the supplied benchmark passage.
- The benchmark uses automated metrics for visual quality and coherence, audio fidelity, and audio-visual synchronization and consistency.The supplied table descriptions also organize comparisons into video quality, audio fidelity, and synchronization categories.
C.2. Evaluation Metrics
Harmony-Bench evaluation measures visual quality and coherence, audio fidelity, and audio-visual synchronization using complementary automated metrics.
- Visual Quality and Coherence: Harmony-Bench evaluates video quality through aesthetic quality, imaging quality, and motion-dynamics metrics.AQ uses aesthetic-predictor-v2-5, IQ uses MUSIQ, and temporal behavior is assessed through motion dynamics.
- Evaluation Scope: Tables 6 and 7 organize comparisons across video quality, audio fidelity, and synchronization, including a dedicated Chinese-speech evaluation.Best and second-best results are marked in the reported tables.
- Audio Fidelity: Speech audio fidelity is measured with WER, while semantic audio-text alignment is quantified by the IB-A Score.WER compares Whisper-large-v3 transcriptions with ground-truth transcripts.
- Audio-Visual Synchronization: Audio-visual synchronization is assessed using Sync-C, Sync-D, DeSync Score, and ImageBind similarity.These metrics cover lip-sync accuracy, temporal misalignment, and overall cross-modal consistency.
D. More Quantitative Comparisons
Quantitative comparisons across environmental-sound, complex-scene, and Chinese-speech settings report stronger fidelity, dynamism, and synchronization for Harmony than competing methods.
- Cross-Dataset Results: Harmony consistently achieves the lowest WER and best audio-visual synchronization scores while maintaining strong video dynamism across diverse datasets.The authors contrast this combination with competitors whose favorable identity scores can arise from nearly static videos.
- Cross-Lingual Speech: On the Chinese speech test set, Harmony achieves substantially lower WER and markedly better synchronization scores than the compared models.Because WER tokenization is imperfect for Chinese, the authors emphasize relative performance between models.
E. More Qualitative Comparisons
Qualitative comparisons show Harmony producing more natural motion and tighter synchronization for speech and environmental sounds, while also supporting voice cloning without visual degradation.
- Human Speech: Harmony generates fluid speech-video motion with clear audio and precise lip synchronization, unlike competitors’ static or minimally dynamic talking heads.The comparison is illustrated in Figure 7 against Ovi and UniVerse-1.
- Environmental Sounds: For environmental sounds, Harmony combines dynamic video, realistic motion, high-fidelity audio, and synchronization with visual events.Competing outputs include unstable audio, static scenes, and mismatched wave and water motion.
- Voice Cloning: A 1–3 second reference audio clip supplies voice timbre for cloning while preserving synchronized lip movement and visual coherence.The reference is encoded into a compact latent representation before generation.
- Audio-Driven Generation: Audio-driven video generation is trained as a native capability by conditioning video synthesis on clean, non-noisy audio latents.Qualitative results show natural facial expressions, coherent head movements, and synchronization with target speech.
H. More qualitative results
Additional qualitative results demonstrate synchronized human-speech videos, diverse artistic styles, and complex ambient sounds across multiple visual and acoustic settings.
- Human Speech: Harmony maintains precise lip synchronization while generating realistic talking heads with natural facial expressions, coherent movements, and clear speech.These results are presented in Figure 11.
- Diverse Visual Styles: Harmony produces temporally coherent motion across distinct aesthetics, including Disney-style animation and traditional ink wash painting.The stylized outputs retain sharp details and vibrant colors consistent with their target aesthetics.
- Diverse Ambient Sounds: Harmony generates diverse ambient sounds, from fireworks to music, with high fidelity and synchronization to corresponding visual sources.The examples are presented in Figure 13.
- Overall Versatility: Across these examples, Harmony combines synchronized speech, varied visual styles, and context-aware soundscapes into expressive audio-visual content.The authors characterize the combined capabilities as evidence of broad generation versatility.