Source-linked AI summary
LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision
Chunyu Li, Chao Zhang, Weikai Xu, Jingyu Lin, Jinghui Xie, Weiguo Feng, Bingyue Peng, Cunjian Chen, Weiwei Xing
TL;DR
Audio-conditioned LDMs can generate lifelike talking videos but achieve suboptimal lip-sync accuracy because they learn visual shortcuts instead of audio-visual correlations. LatentSync adds SyncNet supervision, StableSyncNet, and TREPA; StableSyncNet reaches 94% accuracy on HDTF, and the method surpasses state-of-the-art approaches across metrics on HDTF and VoxCeleb2.
Problem
Audio-conditioned LDMs applied to lip sync may learn visual-visual shortcuts rather than the audio-visual correlations needed for accurate synchronization.
Method
LatentSync integrates SyncNet supervision into audio-conditioned LDMs, introduces StableSyncNet for stable convergence, and uses TREPA to improve temporal consistency.
Results
94% accuracy on the HDTF test set was achieved by StableSyncNet, and LatentSync surpasses state-of-the-art lip-sync approaches across metrics on HDTF and VoxCeleb2.
Takeaways & Limitations
The framework provides an end-to-end audio-conditioned latent-diffusion approach for lifelike, high-resolution lip sync with improved temporal consistency.
Takeaways & Limitations
Subtly different sounds may map to the same motion representation, losing nuanced expressions linked to speech emotion.
Abstract
from arXiv · showhide
End-to-end audio-conditioned latent diffusion models (LDMs) have been widely adopted for audio-driven portrait animation, demonstrating their effectiveness in generating lifelike and high-resolution talking videos. However, direct application of audio-conditioned LDMs to lip-synchronization (lip-sync) tasks results in suboptimal lip-sync accuracy. Through an in-depth analysis, we identified the underlying cause as the "shortcut learning problem", wherein the model predominantly learns visual-visual shortcuts while neglecting the critical audio-visual correlations. To address this issue, we explored different approaches for integrating SyncNet supervision into audio-conditioned LDMs to explicitly enforce the learning of audio-visual correlations. Since the performance of SyncNet directly influences the lip-sync accuracy of the supervised model, the training of a well-converged SyncNet becomes crucial. We conducted the first comprehensive empirical studies to identify key factors affecting SyncNet convergence. Based on our analysis, we introduce StableSyncNet, with an architecture designed for stable convergence. Our StableSyncNet achieved a significant improvement in accuracy, increasing from 91% to 94% on the HDTF test set. Additionally, we introduce a novel Temporal Representation Alignment (TREPA) mechanism to enhance temporal consistency in the generated videos. Experimental results show that our method surpasses state-of-the-art lip-sync approaches across various evaluation metrics on the HDTF and VoxCeleb2 datasets.
1. Introduction
LatentSync addresses suboptimal lip-sync accuracy in audio-conditioned LDMs by countering shortcut learning with SyncNet supervision, StableSyncNet, and TREPA. The method targets audio-visual correlation, SyncNet convergence, and temporal consistency in high-resolution talking videos.
- Task scope: Lip sync regenerates a speaker’s lip movements from audio while preserving head pose and personal identity, with applications including visual dubbing, virtual avatars, and video conferencing.The task is framed as video editing rather than unrestricted portrait animation.
- Motivation: Audio-conditioned LDMs can learn visual shortcuts from masked frames instead of the audio-visual correlations required for accurate lip sync.Without SyncNet supervision, lip-sync accuracy varies with mask size because the model shifts between visual and audio cues.
- StableSyncNet: 94% accuracy on the HDTF test set was achieved by StableSyncNet after empirical studies of SyncNet architecture, training hyperparameters, and preprocessing.The authors report that SyncNet convergence strongly affects the effectiveness of supervision and the supervised model’s lip-sync accuracy.
- Approach: LatentSync incorporates SyncNet supervision into audio-conditioned latent diffusion models to explicitly enforce audio-visual correlation learning.The authors explored decoded pixel-space and latent-space supervision because applying SyncNet effectively to latent diffusion remained unclear.
- Temporal consistency: TREPA is introduced to improve temporal consistency and reduce flickering in high-frequency details such as teeth, lips, and facial hair.The contribution is motivated by observed flickering artifacts in generated talking videos.
2. Related Work
Related work spans pixel-space diffusion, multi-stage diffusion or VAE pipelines, GAN-based generators, and video-to-video lip-sync systems. Audio-driven portrait animation methods differ from lip sync because they use image-to-video generation rather than preserving non-mouth regions through video editing.
- Diffusion-based methods: Diffusion-based lip-sync methods include pixel-space systems such as Diff2Lip [28] and DrivenVideoEditing [3], alongside multi-stage approaches such as MyTalk [48], StyleSync [53], and DiffDub [25].The multi-stage methods separate audio-to-motion or semantic-latent conversion from motion-to-image generation.
- Non-diffusion-based lip sync: Wav2Lip [29] established pretrained SyncNet supervision for lip-sync generator training, while later systems use quantized spaces, StyleGAN2 [22], reenactment pipelines, or feature-map deformation.Examples include, StyleSync [14], VideoReTalking [8], DINet [52], and MuseTalk [50].
- LatentSync: LatentSync uses an audio-conditioned LDM with Whisper [32] audio embeddings, reference and masked frames, and pixel-space TREPA, LPIPS [49], and SyncNet [29] losses.The framework integrates audio through U-Net [34] cross-attention and processes concatenated visual and latent inputs.
- Task distinction: Lip sync is a video-to-video editing task that preserves non-mouth regions, whereas audio-driven portrait animation uses image-to-video generation and may alter head movement or facial expressions.This distinction explains why portrait-animation methods cannot be directly applied to lip sync.
3. Method
LatentSync uses video-to-video latent diffusion with audio cross-attention, fixed masking, SyncNet supervision, and temporal representation alignment to generate synchronized, temporally consistent videos.
- 3.1. LatentSync Framework: LatentSync combines temporal video-to-video inpainting, reference frames, and a 13-channel U-Net input to preserve facial identity during generation.The channels include noise latent, mask, masked frame, and reference frame inputs.
- 3.1. LatentSync Framework: Whisper extracts surrounding-frame audio features, which enter the U-Net through native cross-attention to provide broader temporal audio context.For each generated frame, the method bundles audio from several surrounding frames.
- 3.1. LatentSync Framework: Affine face frontalization and a fixed whole-face mask reduce visual-visual shortcuts, while avoiding landmark-based mask motion that could reveal lip movements.The mask is fixed in position and shape, and affine transformation helps the model learn facial features in side-profile views.
- 3.2. Two-Stage Training Strategy: SyncNet supervision is applied through either decoded pixel-space training or latent-space training, with latent-space supervision showing inferior convergence in empirical analysis.Pixel-space supervision requires storing VAE-decoding activations, motivating a two-stage strategy that first learns visual features and then audio-visual correlations.
- 3.2. Two-Stage Training Strategy: The second training stage optimizes SyncNet, LPIPS, and TREPA objectives to combine lip synchronization, visual quality, and temporal consistency.TREPA aligns temporal representations of generated and ground-truth sequences using a self-supervised video encoder and normalized mean squared error.
4. Empirical Studies on SyncNet Convergence
The study identifies batch size, architecture, embedding dimension, input-frame count, and preprocessing as factors governing SyncNet convergence. StableSyncNet with selected settings reaches 94% accuracy on out-of-distribution HDTF, surpassing the previous 91% result.
- Batch size: A batch size of 1024 converges faster and more stably with lower final validation loss, whereas batch size 128 can remain stuck at 0.69.Batch size 256 can converge but exhibits substantial training-loss oscillations.
- Architecture: StableSyncNet uses redesigned U-Net-based audio and visual encoders and maintains lower training and validation losses than the compared architectures.The design retains residual and self-attention blocks, adjusts downsampling for input sizes, and removes cross-attention.
- Embedding dimension: An embedding dimension of 2048 is optimal for 256 × 256 inputs, while 512 lacks semantic capacity and 4096 or 6144 produces sparse representations that impede convergence.The embedding-dimension comparison is conducted with batch size 512, 16 frames, and the StableSyncNet architecture.
- Number of frames: Using 16 input frames supports convergence, whereas 25 frames can trap training near 0.69 without a meaningful final validation-loss advantage.The frame-count comparison uses batch size 512, the StableSyncNet architecture, and embedding dimension 2048.
- Overall findings: 94% HDTF accuracy surpasses the previous 91% result after training StableSyncNet with the identified optimal settings.The model is trained on VoxCeleb2 and evaluated on HDTF, with VoxCeleb2 validation loss around 0.18.
- Configuration and preprocessing: The selected StableSyncNet configuration uses batch size 1024, 16 frames, 2048-dimensional embeddings, adapted SD U-Net encoders, and offset adjustment after affine transformation.The study identifies batch size, input-frame count, and preprocessing as primary convergence factors.
5. Experiments
LatentSync is evaluated on VoxCeleb2 and HDTF using visual-quality, lip-sync, and temporal-consistency metrics, with comparisons, SyncNet ablations, and TREPA studies. It surpasses other methods in reported lip-sync and visual-quality measures, while SyncNet supervision and TREPA improve key outcomes.
- Quantitative comparisons: 94% StableSyncNet accuracy, together with audio cross-attention, gives LatentSync significantly higher lip-sync accuracy than competing methods on HDTF and VoxCeleb2.The authors also attribute its visual-quality advantage to Stable Diffusion and its FVD advantage to temporal layers and TREPA.
- Qualitative comparisons: LatentSync preserves clearer videos and facial identity than Wav2Lip, VideoReTalking, Diff2Lip, and MuseTalk in cross-generation examples.Wav2Lip is described as accurate but blurry; other baselines show artifacts, low resolution, or weaker facial-feature preservation.
- SyncNet ablations: Removing SyncNet supervision substantially degrades lip-sync performance, while decoded pixel-space supervision outperforms latent-space supervision for lip-sync accuracy and temporal consistency.The latent-space disadvantage is associated with poor SyncNet convergence and information loss in VAE-compressed inputs.
- SyncNet ablations: Higher lip-sync accuracy accompanies improved temporal consistency because the audio window contains rich temporal information and the model uses audio more effectively.This relationship is reported as an observed association in the ablation analysis.
- TREPA ablation: TREPA improves both temporal consistency and visual quality in the HDTF ablation.The authors attribute this improvement to VideoMAE-v2 representations that encode visual and temporal information.
6. Conclusion
The paper introduces LatentSync as an audio-conditioned latent-diffusion lip-sync method, identifies shortcut learning, studies SyncNet convergence, and adds TREPA for temporal consistency. Its conclusion emphasizes addressing pixel-space diffusion limitations, information loss in two-stage generation, and temporal artifacts.
- Conclusion: LatentSync combines audio-conditioned latent diffusion, SyncNet supervision, StableSyncNet convergence analysis, and TREPA to address lip-sync and temporal-consistency challenges.The conclusion presents these as the paper’s central contributions.
- Conclusion: The method targets pixel-space diffusion’s low sampling speed and limited high-resolution generation, as well as information loss from two-stage generation.These limitations motivate the latent-diffusion framework described in the conclusion.
A. SyncNet Training Loss Proof
The proof analyzes a non-converging SyncNet whose predicted positive and negative probabilities collapse toward 0.5. This explains why its training loss decreases initially and then becomes stuck.
- Derivation setup: The derivation begins from positive and negative sample probabilities and expresses the loss using A positive and B negative samples in a batch.The supplied proof setup defines the sample counts and the probability-distribution terms used in the loss calculation.
- Observed behavior: SyncNet predictions collapse toward q(x_i = 1) ≈ q(x_i = 0) ≈ 0.5 for every sample during late training.The figure shows dispersed probabilities early, followed by concentration around 0.5.
- Interpretation: A non-converging SyncNet cannot distinguish positive from negative samples and therefore outputs approximately 0.5 to reduce its training loss.The authors explain this as the loss-minimizing behavior of an ineffective classifier.
- Loss consequence: The training loss falls initially but then stalls near 0.69 as the classifier settles into the uniform 0.5 prediction.The passage attributes the plateau to the non-converging model’s inability to learn the distinction between sample types.