PaperlayerView exact revision on arXiv

Source-linked AI summary

Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

Paul Hyunbin Cho, Jinhyuk Jang, SeokYoung Lee, Joungbin Lee, Siyoon Jin, Heeseong Shin, Jung Yi, Yunjin Park, Chulmin Park, Seungryong Kim

arXiv:2606.11180v1cs.CV

TL;DR

Diffusion-based V2V lip synchronization is visually strong but too costly for real-time use because of bidirectional attention and many denoising steps. Lip Forcing distills a 14B bidirectional teacher into causal students using trajectory-derived guidance, two-step inference, and SyncNet reward supervision. The 1.3B student reaches 31 FPS, while the 14B student is 39.8× faster than its teacher, with sub-millisecond time-to-first-frame at both scales.

  • Problem

    Diffusion-based V2V lip synchronization has strong fidelity and alignment, but high inference cost limits practical latency-sensitive deployment.

  • Method

    Lip Forcing distills a 14B bidirectional video-diffusion teacher into causal students using Sync-Window DMD, a two-step schedule, and a SyncNet-based reward.

  • Results

    The 1.3B student reaches 31 FPS and is 17.6× faster than its same-scale bidirectional model, while the 14B student is 39.8× faster than its teacher; both have sub-millisecond TTFF.

  • Takeaways & Limitations

    Lip Forcing enables real-time streaming lip synchronization with causal two-step students distilled from a bidirectional teacher.

  • Takeaways & Limitations

    The method deliberately favors reference fidelity over some audio-visual synchronization, trailing the strongest baselines on Sync-C.

Abstract

from arXiv · show

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.

1 Introduction

Lip Forcing addresses the deployment gap in diffusion-based V2V lip synchronization by distilling a bidirectional teacher into causal, few-step students. Its analysis-driven recipe combines trajectory-aware guidance, two-step inference, and SyncNet supervision, achieving real-time streaming at one student scale.

  • Diffusion-based V2V lip synchronization improves visual fidelity and audio-visual alignment but remains costly for latency-sensitive deployment.Applications include offline dubbing, live translation, virtual avatars, and interactive agents.
  • Full-sequence self-attention scales quadratically with clip length, while tens of denoising steps further compound inference cost.Autoregressive diffusion and few-step distillation address these two computational factors.
  • Lip Forcing distills a 50-step bidirectional teacher into a two-step streaming student using trajectory analysis of the CFG fidelity–sync tradeoff.No-CFG preserves reference fidelity, whereas CFG improves synchronization mainly within a mid-trajectory band.
  • The method combines Sync-Window DMD, an analysis-derived two-step inference schedule, and a SyncNet-based reward.CFG is enabled only during training timesteps inside the sync-favoring band, while inference uses two model calls without inference-time CFG.
  • 31 FPS makes the 1.3B student 17.6× faster than its same-scale bidirectional model and crosses the 25 FPS real-time threshold.The 14B student runs 39.8× faster than its teacher, and both scales achieve sub-millisecond time-to-first-frame.

2 Related Work

Related work spans efficient GAN-based V2V lip synchronization, diffusion-based generation, and causal autoregressive video diffusion. Recent distillation methods make causal few-step students practical through techniques such as self-rollouts, stabilization mechanisms, and reward weighting.

  • V2V lip synchronization edits existing video to match target audio while preserving identity, pose, and background.Early GAN-based methods offered efficient inference but suffered from blur and temporal inconsistency.
  • Autoregressive video diffusion generates frames or chunks sequentially with KV caching, enabling streaming inference.Recent work distills bidirectional teachers into causal few-step students using Distribution Matching Distillation and Self Forcing.
  • Prior extensions stabilize autoregressive extrapolation with sink-frames, rolling-reference attention, or related mechanisms.Reward-weighted variants additionally reweight per-sample DMD gradients using task-specific rewards.

3 Preliminaries

The preliminaries describe rectified-flow generation, causal autoregressive sampling, and DMD with classifier-free guidance. Teacher trajectory analysis motivates selecting guidance and landing schedules rather than applying one fixed configuration throughout denoising.

  • Rectified flow: Rectified flow models generation as deterministic transport from Gaussian noise to data using a learned velocity field.The backward flow operator Ψ updates an intermediate state toward an earlier timestep.
  • Rectified flow: At continuous time t, t=1 denotes noise and t=0 denotes data; the teacher uses a fixed 50-step shifted ODE schedule.Discrete step index j and continuous timestep τj are distinguished, with τ0=0.999 and τ50=0.
  • Autoregressive diffusion: A causal student generates each frame or chunk conditioned on its own previous clean predictions and conditioning inputs.A K-call student uses a subset of the teacher’s ODE indices, with K much smaller than 50.
  • DMD and guidance: DMD re-noises the student’s clean prediction and updates it using the difference between a frozen teacher score and a learned fake-score network.Self Forcing samples causal context from the student’s own rollout to reduce train-test exposure mismatch.
  • DMD and guidance: Classifier-free guidance combines unconditional and conditional teacher scores, with s=1.0 denoting no-CFG and s>1 denoting CFG-guided sampling.Trajectory analysis compares guidance schedules and evaluates mixed schedules by their second-step landing index.

4 Method

Lip Forcing distills a bidirectional 14B lip-sync teacher into a two-step causal student using trajectory analysis to balance reference fidelity and synchronization. The resulting recipe combines Sync-Window DMD, an analysis-derived landing schedule, and SyncNet reward supervision.

  • Overview: Lip Forcing distills a high-fidelity bidirectional teacher into a two-step causal student for streaming lip synchronization.The framework first pretrains the causal student, then applies Self Forcing DMD with three trajectory-analysis-derived modifications.
  • Bidirectional teacher trajectory analysis: CFG improves audio-visual synchronization at the cost of reference fidelity, whereas no-CFG inference favors reference fidelity.The tradeoff means no fixed CFG scale among those tested optimizes both metrics.
  • Sync-Window DMD: Sync-Window DMD gates teacher CFG to the mid-trajectory window 20 ≤j ≤40 instead of using a fixed guidance scale at every timestep.No-CFG predictions preserve mouth-region fidelity, while CFG guidance is applied where it improves Sync-C for lip articulation.
  • Two-step inference schedule: The two-step inference schedule uses ODE indices JLF = (0, 30), with the second step chosen to prioritize fidelity and the remaining sync gap reduced by reward supervision.The student makes two denoising calls per chunk, plus one pass for clean-latent KV caching, and uses no inference-time CFG.
  • SyncNet-based reward: A SyncNet confidence score between conditioning audio and the decoded clean prediction reweights the DMD generator gradient to add explicit synchronization supervision.The clean prediction is decoded by the frozen Tiny AutoEncoder before SyncNet scoring; β = 2 controls reward strength.

5 Experiments

Experiments evaluate Lip Forcing’s streaming efficiency, fidelity, synchronization, and user-perceived quality on HDTF, then ablate its guidance, reward, and inference-step choices. The results show a strong fidelity–sync tradeoff: the method substantially improves efficiency and fidelity while accepting some Sync-C loss, with user ratings remaining competitive.

  • Main comparison: Sub-millisecond TTFF accompanies 17.6× and 39.8× speedups over same-init bidirectional OmniAvatar-LS models at 1.3B and 14B.The 14B student is also 4.7× faster than LatentSync.
  • Main comparison: 31 FPS makes Lip Forcing (1.3B) the fastest diffusion method and exceeds the 25 FPS playback rate of the test videos.Every multi-step diffusion baseline remains below the real-time threshold.
  • Main comparison: Lip Forcing trades a Sync-C deficit for substantially lower FID and FVD against single-pass baselines, while MuseTalk leads on CSIM and SSIM.Wav2Lip and VideoReTalking exceed ground-truth Sync-C, suggesting possible SyncNet-expert overfitting at the expense of perceptual realism.
  • Main comparison: Against multi-step diffusion baselines, the 14B model posts the best FVD overall and improves over OmniAvatar-LS on FVD, SSIM, and CSIM.At 1.3B, it is on par with LatentSync on FVD and dominates Diff2Lip and X-Dub on fidelity metrics.
  • Ablations: The windowed CFG schedule substantially improves FVD over static CFG, while the SyncNet reward consistently improves Sync-C in both CFG settings.Together, SW-DMD recovers fidelity and the explicit reward narrows the synchronization gap introduced by windowing.
  • Ablations: The analysis-derived two-step landing at j1=30 closes most of the 1-step versus 4-step FVD gap at half the 4-step inference cost.The landing is a direct fidelity–sync control: j1=25 favors synchronization, j1=37 favors FVD, and j1=30 balances both.
  • User study: In the user study, Lip Forcing exceeds all baselines on quality, identity, and naturalness MOS, and matches the highest-performing baseline on synchronization.The study used 30 clips from HDTF and TalkVid with four 5-point Likert items.

6 Conclusion

The paper concludes that Lip Forcing distills a bidirectional diffusion teacher into a two-step causal streaming framework for lip synchronization. Its efficiency and quality results support real-time deployment, while the supplementary material documents the teacher, method, experiments, qualitative results, limitations, and broader impact.

  • Conclusion: Lip Forcing distills a bidirectional video-diffusion teacher into a two-step causal autoregressive student using trajectory analysis, SW-DMD, an analysis-derived schedule, and a SyncNet reward.The framework is distilled from a single 14B OmniAvatar-based teacher.
  • Conclusion: The reported speed and quality results bring streaming lip synchronization within reach of latency-sensitive applications such as live translation, virtual avatars, and interactive agents.The paper also presents trajectory-aware diagnosis as a procedure for adapting conditional-diffusion lip-sync teachers to few-step distillation.
  • Supplementary material: The appendix provides extended details on the OmniAvatar teacher, trajectory analysis, method implementation, experiments, additional qualitative results, limitations, and societal impact.Supplementary experiments include additional benchmarks, long-video evaluation, and cross-identity audio evaluation.

B.3 Training data and preprocessing

Training uses a shared audiovisual preprocessing pipeline and filtered clip pool for teacher finetuning and student distillation. The pipeline standardizes frame/audio rates, face alignment, quality filtering, and trajectory-analysis inputs.

  • Three audiovisual datasets contribute diverse training material, including 50K VoxCeleb2 clips, HDTF talking-face videos, and Hallo3 talking-head and wild-scene videos.
  • Videos are resampled to 25 fps and audio to 16 kHz, then segmented into 5–10 s clips at shot boundaries.
  • Faces are aligned with InsightFace landmarks and resized to 512×512 for model input.
  • SyncNet confidence below 3 and HyperIQA scores below 40 are filtered, leaving approximately 30K clips for both training stages.
  • Each training example uses an 81-frame input window, with a separately sampled reference window from the same source clip.
  • The 14B model serves as the distillation teacher, while the 1.3B model supplies student initialization and a same-scale baseline.

C.2 Full 4-metric trajectory analysis

Additional metrics and alternate CFG-drop experiments reproduce the trajectory analysis: guidance separates fidelity from synchronization, while a single CFG transition performs best near the mid-trajectory landing window.

  • The full four-metric analysis confirms the CFG fidelity–sync tradeoff and schedule decomposition beyond mouth LPIPS and Sync-C.
  • The audio-only trajectory matches the text+audio condition within sub-0.2 Sync-C points and within 0.02 on SSIM and LPIPS.
  • Under audio-only CFG drop, guided and no-CFG trajectories retain the same separation across SSIM, LPIPS, Sync-C, and Sync-D.
  • The no-CFG→CFG cell remains statistically flat in Sync-D across landing steps 25–32, supporting j1=30 as a representative operating point.The paired test between j1=25 and j1=30 gives p=0.13 with n=10.
  • Cells sharing the initial CFG setting converge on reference metrics by mid-trajectory, while single-CFG cells nearly close the synchronization gap around landing step 30.

D.1 Hyperparameters and training details

Training combines Diffusion Forcing pretraining with self-causal DMD distillation, using the analysis-derived two-call schedule, windowed teacher guidance, and SyncNet reward weighting.

  • Training stages: Stage 1 runs Diffusion Forcing pretraining, while Stage 2 performs Self Forcing DMD distillation on the student’s causal rollout.
  • Optimization: Both stages use the filtered dataset, AdamW, bf16 mixed precision, gradient clipping at 10.0, and an effective batch size of 64.
  • Timestep sampling: Stage 1 independently noises each three-latent-frame chunk at a timestep from the shifted-ODE inference grid.
  • Distillation: Stage 2 rolls out the student with K=2 causal denoising calls at JLF=(0,30), then re-noises the clean prediction for DMD supervision.
  • Architecture and rollout: The 14B student uses LoRA adapters while audio-conditioning projections and patch embeddings are fully finetuned; rollout chunks contain three latent frames.
  • Reward: The SyncNet reward applies a forward-only multiplicative weight to the DMD generator gradient, with SyncNet and decoder components detached during backpropagation.

E.1 Compute and efficiency methodology

Efficiency evaluation measures streaming throughput and time-to-first-frame under a uniform timing protocol across Lip Forcing and released baselines. The 1.3B model exceeds the 25 FPS playback target, while causal caching bounds memory independently of rollout length.

  • Measurement protocol: Throughput and time-to-first-frame are measured from the first VAE encode through the first chunk’s final VAE decode on one NVIDIA H100 80 GB GPU.
  • Baselines: All baselines use publicly released code and checkpoints at default settings without architectural or weight modifications.
  • Pareto efficiency: Lip Forcing’s 1.3B and 14B variants occupy the meaningful throughput–FVD Pareto frontier; Wav2Lip is the only other frontier point but has an approximately 3.5× FVD penalty versus Lip Forcing 14B.
  • Streaming throughput: 31.58 FPS places the 1.3B variant above the 25 FPS playback rate at 512×512 with two-step inference, torch.compile, and TAE decoding.
  • Memory: A sink-plus-window KV cache of 1+6=7 latent frames keeps peak allocated memory at 8.78 GB for 1.3B and 40.63 GB for 14B.
  • Evaluation scope: Long-video evaluation tests horizons up to 6 minutes, while held-out Hallo3 and TalkVid sets probe out-of-domain and self-driven generalization.

F Qualitative Results

The paper reports additional qualitative results and a cross-identity evaluation on HDTF, where sync metrics are used because pixel-aligned ground truth is unavailable.

  • Additional qualitative results are provided for the Hallo3, HDTF, and TalkVid test sets.
  • Cross-identity evaluation on HDTF pairs source video with audio from a different speaker and reports sync metrics only.Pixel-aligned ground truth does not apply in this setting.

G Limitations

The method’s trajectory-based recipe has scope boundaries: its cutoffs are characterized on one teacher lineage, the smaller student trades fidelity for speed, and the reward objective remains imperfect.

  • The trajectory recipe assumes a teacher with a CFG fidelity–sync tradeoff and a sync-favoring denoising band.Teachers without this structure require re-running the diagnostic methodology to identify appropriate cutoffs.
  • The specific trajectory structure and schedule cutoffs are characterized on a single 14B OmniAvatar-based teacher and are not claimed to transfer across architectures.The methodology, rather than the specific cutoffs, is presented as transferable.
  • The 1.3B student trails the 14B student on full-frame fidelity despite crossing the 25 FPS playback rate.The paper positions 1.3B for streaming-constrained deployment and 14B for applications prioritizing lip-sync fidelity.
  • SyncNet reward optimization can drift from perceptual realism, so the paper caps reward strength at β=2 and reports both sync and full-frame fidelity metrics.A more principled audio-visual alignment objective remains future work.

H Broader Impact

Lip synchronization supports accessibility, creative production, and interactive agents, but the same efficiency can facilitate deepfakes and lower the barrier to misuse; provenance and authentication are recommended mitigations.

  • Audio-driven lip synchronization can support live captioning, underserved-language dubbing, film and game production, and human-computer interaction agents.
  • The technology can fabricate manipulated video of real individuals, with possible consequences for misinformation, fraud, and non-consensual content.
  • Higher streaming throughput reduces the marginal cost of manipulated content relative to offline pipelines and may lower the barrier to misuse.The paper identifies this as the same property that enables legitimate live applications.
  • The authors recommend provenance signaling and user authentication, complemented by detection research trained on lip-sync artifacts.
  • The study uses four publicly released audio-visual datasets under their original authors’ non-commercial research terms.The datasets are VoxCeleb2, Hallo3, HDTF, and TalkVid.
  • Baseline systems are run from publicly released code and checkpoints under the licenses associated with each release.
Loading 2606.11180v1…