PaperlayerView exact revision on arXiv

Source-linked AI summary

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

Rongxiang Zhang, Songhua Liu

arXiv:2608.00079v1cs.CVcs.SD

TL;DR

Existing talking-head generators trade latency against quality, limiting real-time, long-form synthesis. LeapTalk uses Brownian-bridge distillation for stable one-step generation, achieving competitive fidelity and lip synchronization at up to 200 FPS.

  • Problem

    Existing methods trade latency against quality: diffusion approaches are computationally expensive and short, while autoregressive methods accumulate errors and identity drift.

  • Method

    LeapTalk uses Brownian-bridge data-to-data transport, persistent identity anchoring, heterogeneous SNR-aligned distillation, and audio-driven guidance for one-step streaming generation.

  • Results

    Up to 200 FPS, LeapTalk achieves stable one-step generation with competitive fidelity, lip synchronization, and long-term identity consistency.

  • Takeaways & Limitations

    LeapTalk breaks the latency–quality trade-off for real-time, streaming, open-ended talking-head generation within the reported evaluation scope.

  • Takeaways & Limitations

    The heterogeneous distillation formulation assumes teacher and student share an identical forward process, but LeapTalk violates this assumption because their trajectories differ.

Abstract

from arXiv · show

Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation $Φ(τ)$, which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/

Introduction

LeapTalk addresses the latency-quality trade-off in talking-head generation with stable, open-ended real-time streaming, synthesizing each video chunk in one step while preserving synchronization, facial detail, and identity consistency. Its bridge-based design uses Brownian-bridge data-to-data transport with a persistent visual anchor to reduce error accumulation and identity drift.

  • Limitations: Existing methods trade generation quality against latency: diffusion approaches are computationally expensive and typically produce only 5–10s clips, while long-form StableAvatar synthesis is offline.Autoregressive methods improve efficiency through streaming but rely on previously generated content, creating error-accumulation concerns.
  • LeapTalk: LeapTalk enables stable, open-ended real-time talking-head streaming by synthesizing each video chunk in 1 step while preserving accurate lip synchronization, fine-grained facial details, and identity consistency.The framework is described as the first approach to provide this combination of open-ended streaming and one-step chunk synthesis.
  • Method: LeapTalk reformulates talking-head generation as data-to-data transport through a Brownian bridge and introduces Bridge Forcing with a persistent visual anchor I to mitigate long-term drift and error accumulation.The fixed identity reference grounds the generation trajectory while heterogeneous DMD supports training across differing generative paradigms.
  • Results: Up to 200 FPS on a single GPU is achieved with stable one-step inference, alongside high visual fidelity and long-term identity consistency.The reported contribution specifies the throughput and hardware condition for the real-time streaming framework.

Preliminaries

The preliminaries define probability-path models as continuous-time transport from a source distribution to a target distribution, including stochastic SDEs and deterministic Rectified Flow. Brownian bridges instead connect fixed data endpoints while retaining middle-path stochasticity, providing the basis for paired data-to-data modeling.

  • Probability-path models: Probability-path models transport a source distribution p0 to a target distribution p1 over continuous time.The common formulation is an SDE with velocity field v, diffusion strength σ(t), and Brownian motion Wt.
  • Rectified Flow: Rectified Flow is the deterministic limit of the SDE with σ(t) = 0, typically transporting Gaussian noise x0 ∼N(0, I) to data x1 ∼p1.Its corresponding target velocity is constant: ut = x1 −x0.
  • Brownian bridge: Brownian bridges connect fixed endpoints (x0, x1) while retaining stochasticity along the path and injecting uncertainty only in its middle.The bridge preserves both endpoints and is effective for paired data-to-data problems.
  • Bridge training and prior models: A neural network vθ(Xt, t) can be trained by velocity matching using t ∼U(0, 1) and Xt sampled from the bridge.Prior bridge-based models rely on multi-step sampling and offline inference, whereas LeapTalk uses heterogeneous distillation for stable one-step streaming generation with long-term consistency.

Methods

LeapTalk replaces noise-to-data autoregression with reference-anchored Brownian-bridge generation and trains a one-step student through SNR-aligned heterogeneous distillation. Motion-prefix stitching, audio-driven CFG, bridge losses, and autoregressive self-rollouts support identity preservation, temporal continuity, lip synchronization, and inference consistency.

  • Bridge Forcing: Bridge Forcing models talking-head generation as reference-to-frame data transport, replacing repeated noise-to-data reconstruction that causes long-sequence identity drift and visual degradation.The Brownian bridge remains anchored to the reference image throughout autoregressive generation.
  • Chunk Construction: Each chunk repeats the reference image for identity consistency and replaces its first K frames with the previous chunk’s last K frames to preserve motion continuity.The resulting prefix replacement anchors identity at every chunk while smoothly stitching consecutive chunks.
  • Heterogeneous Distillation: SNR-aligned time transformation t = Φ(τ) resolves mismatched noise levels between the flow-matching teacher and Brownian-bridge student, enabling consistent score supervision.The transformation aligns teacher and student SNRs before evaluating their scores in heterogeneous distillation.
  • Audio-driven CFG: Audio-driven classifier-free guidance replaces the teacher score during one-step distillation to compensate for suppressed high-frequency dynamics, degraded lip synchronization, and static motion.The unconditional branch uses zero audio input, while α controls guidance strength.
  • Bridge Training and Architecture: The student-side fake score network is trained along the residual bridge trajectory, while reconstruction and LPIPS losses, self-rollouts, and a shared Wan2.1-T2V-1.3B backbone connect training to inference.Self-Forcing sequentially generates chunks conditioned on previously generated history under the bridge formulation.

Experiments

Experiments show that LeapTalk preserves identity and lip motion during long-video streaming, while ablations demonstrate the necessity of its bridge formulation, SNR-aligned transformation, and audio-driven CFG. Hyperparameter studies identify α=1.6 and λperc=4.0 as effective defaults for balancing motion and reconstruction quality.

  • Qualitative Results: LeapTalk maintains stable identity and lip motion as long-video streaming progresses, unlike baselines that exhibit identity drift or lip-sync errors.The similarity curve remains high and stable, supporting long-duration and potentially open-ended generation.
  • Ablation Study: FID rises from 21 to 217 and Sync-C falls from 8.38 to 7.16 when the Brownian Bridge formulation is removed, with identity drift and artifacts appearing.This ablation attributes the identity-consistency improvement primarily to the Brownian Bridge.
  • Ablation Study: Removing SNR-aligned time transformation yields FID 378, while removing audio-driven CFG produces Sync-C 4.34 and Sync-D 10.21.These results indicate that noise alignment supports fidelity and audio-driven CFG preserves fine-grained mouth dynamics under one-step inference.
  • Hyperparameter Analysis: α=1.6 and λperc=4.0 are selected as defaults because stronger guidance or perceptual supervision eventually degrades motion or visual quality.Guidance around α=1.6 balances lip-sync and expressiveness, while λperc=4.0 precedes degradation observed at λperc=8.0.

User Study and Further Discussions

A 30-participant user study consistently preferred LeapTalk across identity consistency, visual quality, lip-sync accuracy, and overall preference. TAEHV substantially reduces computation and memory versus WanVAE while preserving comparable one-step quality, with slight fine-region blurriness mitigated by two-step inference.

  • User Study: LeapTalk was consistently preferred across identity consistency, visual quality, lip-sync accuracy, and overall preference in a 30-participant user study.Participants compared LeapTalk with another method and selected the best result across four criteria.
  • Effect of different Autoencoders: TAEHV greatly reduces computation and memory compared with Conv3D-based WanVAE while preserving comparable structure, identity, and motion quality under one-step inference.This supports a favorable efficiency–quality trade-off for real-time deployment.
  • Effect of different Autoencoders: Slight blurriness in fine regions such as lips is TAEHV’s main degradation and can be alleviated by increasing inference from 1 to 2 steps.The reported limitation concerns one-step inference and is improved by using two steps.

Conclusion

LeapTalk enables real-time, streaming, open-ended talking-head generation by using a Brownian bridge to stabilize long-term synthesis and a heterogeneous distillation scheme for high-quality one-step generation. It achieves competitive fidelity and lip synchronization at up to 200 FPS.

  • Framework: LeapTalk is a framework for real-time, streaming, open-ended talking-head generation.The framework targets open-ended synthesis in real-time streaming settings.
  • Stability: Brownian bridge formulation reduces error accumulation and stabilizes long-term synthesis.Generation is formulated as a Brownian bridge process to improve long-term stability.
  • Distillation: SNR-aligned time transformation and audio-driven CFG augmentation enable distillation across mismatched paradigms and high-quality one-step generation.These components are integrated into a heterogeneous DMD scheme for effective cross-paradigm distillation.
  • Results: 200 FPS is the reported upper speed at which LeapTalk achieves competitive fidelity and lip synchronization.The result combines high generation speed with competitive visual fidelity and lip synchronization.

Appendix · Related Works · Long Video Generation

Long-video generation remains difficult because most video models target only 5–10-second clips, while extending them without drift is challenging. Existing approaches address this through noise rescheduling, inference-time corruption during training, and architectural or training strategies.

  • Long Video Generation: Most video generation models are limited to short clips of 5–10 seconds.The passage cites Brooks et al. 2024, Zheng et al. 2024, Wan et al. 2025, and Gao et al. 2025.
  • Long Video Generation: Extending video generation models to longer durations without drift remains challenging.
  • Long Video Generation: Early long-video methods rely on training-free noise rescheduling.The passage attributes this line of work to Qiu et al. 2023 and Kim et al. 2024.
  • Long Video Generation: Later methods simulate inference-time corruption during training.The passage attributes this approach to Chen et al. 2024 and Ruhe et al. 2024.
  • Long Video Generation: Noise rescheduling represents an inference-stage strategy for addressing long-duration generation.
  • Long Video Generation: Inference-time corruption represents a training-stage strategy for addressing long-duration generation.

Audio-driven Talking Head Generation

Audio-driven talking-head methods achieve high visual fidelity but face a fundamental efficiency and stability challenge. Diffusion Transformers are latency-intensive and short-clip, while streaming approaches risk error accumulation, identity drift, and detail degradation.

  • Limitations of Existing Methods: DiT-based methods achieve remarkable visual fidelity but generate only short clips and incur high latency from multi-step denoising.The computational burden of multi-step denoising limits both clip length and efficiency.
  • Limitations of Existing Methods: StableAvatar enables infinite-length synthesis but operates entirely offline, preventing real-time streaming generation.Its infinite-length capability does not resolve the offline-generation constraint.
  • Limitations of Existing Methods: High-speed streaming methods suffer from error accumulation, identity drift, and detail degradation because flow matching starts from noise.The noise-to-data formulation creates instability over long sequences.

An Introduction to Brownian Bridge … Training Algorithm of heterogeneous DMD

The section formulates LeapTalk’s Brownian-bridge transport, derives an SNR-aligned teacher–student time transformation, and specifies heterogeneous DMD training. It also reports motion, streaming, resolution, and challenging-condition evaluations supporting stable, efficient generation.

  • An Introduction to Brownian Bridge: The Brownian bridge conditions a process from Xsrc to Xtgt, with conditional mean (1 −τ)Xsrc + τXtgt and variance τ(1 −τ)I.An equivalent time-reversed parameterization assigns τ = 1 to the source state and τ = 0 to the target state.
  • Transformation: SNR alignment enforces SNRTeacher(t) = SNRStudent(τ), yielding the transformation t = Φ(τ) for heterogeneous distillation.The mapping preserves τ →0 ⇒t →0, τ →1 ⇒t →1, and satisfies Φ(0.5) = 0.5, aligning equivalent noise levels.
  • CFG Ablation: Pose diversity increases with audio CFG scale, while Beat Align Score gradually decreases as CFG grows.The ablation uses Hopenet-based Yaw/Pitch/Roll Std and Average Std together with Beat Align Score to assess motion and alignment.
  • Streaming Chunk Size and Playback: 33 frames achieves the best Tgen/Tchunk ratio of 0.20, generating the next chunk within 20% of the current chunk’s playback time.Because streaming requires Tgen/Tchunk < 1, this setting leaves margin for continuous gap-free playback.
  • Resolution and Inference Speed:: Main experiments use 512 × 512 resolution, while Table 9 reports FPS and latency across resolutions on a single A100 GPU.The reported 200 fps result was measured on an H200 rather than an A100.
  • Additional Results in Challenging Scenarios: Bridge Forcing produces stable and coherent animations under non-human faces, artistic portraits, side views, low light, occlusions, and non-photorealistic objects.These qualitative tests cover substantial variation in geometry, texture, and illumination.
  • Training Algorithm of heterogeneous DMD: Heterogeneous DMD training generates a student chunk and teacher target, computes generator losses, then updates the generator and fake score estimation model.The generator minimizes distribution matching, bridge, and LPIPS losses, while the fake score model improves score estimation.
  • Training Algorithm of heterogeneous DMD: The algorithm uses 1-step student generation, multi-step teacher rollouts, SNR-aligned bridge sampling, audio-driven CFG, spatially weighted losses, and autoregressive history updates.It updates Gθ with LG = LDMD + λbrLBridge + λlpipsLLPIPS, updates Vfake with Lcritic, and detaches history chunks for subsequent generation.
Loading 2608.00079v1…