PaperlayerView exact revision on arXiv

Source-linked AI summary

Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

Yubo Huang, Hailong Guo, Fangtai Wu, Weiqiang Wang, Shifeng Zhang, Shijie Huang, Qijun Gan, Lin Liu, Sirui Zhao, Enhong Chen, Jiaming Liu, Steven Hoi

arXiv:2512.04677v6cs.CV

TL;DR

Real-time streaming and infinite-length avatar generation remain difficult for large diffusion models because of sequential denoising and long-horizon drift. Live Avatar combines causal few-step distillation, long-horizon strategies, and timestep-forcing pipeline parallelism; it reports 45 FPS with a TTFF of 1.21 s at 14B scale. Real-time refers to throughput exceeding playback speed, not conversational latency.

  • Problem

    Large diffusion models face a speed–fidelity trade-off and long-horizon consistency challenges in real-time, streaming avatar generation over unbounded durations.

    Source: §1 · pp. 1–2

  • Method

    Live Avatar uses two-stage causal few-step distillation, long-horizon strategies, and timestep-forcing pipeline parallelism across GPUs.

    Source: §1 · p. 3, §4.2 · p. 7, §4.3 · p. 8

  • Results

    45 FPS with a TTFF of 1.21 s on 5 H100 GPUs, with stable generation beyond 10,000 seconds at 14B scale.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Live Avatar is presented as the first practical real-time streaming of a 14B diffusion model for infinite-length audio-driven avatar generation.

    Source: §6 · p. 14

    “Real-time” denotes generation throughput exceeding playback speed, not end-to-end conversational latency.

    Source: §1 · p. 2

Abstract

from arXiv · show

Audio-driven avatar interaction demands real-time, streaming, and infinite-length generation -- capabilities fundamentally at odds with the sequential denoising and long-horizon drift of current diffusion models. We present Live Avatar, an algorithm-system co-designed framework that addresses both challenges for a 14-billion-parameter diffusion model. On the algorithm side, a two-stage pipeline distills a pretrained bidirectional model into a causal, few-step streaming one, while a set of complementary long-horizon strategies eliminate identity drift and visual artifacts, enabling stable autoregressive generation exceeding 10000 seconds. On the system side, Timestep-forcing Pipeline Parallelism (TPP) assigns each GPU a fixed denoising timestep, converting the sequential diffusion chain into an asynchronous spatial pipeline that simultaneously boosts throughput and improves temporal consistency. Live Avatar achieves 45 FPS with a TTFF of 1.21\,s on 5 H800 GPUs, and to our knowledge is the first to enable practical real-time streaming of a 14B diffusion model for infinite-length avatar generation. We further introduce GenBench, a standardized long-form benchmark, to facilitate reproducible evaluation. Our project page is at https://liveavatar.github.io/.

1 Introduction

Real-time streaming of large diffusion models over unbounded durations faces both long-horizon consistency and speed–fidelity challenges. Live Avatar combines causal few-step distillation, long-horizon strategies, and pipeline parallelism to address them.

1 Introduction

  • Long-horizon generation is challenged by compounding errors that cause identity drift and color artifacts, while prior approaches remain limited in duration or costly at 14B scale.Self-Forcing still degrades under minute-level rollout; LongLive is designed for text-to-video and is too costly to scale to 14B models.Source: §1 · p. 2
  • Noisy KV-cache history, a persistent sink frame, and additional strategies support stable generation beyond 10,000 seconds while addressing accumulated artifacts and distribution drift.Training uses approximately 3-second clips, while stable extrapolation extends beyond 10,000 seconds.Source: §1 · pp. 2–3
  • A two-stage framework adapts a pretrained bidirectional diffusion model into a causal, few-step streaming model, using motion-frame scaffolding to bridge stages and speed distillation 5×.The framework addresses the challenge of streaming with a large diffusion model without compromising visual fidelity.Source: §1 · pp. 2–3
  • 45 FPS with a TTFF of 1.21 s on 5×H100 GPUs; Live Avatar combines streaming, real-time, and infinite-length generation with a 14B diffusion model.The paper presents this as practical real-time streaming at 14B scale.Source: §1 · pp. 2–3, Table 1 · p. 2
  • Timestep-forcing Pipeline Parallelism assigns each GPU a fixed denoising timestep, breaking the sequential sampling bottleneck; real-time here means throughput exceeding playback speed.The reported real-time definition does not refer to end-to-end conversational latency.Source: §1 · pp. 2–3

2 Related Work

Prior work addresses streaming and long-video generation, audio-driven avatar synthesis, and diffusion distillation, but the cited approaches do not achieve real-time, streaming, infinite-length avatar generation with large-scale diffusion models.

Streaming and Long Video Generation

  • Streaming and long-video methods improve quality or temporal consistency, but none achieve real-time streaming long-video generation with large-scale diffusion models.Self-Forcing [18] still suffers exposure bias and fails at minute-level video; LongLive [50] is text-to-video focused and inefficient to scale to 14B models.Source: §2 · pp. 3–4

Audio-driven Avatar Video Generation

  • Avatar-generation work spans GANs and 3D motion models, diffusion and ReferenceNet methods [54], DiT-based methods [8] [15] [23] [26] [31], and autoregressive approaches [1] [4] [48].The cited avatar approaches do not achieve real-time, streaming, infinite-length generation with large-scale diffusion models.Source: §2 · pp. 3–4

Diffusion Distillation

  • Diffusion distillation includes trajectory distillation [11], Consistency Models [27] [57], and DMD [30]; CausVid, Self-Forcing [7] [18], and LongLive [50] use DMD-based few-step distillation for streaming video.The passage notes that recent analysis suggests DMD functions analogously to reinforcement learning, with the pretrained diffusion model serving as a reward signal [28].Source: §2 · p. 4

3 Preliminaries

Video diffusion constructs noisy latents by interpolating clean video latents with Gaussian noise and trains a model to predict velocity. Distribution Matching Distillation (DMD) distills a pretrained teacher into a few-step student by matching their timestep-wise distributions.

3.1 Video Diffusion Models

  • Flow matching interpolates clean latents and Gaussian-noise latents according to a schedule, and trains the model to predict the target velocity.Video is encoded into temporal latents using a causal 3D VAE, while conditioning embeddings are supplied to the diffusion backbone.Source: §3.1 · p. 4

3.2 Distribution Matching Distillation

  • DMD minimizes the reverse KL divergence between the few-step student distribution and the teacher-produced ground-truth distribution at each timestep.The student generator and fake-score model are alternately updated; multi-step distillation samples an intermediate state from the student's trajectory as the starting point.Source: §3.2 · pp. 4–5
  • Figure 1 contrasts diffusion-forcing pretraining with motion-frame scaffolding and self-forcing distillation that replaces those frames with a rolling KV cache.In Stage 2, noise-level consistency is enforced between the cache and noisy latents.Source: Figure 1 · p. 5

4 The Live Avatar Framework

Live Avatar combines causal block-wise generation and two-stage distillation with long-horizon stabilization strategies and a multi-GPU denoising pipeline. Its framework uses a persistent sink for appearance, noisy history for motion context, and timestep-assigned GPUs for streaming inference.

4.1 Model Architecture

  • Autoregressive generation synthesizes consecutive noisy-frame blocks conditioned on cached prior blocks, a sink frame, audio, and prompt embeddings.The noisy block and attended KV cache share a noise level, a design the paper identifies as important for generation quality and inference speed.Source: §4.1 · p. 6

4.2 Model Training

  • Stage 1 uses noisy motion frames as scaffolding; Stage 2 replaces them with a rolling cache in causal few-step self-forcing distillation, yielding a 5× convergence speedup.The distilled student omits the clean-cache refresh pass, so its KV cache contains noisy representations.Source: §4.2 · pp. 6–7

4 The Live Avatar Framework

  • The framework addresses long-run degradation associated with conditioning drift, distribution drift, and accumulated errors in rolling-cache generation.The section introduces the model architecture, training framework, long-video strategies, and TPP as the components of Live Avatar.Source: §4 · p. 6, §4.3 · p. 7

4.3 Long Video Generation

  • Noisy-only KV history suppresses inherited artifacts while retaining motion cues, and its compact latent distribution mitigates out-of-distribution drift.These strategies target error accumulation and distribution drift among the long-horizon failure modes described for autoregressive generation.Source: §4.3 · pp. 7–8
  • Adaptive Attention Sink replaces the real reference image with the first generated latent, while Rolling RoPE adjusts the sink frame's relative position to the current noisy states.The paper also reports that matching noise levels between the noisy block and attended KV cache reduces inter-frame flickering.Source: §4.3 · p. 8

4.4 Timestep-forcing Pipeline Parallelism

  • TPP assigns each GPU a denoising timestep, transforming sequential sampling into an asynchronous pipeline whose throughput depends on one denoising forward.After warm-up, GPUs process blocks simultaneously, reuse local KV caches, and pass only latents between stages; the VAE decoder runs on a dedicated GPU. Conventional sequential parallelization reaches 6 FPS on four GPUs, compared with 5 FPS on one GPU.Source: §4.4 · pp. 8–9, Figure 2 · p. 9

5 Experiments

Experiments evaluate Live Avatar on short- and long-form GenBench data, comparing it with existing methods and testing inference efficiency and long-horizon stability components. Results show stable long-video quality, faster inference, and distinct degradation patterns when stability components are removed.

5.1 Experimental Settings

  • Evaluation trains on 400K AVSpeech clips and tests OOD performance on GenBench’s 100 short and 15 over-five-minute diverse samples, using perceptual, synchronization, and identity metrics.Experiments use 720×400 resolution and 84 frames; reported metrics reflect the optimized inference configuration.Source: §5.1 · p. 10

5.2 Comparison with Existing Methods

  • On GenBench-ShortVideo, Live Avatar achieves the best IQA, competitive ASE, comparable synchronization, and 45.2 FPS versus Ditto’s 21.8 FPS.The paper reports roughly 70× Ditto’s parameters for Live Avatar.Source: §5.2 · p. 11
  • On GenBench-LongVideo, Live Avatar retains ASE 3.44→3.42, IQA 4.51→4.76, and Dino-S 0.96→0.97, with the best overall visual and identity quality as most baselines degrade.Figure 3 shows stable fidelity over time for Live Avatar, while most baselines, especially OmniAvatar and WanS2V, visibly degrade.Source: §5.2 · pp. 11–12, Table 2 · p. 11, Figure 3 · p. 12

5.3 Ablation Study

  • TPP more than doubles throughput, and VAE parallelization plus kernel optimizations bring the full system to 45.2 FPS and 1.21 s TTFF on 5 H100 GPUs.DMD reduces NFE from 80 to 5; TPP removes the clean-cache pass, reducing NFE from 5 to 4.Source: §5.3 · p. 12, Table 3 · p. 12
  • Long-horizon ablations show that removing History Corrupt, AAS, or Rolling RoPE is associated with quality loss, color desaturation, or identity drift.Without History Corrupt, ASE/IQA fall to 2.90/3.88 and Dino-S to 0.81; without AAS, ASE is 3.13, and without Rolling RoPE, Dino-S changes from 0.97 to 0.86.Source: §5.3 · pp. 12–13, Figure 4 · p. 13
  • Increasing denoising steps from 2 to 4 improves Sync-C from 6.41 to 7.03 while FPS remains 45.2; TTFF rises from 0.68 s to 1.21 s.The authors use 4 steps by default.Source: §5.3 · p. 13, Table 5 · p. 13
  • On GenBench-Long, only timestep-forcing avoids severe flickering among the tested KV-cache schedules, with T.Flicker 0.971, and it is the only schedule compatible with TPP.Clean-KV degrades over long horizons; fixed noise improves quality but has T.Flicker 0.876/0.891.Source: §5.3 · p. 13, Table 6 · p. 14
  • With other hyperparameters aligned, motion-frame scaffolding reaches Stage 2 convergence in about 500 steps versus about 2,500 without it, a 5× speedup with no final-quality loss.Source: §5.3 · p. 14

6 Conclusion

Live Avatar combines causal few-step streaming, long-horizon stability strategies, and pipeline parallelism to support real-time generation with a 14-billion-parameter diffusion model. The supplementary material outlines additional evaluations, implementation details, optimizations, and ethical considerations.

6 Conclusion

  • Live Avatar is presented as the first practical real-time streaming system for infinite-length avatar generation with a 14-billion-parameter diffusion model.Its two-stage training adapts a pretrained bidirectional model into a causal, few-step model; TPP reaches 45 FPS with a 1.21 s TTFF on 5 H100 GPUs, while three strategies support generation beyond 10,000 seconds.Source: §6 · p. 14
  • The long-horizon strategies rely on avatar interactions’ largely static scene, and roughly 3 s end-to-end latency does not yet meet seamless bidirectional interaction requirements.The authors note that native long-duration training could make the static-scene prior unnecessary.Source: §6 · p. 14

A Overview of Supplementary Material

  • Supplementary experiments cover 10,000-second extrapolation, in-domain AV-Speech comparisons, a user study, and block-size ablations.The document also details hardware and runtime metrics.Source: A Overview of Supplementary Material · p. 19
  • The supplementary material describes inference optimizations, method pseudocode and implementation procedures, additional visual results, and ethical considerations.Listed optimizations include FP8 quantization, fused attention, streaming VAE decoding, and model offloading.Source: A Overview of Supplementary Material · p. 19

C.1 Extending Autoregressive Generation to 10,000 Seconds

The model is stress-tested autoregressively to 10,000 seconds despite training only on 5-second clips. Across sampled segments, the reported metrics remain nearly unchanged and the authors report no observable quality decay or identity instability.

C.1 Extending Autoregressive Generation to 10,000 Seconds

  • The 10,000-second rollout extends far beyond training: the model saw only 5-second clips and short-range RoPE shifts during training.At inference, 10,000 seconds corresponds to RoPE positions around 40k, about two orders of magnitude beyond training positions; GenBench-LongVideo audio samples are looped to fill the duration.Source: §C.1 · pp. 19–20
  • The fully autoregressive, block-wise rollout uses accumulated KV caches and History Corrupt, Adaptive Attention Sink, and Rolling RoPE throughout.The procedure follows Self-Forcing [18] and loops each approximately seven-minute audio sample to maintain continuous conditioning.Source: §C.1 · p. 20
  • At 10,000 seconds, perceptual quality, audio–visual synchronization, and semantic consistency remain nearly unchanged, with no observable quality decay or identity instability.The report compares segments at 0–10, 100–110, 1000–1010, and 10000–10010 seconds; Figure 6 shows a representative sequence.Source: §C.1 · p. 20, Table 7 · p. 20

C.2 Additional Comparison with Existing Methods

The additional evaluations report competitive or superior in-domain AV-Speech performance and compare user ratings across perceptual dimensions. The user study favors Live Avatar on synchronization and consistency, while WanS2V leads on naturalness.

C.2 Additional Comparison with Existing Methods

  • Live Avatar performs competitively or better on most metrics in the in-domain AV-Speech test, including against the strong AV-Speech-trained OmniAvatar baseline.The evaluation uses 50 unseen clips sampled from 5% of held-out training videos and adds FID and FVD because ground-truth videos are available.Source: §C.2 · pp. 20–21, Table 8 · p. 22

C.3 User Study

  • The double-blind study rates Naturalness, Synchronization, and Consistency; Live Avatar has the highest Synchronization and Consistency scores, while WanS2V leads on Naturalness.Synchronization is rated as holistic audiovisual coherence rather than frame-level lip alignment.Source: §C.3 · pp. 21–22, Table 9 · p. 22
  • The authors attribute slightly lower perceived naturalness for Live Avatar to accelerated motion tempo, while reporting that synchronization and identity consistency are not compromised.The reported ratings are averaged across 20 participants and normalized to a 0–100 scale.Source: §C.3 · p. 22
  • Qualitative examples show OmniAvatar’s exaggerated motion compromising identity preservation and EchoMimic V2’s fixed posture distorting non-standard reference poses.The paper links OmniAvatar’s behavior to lower Naturalness ratings and describes EchoMimic V2 as ignoring the reference body pose.Source: §C.3 · p. 22, Figure 7 · p. 23

C.4 Effect of Block Size

The block-size ablation finds that smaller blocks reduce synchronization and motion dynamics, while block size 3 offers a quality–latency trade-off against size 4.

C.4 Effect of Block Size

  • The ablation compares latent frames per block on GenBench-ShortVideo, with every variant trained end-to-end using identical hyperparameters.Source: Table 10 · p. 23
  • Reducing block size to 1 or 2 drops Sync-C from 7.03 to 4.12 or 5.38, while ASE and IQA decline more mildly and Dino-S stays essentially unchanged.The smaller blocks also produce near-static expressions and minimal gesture variation.Source: §C.4 · p. 23
  • Block size 3 is the default: size 4 is comparable in performance but has a higher TTFF of 1.21 s versus 0.94 s.Size 4 slightly leads on IQA and Sync-D, while size 3 is marginally better on ASE and Sync-C.Source: Table 10 · p. 23, §C.4 · p. 24

D Additional Evaluation Details

The evaluation uses common hardware and resolution settings where supported, with specified exceptions for methods lacking parallel inference or flexible resolutions. Runtime reporting distinguishes end-to-end throughput from first-frame latency and evaluates FPS on long sequences.

Inference Configuration

  • Methods use identical single-H100-node conditions where possible, with single-GPU inference for EchoMimic V2 and Ditto and a 512 × 512 crop for Hallo3.Other models use 720 × 400 resolution; EchoMimic V2 is excluded from long-video comparisons because its per-frame skeleton templates prevent long-duration inference.Source: D Additional Evaluation Details · p. 24

Runtime Metrics

  • FPS includes denoising, VAE decoding, and CPU-side processing, while TTFF adds random arrival latency to first-frame denoising and decoding.Random arrival latency depends on output frame rate, so TTFF is coupled with FPS.Source: D Additional Evaluation Details · p. 24
  • FPS is measured on GenBench-LongVideo, where long-sequence testing provides more stable runtime estimates.TTFF spans audio-signal arrival through visual output.Source: D Additional Evaluation Details · p. 24

E Kernel-Level Optimizations

Kernel-level optimizations improve streaming throughput, while the algorithm details specify training and inference mechanisms for long-horizon generation. These include noisy-cache training, an adaptive identity anchor, rolling positional indices, and multi-GPU execution.

E Kernel-Level Optimizations

  • The optimization suite combines tiered attention kernels, ahead-of-time graph compilation, and merged LoRA weights to reduce inference overhead.FlashAttention-3 is primary on Hopper GPUs, with FlashAttention-2 as fallback; graph compilation enables operator fusion and kernel auto-tuning.Source: E Kernel-Level Optimizations · pp. 24–25
  • Disabling FP8 changes throughput from 45.2 to 36.0 FPS, with only marginal quality differences; FP8 reduces per-GPU VRAM from ∼80 GB to under 48 GB.IQA improves slightly without FP8 while other metrics remain virtually unchanged; Table 11 compares FP8 and FP16 quality and throughput.Source: E Kernel-Level Optimizations · pp. 24–25, Table 11 · p. 25
  • Causal VAE decoding caches the latest 2 frames per convolution layer, while a dedicated GPU can decode blocks in parallel with DiT computation.CPU offloading frees VRAM for the DiT denoising loop in single-GPU deployment, and KV caches can optionally be offloaded between clips.Source: E Kernel-Level Optimizations · pp. 25–26
  • ∼2.5× peak and ∼3× average FPS improvement over the unoptimized baseline yields stable 45+ FPS on 5×H100 and 18 FPS with 3.12 s TTFF on 5×H20.The 5×H100 result uses 4-step sampling.Source: E Kernel-Level Optimizations · p. 26

F Algorithm Details

  • Two-stage training moves from block-wise noise scheduling and causal attention to self-forcing with rolling KV states, called History Corrupt.Block-wise gradient accumulation partitions the backward graph and accumulates gradients across steps to reduce peak memory while preserving training behavior.Source: F Algorithm Details · p. 26
  • AAS replaces the reference-image sink with the first generated latent, keeping the persistent identity anchor on the model’s generation manifold without extra VAE encoding or decoding.The updated sink is used for subsequent blocks.Source: F Algorithm Details · p. 26
  • Rolling RoPE updates the sink frame’s positional index to stay slightly ahead of the current noisy block, keeping relative distance within the training-time offset range.The update addresses growing sink-to-block distance during rollout, which can degrade attention to the sink frame.Source: F Algorithm Details · pp. 26–28, Figure 9 · p. 28
  • TPP assigns each GPU a denoising timestep, and after warmup most GPU time is spent on DiT computation with only sporadic idle gaps.Figure 8 contrasts baseline, AAS, timestep-forcing, and the combined configuration; Figure 10 visualizes per-GPU computation and waiting periods.Source: Figure 8 · p. 27, F Algorithm Details · p. 28, Figure 10 · p. 29

G Additional Visual Results

Additional examples show long-duration avatar generation for three subjects, with stable identity and coherent appearance across the reported time points. The authors also identify impersonation risk and recommend responsible deployment safeguards.

G Additional Visual Results

  • Across three subjects, generated frames remain visually aligned with the reference image at 10 s, 100 s, 1000 s, and 10000 s without temporal drift or degradation.The examples also show consistent lip movements and coherent visual appearance beyond the training horizon.Source: G Additional Visual Results · pp. 33–34
  • The long-duration examples follow audio-driven motion patterns while maintaining stable identity and visual coherence.The reported frames span 10 s through 10000 s.Source: G Additional Visual Results · p. 34

H Ethical Consideration

  • The authors identify impersonation risk for high-fidelity avatars and recommend access control and watermarking for responsible deployment.Training and evaluation identity data is reported as collected with permission.Source: H Ethical Consideration · p. 34

Related papers

Loading 2512.04677v6…