Source-linked AI summary
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
Yuxuan Zhang, Haozhong Xiong, Yubo Huang, Jiayi Song, Jinpeng Yu, Haofan Wang, Jiaming Liu, Ruihua Huang, Liwei Wang
TL;DR
Real-time, stable long-form human animation remains difficult because prior diffusion systems are offline or degrade over extended streams. LiveAnimate combines causal training, three-step distillation, and bounded pose-retrieval attention, sustaining nearly constant perceptual quality and identity at 19.63 FPS on two H100 GPUs over three minutes.
Problem
Prior full-body diffusion systems do not simultaneously support streaming generation, real-time inference, and stable long-form output for responsive interaction.
Method
LiveAnimate converts a bidirectional video DiT into a three-step block-causal generator and uses Pose-Retrieval Sink Attention to preserve appearance with bounded historical context.
Results
19.63 FPS on two H100 GPUs accompanies nearly constant perceptual quality and identity across a three-minute rollout.
Takeaways & Limitations
LiveAnimate establishes a supported operating point for interactive full-body animation combining real-time generation, long-form duration, and stable quality.
Abstract
from arXiv · showhide
Pose-driven human animation synthesizes a video of a target person from a single reference image and a driving pose stream. Real-time generation is essential for interactive applications such as live streaming, telepresence, and virtual avatars, yet diffusion-based systems require minutes to hours per clip, precluding responsive interaction. We present LiveAnimate, to our knowledge the first animation system to combine real-time streaming with stable long-form generation at billion scale, built on a 14B-parameter video Diffusion Transformer (DiT). A two-stage training pipeline first adapts a pretrained bidirectional DiT into a block-causal autoregressive generator through Reference-Anchored Teacher-Forcing Adaptation, and then reduces the sampling budget to three steps through Block-wise Self-Forcing Distillation. To preserve appearance over extended streams, we introduce Pose-Retrieval Sink Attention (PR-Sink), a bounded KV-cache mechanism combining a Static Sink that permanently anchors the first generated block, a Dynamic Sink that holds a pose-retrieved historical block, and a three-slot Rolling Window. When a pose recurs, PR-Sink restores the relevant appearance context without retaining the entire sequence, so memory and per-block latency remain constant regardless of stream duration. Together with Ulysses sequence parallelism and operator fusion, these designs enable 19.63\,FPS streaming inference on two NVIDIA H100 GPUs. On a three-minute benchmark, LiveAnimate maintains nearly constant perceptual quality and identity from the first 30 seconds to the final minute, while prior systems degrade substantially or require hours of offline computation for the same rollout. These results establish a new operating point in quality, latency, and duration for interactive full-body animation.
1. Introduction
LiveAnimate addresses the lack of responsive, stable long-form pose-driven human animation by combining real-time streaming with a 14B-parameter causal DiT. Its two-stage training pipeline converts a pretrained bidirectional DiT into a block-causal generator and reduces the sampling budget to three steps, achieving ∼20 FPS on 2×H100 GPUs.
- Motivation: Pose-driven human animation must preserve appearance fidelity, pose accuracy, and temporal coherence over extended sequences while conditioning on a single reference image.The task supports applications including virtual try-on, digital human creation, live streaming, and telepresence.
- Problem: Prior methods are offline, requiring minutes to hours per clip, and no prior full-body diffusion system simultaneously supports streaming generation, real-time inference, and stable long-form output.Most methods generate fixed-length clips without processing open-ended pose streams; EverAnimate requires 20 denoising steps per chunk.
- Contribution: LiveAnimate is presented as the first system to enable real-time streaming human animation with stable long-form generation at billion scale using a 14B-parameter causal DiT.The system is designed to address the gap between interactive streaming and stable long-form generation.
- Training pipeline: Reference-Anchored Teacher-Forcing Adaptation converts a pretrained bidirectional DiT into a block-causal generator by conditioning each training block on ground-truth clean history.A global Ref Sink exposes the reference-image latent to all generated blocks and differs from generated-history sinks used during inference.
- Training pipeline: Three steps: Block-wise Self-Forcing Distillation reduces the sampling budget through self-forcing rollout and one-block-at-a-time replay with Block-wise DMD Optimization.The method builds on Self Forcing and performs the rollout without gradient tracking.
- Results: ∼20 FPS on 2×H100 GPUs: LiveAnimate achieves real-time streaming human animation with stable long-form generation at billion scale.This is stated as a summary contribution of the system.
2. Related Work
Prior work has advanced video diffusion from latent architectures to billion-scale DiTs and developed diffusion-based pose-driven human animation. Autoregressive video methods mitigate long-horizon rollout drift, but the cited approaches do not address pose-aware streaming.
- Video diffusion backbones: Video diffusion progressed from pixel-space U-Nets to latent-space models and billion-scale Diffusion Transformers, including Wan’s 14B-parameter backbone.Latent-space video models use compressed VAE representations for more manageable high-resolution generation.
- Video diffusion backbones: Wan2.2-Animate-14B provides the foundation for adapting offline clip generation to causal, real-time streaming.The paper specifically builds on this 14B-parameter animation model.
- Pose-driven human animation: Pose-driven animation evolved from real-time GAN warping methods to diffusion systems using reference appearance, pose conditioning, and temporal attention.GAN-based methods can produce artifacts under large pose changes; later systems include Animate Anyone, MagicAnimate, MusePose, and Champ.
- Long-horizon video generation: Autoregressive chunk-wise generation extends video beyond the training horizon, but repeated rollouts can accumulate visual degradation and semantic drift.Diffusion Forcing, CausVid, Causal Forcing, Rolling Forcing, and Context Forcing address aspects of autoregressive training or long-video consistency.
- Long-horizon video generation: Existing long-video methods mitigate rollout drift through error recycling, long-context conditioning, or refreshed attention sinks, but they do not address pose-aware streaming.The cited methods include Stable Video Infinity and Rolling Sink.
3. Method
LiveAnimate converts a bidirectional video DiT into a causal streaming generator through two-stage training, then maintains long-horizon context with bounded pose-retrieval attention. Clean KV updates, fixed-capacity memory, and position-consistent reuse support constant-cost autoregressive generation.
- Reference-Anchored Teacher-Forcing Adaptation: Reference-Anchored Teacher-Forcing Adaptation converts a bidirectional DiT into a block-causal autoregressive generator using clean ground-truth history during Stage 1.A global Ref Sink stores clean reference-image KV states, preserving identity and appearance throughout generation.
- Block-wise Self-Forcing Distillation: 3 steps replace the original 50 denoising steps per block through block-wise self-forcing distribution matching distillation.A two-pass block-wise replay trains each block under student-induced history while retaining backward activations only for the currently replayed block.
- Pose-Retrieval Sink Attention: PR-Sink combines a permanent first-generated-block Static Sink, a pose-retrieved historical Dynamic Sink, and a three-slot Rolling Window.Reference-image tokens remain in a separate global Ref Sink, while Clean KV Update writes denoised historical states at the clean timestep.
- Pose-Retrieval Sink Attention: A 558-dimensional fingerprint represents temporal pose configuration for retrieval, with zero padding on failed detections and confidence retained otherwise.The pose bank favors diverse representatives during its first 20 blocks, then reuses selected contexts; retrieval excludes block b −1 because it is already in the Rolling Window.
- Pose-Retrieval Sink Attention: Position-consistent KV reuse caches Kraw before RoPE with V, allowing retrieved historical blocks to be reused despite different temporal positions.PR-Sink is the union of the pose-conditioned Dynamic Sink and permanent first-block Static Sink, not either component alone.
4. Experiments
Experiments evaluate LiveAnimate on a three-minute benchmark using perceptual, identity, distributional, temporal, latency, and throughput measures. LiveAnimate achieves strong short-horizon quality and stable long-horizon generation, while qualitative comparisons and ablations expose the benefits of its sinks and self-forcing distillation.
- Evaluation protocol: The three-minute benchmark contains 24 reference-image and driving-video pairs spanning in-the-wild and controlled long-horizon full-body motion.The in-the-wild portion covers diverse motions, appearances, and backgrounds; the controlled portion derives sequences from X-Dance.
- Short-horizon quality: LiveAnimate achieves the best initial 0–30 s ASE (2.823) and IQA (4.047), showing strong perceptual quality under three-step inference.The evaluation also measures DINO-S, FID, and V-MAE, with higher-is-better ASE, IQA, and DINO-S and lower-is-better FID and V-MAE.
- Long-horizon stability: LiveAnimate’s IQA remains essentially unchanged as the rollout grows, whereas prior systems degrade substantially or require hours of offline computation for three-minute sequences.SCAIL and UniAnimate-DiT achieve higher DINO-S but exhibit inter-frame flickering, while EverAnimate is not real-time.
- Qualitative comparison: LiveAnimate remains stable under large pose changes, while One-to-All collapses, UniAnimate-DiT and SCAIL flicker, and Wan-Animate and EverAnimate develop color or background drift.For subtler upper-body motion, EverAnimate and LiveAnimate remain stable and consistent, but only LiveAnimate operates in real time.
- Qualitative ablation: Removing the Static Sink causes major subject and background changes, while removing the Dynamic Sink introduces localized appearance errors when poses recur.Pose-matched retrieval alone cannot provide a time-invariant identity anchor.
- Quantitative ablation: Teacher forcing lowers IQA from 3.849 to 3.438 and DINO-S from 0.780 to 0.679, while two sampling steps reduce final DINO-S to 0.783.Four steps produce the strongest frame-level scores but require an additional denoising pass; PR-Sink retrieval and bank maintenance add only 0.542%.
5. Conclusion
LiveAnimate combines real-time streaming with stable long-form pose-driven human animation. Its two-stage training and bounded-cache attention enable efficient generation while preserving long-range appearance context.
- LiveAnimate combines real-time generation with stable long-form quality for streaming pose-driven human animation.
- A two-stage pipeline converts a pretrained bidirectional DiT into a block-causal generator and distills sampling to three steps.The pipeline is trainable on a single 8×80GB GPU node.
- Pose-Retrieval Sink Attention maintains long-range appearance context using a bounded cache.
Supplementary Material · 6. Additional Qualitative Results
LiveAnimate’s supplementary qualitative results evaluate seven additional three-minute streams across varied identities, scenes, framing, and motion. The examples show recognizable subjects with stable clothing, backgrounds, and appearance throughout long rollouts while following substantial pose changes.
- 6. Additional Qualitative Results: Seven additional three-minute examples extend the qualitative evaluation across diverse identities, clothing, backgrounds, camera framing, and motion patterns.Generated frames are shown with the reference image and driving pose at 20-second intervals from 20 to 180 seconds.
- 6. Additional Qualitative Results: Despite large changes in limb configuration and body orientation, generated subjects remain recognizable across the sampled timestamps.The first two Figure 8 examples emphasize full-body changes, while the last two focus on upper-body motion, facial variation, and gestures near the face.
- 6. Additional Qualitative Results: Clothing and backgrounds do not exhibit progressive changes across the sampled timestamps despite varied motion patterns and scene conditions.This qualitative observation covers both full-body and upper-body examples.
- 6. Additional Qualitative Results: Figure 9 tests appearance preservation on three three-minute sequences covering dim indoor, bright outdoor, and close-up portrait settings.The sequences include indoor full-body motion, outdoor dancing, and frequent hand gestures, with outputs sampled at 20-second intervals.
- 6. Additional Qualitative Results: Figure 8 evaluates four three-minute sequences spanning full-body motion with large pose changes and upper-body motion involving facial expressions and hand gestures.The figure samples driving pose signals and outputs every 20 seconds from 20 to 180 seconds.
- 6. Additional Qualitative Results: Across Figure 9’s rollout, LiveAnimate maintains the subject’s appearance and background while responding to substantial changes in body configuration.The figure includes indoor full-body motion, an outdoor dance sequence, and an upper-body subject performing frequent hand gestures.