PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaLive! Expressive Portrait Image Animation for Live Streaming

Zhiyuan Li, Chi-Man Pun, Chen Fang, Jue Wang, Xiaodong Cun

arXiv:2512.11253v1cs.CV

TL;DR

Diffusion-based portrait animation often prioritizes visual quality over inference efficiency, while chunk-wise generation can add latency or accumulate errors over long videos. PersonaLive combines hybrid motion control, appearance distillation, and autoregressive micro-chunk streaming with sliding training and historical keyframes. The authors report state-of-the-art performance and up to 7–22× speedup over prior diffusion-based portrait animation models.

  • Problem

    Diffusion-based portrait animation methods focus on visual quality and motion consistency while overlooking inference efficiency, and chunk-wise approaches can add latency or accumulate errors in long videos.

    Source: §1 · p. 2

  • Method

    PersonaLive combines implicit facial representations and 3D implicit keypoints, fewer-step appearance distillation, and autoregressive micro-chunk streaming with sliding training and historical keyframes.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    PersonaLive achieves state-of-the-art performance with up to 7–22× speedup over prior diffusion-based portrait animation models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that PersonaLive supports low-latency real-time streaming and stable long-term generation with enhanced temporal coherence.

    Source: §3 · p. 4, §3.3 · p. 5, §5 · p. 8

    The model is trained primarily on human facial data and struggles to generalize to non-human portraits such as cartoons or animals.

    Source: §5 · p. 9

Abstract

from arXiv · show

Current diffusion-based portrait animation models predominantly focus on enhancing visual quality and expression realism, while overlooking generation latency and real-time performance, which restricts their application range in the live streaming scenario. We propose PersonaLive, a novel diffusion-based framework towards streaming real-time portrait animation with multi-stage training recipes. Specifically, we first adopt hybrid implicit signals, namely implicit facial representations and 3D implicit keypoints, to achieve expressive image-level motion control. Then, a fewer-step appearance distillation strategy is proposed to eliminate appearance redundancy in the denoising process, greatly improving inference efficiency. Finally, we introduce an autoregressive micro-chunk streaming generation paradigm equipped with a sliding training strategy and a historical keyframe mechanism to enable low-latency and stable long-term video generation. Extensive experiments demonstrate that PersonaLive achieves state-of-the-art performance with up to 7-22x speedup over prior diffusion-based portrait animation models.

1. Introduction

The introduction identifies efficiency and long-term streaming as obstacles to applying diffusion-based portrait animation in live settings. PersonaLive combines hybrid motion control, fewer-step appearance distillation, and autoregressive micro-chunk generation to address them.

1. Introduction

  • Current diffusion methods require costly denoising, while chunk-wise approaches either add latency through overlapping frames or risk error accumulation during long videos.Source: §1 · p. 2
  • Up to 7–22× speedup over prior diffusion-based portrait animation models accompanies PersonaLive’s state-of-the-art performance.Source: §1 · p. 2
  • Implicit facial representations and 3D implicit keypoints jointly control facial dynamics and head movement, with 3D keypoints offering more flexible head-motion control than 2D landmarks or motion frames.Source: §1 · p. 2
  • Appearance distillation removes redundant refinement after structure and motion are established, improving inference efficiency without compromising visual quality.Source: §1 · p. 2
  • Autoregressive micro-chunk generation uses sliding training and historical keyframes to mitigate exposure bias and error accumulation for stable long-term streaming.Source: §1 · p. 2

2. Related Work

Related work covers diffusion-based portrait animation, long-term video generation, and diffusion acceleration. The authors position PersonaLive as addressing efficiency and streaming gaps across these areas.

Diffusion-based Portrait Animation

  • Portrait animation extends pretrained diffusion models with structural motion conditions through ControlNet or PoseGuider, while implicit facial representations target fine-grained expression dynamics [28] [46] [54] [63].Source: §2 · p. 3

Long-term Portrait Animation

  • Long-term methods use prompt traveling, keyframe-guided interpolation, or shifted windows, but existing streaming approaches can add training overhead and accumulate errors [20] [41] [53].Source: §2 · p. 3

Diffusion Model Acceleration

  • Diffusion acceleration reduces sampling steps through approaches such as ADD, LCM, DMD, and DMD2, while portrait-animation applications of distillation have received little attention [35] [27] [58] [57].Source: §2 · pp. 3–4

3. Method

PersonaLive’s three-stage pipeline combines hybrid motion control with fewer-step appearance distillation and autoregressive micro-chunk streaming. The stages target expressive motion, efficient denoising, and stable long-term generation.

3.1. Image-level Hybrid Motion Training

  • Hybrid motion control combines implicit facial representations for local facial dynamics with 3D implicit keypoints for global pose, position, and scale, injecting these signals through cross-attention and PoseGuider.The keypoints are mapped to pixel space before being injected into the denoising backbone.Source: §3.1 · p. 4

3.2. Fewer-step Appearance Distillation

  • Appearance distillation compresses redundant later appearance refinement into fewer denoising steps, using random step sampling and MSE, LPIPS, and adversarial losses for supervision.Gradients pass only through the final denoising step, while stochastic step sampling provides supervision across middle timesteps.Source: §3.2 · p. 4

3.3. Micro-chunk Streaming Video Generation

  • Micro-chunk streaming uses progressively higher noise levels across chunks, emits clean frames as the window slides, and appends a new noisy chunk without overlap.Sliding training simulates inference errors; motion-interpolated initialization aligns the first window with training, while motion-selected historical keyframes provide appearance cues for temporal consistency.Source: §3.3 · p. 5

4. Experiments

PersonaLive is evaluated for reconstruction, cross-reenactment, inference efficiency, and streaming-component ablations. The reported results combine competitive animation quality with fast inference and stable long-term coherence.

4. Experiments

  • Training uses VFHQ, NerSemble, and DH-FaceVid-1K; evaluation includes TalkingHead-1KH and LV100, a benchmark of 100 portraits and 100 unseen long videos.Videos are processed at 25 fps and cropped to 512 × 512; stage 2 and 3 use 4 denoising steps.Source: §4 · pp. 5–6

4.1. Evaluations and Comparisons

  • 15.82 FPS and 0.253 s average latency; PersonaLive matches identity and motion performance comparably while achieving the best FVD and tLP scores.Replacing the standard VAE decoder with TinyVAE raises speed to 20 FPS.Source: Table 1 · p. 7, §4.1 · p. 7
  • PersonaLive achieves competitive or superior performance across all reconstruction metrics despite using significantly fewer denoising steps.The evaluation uses each test video's first frame as reference and the remaining frames as driving inputs and ground-tr truth targets.Source: §4.1 · p. 7

4.2. Ablation Studies

  • At 4 steps without CFG, appearance distillation improves quality over no distillation, and adding adversarial loss restores realistic detail without CFG.Distillation without GAN leaves outputs overly smooth; using CFG reaches only 9.5 FPS.Source: Figure 6 · p. 7, §4.2 · p. 7
  • Removing sliding training creates a train-inference mismatch, rapid error accumulation, temporal collapse, and an ID-SIM drop to 0.549.The model then trains only on ground-truth-constructed noisy inputs and does not learn to correct its own prediction drift.Source: §4.2 · pp. 7–8, Figure 7 · p. 8
  • Historical keyframes suppress long-sequence drift in regions such as clothing and improve temporal coherence, with a slight decrease in ID-SIM.The reported trade-off is considered acceptable given the temporal-coherence improvement.Source: Figure 7 · p. 8, §4.2 · p. 8
  • Removing motion-interpolation initialization introduces appearance distortions at sequence onset because inference abruptly transitions from reference motion to driving motion.The artifacts are attributed to a mismatch between training and inference.Source: Figure 7 · p. 8, §4.2 · p. 8
  • Reducing chunk size from 4 to 2 slightly improves temporal consistency but degrades identity similarity; chunk-wise causal attention preserves similar motion accuracy with mildly lower identity similarity.Smaller chunks narrow the temporal receptive field, and the figure shows more artifacts in later frames.Source: Figure 7 · p. 8, §4.2 · p. 8

5. Conclusion

PersonaLive combines hybrid motion control, appearance distillation, and micro-chunk streaming with sliding training and historical keyframes. The conclusion also identifies temporal-redundancy and out-of-domain generalization limits.

5. Conclusion

  • PersonaLive uses a three-stage strategy combining hybrid motion control, fewer-step appearance distillation, and micro-chunk streaming with sliding training and historical keyframes.The framework is reported to support real-time, low-latency animation and stable long-term generation.Source: §5 · p. 8
  • The framework does not explicitly exploit temporal redundancy across consecutive frames, which could improve inference efficiency and allow longer denoising windows.These are potential gains identified for future work.Source: §5 · p. 8
  • Training primarily on human facial data limits generalization to non-human portraits such as cartoons and animals, where eyes or mouths may blur or distort.The conclusion points to scalability and applicability in real-world scenarios as future research directions.Source: §5 · p. 9

A. Preliminary: Latent Diffusion Model

The preliminary section describes latent diffusion as progressively corrupting image latents with Gaussian noise and training a U-Net to predict and remove that noise. Portrait animation conditions the process on reference appearance and driving-video motion features.

A. Preliminary: Latent Diffusion Model

  • An encoder maps an image to a latent representation, which is then progressively corrupted by Gaussian noise.Latent diffusion performs the diffusion process in a compact latent space to improve efficiency.Source: A. Preliminary: Latent Diffusion Model · p. 12
  • A predefined noise schedule sets the diffusion horizon, and a U-Net denoiser is trained to predict and remove the added noise.The stated time horizon is t ∈ [0, 1000].Source: A. Preliminary: Latent Diffusion Model · p. 12
  • In portrait animation, conditioning combines appearance features from a reference portrait with motion features derived from the driving video.The input also uses a multi-frame latent window.Source: A. Preliminary: Latent Diffusion Model · p. 12

B. Experimental Details

PersonaLive is trained in three stages, progressing from image-level hybrid motion training through appearance distillation to temporal modeling. The section also describes motion initialization and the LV100 long-term evaluation data.

B. Experimental Details

  • Training proceeds through image-level hybrid motion training, fewer-step appearance distillation, and temporal modeling with only temporal attention layers trainable.The stages use 30K, 30K, and 10K iterations, respectively; Stage 3 slides over 40-frame sequences.Source: B. Experimental Details · p. 12
  • LV100 contains 100 unseen videos of at least one minute at 25 FPS and 100 in-the-wild reference portraits.The videos come from online platforms, while the portraits are sourced from ChatGPT-5, Doubao, and Pexels.Source: B. Experimental Details · p. 12, Figure 9 · p. 13
  • The implicit 3D keypoint extractor produces 21 canonical keypoints, from which stable landmarks encode global head pose, scale, and spatial configuration.These selected landmarks serve as a global motion prior in hybrid motion control.Source: B. Experimental Details · p. 12, Figure 10 · p. 13
  • Initial-window motion signals are interpolated between the reference image and the first driving frame.The keypoint interpolation uses rotation, translation, and scale parameters; rotation is represented using pitch, yaw, and roll.Source: B. Experimental Details · pp. 12–13

C. More Ablations

The ablations examine hybrid motion signals, the historical-keyframe motion threshold, and VAE decoder choice. They describe trade-offs among motion control, temporal stability, identity preservation, visual quality, and inference speed.

Hybrid motion signals

  • Implicit 3D keypoints control global head movements, while facial motion embeddings primarily control fine-grained expressions.The keypoints have higher priority than pose-related cues in the facial motion embedding.Source: C. More Ablations · pp. 13–14

Motion threshold

  • At τ = 17, the historical-keyframe mechanism offers the reported best trade-off between identity preservation and temporal coherence.Smaller thresholds update the history bank more frequently, improving FVD and tLP but lowering ID-SIM; larger thresholds better preserve identity but slightly reduce temporal stability.Source: C. More Ablations · p. 14, Table 3 · p. 14

VAE decoder

  • TinyVAE accelerates inference to up to 20 FPS but degrades visual quality, while SD VAE has the best overall performance with competitive speed.SVD VAE does not improve temporal consistency and reduces runtime efficiency.Source: C. More Ablations · p. 14, Table 4 · p. 14

D. More results

Additional visualizations cover self-reenactment and cross-reenactment, while long-avatar videos illustrate long-term stability and consistency.

D. More results

  • Self-reenactment and cross-reenactment visualizations are presented as further evidence of PersonaLive's robustness and generalization.The long-avatar video results highlight stability and consistency over long sequences.Source: D. More results · p. 14
  • Long-avatar videos show synthesized sequences over extended durations.The section presents four sets of long-video results.Source: D. More results · p. 14

E. Ethics Statement.

The authors frame the work as academic and creative research and acknowledge potential misuse in deceptive or non-consensual synthetic media.

E. Ethics Statement.

  • The work is developed for academic and creative research.The authors acknowledge its potential misuse in generating deceptive or non-consensual synthetic media.Source: E. Ethics Statement · p. 14
  • The authors call for generated content to be clearly marked as artificial and for use to follow ethical and legal standards.The statement specifically notes the potential for deceptive or non-consensual synthetic media.Source: E. Ethics Statement · p. 14

Related papers

Loading 2512.11253v1…