PaperlayerView exact revision on arXiv

Source-linked AI summary

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, Di Zhang

arXiv:2407.03168v2cs.CV

TL;DR

Portrait animation seeks lifelike, controllable video from a single image, but existing diffusion approaches can be expensive and difficult to control precisely. LivePortrait extends implicit-keypoint-based video-driven animation with scaled training, improved modeling, and lightweight stitching and retargeting modules. It reports 12.8ms generation on an RTX 4090 while outperforming heavy diffusion-based methods, with limitations under large pose variation and significant shoulder movement.

  • Problem

    Portrait animation requires realistic, expressive synthesis from a static image while existing diffusion-based methods can be computationally expensive and lack precise controllability.

  • Method

    LivePortrait extends implicit-keypoint-based video-driven animation with large-scale mixed image-video training, improved motion modeling and objectives, plus lightweight stitching and retargeting modules.

  • Results

    12.8ms generation on an RTX 4090 GPU with PyTorch is reported alongside performance surpassing other heavy diffusion-based methods.

  • Takeaways & Limitations

    Compact implicit keypoints can support blendshape-like control, while stitching and retargeting improve alignment and targeted facial-expression control with negligible overhead.

  • Takeaways & Limitations

    The model struggles with cross-reenactment involving large pose variations and can produce jitter when driving videos contain significant shoulder movements.

Abstract

from arXiv · show

Portrait Animation aims to synthesize a lifelike video from a single source image, using it as an appearance reference, with motion (i.e., facial expressions and head pose) derived from a driving video, audio, text, or generation. Instead of following mainstream diffusion-based methods, we explore and extend the potential of the implicit-keypoint-based framework, which effectively balances computational efficiency and controllability. Building upon this, we develop a video-driven portrait animation framework named LivePortrait with a focus on better generalization, controllability, and efficiency for practical usage. To enhance the generation quality and generalization ability, we scale up the training data to about 69 million high-quality frames, adopt a mixed image-video training strategy, upgrade the network architecture, and design better motion transformation and optimization objectives. Additionally, we discover that compact implicit keypoints can effectively represent a kind of blendshapes and meticulously propose a stitching and two retargeting modules, which utilize a small MLP with negligible computational overhead, to enhance the controllability. Experimental results demonstrate the efficacy of our framework even compared to diffusion-based methods. The generation speed remarkably reaches 12.8ms on an RTX 4090 GPU with PyTorch. The inference code and models are available at https://github.com/KwaiVGI/LivePortrait

1. Introduction

LivePortrait targets realistic, expressive animation of static portraits while improving inference efficiency and precise controllability. It extends implicit-keypoint-based video-driven methods rather than relying on computationally expensive diffusion-based approaches.

  • LivePortrait aims to animate static portrait images realistically and expressively with high inference efficiency and precise controllability.
  • Diffusion-based portrait animation methods achieve impressive quality but are usually computationally expensive and lack precise controls such as stitching.
  • The framework scales training to about 69 million high-quality portrait images and combines image-video training with architectural and objective improvements.
  • LivePortrait adds stitching and two retargeting modules using a small MLP with negligible computational overhead to enhance controllability.
  • The framework develops implicit-keypoint-based video-driven portrait animation to balance generalization ability, computational efficiency, and controllability.

2. Related Work

Video-driven portrait animation methods comprise non-diffusion and diffusion-based approaches. Non-diffusion methods use implicit keypoints and optical flow, while diffusion-based methods incorporate diffusion models with varied motion representations and conditioning strategies.

  • Recent video-driven portrait animation methods are divided into non-diffusion-based and diffusion-based categories.
  • Non-diffusion-based methods: Implicit-keypoint-based non-diffusion methods use implicit keypoints as intermediate motion representations and warp source portraits with driving images through optical flow.
  • Non-diffusion-based methods: FOMM models motion near each keypoint with local affine transformations, while MRAA uses PCA-based motion estimation and Face vid2vid introduces 3D implicit keypoints.
  • Diffusion-based methods: Diffusion models synthesize samples from Gaussian noise by iteratively removing noise, with latent diffusion transferring computation into compressed latent space.
  • Diffusion-based methods: Diffusion-based portrait animation methods combine pretrained implicit-keypoint models, 3DMM guidance, identity adapters, spatial conditions, or original driving videos.

3. Methodology

LivePortrait extends face vid2vid with a two-stage framework that improves generalization, expressiveness, and controllability while retaining efficient implicit-keypoint animation. The method combines enhanced base-model training with lightweight stitching and retargeting modules.

  • Base framework: The framework uses appearance and motion extraction, keypoint-based warping, and decoding to transform driving-video motion into an animated portrait.Face vid2vid extracts source appearance features and implicit keypoints, warps the source feature volume, and decodes the result into image space.
  • Base model training: High-quality data curation combines public datasets, 4K portrait videos, talking-head videos, styled portraits, and private LightStage data.The training collection contains 69M filtered video frames from about 18.9K identities and 60K static styled portraits.
  • Base model training: Mixed image-video training addresses poor generalization from realistic portrait videos to styled portraits by adding abundant styled portrait images.The authors collected approximately 60K styled images, compared with about 1.3K styled video clips from fewer than 100 identities.
  • Base model training: The upgraded motion model unifies canonical keypoint detection, head-pose estimation, and expression deformation, while the SPADE decoder improves generation capacity.ConvNeXt-V2-Tiny serves as the backbone, and PixelShuffle is inserted for efficiency.
  • Base model training: Scalable motion transformation explicitly incorporates source and driving scale factors, balancing expression flexibility against drivability.The authors report that scale-orthographic projection can produce overly flexible deformations and texture flickering across identities.
  • Base model training: Landmark-guided optimization targets subtle eye and lip expressions, while cascaded perceptual, GAN, and identity losses improve texture quality and source identity preservation.The selected landmarks come from the eyes and lips, with N set to 10; cascaded discriminators operate globally and on face and lip regions.
  • Stitching and retargeting: Stage II adds stitching and two retargeting modules based on compact implicit keypoints, using small MLPs for controllable face editing with negligible overhead.Stitching corrects spatial misalignment and supports larger images or multiple faces; eye and lip retargeting address cross-identity expression mismatches.

4. Experiments

Experiments evaluate LivePortrait on self- and cross-reenactment, comparing qualitative fidelity, quantitative quality and motion accuracy, and controllability modules. The results show stable motion transfer, source-identity preservation, competitive quality, and benefits from stitching and retargeting.

  • Experimental setup: The experiments compare LivePortrait with non-diffusion and diffusion-based baselines using image-quality, motion-accuracy, and identity-related benchmarks.Evaluation covers self-reenactment on TalkingHead-1KH and VFHQ and cross-reenactment using FFHQ source portraits.
  • Self-reenactment: LivePortrait faithfully transfers lip movements, eye gazes, appearance details, and poses, including stable animation under large poses.The qualitative self-reenactment results are presented in the original image space after stitching.
  • Self-reenactment: LivePortrait slightly outperforms previous diffusion-based methods in self-reenactment generation quality and achieves better eyes motion accuracy than other methods.The reported comparison includes FADM, AniPortrait, and X-Portrait.
  • Cross-reenactment: Cross-reenactment transfers delicate lip and eye motions while preserving source appearance and remains stable under large poses and small face regions.Stitching also supports animation of multi-person inputs or full-body images when the reference face is relatively small.
  • Cross-reenactment: LivePortrait outperforms prior diffusion- and non-diffusion-based methods in cross-reenactment generation quality and motion accuracy, with stated exceptions for FID and CSIM.The passage reports better FID for AniPortrait on TalkingHead-1KH and better CSIM for X-Portrait on both datasets.
  • Ablation study: Stitching aligns animated shoulders with the cropped source and prevents visually apparent shoulder misalignment after pasting into the original image space.The ablation compares results without stitching against stitched results while preserving motion and appearance.
  • Ablation study: Eyes retargeting enables eye motion control across differing source and driving eye sizes, matching the driving video’s eye-closing motion.The module accepts an eyes-open scalar and source eye condition, while leaving the remaining reference image unaffected.
  • Ablation study: Lip retargeting drives lips between closed and fully open states using a lip-open scalar without affecting the remaining source image.The reported results also include reasonable behavior for an out-of-training-distribution scalar and tongue generation when lips are widely open.

5. Conclusion

LivePortrait targets realistic, expressive, efficient, and controllable portrait animation, reaching 12.8ms generation on an RTX 4090 with PyTorch while outperforming heavy diffusion-based methods. The authors note remaining cross-reenactment and shoulder-motion limitations, alongside deepfake-related risks and detectable visual artifacts.

  • 12.8ms generation on an RTX 4090 GPU with PyTorch, while simultaneously outperforming other heavy diffusion-based methods.
  • The framework targets realistic, expressive portrait animation with high inference efficiency and precise controllability.
  • The model struggles in cross-reenactment scenarios with large pose variations and can produce jitter during significant shoulder movements.
  • Portrait animation poses social risks including deepfake misuse, while some synthesized visual artifacts may aid detection.

A. Benchmark Metric Details

The benchmark evaluates perceptual similarity, expression and pose alignment, eyeball direction, identity preservation, and distributional realism using defined metrics and specified evaluation data processing. Additional figures show audio-driven lip control and animal animation from human driving videos.

  • LPIPS measures perceptual similarity between animated and driving images using an AlexNet-based metric.
  • AED is the mean L1 distance between animated and driving expression parameters extracted by SMIRK.The parameters include facial movement, eyelid, and jaw pose parameters.
  • APD is the mean L1 distance between animated and driving pose parameters extracted by SMIRK.
  • MAE measures eyeball direction error as the mean angular error between predicted animated and driving eyeball direction vectors.
  • CSIM measures identity preservation through cosine similarity between pretrained face-recognition embeddings, with pairings differing between self- and cross-reenactment.
  • FID compares animated-image distributions with real-image distributions, using specified FFHQ subsets for TalkingHead-1KH and VFHQ.
  • Evaluation sequences sample 24 frames from TalkingHead-1KH videos and 6 frames from VFHQ videos, while X-Portrait cross-reenactment uses longer driving sequences.
  • Audio-driven examples show accurate lip movements with stitching applied, while Ragdoll cat and Corgi dog animations use motions derived from human videos.

B. Qualitative Results on Multi-person Portrait

LivePortrait extends to multi-person portrait animation, using its stitching ability to animate each person separately within a portrait.

  • Each person in a multi-person portrait can be animated separately using the model’s stitching ability.

C. Audio-driven Portrait Animation

The video-driven model can be extended to audio-driven portrait animation by deriving expression and head-pose motions from audio. Whisper encodes audio features, and a transformer-based framework autoregresses the motions.

  • The model extends to audio-driven portrait animation by regressing or generating expression deformations and head poses from audio inputs.
  • Whisper encodes audio into sequential features, while a transformer-based framework following FaceFormer autoregresses the motions.

D. Generalization to Animals

LivePortrait generalizes beyond single human portraits: fine-tuning with a small animal dataset enables animation of cats and dogs, while the framework also supports multi-person inputs driven by human videos.

  • D. Generalization to Animals: Fine-tuning on a small animal-portrait dataset combined with the original data enables generalization to cats and dogs.The animal fine-tuning stage removes human-specific losses for head pose, lips, and face identity.
  • D. Generalization to Animals: The model can animate each subject in a group photo while applying stitching to the animated outputs.
  • D. Generalization to Animals: Human driving videos can animate animal portraits after this fine-tuning procedure.

E. Portrait Video Editing

LivePortrait extends portrait animation to source videos by transforming source and driving implicit keypoints, allowing head-region edits while retaining non-head content and temporal consistency.

  • E. Portrait Video Editing: The model edits the head region of a source video while minimally sacrificing the source video's temporal consistency.
  • E. Portrait Video Editing: The keypoint transformation combines source and driving pose, expression, scale, and translation information for edited-frame generation.The displayed transformation is introduced for source and driving implicit keypoints, with source-frame symbols defined in the accompanying prose.
  • E. Portrait Video Editing: Averaging adjacent driving expression offsets smooths the edited motion across frames.The smoothing term averages the expression offsets of the i-th and (i + 1)-th driving frames.
  • E. Portrait Video Editing: With stitching, edited frames inherit expressions from corresponding driving frames while preserving non-head regions from corresponding source frames.
Loading 2407.03168v2…