PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention

You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, Linjie Luo

arXiv:2403.15931v4cs.CVcs.AI

TL;DR

Portrait animation seeks to transfer a driving video's expressions and head motion to a static portrait, while prior approaches can struggle with subtle or extreme motion and identity leakage. X-Portrait uses diffusion rendering with cross-identity RGB motion control and local facial guidance; the authors report strong performance across varied portraits and driving motions, with limitations tied to the reenactment network and source-image appearance context.

  • Problem

    Prior portrait-animation methods can miss subtle or extreme expressions, while explicit motion controls may lose expression detail and carry driving-face semantics into the source identity.

    Source: §1 · p. 2

  • Method

    X-Portrait combines diffusion-based rendering with cross-identity RGB motion control and local facial guidance to animate a source portrait from driving video.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    X-Portrait surpasses state-of-the-art portrait-animation baselines quantitatively and qualitatively and demonstrates effectiveness on diverse portraits and driving motions.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The method produces fine-tuning-free portrait animations across diverse portraits and expressive driving sequences while balancing motion expressiveness, identity preservation, and robustness.

    Source: §1 · p. 3, §5 · p. 8

    The reenactment network may not activate for exceptionally challenging expressions, limiting X-Portrait's ability to learn those motion distributions.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We propose X-Portrait, an innovative conditional diffusion model tailored for generating expressive and temporally coherent portrait animation. Specifically, given a single portrait as appearance reference, we aim to animate it with motion derived from a driving video, capturing both highly dynamic and subtle facial expressions along with wide-range head movements. As its core, we leverage the generative prior of a pre-trained diffusion model as the rendering backbone, while achieve fine-grained head pose and expression control with novel controlling signals within the framework of ControlNet. In contrast to conventional coarse explicit controls such as facial landmarks, our motion control module is learned to interpret the dynamics directly from the original driving RGB inputs. The motion accuracy is further enhanced with a patch-based local control module that effectively enhance the motion attention to small-scale nuances like eyeball positions. Notably, to mitigate the identity leakage from the driving signals, we train our motion control modules with scaling-augmented cross-identity images, ensuring maximized disentanglement from the appearance reference modules. Experimental results demonstrate the universal effectiveness of X-Portrait across a diverse range of facial portraits and expressive driving sequences, and showcase its proficiency in generating captivating portrait animations with consistently maintained identity characteristics.

1 INTRODUCTION

Portrait animation aims to transfer a driving video's head poses and expressions to a still portrait, including in-the-wild styles and wide-ranging motion. X-Portrait uses diffusion-based rendering and RGB motion controls to address limitations of coarse controls and improve identity-preserving animation.

1 INTRODUCTION

  • Earlier warping-and-rendering methods can miss subtle or extreme expressions and produce limited-resolution, blurry, or distorted results under out-of-domain portraits and long-range head motion.Their motion fields may rely on landmarks, neural keypoints, or latent codes.Source: §1 · p. 2
  • Explicit pose controls such as landmarks, skeletons, and dense pose may lose expression detail, depend on detector accuracy, and carry driving-face semantics into the source identity.The resulting cross-identity appearance leakage can affect source characteristics.Source: §1 · p. 2
  • X-Portrait uses diffusion priors with RGB-driven ControlNet motion, cross-identity training images, and local facial attention to transfer motion while mitigating appearance leakage.The local control focuses especially on movements around the eyes and mouth.Source: §1 · p. 2
  • Across benchmarks with hundreds of expressive driving videos, X-Portrait works on out-of-domain portraits and motions without fine-tuning and surpasses state-of-the-art baselines quantitatively and qualitatively.The reported strengths include visual fidelity, identity resemblance, and motion accuracy.Source: §1 · pp. 2–3

2 RELATED WORK

Prior portrait-animation work spans warping-and-rendering GAN methods and newer diffusion-based approaches. X-Portrait positions direct video-driven control as an alternative to explicit motion representations.

2.1 GAN-based Portrait Animation

  • GAN-based portrait animation commonly warps source identity features using estimated motion, then renders frames; variants improve motion representations or resolution, but many focus on talking scenarios and explicit feature warping.Examples include 3D keypoints, depth, thin-plate splines, 3D morphable models, and megapixel generation.Source: §2.1 · p. 3

2.2 Diffusion-based Portrait Animation

  • Diffusion-based portrait-animation methods combine reference-image appearance features, ControlNet-style structural conditions, and temporal attention in pretrained UNets.These approaches use signals including skeletons, facial landmarks, or dense pose.Source: §2.2 · p. 3
  • Explicit diffusion motion signals may miss original motion and depend on third-party detector accuracy, whereas X-Portrait drives portraits directly from video without intermediate explicit motion representations.The cited concerns include potential reductions in animation expressiveness and stability.Source: §2.2 · p. 3

3 METHOD

X-Portrait animates a single source portrait with a driving video while aiming to preserve source identity and background and follow the driving pose and expression. Its method combines diffusion-based rendering, cross-identity RGB motion control, local facial controls, and identity-preservation measures.

3 METHOD

  • Given a single source portrait, X-Portrait generates a sequence conditioned on driving-video frames, retaining source identity and background while following driving head pose and facial expression.The framework builds on a frozen latent diffusion backbone with appearance, motion, and temporal modules.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Preliminaries

  • The latent diffusion model learns reverse denoising in latent space, using source-image appearance and driving-image motion conditions rather than text control.An appearance reference module, ControlNet-style motion module, and temporal module handle appearance, structural control, and cross-frame correspondence.Source: §3.1 · pp. 3–4

3.2 Implicit Cross-Identity Motion Control

  • A reenactment network generates cross-identity RGB control images for training, teaching the motion module to derive identity-disentangled motion and reducing driving-signal appearance leakage.At inference, the driving video is used directly without preprocessing; the reenactment generator may be less reliable for extreme expressions and long-range head motion.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 Enhanced Attention to Local Facial Movements

  • An auxiliary ControlNet receives masked 128 × 128 eye-and-mouth patches to focus denoising guidance on local facial structure and subtle motion.The control-image generator is also fine-tuned with local-patch L1 and VGG perceptual reconstruction losses.Source: §3.3 · p. 5

3.4 Identity Preservation

  • The appearance reference module extracts source features for UNet transformer blocks, while random heterogeneous scaling misaligns control and target structures to encourage reliance on source identity.The stated scaling range is [0.9, 1.1]; excessive misalignment can hinder control-module learning.Source: §3.4 · p. 6
  • When multiple reference images are available, their appearance features can be concatenated through the reference module to provide more context and better-retained identity attributes.The method requires no network modification for this multiple-reference use.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate X-Portrait on varied portrait-animation data and report stronger motion transfer, identity preservation, and image quality than the compared baselines. Ablations identify the contributions of local control and scaling-augmented training, while the limitations discussion ties difficult-to-learn expressions to failures in the base reenactment method.

4.1 Implementation Details

  • Training uses 550 subjects, while evaluation includes 100 in-the-wild portraits and 200 licensed test videos with varied expressions and head poses.The training data includes recordings of 42 expressions and 20-minute talkings; images are cropped to 512 × 512.Source: §4.1 · p. 6
  • The model freezes the SD 1.5 backbone and trains reference, control, and motion modules sequentially for 30K steps each, using 16 video frames per step.Training uses AdamW with a learning rate of 10^-5.Source: §4.1 · p. 6
  • At inference, source-image forward diffusion initializes noise, and prompt traveling enhances temporal smoothness; 24 frames can be generated in 30 seconds on an A10 GPU.The reported timing uses 10 steps, and the method does not rely on F during inference.Source: §4.1 · p. 6

4.2 Evaluations and Comparisons

  • X-Portrait is reported to outperform baselines in image quality and motion accuracy, with strong perceptual quality, motion precision, identity similarity, and transfer of subtle and extreme expressions across styles.The quantitative assessment also reports superior identity preservation and motion accuracy to MagicDance; the qualitative comparison describes wide-range head motion and precise resemblance to the reference.Source: Figure 4 · p. 7, §4.2 · pp. 7–8

4.3 Ablation Studies

  • Ablations associate self-driven training with reduced identity resemblance, removing local control with weaker expression detail, and removing scaling-augmented training with identity drift toward the driver.The local-control ablation has slightly better identity similarity, attributed to more rigid local facial movements.Source: §4.3 · p. 8

4.4 Limitations and Future Work

  • When the base reenactment method F produces no correlated motion clues, difficult expressions can be filtered during training, making their distributions challenging for X-Portrait to learn.The authors identify improved gesture animation, refined diffusion models, and advanced spatiotemporal attention to reduce jitter as future directions.Source: §4.4 · p. 8, Figure 5 · p. 8

5 CONCLUSION

The conclusion presents X-Portrait as an SD-based portrait-animation framework for transferring driving expressions and head poses. It highlights cross-identity training and local control, and reports effectiveness on generalized source portraits and driving motions.

  • X-Portrait is an SD-based framework for transferring driving facial expressions and head poses to portrait animations.Source: §5 · p. 8
  • Cross-identity driving inputs in training support a balance of motion expressiveness, identity preservation, and animation robustness.Source: §5 · p. 8
  • A local control module directs attention to subtle facial-expression details that matter for emotion conveyance.Source: §5 · p. 8
  • The authors report effectiveness on generalized source portraits and driving motions.Source: §5 · p. 8

Motion Attention (Supplementary)

The supplied passage provides the paper's arXiv identifier, version, and date.

A INFERENCE WITH MULTIPLE REFERENCE IMAGES

The appearance reference module supplies source-image features that the backbone UNet self-attention blocks query. The paper states this design extends naturally to multiple source images, which improve personalized appearance features.

B CONTROL AND GENERALIZATION MECHANISM

X-Portrait combines a pre-trained latent diffusion backbone with separate appearance-reference and motion-control mechanisms. The learned controls use structural motion clues to guide denoising and can adapt to more precise expressions and poses under ground-truth motion supervision.

B CONTROL AND GENERALIZATION MECHANISM

  • SD 1.5 supplies the synthesis framework, while a reference module cross-queries source appearance features to establish localized spatial correspondences with the output.The learned latent space also supports generalization to out-of-domain appearances.Source: B CONTROL AND GENERALIZATION MECHANISM · p. 12
  • Additive attentions inject structural motion guidance into U-Net denoising, using conditional control images generated by FaceVid2Vid Plus as clues rather than supervision targets.The control images need not be strictly photo-realistic or pixel-wise accurate if they provide structural motion clues.Source: B CONTROL AND GENERALIZATION MECHANISM · p. 12
  • Although generated control images may degrade on challenging expressions, control modules can learn their motion clues and adapt to more precise expressions and poses when supervised with ground-truth motions.Source: B CONTROL AND GENERALIZATION MECHANISM · p. 12

C USER STUDY

In a user study, most collected responses favored X-Portrait over Face Vid2vid Plus for matching expressions and poses. The paper also shows that using multiple reference images improves captured personalized appearance traits.

C USER STUDY

  • Using both reference images rather than one produced animations with better-captured personalized appearance traits, including differences in hair, ears, and face shape.Source: Figure 1 · p. 12
  • 129 out of 155 responses (83.23%) affirmed X-Portrait surpasses Face Vid2vid Plus in expressions and poses.Source: C USER STUDY · p. 12
  • Thirty-one participants completed five questions, selecting synthesized videos that most closely resembled the driving video's expression; Figure 2 labels the compared outputs A and B.Source: C USER STUDY · p. 12

Related papers

Loading 2403.15931v4…