Source-linked AI summary
Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, Qifeng Chen
TL;DR
Portrait animation must transfer target expressions while preserving the reference identity and temporal fidelity, but existing motion representations can misalign expressions or cause identity leakage. Follow-Your-Emoji extends Stable Diffusion with expression-aware landmarks, facial fine-grained loss, dedicated expression data, and progressive generation, achieving strong control across freestyle portraits and reported superiority on EmojiBench.
Problem
Portrait animation requires accurate expression transfer alongside reference-identity preservation and temporal consistency, while existing 2D landmarks can misalign target expressions and leak identity.
Method
Follow-Your-Emoji extends Stable Diffusion with expression-aware landmarks, facial fine-grained loss, an expression-focused training dataset, and progressive generation for long-term animation.
Results
EmojiBench evaluations report quantitatively and qualitatively better performance than existing baselines, including visual fidelity, identity preservation, and precise motion rendering across out-of-domain portraits and motions.
Takeaways & Limitations
The framework supports fine-controllable and expressive animation of freestyle portraits, including humans, cartoons, sculptures, and animals, with stable long-term synthesis.
Abstract
from arXiv · showhide
We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences. The main challenge of portrait animation is to preserve the identity of the reference portrait and transfer the target expression to this portrait while maintaining temporal consistency and fidelity. To address these challenges, Follow-Your-Emoji equipped the powerful Stable Diffusion model with two well-designed technologies. Specifically, we first adopt a new explicit motion signal, namely expression-aware landmark, to guide the animation process. We discover this landmark can not only ensure the accurate motion alignment between the reference portrait and target motion during inference but also increase the ability to portray exaggerated expressions (i.e., large pupil movements) and avoid identity leakage. Then, we propose a facial fine-grained loss to improve the model's ability of subtle expression perception and reference portrait appearance reconstruction by using both expression and facial masks. Accordingly, our method demonstrates significant performance in controlling the expression of freestyle portraits, including real humans, cartoons, sculptures, and even animals. By leveraging a simple and effective progressive generation strategy, we extend our model to stable long-term animation, thus increasing its potential application value. To address the lack of a benchmark for this field, we introduce EmojiBench, a comprehensive benchmark comprising diverse portrait images, driving videos, and landmarks. We show extensive evaluations on EmojiBench to verify the superiority of Follow-Your-Emoji.
1. introduction
Follow-Your-Emoji addresses portrait-animation failures in motion alignment, identity preservation, expression control, and long-term stability by extending Stable Diffusion with specialized motion, loss, data, and generation strategies.
- Motivation: Existing GAN-based pipelines often produce unrealistic content and artifacts because of limited generation quality and inaccurate flow-field motion representation.These pipelines warp reference features before GAN-based rendering and completion.
- Motivation: Stable-Diffusion-based methods improve generation but can distort uncommon-domain portraits because their motion signals remain insufficiently robust.Reported affected domains include cartoons, sculptures, and animals.
- Method: Follow-Your-Emoji introduces expression-aware landmarks to align target motion with the reference portrait during inference and avoid identity leakage.The landmarks are obtained by projecting 3D keypoints from MediaPipe.
- Method: A progressive generation strategy extends the method to long-term animation with fidelity and stability.This is presented as an additional capability of the framework.
- Evaluation: EmojiBench contains 410 varied portrait-animation videos, and evaluations report better quantitative and qualitative performance than existing baselines.The reported advantages include visual fidelity, identity preservation, and precise motion rendering, including outside-domain portraits and motions.
2. Related Work
Related work progresses from GAN-based portrait animation and specialized driving signals toward diffusion models that leverage Stable Diffusion for image and video generation.
- GAN-based animation: Earlier portrait-animation approaches primarily used GANs with warping and rendering to generate plausible motion from estimated facial or head motion.These methods also include video reenactment and driving signals such as 3D facial priors and audio.
- GAN-based animation: GAN-based and related approaches struggle with high-quality facial details and diverse domain styles, particularly outside talking scenarios.The cited driving signals include 3D facial priors and audio.
- Diffusion models: Diffusion models have achieved strong performance across image generation, image editing, video generation, and video editing.Latent diffusion further improves performance by carrying out diffusion in latent space.
- Diffusion models: Recent portrait-animation methods extend Stable Diffusion with temporal information, as exemplified by AnimateDiff, MagicVideo, VideoCrafter, and ModelScope.These systems adapt foundation diffusion models to video-generation settings.
3. Preliminaries
The preliminaries describe latent diffusion and the modular structure used to adapt Stable Diffusion for portrait animation, combining appearance, temporal, motion, and image-prompt conditioning.
- Latent diffusion: Latent diffusion reformulates diffusion and denoising in latent space rather than image space for stable and fast training.A VAE projects RGB images into latent space before diffusion.
- Latent diffusion: A UNet learns reverse denoising in latent space using self-attention and cross-attention, with cross-attention injecting conditioning information.The described conditioning is textual embedding in the Stable Diffusion formulation.
- Latent diffusion: The latent-diffusion training objective predicts ground-truth noise from a noisy latent embedding at timestep t under a condition embedding.The passage defines z as the latent embedding, ϵθ as predicted noise, ϵ as ground-truth noise, c as conditioning, and ᾱ_t as the diffusion coefficient.
- Portrait animation with diffusion: Portrait-animation adaptations of Stable Diffusion use Appearance Net, Temporal Attention, Control Motion Injection, and Image Prompt Injection.These modules respectively encode identity and background, preserve temporal correspondence, map motion controls, and replace text conditioning with reference-image tokens.
4. Method
Follow-Your-Emoji combines expression-aware landmarks, identity-preserving diffusion components, facial fine-grained supervision, and progressive generation to animate portraits with controlled expressions and stable long videos.
- Framework: The framework injects landmark features into the UNet alongside appearance conditioning and temporal attention to transfer expressions while preserving identity and frame consistency.A landmark encoder extracts motion features, while the appearance net, image prompt injection, and temporal attention support identity and temporal coherence.
- Expression-Aware Landmark: Expression-aware landmarks provide motion control that preserves facial features, captures iris movement, and enables canonical-space alignment to the reference portrait.The representation projects MediaPipe 3D keypoints while discarding facial contours and retaining facial features and iris relationships.
- Facial Fine-Grained Loss: Facial fine-grained loss uses expression and facial masks to focus training on facial content relevant to expression generation and identity preservation.Expression masks dilate expression-aware landmark points, while facial masks connect projected MediaPipe facial keypoints.
- EmojiBench: EmojiBench contains portrait examples spanning high expression diversity, exaggeration, and varied visual styles for evaluating freestyle portrait animation.The benchmark examples include diverse portrait styles and motion conditions relevant to uncommon-domain portraits.
- Progressive Generation: Progressive generation first predicts keyframes and then interpolates long-term animation from coarse to fine, addressing temporal-consistency degradation from overlapping clip merging.The method masks intermediate latent frames during training to simulate keyframe-based long-video generation.
5. Experiment
Experiments evaluate Follow-Your-Emoji on diverse portraits, benchmark comparisons, and ablations. The method shows stronger expression generation, identity preservation, and motion quality, while its facial fine-grained loss improves complementary facial and expression capabilities.
- 5.2. EmojiBench: EmojiBench contains 410 portraits across cartoon, real-human, and animal domains, plus 45 driving videos covering diverse head motions and facial expressions.Videos contain approximately 150 frames each and include expressions such as frowning, crossed eyes, and pouting.
- 5.3.1 Qualitative results.: Compared with GAN-based and diffusion-based baselines, Follow-Your-Emoji better handles large pose changes, subtle expressions, and identity preservation for uncommon-style portraits.GAN methods show artifacts under large head rotations, while MagicDance and FADM transfer expressions better but do not preserve identity as well.
- 5.3.2 Quantitative results.: User studies report that the approach outperforms previous baselines in temporal coherence, identity preservation, and motion quality.Quantitative evaluation covers self and cross reenactment on 256 × 256 test images.
- 5.4. Ablation Study: The experiments compare alternative motion representations to analyze the effectiveness of expression-aware landmarks in portrait animation.The evaluated variants include 2D landmarks, landmarks with facial contours, and landmarks without pupil points.
- 5.4. Ablation Study: Removing the facial aspect of FFG loss weakens identity and appearance preservation, whereas removing its expression aspect harms subtle expression changes such as pupil movement.The ablation study evaluates both expression and facial components of the loss.
6. Conclusion
The paper presents Follow-Your-Emoji as a diffusion framework for controllable freestyle portrait animation. It combines expression-aware landmarks, facial fine-grained loss, progressive generation, a dedicated training dataset, and EmojiBench to support expressive and stable animation evaluation.
- 6. Conclusion: Follow-Your-Emoji combines expression-aware landmarks and facial fine-grained loss to improve expression generation and identity preservation.The framework also uses progressive generation for stable long-term animation.
- 6. Conclusion: The authors introduce a training dataset with 18 exaggerated expressions and 20-minute real-human videos from 115 subjects.
- 6. Conclusion: EmojiBench is introduced as a comprehensive benchmark for evaluating generalized reference portraits and driving motions.