Source-linked AI summary
GANimation: Anatomically-aware Facial Animation from a Single Image
Albert Pumarola, Antonio Agudo, Aleix M. Martinez, Alberto Sanfeliu, Francesc Moreno-Noguer
TL;DR
Discrete emotion categories restrict facial-expression GANs despite expressions arising from continuous combinations of muscle movements. The paper conditions a GAN on Action Unit activations, trains it without paired target images, and uses attention to focus transformations on relevant regions. It reports anatomically coherent continuous expression synthesis that compares favorably with conditioned GANs and handles complex backgrounds and illumination, while failures remain for some non-human distributions and extreme expressions.
Problem
Discrete emotion categories restrict facial-expression generation, whereas facial movements comprise many anatomically coordinated Action Unit combinations.
Method
The model conditions a GAN on continuous Action Unit activations, trains unsupervised without paired target images, and uses attention to focus transformations on relevant facial regions.
Results
The method synthesizes anatomically coherent expressions continuously, compares favorably with conditioned GANs, and handles images with complex backgrounds and illumination conditions.
Takeaways & Limitations
Action Unit conditioning enables a wider range of anatomically feasible expressions and smooth facial movement transitions in images and video sequences.
Takeaways & Limitations
Failures include attention errors for extreme input expressions and artifacts on non-human anthropomorphic or animal distributions.
Abstract
from arXiv · showhide
Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.
1 Introduction
The paper addresses the limitation of discrete-expression facial GANs by conditioning generation on anatomically grounded Action Units, enabling continuous expression synthesis from a single image. The resulting method is designed for anatomically coherent animation and images in the wild.
- Problem: StarGAN-style facial expression synthesis is limited to a discrete set of attributes determined by dataset annotation granularity.RaFD, for example, provides eight binary facial-expression labels.
- Results: Given one input image, gradually changing Action Unit magnitudes produces a smiling-like expression with controllable activation.The system also handles unnatural illumination conditions in the demonstrated examples.
- Motivation: Facial expressions arise from coordinated muscle actions, with 30 anatomically related Action Units producing more than 7,000 observed combinations.The Facial Action Coding System represents expressions through anatomically related muscle contractions.
- Approach: The proposed model conditions a GAN on a vector encoding each Action Unit’s presence or magnitude rather than on images from a discrete expression domain.The model uses 200,000 EmotioNet images annotated with Action Unit activations and is trained unsupervised.
- Results: The method generates anatomically coherent facial expressions in a continuous domain and handles images in the wild with complex backgrounds and illumination conditions.The paper reports favorable comparisons with other conditioned GAN schemes in visual quality and generation possibilities.
2 Related Work
Prior GAN research supports conditional and unpaired image generation, but facial-editing systems generally condition on discrete emotion categories. This paper instead uses a continuous embedding of muscle movements to model anatomically possible expressions and smooth transitions.
- GANs and conditional generation: GANs generate realistic images through adversarial training, while conditional GANs incorporate conditions such as text, class information, or image-based constraints.The paper uses the continuous Earth Mover Distance metric for training its model.
- Unpaired Image-to-Image Translation: Unpaired image-to-image translation methods learn mappings from marginal domain data while preserving aspects of the original image content.Related approaches include cycle-consistency methods such as CycleGAN, DiscoGAN, and StarGAN.
- Face Image Manipulation: Recent face-editing approaches work with images in the wild but are conditioned on discrete emotion categories such as happy, neutral, and sad.The paper positions its method as integrating skin-and-muscle modeling with modern deep-learning machinery.
- Face Image Manipulation: The proposed model learns a GAN conditioned on a continuous muscle-movement embedding, allowing a large range of anatomically possible expressions and smooth facial transitions.This contrasts with conditioning based on discrete emotion categories.
3 Problem Formulation
The problem formulation represents an input face and target expression using Action Unit vectors, then seeks an image-to-image mapping that produces the target expression. Training is unsupervised and does not require paired identities or known target images.
- Expression representation: Each facial expression is encoded as an N-dimensional Action Unit vector whose normalized values between 0 and 1 represent muscle-activation magnitudes.This continuous representation supports interpolation between expressions and rendering realistic, smooth facial expressions.
- Problem definition: The goal is to learn a mapping M that translates an input image Iyr into an output image Iyg conditioned on a target Action Unit vector yg.The supplied formulation identifies the input image, target condition, and desired output relationship.
- Model formulation: The architecture uses a generator to transform expressions and a critic to assess generated-image quality and expression fulfillment.The generator is applied in both directions, while the critic is WGAN-GP based.
- Training setting: Training is unsupervised and requires neither same-person image pairs with different expressions nor the expected target image.The target vectors are randomly generated in the described training setup.
4 Our Approach
The approach uses a bidirectional conditional generator and critic to transform facial expressions while preserving identity and image realism. Attention masks, expression conditioning, adversarial training, and identity constraints jointly guide synthesis.
- Network Architecture: The generator maps an input face to a desired expression and then reconstructs the original expression through bidirectional application.The critic evaluates generated-image realism and expression fulfillment.
- Network Architecture: Attention makes the generator modify expression-relevant regions while preserving hair, glasses, hats, jewelry, and other static elements.It predicts a color mask C and attention mask A instead of directly regressing a full image.
- Network Architecture: The rendered image combines the transformed color output and original image using the attention mask: I_yf = (1 − A) · C + A · I_yo.A controls each pixel’s contribution from the color transformation versus the original image.
- Network Architecture: A PatchGAN-like critic assesses local patch realism and uses an auxiliary regression head to estimate action-unit activations.The regression head supplies conditioning feedback for the desired expression.
- Learning the Model: The full objective combines image adversarial, attention, conditional expression, and identity losses.These terms respectively target realistic images, non-saturating smooth attention, desired expressions, and preservation of person texture identity.
- Learning the Model: WGAN-GP replaces the standard GAN divergence with Earth Mover Distance and constrains the critic through a gradient-penalty formulation.The critic is constrained to the set of 1-Lipschitz functions.
- Learning the Model: Attention regularization prevents masks from saturating, while total variation encourages smooth spatial color transformations.Without regularization, an attention mask can become one and leave the input unchanged.
- Learning the Model: The conditional expression loss trains generated images toward target action-unit activations, and cycle-consistency identity loss preserves the input person without ground-truth supervision.The identity loss penalizes differences between the original image and its reconstruction.
5 Implementation Details
The model uses a modified image-to-image generator and is trained on a 200,000-image EmotioNet subset with specified optimization settings. Evaluation varies individual action-unit intensities while checking identity preservation.
- Implementation Details: The generator uses parallel final convolutional layers for color and attention masks, and instance normalization improves training stability.Its architecture builds on a Johnson et al. image-to-image mapping variation.
- Implementation Details: Training uses 200,000 EmotioNet samples, Adam with learning rate 0.0001, batch size 25, and 30 epochs with decay during the final 10 epochs.The critic is updated five times for each generator update.
6 Experimental Evaluation
The evaluation tests AU editing, comparisons with conditional GAN baselines, expression-space sampling, wild-image handling, and failure cases. Results show continuous, anatomically feasible transformations, localized attention, identity preservation, and robustness to varied visual conditions, with limitations under insufficiently represented inputs.
- Evaluation overview: The evaluation covers single and multiple AU editing, discrete-emotion comparison, expression variability, images in the wild, and model limitations.The experiments assess both core editing behavior and generalization beyond standard cropped facial images.
- Single Action Units Edition: Four intensity levels—0, 0.33, 0.66, and 1—produce progressively stronger AU movements while preserving identity at zero intensity.The model generates an identical copy of the input when the requested AU intensity is 0.
- Single Action Units Edition: Mouth-related AUs 10, 12, 15, and 25 do not affect eye or eyebrow muscles, while upper-face AUs avoid changing mouth muscles.This indicates that the generator learns independence between facial muscle clusters during isolated AU editing.
- Attention Model: Attention masks focus on AU-relevant facial regions, copy background pixels from the original image, and reduce the need to estimate irrelevant color values.The attention mechanism is learned without supervision and supports later processing of images in the wild.
- Simultaneous Edition of Multiple AUs: Linear interpolation between source and target AU configurations yields smooth, consistent transformations while handling multiple AUs, challenging lighting, and unseen avatar-like distributions.The interpolation uses αy_g + (1 − α)y_r between the original and target expressions.
- Qualitative comparison: Compared with DIAT, CycleGAN, IcGAN, and StarGAN, the model provides a favorable trade-off between visual accuracy and spatial resolution.The comparison uses input images and seven generated facial expressions; some StarGAN outputs show blur.
- High Expressions Variability: Conditioning one source image with 14 AUs produces a wide variety of anatomically feasible expressions while conserving the person’s identity.The experiment samples many face configurations from the same input image.
- Images in the Wild: On film and television frames, attention enables smooth merging of generated faces with original backgrounds while retaining high-resolution images.Examples include frames from “Pirates of the Caribbean” and “Game of Thrones.”
7 Conclusions
The paper presents a fully unsupervised GAN for anatomically consistent facial animation in the wild, using Action Units to control expression deformations. Attention mechanisms help process distracting backgrounds and illumination artifacts, while evaluations report promising smooth expression transitions.
- The proposed GAN performs face animation in the wild and can be trained fully unsupervised.
- Action Units parameterize anatomically consistent face deformations and enable a wide range of expressions through interpolation.
- An embedded attention model focuses processing on regions relevant to each expression, supporting images with distracting backgrounds and illumination artifacts.
- Evaluations on EmotioNet, RaFD, and movie images report promising results, including smooth transitions between different expressions.