Source-linked AI summary
ExprGAN: Facial Expression Editing with Controllable Expression Intensity
Hui Ding, Kumar Sricharan, Rama Chellappa
TL;DR
Facial expression editing has been limited by low-resolution synthesis, paired-data requirements, and control restricted to expression categories. ExprGAN combines an encoder-decoder with an expression controller and incremental training to edit expressions with continuously adjustable intensity. The paper reports applications including expression transfer, image retrieval, and improved expression-recognition training.
Problem
Existing facial expression editing methods may require paired data or produce low-resolution images, and generally control expression categories without continuously controlling intensity.
Method
ExprGAN combines a GAN-based encoder-decoder with a real-valued expression controller and a three-stage incremental learning scheme for small datasets.
Results
ExprGAN continuously controls expression intensity, synthesizes multiple styles, and supports expression editing, expression transfer, and data augmentation for improved expression recognition.
Takeaways & Limitations
The learned disentangled identity and expression representations enable applications beyond editing, including expression transfer, image retrieval, and expression-recognition data augmentation.
Abstract
from arXiv · showhide
Facial expression editing is a challenging task as it needs a high-level semantic understanding of the input face image. In conventional methods, either paired training data is required or the synthetic face resolution is low. Moreover, only the categories of facial expression can be changed. To address these limitations, we propose an Expression Generative Adversarial Network (ExprGAN) for photo-realistic facial expression editing with controllable expression intensity. An expression controller module is specially designed to learn an expressive and compact expression code in addition to the encoder-decoder network. This novel architecture enables the expression intensity to be continuously adjusted from low to high. We further show that our ExprGAN can be applied for other tasks, such as expression transfer, image retrieval, and data augmentation for training improved face expression recognition models. To tackle the small size of the training database, an effective incremental learning scheme is proposed. Quantitative and qualitative evaluations on the widely used Oulu-CASIA dataset demonstrate the effectiveness of ExprGAN.
Introduction
ExprGAN addresses limitations in facial expression editing by enabling multiple target-expression styles with continuously controllable intensity, while preserving identity through disentangled representations. Its encoder-decoder architecture, expression controller, and incremental training strategy support editing on small, unpaired datasets and related applications.
- Motivation and Contributions: Existing methods generally change only expression categories, whereas ExprGAN continuously controls intensity from weak to strong and synthesizes multiple styles.The model does not require training data with intensity values.
- Model Architecture: The expression controller generates a real-valued expression code conditioned on the expression label, allowing intensity information beyond a deterministic one-hot vector.A regularizer network maximizes conditional mutual information so code dimensions capture different intensity variations.
- Model Architecture: ExprGAN uses an encoder-decoder generator whose learned identity and expression representations bridge encoding and reconstruction.The framework also includes adversarial training, an expression controller, and a regularizer network.
- Model Architecture: Multi-layer perceptual and pixel-wise L1 losses preserve facial structure, while adversarial networks on the encoder and decoder improve photo-realism.These components are paired with incremental learning because the training set is small.
- Applications: ExprGAN explicitly disentangles identity and expression representations, supporting expression transfer, image retrieval, and expression-recognition data augmentation.The paper presents these capabilities as applications of the learned representations.
- Training Strategy: The authors propose a three-stage incremental training strategy to train ExprGAN on a relatively small dataset without requiring paired samples.The stated stages are controller learning, image reconstruction, and image refining.
Related Works
Prior facial-expression editing methods include graphics-based manipulation, deep generative models, and conditional GAN variants. Their limitations motivate ExprGAN’s unified encoder-decoder design and multi-category expression control.
- Deep Generative Models: GANs learn data distributions through a minimax generator-discriminator game, while VAEs use an encoder-decoder with a latent representation and reconstruction objective.ExprGAN adopts an autoencoder structure within a GAN-based framework.
- Earlier Approaches: Earlier approaches used 3D models, 2D expression mapping, flow-based warping, deep belief networks, and higher-order Boltzmann machines.These methods model expression through image manipulation or learned relationships among identity and expression factors.
- Related GAN Methods: CFGAN uses a filter module for face-attribute control, whereas ExprGAN unifies the encoder and decoder for image editing and supports multiple expression categories.CFGAN requires a separately trained encoder, and its attribute filter is mainly designed for a single class.
Proposed Method
ExprGAN combines an encoder-decoder generator with expression control, identity preservation, adversarial refinement, and incremental training. Its continuous expression code enables manipulation of expression type, style, and intensity.
- Expression Controller Module: The expression controller converts a one-hot label and random input into a continuous code whose positive and negative values distinguish the selected class from other classes.For the selected expression, ci is positive within 0 and 1; other expression components have values from -1 to 0.
- Expression Controller Module: Manipulating the expression code at test time provides control over diverse facial-expression styles and intensities.The paper states that this flexibility increases controllability over synthesizing diverse styles and intensities.
- Regularizer on Expression Code: A mutual-information regularizer encourages dimensions of the expression code to capture different factors of expression-intensity variation.The regularizer maximizes I(c; x̂|y) between the generated image and expression code.
- Generator Network: ExprGAN encodes an input face into an identity-preserving representation and decodes it with an expression code to reconstruct or edit the image.The generator uses an autoencoder structure, with Gdec producing x̂ = Gdec(Genc(x), c).
- Training Objectives: Pixel-wise reconstruction, perceptual identity, image adversarial, identity-representation adversarial, and total-variation losses constrain reconstruction, identity, realism, latent structure, and artifacts.Dimg improves photorealistic textures, Dz reduces holes in the identity representation, and total variation reduces spike artifacts.
- Incremental Training: Because joint optimization performs poorly on the small dataset, ExprGAN uses three-stage incremental training: controller learning, image reconstruction, and image refining.The stages progressively train conditional generation, disentangled representations, and photorealistic refinement.
Experiments
Experiments on Oulu-CASIA evaluate ExprGAN for expression editing, continuous intensity control, transfer, generation, augmentation, and representation quality. The results show identity-preserving, detailed synthesis, controllable intensity, useful transfer and retrieval representations, and improved recognition with sufficient synthetic data.
- Facial Expression Editing: ExprGAN preserves facial identity and fine details while producing natural expressions, whereas CAAE outputs blurry images lacking fine details.
- Facial Expression Editing: Five intensity levels produce expressions ranging from weak to strong, and setting all expression-code dimensions negative generates neutral faces absent from training.
- Facial Expression Transfer: Expression transfer preserves the target expression while retaining the source identity, including generating teeth absent from the source face when transferring Happy to Angry.
- Face Image Generation for Data Augmentation: ExprGAN generates realistic subjects across six expression categories, with diverse expressions within the same class.
- Face Image Generation for Data Augmentation: 84.72% recognition accuracy with 30K synthetic images improves over 77.78% with 3K images, reducing relative error by 31.23%.Performance begins to saturate when 60K synthetic images are used.
- Feature Visualization: Identity features separate most subjects, while expression-code retrieval finds more similar expression styles than pixel-space or label-based retrieval.
Conclusions
ExprGAN enables continuous facial-expression intensity control while disentangling identity and expression representations. The model also supports multiple applications and incremental training on small datasets.
- ExprGAN continuously controls the intensity of a transformed facial expression.
- The model explicitly learns disentangled identity and expression representations.
- ExprGAN supports expression editing, expression transfer, and data augmentation for face-expression recognition.
- An incremental learning scheme enables training on small datasets.