PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Guided Adversarial Facial Expression Synthesis

Lingxiao Song, Zhihe Lu, Ran He, Zhenan Sun, Tieniu Tan

arXiv:1712.03474v1cs.CV

TL;DR

Facial expression synthesis remains difficult because large, nonlinear geometry variations challenge controllable, realistic, identity-preserving generation. G2-GAN conditions facial synthesis on geometry and jointly trains expression-removal and expression-synthesis subnetworks, producing compelling perceptual results and supporting expression transfer and expression-invariant recognition.

  • Problem

    Existing methods have limited control over complex expression geometry, while warping methods struggle with unseen facial components and new faces.

  • Method

    G2-GAN uses facial geometry as a control condition and jointly trains paired GAN subnetworks for expression removal and synthesis in a mapping cycle.

  • Results

    Experiments on two facial expression databases demonstrate photo-realistic and identity-preserving expression images.

  • Takeaways & Limitations

    The paired geometry-guided framework supports facial expression transfer and expression-invariant face recognition.

  • Takeaways & Limitations

    Warping-based alternatives cannot reliably recover unseen facial components or synthesize realistic images for new faces.

Abstract

from arXiv · show

Facial expression synthesis has drawn much attention in the field of computer graphics and pattern recognition. It has been widely used in face animation and recognition. However, it is still challenging due to the high-level semantic presence of large and non-linear face geometry variations. This paper proposes a Geometry-Guided Generative Adversarial Network (G2-GAN) for photo-realistic and identity-preserving facial expression synthesis. We employ facial geometry (fiducial points) as a controllable condition to guide facial texture synthesis with specific expression. A pair of generative adversarial subnetworks are jointly trained towards opposite tasks: expression removal and expression synthesis. The paired networks form a mapping cycle between neutral expression and arbitrary expressions, which also facilitate other applications such as face transfer and expression invariant face recognition. Experimental results show that our method can generate compelling perceptual results on various facial expression synthesis databases. An expression invariant face recognition experiment is also performed to further show the advantages of our proposed method.

1. Introduction

The paper introduces G2-GAN, which uses facial geometry to control photo-realistic, identity-preserving expression synthesis. Paired generators perform expression removal and synthesis, supporting expression transfer and related recognition applications.

  • Core approach: G2-GAN uses facial geometry as a controllable condition for synthesizing photo-realistic, identity-preserving facial expressions.Geometry is represented as feature points transformed into a heat map and fed to the network with the input face.
  • Controllable synthesis: The geometry-guided architecture generates different expressions from a single image while allowing target expressions to be controlled by facial geometry inputs.
  • Core approach: The framework employs paired GAN subnetworks for the opposite tasks of expression removal and expression synthesis.Together, the subnetworks form a mapping cycle between expressionless and expressioned faces.
  • Geometry modeling: An individual-specific shape model accounts for facial differences and supports facial expression transfer and interpolation.
  • Evaluation: Extensive experiments on two facial expression databases demonstrate photo-realistic and identity-preserving expression images.

2. Related Works

Prior facial expression synthesis methods mainly use warping, example reuse, or generative models, each offering different controls and limitations. Warping methods struggle with unseen facial components and new faces, while generative methods can lack detail and fine-grained control.

  • Method categories: Existing expression synthesis methods are broadly categorized by pixel manipulation into warping or example-based methods and generative-model methods.
  • Warping and example-based methods: Warping-based methods use 2D/3D warping, flow mapping, or image reordering to manipulate facial expressions.
  • Warping and example-based methods: Warping-based methods have difficulty recovering unseen facial components such as skin wrinkles and the inner mouth area, or synthesizing realistic images for new faces.
  • Warping and example-based methods: Example-based methods reuse image patches or retrieve training samples to synthesize desired expressions and unseen faces.
  • Generative models: Generative models provide flexible semantic-level image generation but may produce blurry or low-resolution images and offer limited fine-grained control.
  • Generative models: GAN-based approaches motivate the paper’s framework for generating photo-realistic images with high-quality local details.

3. Methods

G2-GAN guides facial expression synthesis with controllable facial geometry while preserving identity through paired generators, cycle consistency, and identity-aware losses. Its shape model separates identity-related geometry from expression-related changes and supports expression interpolation from a single image.

  • Motivation: Existing deep-learning methods often support only limited semantic expressions and provide weak control over expression intensity.Facial geometry offers a continuous control signal for representing expression variations beyond labels such as smile or angry.
  • Geometry-guided framework: G2-GAN uses fiducial-point heatmaps to condition synthesis, mapping an expressionless face and target geometry to an expressioned face.The paired generators GE and GN perform expression synthesis and removal, respectively, with heatmaps serving as control or auxiliary annotation.
  • Training objectives: The generators are trained with adversarial, pixel, cycle-consistency, and identity-preserving losses combined into a weighted objective.Cycle consistency links neutral-to-expression and expression-to-neutral mappings, while a face-recognition feature extractor enforces identity consistency.
  • Geometry manipulation: Facial geometry can be adjusted continuously through fiducial-point locations to obtain expressions with different intensities and support expression editing applications.The framework uses geometry manipulation for expression transfer and can synthesize different expressions from one neutral image using target shape parameters.
  • Geometry modeling: The shape model constrains facial geometry using PCA-derived parameters while distinguishing identity variation from expression variation.The individual-specific model replaces the population mean shape with an identity-specific neutral shape, leaving shape parameters to represent expression changes.

4. Experiments

Experiments on CK+ and Oulu-CASIA evaluate G2-GAN for expression editing, transfer, interpolation, and expression-invariant face recognition. The method produces identity-preserving edits, supports controllable expression intensity, and improves recognition performance when expression removal is used as normalization.

  • Experimental setup: Experiments cover single-image expression synthesis and removal, face transfer, expression interpolation, and expression-invariant face recognition on CK+ and Oulu-CASIA.The evaluation uses two facial expression databases and includes both qualitative and quantitative analyses.
  • Facial expression editing: G2-GAN generates compelling identity-preserving faces for desired expressions on both testing datasets, including detailed mouth and teeth textures.CK+ results show better low-level image quality than Oulu-CASIA because its images have higher resolution.
  • Quantitative evaluation: Cycle consistency and identity-preserving losses both improve expression-editing performance, with their combination producing the best result.PSNR is computed on the luminance channel, while SSIM is computed across the RGB channels.
  • Facial expression transfer: Expression transfer preserves identity-irrelevant attributes such as eyeglasses and hair while producing subject-specific local deformations.Examples are arranged as input, expression removal, and transferred-expression results for each subject.
  • Facial expression interpolation: Interpolation produces unseen expressions with ascending intensities and captures expression-specific details, including wrinkles, brow movement, mouth deformation, and smile-related teeth.The interpolation experiment has no ground truth and uses fiducial-point locations from a pretrained shape dictionary.
  • Expression-invariant face recognition: Using expression removal as a normalization module further improves face-recognition performance, especially at lower false-acceptance rates.Removing the identity-preserving loss can cause slight drops relative to recognition on original images, supporting its role in face editing.

5. Conclusions

The paper develops a geometry-guided adversarial framework for photo-realistic facial expression synthesis and operation-friendly target specification. Its paired subnetworks form a cycle between expressionless and expressed faces, supporting expression transfer and expression-invariant face recognition.

  • 5. Conclusions: G2-GAN uses facial geometry to guide photo-realistic face synthesis and provide an operation-friendly way to specify target expressions.An individual-specific shape model accounts for differences between subjects.
  • 5. Conclusions: Paired facial editing subnetworks jointly remove and synthesize expressions, forming a mapping cycle between expressionless and expressed faces.The cycle supports expression transfer and expression-invariant face recognition.
  • 5. Conclusions: Extensive experimental results demonstrate the effectiveness of the proposed method for facial expression synthesis.

A.1. Comparison experiment with ExprGAN [6]

The comparison uses the same facial expression synthesis experiments as ExprGAN on Oulu-CASIA. G2-GAN better preserves identity and local details while generating more faithful neutral and expressive faces under the described cropping setup.

  • A.1. Comparison experiment with ExprGAN [6]: The comparison uses the same facial expression synthesis experiments and source images as ExprGAN on the Oulu-CASIA dataset.ExprGAN synthesizes six annotated expression classes with ascending intensities and can also generate neutral faces.
  • A.1. Comparison experiment with ExprGAN [6]: G2-GAN preserves identity information and local details better than ExprGAN during expression transformation.Examples include preserving hair and beard details despite the larger face areas covered in the authors’ cropping setup.
  • A.1. Comparison experiment with ExprGAN [6]: G2-GAN generates neutral faces with less identity loss than ExprGAN, whose recovered neutral faces differ from the ground-truth images.
  • A.1. Comparison experiment with ExprGAN [6]: G2-GAN generates fine details such as frown- and pout-related wrinkles, whereas ExprGAN results tend to lack these details.

A.2. Expression synthesis with controlled eye status

Direct manipulation of facial geometry enables G2-GAN to control eye status and synthesize special expressions while preserving identity. Results on CK+ and additional datasets demonstrate perceptual quality, fine-grained control, and diverse outputs.

  • A.2. Expression synthesis with controlled eye status: Facial geometry provides an intuitive control mechanism for specifying target expressions, including special configurations such as a lopsided grin with one eye open.The controlled-eye experiment uses CK+ because Oulu-CASIA images are low resolution.
  • A.2. Expression synthesis with controlled eye status: G2-GAN generates compelling perceptual results while preserving identity across synthesized eye-status variations.Figure 10 presents examples with gradually changing eye closure.
  • A.2. Expression synthesis with controlled eye status: Direct manipulation of face geometry enables fine-grained control of eye status that is difficult for other generative-model approaches.
  • A.2. Expression synthesis with controlled eye status: The controlled-geometry results indicate operation-friendliness and diversity of synthesized faces, suggesting potential applications for face editing.Additional synthesis results are shown for CK+ and MultiPIE.
Loading 1712.03474v1…