PaperlayerView exact revision on arXiv

Source-linked AI summary

ExpNet: Landmark-Free, Deep, 3D Facial Expressions

Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, Gerard Medioni

arXiv:1802.00542v1cs.CV

TL;DR

Landmark-dependent face methods can struggle with extreme pose, scale, and occlusion, motivating direct estimation of facial expressions from images. ExpNet regresses 29D 3D expression coefficients from image intensities without landmarks, and the authors report better emotion recognition and scale robustness than landmark-based methods.

  • Problem

    Landmark-dependent methods can struggle with extreme out-of-plane pose, scale, and occlusion, limiting their use for unconstrained face images.

    Source: §I · p. 1

  • Method

    ExpNet applies a CNN to face-image intensities to regress 29D 3D expression coefficients without facial landmark detection.

    Source: §I · p. 2, §III · p. 2

  • Results

    ExpNet achieves the strongest emotion-recognition accuracy and greater robustness to scale changes than the compared landmark-based methods.

    Source: A. Quantitative Tests · p. 7

  • Takeaways & Limitations

    Avoiding landmark detection supports expression estimation on face images from extreme viewing conditions that challenge landmark methods.

    Source: §V · p. 7

    ExpNet has weaknesses on strong-intensity expressions such as surprise.

    Source: B. Qualitative Results · p. 7

Abstract

from arXiv · show

We describe a deep learning based method for estimating 3D facial expression coefficients. Unlike previous work, our process does not relay on facial landmark detection methods as a proxy step. Recent methods have shown that a CNN can be trained to regress accurate and discriminative 3D morphable model (3DMM) representations, directly from image intensities. By foregoing facial landmark detection, these methods were able to estimate shapes for occluded faces appearing in unprecedented in-the-wild viewing conditions. We build on those methods by showing that facial expressions can also be estimated by a robust, deep, landmark-free approach. Our ExpNet CNN is applied directly to the intensities of a face image and regresses a 29D vector of 3D expression coefficients. We propose a unique method for collecting data to train this network, leveraging on the robustness of deep networks to training label noise. We further offer a novel means of evaluating the accuracy of estimated expression coefficients: by measuring how well they capture facial emotions on the CK+ and EmotiW-17 emotion recognition benchmarks. We show that our ExpNet produces expression coefficients which better discriminate between facial emotions than those obtained using state of the art, facial landmark detection techniques. Moreover, this advantage grows as image scales drop, demonstrating that our ExpNet is more robust to scale changes than landmark detection methods. Finally, at the same level of accuracy, our ExpNet is orders of magnitude faster than its alternatives.

I. INTRODUCTION

ExpNet regresses 3D expression coefficients directly from image intensities without facial landmark detection. The paper reports stronger emotion discrimination and scale robustness than landmark-based methods.

I. INTRODUCTION

  • Landmark-dependent methods can struggle with extreme out-of-plane pose, low scale, and occlusion, while analysis-by-synthesis methods can be computationally expensive.The introduction also notes difficulties distributing and parallelizing the latter methods on GPUs.Source: §I · p. 1
  • The proposed CNN estimates 3D expression coefficients directly from image intensities without requiring facial landmark detection.The authors describe this as the first such direct, landmark-free CNN approach to their knowledge.Source: §I · p. 2
  • The paper presents many expression-reconstruction examples on faces in challenging unconstrained conditions.Figure 1 highlights out-of-plane rotations, scale changes, and age variation.Source: Figure 1 · p. 1, §I · p. 2
  • ExpNet produces more accurate emotion-recognition results and greater scale robustness than landmark-based methods.The comparisons use the CK+ and EmotiW-17 benchmarks.Source: §I · p. 2

II. RELATED WORK

Expression classification assigns images to discrete emotion or action-unit categories, whereas expression regression estimates parameters describing facial deformation. This work focuses on 3D expression coefficients and decouples their estimation from facial landmark detection.

A. Expression Estimation

  • Emotion classification predicts discrete emotion categories or action units, while expression regression extracts parameters for facial deformation.The paper notes that classification has increasingly used deep networks directly on image intensities rather than landmarks as a proxy.Source: A. Expression Estimation · p. 2
  • Expression-regression parameters can use AAM or blendshape coefficients; this work estimates 3D expression coefficients using the representation described by 3DDFA [49].Unlike 3DDFA, ExpNet decouples coefficient regression from facial landmark detection.Source: A. Expression Estimation · p. 2
  • Decoupling expression regression from landmark detection yields greater robustness to changing image scales than the compared approach.Source: A. Expression Estimation · p. 2

B. Facial Landmark Detection

Facial landmark detectors estimate landmark locations from appearance or model landmark shape and appearance explicitly. The section situates landmark-free expression estimation among methods addressing landmark failure under challenging imaging conditions.

B. Facial Landmark Detection

  • Regression-based landmark detectors estimate locations from facial appearance, while model-based methods explicitly model landmark shape and appearance.Source: B. Facial Landmark Detection · p. 2
  • Landmark estimation can fail under extreme out-of-plane rotation, low scale, or face-box conditions unlike those used to develop the detector.Source: B. Facial Landmark Detection · p. 2
  • 3DDFA [49] learns 3DMM parameters with a CNN but uses iterative analysis-by-synthesis, while CE-CLM [46] uses convolution experts for complex landmark-appearance variation.Source: B. Facial Landmark Detection · p. 2
  • The paper frames landmark-free expression modeling as a way to estimate expressions on images that defy landmark-detection techniques.Source: B. Facial Landmark Detection · p. 2

III. DEEP, 3D EXPRESSION MODELING

The method trains a CNN to estimate facial expression coefficients directly from image intensities, using labels that are difficult to collect by hand. It represents a 3D face as an average shape plus linear shape and expression deformations.

III. DEEP, 3D EXPRESSION MODELING

  • The CNN estimates facial expression coefficients from image intensities, but training requires labeled 29D real-valued coefficient vectors without an easy manual labeling interpretation.Source: §III · p. 2

A. Representing 3D Faces and Expressions

  • ExpNet estimates expression parameters by directly regressing 3DMM expression coefficients, decoupling shape and texture from pose and expression.Standard methods typically use detected facial feature points as constraints when estimating expression coefficients.Source: A. Representing 3D Faces and Expressions · p. 3
  • The face representation combines an average shape with linear combinations of shape and expression components, controlled by coefficients α and η.The shape basis uses 99 BFM components and the expression basis uses 29 components defined by 3DDFA [49].Source: A. Representing 3D Faces and Expressions · p. 3

B. Generating 3D Expression Data

The paper addresses the lack of sufficiently large, varied training data with 29D expression labels by generating labels for in-the-wild face images using estimated 3D shape and pose. Expression coefficients are then obtained by fitting projected 3D shape to detected facial landmarks.

B. Generating 3D Expression Data

  • BU-4DFE’s constrained views, 101 subjects, and six expressions limit both deployment settings and the range of predicted coefficients.The authors note that these limits make it unsuitable as a sufficiently broad training source.Source: B. Generating 3D Expression Data · p. 3
  • Existing landmark benchmarks provide too few examples for training a deep CNN to regress 29D real-valued vectors; 300W has 3,026 training and testing images.The authors propose generating ample in-the-wild faces with 29D expression labels.Source: B. Generating 3D Expression Data · p. 3
  • For CASIA WebFace images, the method estimates 99D 3DMM coefficients, pools shape estimates by subject, estimates each image’s pose, and computes a projection matrix.The projection maps recovered 3D shape to image points; expression coefficients are solved by optimization using detected 2D landmarks and expression-component standard deviations.Source: B. Generating 3D Expression Data · p. 3

C. Training ExpNet to Predict Expression Coefficients

ExpNet is trained to regress expression coefficients obtained from the proposed fitting procedure. The implementation uses a ResNet-101 architecture, while the authors leave smaller architectures untested.

C. Training ExpNet to Predict Expression Coefficients

  • Expression coefficients obtained from Eq. (2) serve as ground-truth labels for training ExpNet to regress coefficients from images.Training uses a standard ℓ2 reconstruction loss between predictions and labels.Source: C. Training ExpNet to Predict… · p. 3
  • ExpNet uses a ResNet-101 architecture; the authors did not test smaller networks and note that a more compact network may work as well.Source: C. Training ExpNet to Predict… · p. 3
  • Training uses SGD with batch size 144, momentum 0.9, weight decay 5e-4, and an initial learning rate of 1e-3.No data augmentation is used; CASIA images are roughly aligned, and their empirical mean is removed.Source: C. Training ExpNet to Predict… · p. 3
  • Unlike shape coefficients, expression parameters vary between images of the same subject.The authors contrast this with Tran et al. [38], who assume a unique shape representation per subject.Source: C. Training ExpNet to Predict… · pp. 3–4

D. Estimating Expressions Coefficients with ExpNet

At test time, ExpNet estimates expression coefficients in a single forward pass on a preprocessed face image.

D. Estimating Expressions Coefficients with ExpNet

IV. EXPERIMENTAL RESULTS

The paper evaluates expression estimates qualitatively and quantitatively, with quantitative tests designed to assess how well the estimates reflect facial emotions. It notes that few prior 3D expression-estimation methods performed quantitative tests.

IV. EXPERIMENTAL RESULTS

  • Quantitative emotion-reflection tests complement qualitative evaluation, which was the primary form of assessment in most prior 3D expression-estimation work.Source: §IV · p. 4

A. Quantitative Tests

ExpNet achieves the strongest emotion-recognition accuracy among the compared methods and is more robust to reduced image scales than landmark-based approaches. The evaluation compares expression estimates through emotion classification on CK+ and EmotiW-17, using the same classifier across methods.

A. Quantitative Tests

  • The evaluation compares 29D expression estimates through emotion classification, using the same K=5 kNN classifier and leave-one-clip-out testing across image scales.The methods are compared on CK+ and EmotiW-17 because no benchmark provides ground-truth expression coefficients.Source: A. Quantitative Tests · p. 4
  • The emotion-classification results are far from state of the art; the stated goal is to compare expression-estimation techniques, not emotion classifiers.The same simple classifier is used for all methods to avoid obscuring differences in expression estimates.Source: A. Quantitative Tests · p. 4
  • On CK+, ExpNet captures surprise, happiness, and disgust well; on EmotiW-17, it performs well on neutral, happy, sad, and angry emotions, but less well on disgust, fear, and surprise.The authors note that disgust, fear, and surprise can look visually similar to anger on EmotiW-17.Source: A. Quantitative Tests · p. 4
  • ExpNet is the most accurate method for emotion recognition and is more robust to scale changes than landmark-based methods.Fig. 4 compares accuracy across resized CK+ and EmotiW-17 images.Source: Figure 4 · p. 5, A. Quantitative Tests · p. 7
  • ExpNet outperforms CLNF by a wide margin in all tests, despite CLNF having generated its training labels.The authors say this suggests ExpNet generalized from its training data to a wider range of viewing conditions and challenges.Source: A. Quantitative Tests · p. 7
  • ExpNet is at least one order of magnitude faster than its alternatives, which require landmark detection and expression optimization rather than a single estimation step.The runtime comparison includes deep methods and landmark-based approaches.Source: Table I · p. 4, A. Quantitative Tests · p. 7

B. Qualitative Results

ExpNet captures subtle expression nuances, while landmark-based alternatives can exaggerate expressions or vary across image scales. Its estimates are less able to handle strong, extreme expressions.

B. Qualitative Results

  • The comparisons render estimates at the original scale and scale 0.2, using a shared 3D shape and coarse facial shapes to emphasize expressions over detail.The figures compare ExpNet with 3DDFA, CE-CLM, and CLNF, and include a reference shape before expressions are added.Source: B. Qualitative Results · p. 7
  • ExpNet captures subtle fear and anger expressions, while 3DDFA may exaggerate or underestimate expressions and CE-CLM and CLNF vary across scales.The figures show the same input at different resolutions and report different estimates from CE-CLM and CLNF.Source: B. Qualitative Results · p. 7
  • ExpNet is less able to handle strong, extreme expressions such as surprise; 3DDFA exaggerates some estimates, while CE-CLM is inconsistent across scales.The failure examples show the limitations of each method on extreme expressions.Source: B. Qualitative Results · p. 7, Figure 7 · p. 7

V. CONCLUSIONS

The paper presents deep 3D expression modeling without facial landmarks and reports greater robustness than widely used landmark-detection methods. It argues that avoiding landmarks supports processing faces in extreme viewing conditions.

V. CONCLUSIONS

  • The proposed deep method models 3D facial expressions without using facial landmarks and is reported as more robust than commonly used landmark-detection methods.The authors suggest landmark detection may be redundant for this task.Source: §V · p. 7
  • Avoiding facial landmark detection supports processing face images captured in extreme viewing conditions that can challenge landmark-detection methods.Source: §V · p. 7
  • The conclusion is consistent with prior results on landmark-free 3D face-shape estimation and 6DoF head alignment.Source: §V · p. 7

Related papers

Loading 1802.00542v1…