PaperlayerView exact revision on arXiv

Source-linked AI summary

EMOCA: Emotion Driven Monocular Face Capture and Animation

Radek Danecek, Michael J. Black, Timo Bolkart

arXiv:2204.11312v1cs.CV

TL;DR

Monocular 3D face reconstruction often misses subtle or extreme emotional expressions because standard losses do not adequately measure emotional content. EMOCA adds perceptual emotion-consistency supervision while retaining DECA-based reconstruction components, yielding stronger expression quality and competitive emotion recognition.

  • Problem

    Existing monocular parametric 3D face reconstruction methods and standard training metrics are insufficient for capturing the full emotional content of subtle or extreme expressions.

  • Method

    EMOCA trains an animatable 3D face model from in-the-wild images using a perceptual emotion consistency loss between input and rendered-image emotion features.

  • Results

    EMOCA achieves 3D reconstruction on par with current methods, significantly improves reconstructed expression quality, and reaches on-par in-the-wild emotion recognition with the best image-based methods.

  • Takeaways & Limitations

    Emotion-aware geometric reconstruction can preserve emotional content while supporting animation and emotion recognition from estimated 3D expression parameters.

  • Takeaways & Limitations

    EMOCA inherits slight image misalignment from its fixed coarse shape encoder, and its emotion-network supervision can produce artifacts with some architectures.

Abstract

from arXiv · show

As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. We address this with EMOCA (EMOtion Capture and Animation), by introducing a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content. We also directly regress levels of valence and arousal and classify basic expressions from the estimated 3D face parameters. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior. The model and code are publicly available at https://emoca.is.tue.mpg.de.

1. Introduction

EMOCA targets a central limitation of monocular 3D face reconstruction: standard methods often fail to recover expressions that preserve the input image’s emotional content. It introduces emotion-aware supervision and achieves stronger expression quality while retaining accurate identity reconstruction and supporting animation.

  • Existing monocular 3D face methods often produce shapes that do not carry the input image’s emotional content.Some face models lack sufficient expressiveness, while landmark, photometric, recognition, and multi-image consistency losses inadequately capture subtle expression cues.
  • EMOCA learns an animatable face model from in-the-wild images without 3D supervision.Its design uses a trained emotion recognition model as supervision during reconstruction training.
  • EMOCA introduces a perceptual emotion consistency loss that encourages similar emotional content in the input and rendered reconstruction.
  • EMOCA significantly outperforms previous state-of-the-art methods in reconstructed expression quality while preserving state-of-the-art identity shape accuracy and ready animatability.
  • EMOCA’s expression parameters support in-the-wild emotion recognition with performance on par with the best image-based methods.
  • The paper contributes an emotion-consistency loss, a geometry-based emotion-recognition framework, and publicly available code and model.

2. Related work

Prior monocular face reconstruction methods use model-based or model-free approaches with diverse supervision, but expressive accuracy remains limited. EMOCA distinguishes itself by using emotion recognition to improve reconstruction rather than only recognizing expressions.

  • Model-free methods regress meshes, voxels, or signed distance functions, while model-based methods reconstruct parameters of statistical face models such as BFM, FaceWarehouse, or FLAME.
  • Many existing approaches rely on explicit 3D supervision or self-supervised keypoint, contour, photometric, recognition, and multi-view constraints.
  • Keypoint and photometric losses are sensitive to misalignment, and keypoint fitting can be especially inaccurate for extreme expressions and head poses.
  • Large multi-view datasets combining many identities with diverse expressions, ethnicities, ages, and imaging conditions are unavailable.
  • Earlier expression-recognition work evolved from hand-designed geometric and appearance features toward end-to-end models trained on large image and audio-visual datasets.
  • Existing 3DMM-based methods generally recognize expressions, whereas EMOCA leverages emotion recognition to reconstruct more expressive 3D faces.

3. Preliminaries

The preliminaries define FLAME, DECA’s reconstruction pipeline, and the emotion-recognition network used by EMOCA. Together, these components provide parametric face geometry, appearance, rendering, detail generation, and affect predictions.

  • Face model: FLAME also provides an appearance model that maps albedo parameters α to a texture map.
  • Face reconstruction: DECA’s coarse encoder predicts FLAME geometry, albedo, spherical-harmonics lighting, and camera parameters from an image.
  • Face reconstruction: DECA’s detail encoder produces a subject-specific detail vector, while its detail decoder combines that vector with expression and jaw-pose parameters to generate dynamic wrinkle displacements.
  • Face reconstruction: The renderer maps coarse or detailed FLAME geometry, appearance, lighting, camera, and optional normal-map inputs to images.
  • Emotion recognition: The emotion-recognition network uses a ResNet-50 backbone trained on AffectNet to predict expression classes, valence, and arousal.

4. Method: EMOCA

EMOCA extends DECA by training a separate expression encoder with emotion-aware supervision while retaining fixed identity-related predictions. Its loss combines perceptual emotion consistency with reconstruction and expression constraints, followed by an animatable detail stage.

  • EMOCA aims to recover single-image 3D face shapes whose expressions convey the full spectrum of emotion.
  • EMOCA adds an expression encoder and keeps DECA’s coarse encoder fixed, retaining predictions for identity shape, pose, albedo, lighting, and camera.
  • Training only the expression encoder removes the need for multiple images per subject, face recognition loss, and landmark reprojection loss, reducing training resources, time, and memory.
  • The total objective combines emotion consistency, photometric, eye-closure, mouth-closure, lip-corner, and expression-regularization losses.
  • Emotion consistency compares emotion features from the input and rendered images, providing perceptual rather than geometric supervision for emotional content.
  • The detail stage fixes the expression encoder and uses expression and jaw-pose parameters to condition an animatable wrinkle-detail decoder.

5. Experiments

EMOCA is evaluated through qualitative reconstruction, emotion recognition, and perceptual consistency, showing improved emotional expression capture over prior 3D methods.

  • Evaluation setup: EMOCA is evaluated qualitatively, through in-the-wild emotion recognition, and perceptually using an Amazon Mechanical Turk study.
  • Emotion recognition: EMOCA outperforms all 3D face reconstruction methods and performs on par with image-based state-of-the-art emotion recognition.
  • Perceptual study: EMOCA achieves higher perceptual classification consistency than prior methods, scoring 0.68 for coarse and 0.65 for detailed meshes.
  • Perceptual study: The perceptual study finds little difference between EMOCA’s coarse and detailed meshes, possibly because added details can include lip and forehead artifacts.
  • Qualitative evaluation: EMOCA’s coarse reconstructions capture input emotional content better than previous methods, while its detailed displacements better capture fine details in highly emotional images than DECA.

6. Discussion and limitations

The discussion identifies limitations in EMOCA’s baseline, image alignment, emotion supervision, and jaw-rotation optimization, while highlighting the importance of its emotion consistency loss.

  • Limitations: EMOCA inherits DECA’s occasional image misalignment because it uses a fixed coarse shape encoder, and expressive faces can also contain misaligned expressions.
  • Limitations: Balancing geometric alignment against emotion similarity and mitigating reconstruction artifacts remain directions for further work.
  • Ablation findings: Ablation results identify the emotion consistency loss as critical for emotion-recognition performance, while AffectNet finetuning provides only marginal gains over DECA’s training data.
  • Emotion supervision: The emotion recognition network must be selected carefully because the initially used pretrained model produced strong artifacts during reconstruction.
  • Optimization: Jaw-rotation optimization is unstable during training, which the authors hypothesize results from the lack of a good prior for jaw rotation.

7. Conclusions

EMOCA reconstructs expressive 3D faces whose geometry better conveys the input emotion while retaining competitive shape accuracy and supporting emotion recognition. The work also identifies applications and risks associated with more realistic facial animation.

  • Conclusions: EMOCA uses deep emotion-recognition features to supervise reconstructed expressions through an emotion similarity loss.The loss provides training supervision for expressions reconstructed from single in-the-wild images.
  • Conclusions: EMOCA matches state-of-the-art methods in 3D face shape reconstruction while outperforming them on reconstructed-expression quality.Its reconstructed expression parameters also outperform existing 3DMM-based methods for in-the-wild emotion recognition and match the best image-based method.
  • Conclusions: The method combines monocular 3D face reconstruction with emotion analysis and may support applications in gaming, movies, AR/VR, and communication.The authors describe these as potential downstream industrial applications.
  • Conclusions: More realistic 3D face acquisition and animation may enable harder-to-detect deep fakes by reproducing individual subtle emotional cues.The authors also note that failures to communicate emotion in avatars can risk misunderstandings.
  • Conclusions: The appendix frames EMOCA’s novelty as applying emotion recognition to face reconstruction, despite deep perceptual losses themselves not being new.The authors state that making this idea work required overcoming substantial design challenges.

B. Implementation details

The implementation evaluates emotion recognition with regression, correlation, concordance, classification, and sign-agreement measures, using image- and 3DMM-based recognition protocols. Training choices include balanced expression sampling, multilayer prediction models, and standard emotion losses without knowledge distillation.

  • Emotion recognition metrics: RMSE measures root mean squared error, while PCC measures prediction–ground-truth correlation and CCC additionally penalizes differences in means.These metrics are used for emotion-analysis evaluation.
  • Emotion recognition metrics: SAGR evaluates whether predicted and ground-truth values have the same sign.The measure is defined using sign agreement between corresponding values.
  • Emotion recognition loss: The emotion-recognition loss combines category cross-entropy with weighted LMSE, LPCC, and LCCC terms.The weights are controlled by α, β, and γ, with the complete combination specified in the implementation details.
  • Training choices: Knowledge distillation is omitted because its improvements are marginal and it substantially complicates training.This differs from the procedure used by Toisoul et al.
  • Image-based emotion recognition: Image-based emotion models use ResNet-50, Swin Transformer, and EmoNet architectures trained on AffectNet with balanced frequencies for seven expression labels.ResNet-50 and Swin Transformer models are pretrained on ImageNet.
  • 3DMM-based emotion recognition: 3DMM-based recognition trains a four-layer MLP to predict valence, arousal, and expression classes from regressed identity and expression parameters.Each hidden layer has size 2048, with Batch Normalization and LeakyReLUs.

C. Qualitative evaluation

The qualitative-evaluation section additionally tests EMOCA on the AFEW-VA dataset, with results reported in Table 3.

  • Qualitative evaluation: EMOCA is additionally evaluated for emotion analysis on AFEW-VA, with results reported in Table 3.The passage identifies the dataset and table but does not state the reported outcomes.

D. Emotion optimization

EMOCA’s emotion optimization extracts emotion features from rendered and target images, then optimizes expression parameters with regularization. The procedure uses ResNet-50 rather than EmoNet and fixes jaw pose because jaw optimization is unstable.

  • Optimization: Emotion retargeting optimizes expression parameters by minimizing feature distance between rendered and target images plus an expression regularizer.The rendered emotion features are A(IR(ψ)), and the target features are A(IT).
  • Emotion network: ResNet-50 is used as the default emotion network because the released EmoNet produces unsuitable optimization results.The comparison includes the original released EmoNet and a self-trained EmoNet.
  • Jaw optimization: Jaw pose is excluded from optimization because optimizing it makes the process unstable and produces unrealistic or incorrect poses.The authors report that fixing jaw pose avoids this instability.
  • Limitations: FLAME lacks a comprehensive jaw-pose prior, while simple hand-crafted priors did not improve optimization.The authors suggest that a prior entangling expression and jaw-pose spaces may be needed.
  • Limitations: Emotion optimization may permit distorted images that achieve the desired emotion feature vector through an adversarial attack on the emotion network.This concern is specifically associated with allowing jaw pose to vary.

E. Perceptual study

The perceptual study evaluates whether reconstructed faces preserve the expressions of real input images through participant classifications. EMOCA’s variants outperform the other reconstruction methods, while supervision architecture and loss weighting materially affect visual quality.

  • Study design: The perceptual study compares participant labels of real images and reconstructions using confusion matrices, with separate comparisons against ground-truth emotion labels.Figure 9 compares real and reconstructed-image labels; Figure 10 compares reconstruction labels with ground truth.
  • Loss weighting: A small emotion-consistency weight captures emotion insufficiently, while high weights produce unnaturally exaggerated expressions.Figures 13 and 14 visualize this trade-off for ResNet-50 and SWIN-B.
  • Ablations: Removing relative landmark losses negatively affects samples, particularly around the mouth region.The analyzed losses include mouth closure, eye closure, and lip-corner distance.

G. Emotional retargeting

EMOCA separates identity and expression geometry, enabling emotional retargeting across different identities. Its reconstructions and animations preserve emotional fidelity, while emotion embeddings support retrieval of faces with similar emotional content.

  • G. Emotional retargeting: EMOCA disentangles coarse identity and expression geometry from identity- and expression-dependent details, allowing its reconstructions to be animated.A source 3D face can be animated using another actor’s video sequence.
  • G. Emotional retargeting: Retargeted faces preserve the source emotion even when identity and person-specific detailed displacements change.Figure 16 compares the input, coarse reconstruction, detailed reconstruction, and emotion retargeting to another identity.
  • Embedding hypothesis: The emotion embedding is hypothesized to place similar expressions close together and dissimilar expressions farther apart, with invariance to pose, identity, and lighting as a desired property.These are stated properties of the emotion-recognition embedding used by the method.
  • Emotion retrieval: The retrieval examples use a 256-dimensional EmoNet feature embedding and nearest-neighbor search by L2 distance.The retrieval is performed on Aff-Wild2 images and compared with retrieval in ground-truth valence-arousal space.
  • Emotion retrieval: EmoNet-feature retrieval produces faces with very similar emotional content, whereas retrieval using annotated valence and arousal is lower quality.The comparison is illustrated in Figures 17 and 18.
Loading 2204.11312v1…