PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning an Animatable Detailed 3D Face Model from In-The-Wild Images

Yao Feng, Haiwen Feng, Michael J. Black, Timo Bolkart

arXiv:2012.04012v2cs.CV

TL;DR

Monocular face reconstruction methods either fail to model animatable expression wrinkles or depend on scan-quality training data. DECA learns person-specific, expression-dependent geometric details from in-the-wild images without paired 3D supervision, achieving state-of-the-art shape reconstruction and enabling animated reconstructions. Its main scope limitations are albedo quality, facial-hair modeling, and the lack of temporal personalization from video.

  • Problem

    Existing methods may produce non-animatable details or require high-quality scans that do not generalize well to unconstrained images.

  • Method

    DECA jointly learns a low-dimensional displacement model and an image regressor, using detail consistency to separate person-specific details from expression-dependent wrinkles.

  • Results

    DECA achieves state-of-the-art shape reconstruction performance and reconstructs animatable detailed faces from single in-the-wild images without 2D-to-3D supervision.

  • Takeaways & Limitations

    The disentangled detail representation supports realistic wrinkle animation, wrinkle transfer, and related applications from single images.

  • Takeaways & Limitations

    Future work is needed for temporal tracking and more personalized models learned from video with continuity of intrinsic wrinkles over time.

Abstract

from arXiv · show

While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks. Qualitative results on in-the-wild data demonstrate DECA's robustness and its ability to disentangle identity- and expression-dependent details enabling animation of reconstructed faces. The model and code are publicly available at https://deca.is.tue.mpg.de.

1 INTRODUCTION

DECA addresses the need for detailed facial geometry that remains animatable, person-specific, and robust to unconstrained images. It learns expression-dependent wrinkles and separates them from identity-dependent details using a detail consistency loss.

  • Existing coarse face models cannot capture expression-dependent wrinkles, which are important for realism and human-emotion analysis.
  • Detailed reconstruction methods require high-quality scans, lack robustness to occlusions, or do not model how wrinkles change with expression.
  • DECA learns an animatable displacement model from in-the-wild images without 2D-to-3D supervision, regressing individual-specific details and expression-dependent wrinkles from one image.
  • A detail consistency loss disentangles person-specific details from expression-dependent wrinkles by swapping detail codes across images of the same identity.
  • DECA reconstructs detailed faces in real time and animates them with realistic adaptive expression wrinkles.The reported speed is about 120fps on a Nvidia Quadro RTX 5000.
  • The paper reports an animatable displacement model, robustness to common image variations, and state-of-the-art shape reconstruction accuracy on two benchmarks.

2 RELATED WORK

Related work includes model-free reconstruction, texture-based detail recovery, optimization and regression methods, and expression-dependent detail models. DECA differs by representing details as animatable geometry learned from ordinary 2D images without paired 3D ground truth.

  • Model-free methods capture varied 3D faces but require explicit 3D supervision from model fitting or synthetic data.
  • Texture-based methods can produce high-fidelity details, but baked shading limits lighting changes and realism.
  • Optimization-based detail reconstruction is slow and can lack robustness to occlusions.
  • Regression-based methods commonly estimate a statistical face model before refining localized details, with some relying on wrinkle statistics from high-resolution scans.
  • Some expression-dependent models represent geometric details in albedo, so detail shading does not adapt to lighting and renderings become unrealistic.
  • DECA learns person-specific, realistically animatable details from single 2D images without paired 3D ground truth, using detail consistency loss as its key distinction.

3 PRELIMINARIES

FLAME provides DECA’s low-dimensional coarse face representation by combining identity, pose, and expression spaces with articulated skinning. Its limited mesh resolution leaves mid-frequency details to DECA’s displacement model.

  • FLAME combines identity shape, expression spaces, linear blend skinning, and pose-dependent corrective blendshapes to articulate the neck, jaw, and eyeballs.
  • The blend-skinning function rotates template vertices around joints using pose parameters and blendweights, with joint locations depending on identity.
  • FLAME’s posed template adds identity shape, pose corrective, and expression blendshapes to the mean template.
  • Given geometry, albedo, lighting, and camera information, the rendering function generates a 2D image from the face model and shaded texture.
  • FLAME supports varied poses, shapes, and expressions from a low-dimensional latent space, but its low mesh resolution omits most mid-frequency details.

4 METHOD

DECA reconstructs detailed 3D faces from in-the-wild images and learns expression-conditioned displacements that preserve person-specific details. Its training combines coarse reconstruction losses with consistency objectives that disentangle identity-dependent details from expression-dependent wrinkles.

  • Coarse reconstruction: DECA regresses coarse geometry, albedo, camera, and lighting parameters from a single face image using an analysis-by-synthesis encoder.The coarse latent code contains FLAME shape, expression, and pose parameters together with albedo, camera, and lighting parameters.
  • Coarse reconstruction: The coarse reconstruction branch combines landmark, eye-closure, photometric, identity, shape-consistency, and regularization losses.The eye-closure loss is less sensitive to projected-face misalignment, while the face-region photometric loss improves robustness to occlusions.
  • Detail reconstruction: Soft symmetry regularizes non-visible UV regions against self-occlusion artifacts, while displacement regularization reduces detail noise.Without soft symmetry, extreme poses expose boundary artifacts in occluded regions.
  • Detail disentanglement: The detail-consistency loss swaps detail codes between same-subject images so expression parameters control wrinkles while person-specific details remain unchanged.This objective uses one image’s expression and jaw parameters with another image’s detail code, requiring the resulting rendered details to remain realistic.

5 IMPLEMENTATION DETAILS

DECA is evaluated qualitatively and quantitatively against coarse and detailed reconstruction methods, with emphasis on robustness, animatable details, and reconstruction accuracy.

  • Implementation details: DECA is trained on VGGFace2, BUPT-Balancedface, and VoxCeleb2, totaling 2 million images.The datasets provide identity labels; FAN-based landmark filtering improves data quality.
  • Qualitative evaluation: DECA reconstructs subject-specific details and wrinkles from single images while remaining robust to partial occlusions.Its coarse FLAME shape represents overall geometry, while the detail model captures mid-frequency facial structure.
  • Qualitative evaluation: DECA qualitatively improves coarse face reconstruction over PRNet, RingNet, Deng et al., FML, and 3DDFA-V2, including face shape, mouth, eyes, and wrinkles.It also reconstructs a full head, whereas several comparison methods produce tightly cropped faces.
  • Qualitative evaluation: DECA is more robust to occlusions than several detailed reconstruction methods, while Extreme3D and Cross-modal reconstruct more detail and FaceScape relies on high-resolution scans.Extreme3D and Cross-modal use static details, which cause artifacts during animation; FaceScape can artifact or fail under occlusion.
  • Qualitative evaluation: Static details produce animation artifacts, whereas DECA’s expression-conditioned details yield plausible animated geometry resembling the reference identity.The comparison is shown for artifacts around the mouth corner and forehead.
  • Quantitative evaluation: 1.18/1.19 median reconstruction error is reported for coarse/detailed shapes on the NoW validation set, with detailed shape performance slightly worse quantitatively.The corresponding mean errors are 1.46/1.47 and standard deviations are 1.25/1.25 for coarse/detailed shapes.

7 LIMITATIONS AND FUTURE WORK

DECA has limitations involving albedo quality, facial-hair modeling, extreme conditions, camera modeling, and training-data trade-offs. Future work includes broader data and temporal personalization.

  • Albedo and appearance: DECA’s detailed-mesh rendering is limited by its BFM-derived albedo model, which lacks sufficiently varied, illumination-free skin appearance.The authors call for an albedo model covering more skin colors and texture details without baked-in shading, specularities, or shadows.
  • Facial hair: Facial hair is not explicitly modeled, causing skin tone to enter lighting and facial hair to be explained through shape deformations.
  • Robustness boundaries: The method can still fail under extreme head pose, lighting, or occlusion, including cases where a hand covers large portions of the face.The authors suggest more diverse training data to address these conditions.
  • Training-data trade-offs: Low-resolution training images improve robustness but can introduce noisy details, while DECA-HR improves visual quality at the cost of robustness.
  • Camera model: DECA’s weak-perspective camera model must be extended with focal length to recover head geometry from selfies; single-image perspective inference remains unsolved.
  • Future work: Future work aims to extend DECA over time for tracking and personalized video models with continuity of intrinsic wrinkles.

8 CONCLUSION

DECA learns animatable facial details from in-the-wild images and reaches state-of-the-art shape reconstruction while separating identity- and expression-dependent wrinkles. This supports single-image facial animation and related applications.

  • DECA is trained on about 2M in-the-wild face images without 2D-to-3D supervision and enables detailed expression capture and animation from single images.
  • DECA reaches state-of-the-art shape reconstruction performance through a shape consistency loss.
  • A novel detail consistency loss disentangles expression-dependent wrinkles from person-specific details.
  • The low-dimensional detail latent space makes fine-scale reconstruction robust to noise and occlusions, supporting animation, wrinkle transfer, and face reenactment.

A OVERVIEW

The supplemental material includes the paper document and a video, alongside implementation details and an extended qualitative evaluation.

  • The supplemental video provides an illustrated summary of the method and animation examples.
  • The supplemental material provides implementation details and an extended qualitative evaluation.

B IMPLEMENTATION DETAILS

DECA is trained on large-scale face datasets, with separate coarse-model pretraining and training stages before detail-model training.

  • Data: DECA uses 2M images drawn from VGGFace2, BUPT-Balancedface, and VoxCeleb2.The sampling includes 950k VGGFace2 images and 550k BUPT-Balancedface images in the described data setup.
  • Training details: The coarse model is pretrained for two epochs with batch size 64 before a further 1.5-epoch training stage using four images per subject.
  • Training details: The training objective uses weighted landmark, photometric, identity, shape-consistency, and expression-related terms with separately specified coefficients.

C EVALUATION C.1 Qualitative comparisons

Qualitative comparisons show that DECA reconstructs overall face shape and fine details effectively on in-the-wild images. Its results are also robust to occlusions, while retargeted details appear realistic under a common expression.

  • DECA better reconstructs overall face shape than the compared existing methods.
  • DECA reconstructs more facial details than coarse reconstruction methods such as 3DDFA-V2.
  • DECA is more robust to occlusions than detail reconstruction methods including FaceScape and Extreme3D.
  • Across more than 200 randomly selected ALFW2000 samples, DECA’s retargeting results appear realistic when details are reposed to a common expression.
Loading 2012.04012v2…