Learning an Animatable Detailed 3D Face Model from In-The-Wild Images

Introduction

• Prior coarse models miss expression-dependent wrinkles, while detailed methods need scans [Chen et al. 2019] or lack occlusion robustness [Abrevaya et al. 2020; Richardson et al. 2017].Earlier expression-dependent detail approaches either rely on scans and do not generalize to unconstrained images [Yang et al. 2020], or put details in appearance maps rather than geometry [Chaudhuri et al. 2020].Source: §1 · p. 2
• DECA learns a UV displacement model from person-specific detail and expression parameters, alongside an image regressor for detail, albedo, shape, expression, pose, and lighting.The model builds on FLAME’s coarse geometry and uses its jaw pose and expression parameters to control displacements.Source: §1 · p. 2
• A detail-consistency loss disentangles stable person-specific features from expression-dependent wrinkles by swapping detail codes between same-identity images and matching rendered results to the inputs.The method relies on the observation that shape and person-specific details remain the same across expressions, while wrinkle expression and intensity change.Source: §1 · p. 2
• Reconstructions run at about 120fps and can be reposed with realistic adaptive wrinkles specific to the person and target expression.The paper illustrates these results with in-the-wild examples, including wrinkles in the forehead and mouth.Source: Figure 1 · p. 1, §1 · p. 2
• DECA achieves state-of-the-art shape reconstruction accuracy on two benchmarks.The model and code are publicly available for research purposes.Source: §1 · p. 2
• DECA’s low-dimensional representation, detail disentanglement, and in-the-wild training support robustness to common occlusions, wide pose variation, and illumination variation.Person-specific details matter because realistic animation requires detailed geometry that can be reposed, not just accurate geometry in the input image.Source: §1 · p. 2

Coarse reconstruction

• Statistical-model methods capture only low-frequency shape, while model-free regressors require explicit 3D supervision and still recover only coarse variations.The latter supervision comes from model fitting or synthetic samples of a statistical face model.Source: §2 · p. 2

Detail reconstruction

• Texture-based detail methods bake shading into texture, whereas DECA represents details as geometry so they respond to lighting changes.Chaudhuri et al. [2020] model geometric details as albedo, so their shading does not adapt to lighting and can look unrealistic.Source: §2 · pp. 2–3
• Optimization-based shape-from-shading methods reconstruct mid-frequency details but are slow, fragile to occlusions, and depend on landmarks that can fail at large viewing angles.These methods first fit a statistical face model, then recover details from monocular images or videos.Source: §2 · p. 3
• Regression-based detail methods refine coarse shapes, but generally leave occluded regions artifact-prone and do not provide animatable details.Tran et al. [2018] address occlusions with face segmentation and example-based hole filling; other approaches include scan-derived wrinkle statistics and per-pixel displacement maps.Source: §2 · p. 3

Animatable detail reconstruction

• Earlier animatable-detail methods learn wrinkle or attribute correlations from high-quality meshes, while FaceScape [Yang et al. 2020] is not robust to occlusions.DECA instead learns from in-the-wild images without paired 3D training data; FaceScape predicts a displacement map from texture after coarse-shape optimization.Source: §2 · p. 3

Novelty

• DECA uses a single image to produce person-specific details that can be realistically animated, without high-resolution scans or paired 3D training data.The paper identifies detail consistency loss as essential to this approach.Source: §2 · p. 3

3 PRELIMINARIES

• FLAME combines linear identity and expression spaces with LBS and pose-corrective blendshapes to generate 5,023-vertex heads from identity, pose, and expression parameters.Its template adds identity-shape, pose-corrective, and expression blendshapes; joint locations depend on identity.Source: §3 · p. 3
• Because FLAME lacks appearance, the Basel Face Model albedo subspace is converted to FLAME’s UV layout to produce UV albedo maps.Source: §3 · p. 3
• An orthographic camera models distant photographs, while spherical harmonics assume distant lighting and Lambertian reflectance; rendering projects the shaded face using mesh and camera parameters.The model represents albedo, surface normals, and shaded texture in UV coordinates.Source: §3 · pp. 3–4
• FLAME generates varied shapes, poses, and expressions, but its low-resolution mesh misses mid-frequency detail, motivating an expression-dependent displacement model for reconstruction and animation.Source: §3 · p. 4

4 METHOD

• DECA learns an expression-conditioned detail model from in-the-wild images, separating static person-specific details from dynamic wrinkles that vary with facial expression.After reconstruction from one image, FLAME expression and jaw-pose parameters can control new wrinkles while person-specific details remain unchanged.Source: §4 · p. 4

4.1 Coarse reconstruction

• A ResNet50 encoder predicts a 236-dimensional code of FLAME shape, expression, albedo, camera, and lighting parameters for analysis-by-synthesis coarse reconstruction.The coarse loss includes landmark, eye-closure, photometric, identity, shape-consistency, and regularization terms; a visible-skin mask makes photometric fitting robust to occlusions.Source: §4.1 · p. 4

4.2 Detail reconstruction

• A 128-dimensional detail code is concatenated with FLAME expression and jaw-pose parameters and decoded into a UV displacement map for detailed geometry.The code controls static person-specific details, while expression and jaw pose control dynamic wrinkles.Source: §4.2 · p. 6
• Detail training combines masked photometric and ID-MRF losses with soft symmetry and L1 displacement regularization to capture high-frequency detail and constrain occluded regions.ID-MRF compares nearest-neighbor feature patches; soft symmetry regularizes non-visible face parts, where extreme poses can otherwise reveal boundary artifacts.Source: §4.2 · p. 6

4.3 Detail disentanglement

• The detail-consistency loss swaps detail codes between images of the same person while retaining each image’s expression and jaw pose, disentangling identity-specific detail from expression-dependent wrinkles.Without this loss, the detail code captures both identity- and expression-dependent details, preventing the reconstruction from being re-posed by changing expression or jaw pose.Source: §4.3 · p. 6

5 IMPLEMENTATION DETAILS

• DECA trains on 2 million images from VGGFace2 [Cao et al. 2018b], BUPT-Balancedface [Wang et al. 2019], and VoxCeleb2 [Chung et al. 2018a].VGGFace2 contains over 8k subjects averaging more than 350 images each; BUPT-Balancedface offers 7k subjects per ethnicity across four ethnicities; VoxCeleb2 contains 145k videos of 6k subjects.Source: §5 · pp. 6–7
• FAN predicts 68 landmarks twice using different face crops, and images with non-matching landmarks are discarded.The datasets provide identity labels for each image.Source: §5 · p. 7
• DECA uses PyTorch [Paszke et al. 2019] and the differentiable Pytorch3D rasterizer, with Adam [Kingma and Ba 2015], a 1e-4 learning rate, 2242 input images, and UV size d=256.Source: §5 · p. 7

6.1 Qualitative evaluation

• Compared with PRNet [Feng et al. 2018b], RingNet [Sanyal et al. 2019], Deng et al. [2019], FML [Tewari et al. 2019], and 3DDFA-V2 [Guo et al. 2020], DECA better reconstructs face shape, mouth, and eyes.DECA also reconstructs a full head, whereas several compared methods output tightly cropped faces; it better reconstructs shape and expression than RingNet [Sanyal et al. 2019].Source: §6.1 · p. 7, Figure 5 · p. 8
• Extreme3D [Tran et al. 2018] and Cross-modal [Abrevaya et al. 2020] reconstruct more detail but are less robust to occlusion and use static details; FaceScape [Yang et al. 2020] produces occlusion artifacts or fails.FaceScape is trained on high-resolution scans, whereas DECA is trained solely on in-the-wild images.Source: §6.1 · p. 7, Figure 6 · p. 9
• DECA’s expression-conditioned details produce plausible animated wrinkles, while animating only the coarse shape with static details creates visible artifacts.The details depend on person-specific parameters and FLAME jaw-pose and expression parameters; animated results resemble reference details of the same identity.Source: §6.1 · p. 7, Figure 4 · p. 8, Figure 7 · p. 9

6.2 Quantitative evaluation

• DECA achieves state-of-the-art coarse-shape reconstruction on NoW, with the lowest mean, median, and standard deviation, and state-of-the-art performance on Feng et al.’s benchmark.The quantitative analysis evaluates coarse shape because no benchmark face dataset has ground-truth shape detail; DECA is trained without paired 3D data.Source: §6.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
• On NoW validation, detailed versus coarse shape errors are median: 1.19/1.18, mean: 1.47/1.46, and std: 1.25/1.25, respectively.The detailed shape improves visual quality but is quantitatively slightly worse than the coarse shape.Source: §6.2 · p. 7
• NoW test errors are median: 1.03/1.16 (f/m), mean: 1.32/1.45 (f/m), and std: 1.16/1.20 (f/m); DECA achieves state-of-the-art performance for both genders.Recovered female shapes are slightly more accurate than male shapes.Source: §6.2 · p. 8

6.3 Ablation experiment

• Without L_dc, identity-specific details and expression-dependent wrinkles are not well disentangled; without L_mrf, wrinkle details are absent and results are overly smooth.With L_dc, raised eyebrows produce forehead wrinkles; with L_mrf, DECA captures detail such as forehead wrinkles.Source: §6.3 · p. 9
• Adding L_id lowers NoW validation mean error to 1.46mm from 1.59mm without it; eye-closure loss and photometric-loss segmentation are also evaluated qualitatively.The qualitative coarse-model comparison also evaluates the eye-closure loss, segmentation on the photometric loss, and identity loss.Source: §6.3 · p. 9

7 LIMITATIONS AND FUTURE WORK

• DECA’s detailed-mesh rendering quality is mainly limited by its BFM-derived albedo, which must omit baked-in shading, specularities, and shadows to separate albedo from geometric details.Future work should learn a high-quality albedo model with varied skin colors and texture details, without illumination effects.Source: §7 · p. 9
• DECA does not explicitly model facial hair, causing skin tone to enter the lighting model and facial hair to be explained through shape deformations.The paper calls for a different approach to model facial hair properly.Source: §7 · p. 9
• DECA can fail under extreme head pose and lighting, and does not handle extreme occlusion such as a hand covering large portions of the face.It tolerates common dataset occlusions; more diverse training data is suggested to address these limitations.Source: §7 · p. 9
• DECA’s many low-resolution training images improve robustness but can introduce noisy details, while the less varied high-resolution data captures more detail but limits robustness and disentanglement.The released DECA-HR model, trained only on high-resolution images, improves visual quality and reduces detail noise, at the cost of robustness to low resolution, extreme poses, and extreme expressions; limited high-resolution data also makes disentanglement difficult.Source: §7 · p. 9
• DECA uses a weak-perspective camera; selfie geometry requires focal-length support, while jointly inferring 3D geometry and focal length from one in-the-wild image under perspective projection remains unsolved and likely needs explicit training supervision [Zhao et al. 2019].Source: §7 · pp. 9–10
• Future work will extend DECA to video for tracking and more personalized individual models, using temporal continuity of intrinsic wrinkles.Source: §7 · p. 10

• DECA achieves state-of-the-art shape reconstruction performance, enabled by a shape-consistency loss.Source: §8 · p. 10
• Trained on about 2M in-the-wild face images without 2D-to-3D supervision, DECA learns an animatable detail model from single images.Source: §8 · p. 10
• A detail-consistency loss disentangles expression-dependent wrinkles from person-specific details, enabling wrinkle transfer and expression-controlled animation.Source: §8 · p. 10
• DECA’s low-dimensional detail latent space makes fine-scale reconstruction robust to noise and occlusions, while its accuracy, reliability, and speed support reenactment and virtual-avatar creation.Source: §8 · p. 10

A OVERVIEW

• The supplement contains implementation details and an extended qualitative evaluation, while its video illustrates the method and presents animation examples.Source: A OVERVIEW · p. 14

B IMPLEMENTATION DETAILS

• Training uses 2 million images from VGGFace2 [Cao et al. 2018b], BUPT-Balancedface [Wang et al. 2019], and VoxCeleb2 [Chung et al. 2018a], plus 50,000 VGGFace2 test images for validation.The training set includes 950,000 VGGFace2 images, 550,000 BUPT-Balancedface images sampled with Asian or African ethnicity labels, and 500,000 VoxCeleb2 frames sampled to vary expression and head pose within subjects.Source: B IMPLEMENTATION DETAILS · p. 14
• Face crops are shifted 5% down and right, then expanded by 10% upward and 20% on the other sides before FAN is run.The passage also says images are discarded based on FAN landmark outputs, but the cutoff is incomplete in the supplied text.Source: B IMPLEMENTATION DETAILS · p. 14
• The coarse model is pretrained for two epochs at batch size 64, then trained for 1.5 epochs at batch size 32 with four images per subject; mouth-corner and nose-tip landmarks receive weight 3.Other mouth and nose landmarks receive weight 1.5, and all remaining landmarks receive weight 1.0.Source: B IMPLEMENTATION DETAILS · p. 14
• The detail encoder and decoder are trained on VGGFace2 and VoxCeleb2 at batch size 6, with three images per subject, while the coarse model is fixed.The detail-model training uses photo, multi-resolution feature, symmetry, detail-consistency, and detail-regularization losses.Source: B IMPLEMENTATION DETAILS · p. 14

C.1 Qualitative comparisons

• Across more than 200 randomly selected ALFW2000 samples, DECA reconstructs more detail than 3DDFA-V2 and is more robust to occlusions than FaceScape and Extreme3D.Source: §C.1 · p. 14, Figure 13 · p. 16
• DECA’s retargeting results on the ALFW2000 comparisons appear realistic.Source: §C.1 · p. 14, Figure 13 · p. 16
• DECA better reconstructs overall face shape than existing methods, adds more detail than coarse methods, and is more robust to occlusions than detail methods.Source: §C.1 · p. 14, Figure 12 · p. 15