Photo-Realistic Facial Details Synthesis from Single Image

• The pipeline transforms an input face image into a proxy 3D face, detailed geometry with displacement, and a re-rendered image.Source: Figure 1 · p. 1

• Single-image methods are sought because dome and LightStage capture systems remain bulky, despite producing high-quality facial geometry.Source: §1 · p. 1
• Existing proxy-based methods can produce over-smoothing and incorrect expressions when few parameters approximate real faces.Source: §1 · p. 1
• Existing methods struggle with correct shape under unseen emotional expressions and lighting, yielding inaccurate geometric details.Source: §1 · p. 2
• The approach predicts emotion to determine an expression-informed proxy because different 3D expressions can share similar 2D landmarks.Source: §1 · p. 2
• DFDN maps image patches to detailed displacement maps using supervised scans and unsupervised in-the-wild images with image-based reconstruction loss.Source: §1 · p. 2

Reconstruction-based Techniques

• Reconstruction systems use stereo, structure-from-motion, or photometric stereo, but texture-poor skin and calibration demands limit accurate wrinkle recovery.Source: §2 · p. 2

Synthesis-based approaches

• 3D morphable models and larger facial databases improved single-image reconstruction, including handling complex expressions.Source: §2 · p. 2

• The processing pipeline separates emotion-driven proxy generation and facial detail synthesis during training, followed by testing on an input image.Source: Figure 2 · p. 3

Synthesis-based approaches

• Current databases still lack mid- and high-frequency details such as wrinkles and pores, while shading compensation remains far below photographic reconstruction quality.Source: §2 · p. 3
• Learning-based proxy and detail methods use CNNs, shape-from-shading, bump maps, volumetric representations, or normal fields, but few generate very fine details.Source: §2 · p. 3

3. Expression-Aware Proxy Generation

• The BFM represents shape, expression, and albedo, with shape and expression determining vertex positions and albedo encoding per-vertex appearance.Source: §3 · p. 3

3.1. Proxy Estimation

• Proxy estimation minimizes weighted landmark reprojection error with shape regularization under an affine camera model.Source: §3.1 · p. 4

3.2. Imposing Expression as Priors

• Different 3D expressions can produce nearly identical 2D landmarks, causing reprojection optimization to fall into ambiguous local minima.Source: §3.2 · p. 4, Figure 3 · p. 4
• The method narrows expression estimation using emotion and physically based appearance features mapped to expression parameters.Source: §3.2 · p. 4
• For a new image, Emotion-Net and an appearance predictor identify the nearest semantic dictionary entry and provide prior expression parameters.Source: §3.2 · p. 4

4.1. Network Architecture

• A two-stage DFDN combines PDIM for medium-frequency wrinkles with PDRM for high-frequency pores and spots.PDIM uses PCA-based displacement estimation, while PDRM refines the lost high-frequency features.Source: §4.1 · pp. 4–5, Figure 4 · p. 5
• Supervised CGAN training uses labeled 3D scans together with unsupervised image-based reconstruction to handle lighting, pose, texture, and expression variation.The reconstruction loss is weighted by η=0.5 and complements the scan-based geometry loss.Source: §4.1 · p. 5

4.2. Geometry Loss

• 256 × 256 texture/displacement patches are represented as linear combinations of 64 PCA basis patches to improve generalization from limited detailed face models.The geometry loss combines PCA-space supervision with per-pixel loss, and training patches are sampled regionally from facial texture maps.Source: §4.2 · p. 6

4.3. Appearance Loss

• Appearance reconstruction estimates proxy geometry, normals, lighting, and albedo, then re-renders the input image to backpropagate reconstruction loss into the displacement map.The method assumes Lambertian reflectance with spherical-harmonic illumination and estimates albedo and environment lighting from sparse proxy vertices.Source: §4.3 · pp. 6–7, Figure 5 · p. 6

Expression Generation

• Our emotion predictor achieves 52.2% test accuracy and produces reasonable expression priors despite facial emotion classification reaching only 60.7% human-annotator accuracy.The authors consider this accuracy sufficient for generating useful expression-informed proxies.Source: §5 · p. 7
• Emotion-driven proxies better match cheeks, nasolabial folds, under-eye bags, jowls, and person-specific expression characteristics than landmark-based 3DFA and ExpNet results.The method exploits global image information, addressing ambiguities that landmarks miss in these regions.Source: §5 · p. 7

Facial Detail Synthesis

• DFDN produces wrinkles resembling real skin more closely than Pix2vertex, FPD, and Extreme3D by combining illumination modeling with facial statistics and wrinkle PCA templates.The comparison is illustrated through close-up synthesized meshes in Fig. 7.Source: §5 · p. 7, Figure 7 · p. 8
• The approach preserves skin details while handling surface noise from eyebrows and beards more effectively than the compared methods.This behavior is shown in the second and sixth rows of Fig. 7.Source: §5 · p. 7

Quantitative Evaluation

• Quantitative evaluation finds much lower errors under strong expressions, especially near the nose and eyebrows where shape deformations are pronounced.Fig. 8 reports errors for proxy-plus-displacement and displacement-only results.Source: §5 · p. 8

Rendering Integration

• The output displacement map integrates easily with existing rendering pipelines and produces high-fidelity results.The authors demonstrate this capability in Fig. 1.Source: §5 · p. 8

Quantitative Evaluation

• Figure 8 shows comparable medium-to-large-scale proxy geometry but much lower fine-detail errors around the forehead and nasolabial folds.The figure also includes displacement-only error maps from the authors’ capture system and USC LightStage [32].Source: Figure 8 · p. 8

Conclusion

• The method produces ultra-high-quality 3D faces with fine geometric details from single images across varied expressions and lighting conditions.The conclusion attributes this capability to combined emotion, expression, appearance, and lighting cues.Source: §6 · p. 8

Limitations and Future Work

• The method has not yet handled occlusions such as hair or glasses, and hard shadows may cause incorrect displacement estimates.The authors suggest ambient-occlusion correction or integrated shadow detection as possible future directions.Source: §6 · p. 8
• Low-resolution images remain outside the method’s capability because geometric detail prediction relies heavily on reliable pixel appearance distributions.Proposed remedies include lighting-aware facial super-resolution and proxy models with deformable geometric details.Source: §6 · p. 8