Source-linked AI summary
Photorealistic Facial Texture Inference Using Deep Neural Networks
Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, Hao Li
TL;DR
Reconstructing a complete photorealistic face texture from one unconstrained image is difficult when the view is partial and resolution and lighting are uncontrolled. The paper blends deep-network feature correlations from high-resolution face textures and optimizes a complete albedo map. Its user study reports renderings visually comparable to Light Stage capture, though exact placement of high-frequency details is uncontrolled.
Problem
Single-image face reconstruction must infer complete, high-resolution appearance from partial views with limited resolution and unknown lighting, while preserving subject likeness and photorealistic detail.
Source: §1 · pp. 1–2Method
The method analyzes visible high-frequency albedo with mid-layer deep-network feature correlations, fits their convex combination from a face-texture database, and iteratively optimizes a complete texture.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach produces plausible mesoscopic details from varied low-resolution inputs, including archival images of historical figures.
Source: §6 · p. 6The multi-scale detail representation does not control the exact appearance or placement of synthesized high-frequency details, so a mole may appear where none exists.
Source: §7 · p. 8
Abstract
from arXiv · showhide
We present a data-driven inference method that can synthesize a photorealistic texture map of a complete 3D face model given a partial 2D view of a person in the wild. After an initial estimation of shape and low-frequency albedo, we compute a high-frequency partial texture map, without the shading component, of the visible face area. To extract the fine appearance details from this incomplete input, we introduce a multi-scale detail analysis technique based on mid-layer feature correlations extracted from a deep convolutional neural network. We demonstrate that fitting a convex combination of feature correlations from a high-resolution face database can yield a semantically plausible facial detail description of the entire face. A complete and photorealistic texture map can then be synthesized by iteratively optimizing for the reconstructed feature correlations. Using these high-resolution textures and a commercial rendering framework, we can produce high-fidelity 3D renderings that are visually comparable to those obtained with state-of-the-art multi-view face capture systems. We demonstrate successful face reconstructions from a wide range of low resolution input images, including those of historical figures. In addition to extensive evaluations, we validate the realism of our results using a crowdsourced user study.
1. Introduction
The paper targets complete, photorealistic 3D face appearance from a single unconstrained image, where partial visibility, low resolution, and unknown lighting make reconstruction difficult. It combines low- and high-frequency albedo representations with feature-correlation analysis and synthesis; the authors report results visually comparable to Light Stage capture.
1. Introduction
- A single unconstrained image can yield high-resolution albedo textures with plausible mesoscopic details, addressing appearance details that linear PCA models cannot capture.The method targets resemblance to the subject despite partial views, limited resolution, and unknown lighting.Source: §1 · pp. 1–2
- Mid-layer deep-network feature correlations are blended through convex combinations to synthesize semantically plausible fine-scale facial details.The representation separates low-frequency albedo from high-frequency details and supports high-resolution texture synthesis without further training of the pre-trained VGG-19 network.Source: §1 · p. 2
- A crowdsourced user study found the method's photorealistic results visually comparable to ground-truth measurements from a Light Stage capture device.Source: §1 · p. 2
- The work introduces a high-fidelity textured 3D face dataset based on high-resolution Chicago Face Database photographs.The authors state that it will be publicly available to the research community.Source: §1 · p. 2
2. Related Work
Prior work spans specialized capture systems, monocular reconstruction, linear face models, texture synthesis, and deep-learning inference. The section identifies limits in single-image texture resolution and in representing mesoscopic facial details and variation.
Facial Appearance Capture
- Specialized capture systems measure reflectance and appearance detail but are restricted to studio environments, while single-image methods compute visible-region textures at input-limited resolution.Multi-view monocular systems can integrate texture maps, but require multiple views.Source: §2 · pp. 2–3
Linear Face Models
- PCA-based morphable face models estimate complete albedo maps from one image, but their linear assumption limits mesoscopic detail and large appearance variations.The section traces linear appearance models from Eigenfaces and active appearance models to morphable face models.Source: §2 · p. 3
Texture Synthesis
- Patch-based synthesis handles stochastic micro-scale textures but lacks facial-configuration cues for mesoscopic details; prior hallucination methods also report blur, missing regions, or high-resolution artifacts.Facial detail enhancement for pores and wrinkles has been demonstrated in the geometric domain.Source: §2 · p. 3
Deep Learning Inference
- Deep Boltzmann inference addresses facial appearance variation, while encoder-decoder in-painting is limited to very small images; style transfer represents style through mid-layer feature correlations [17].The paper proposes these correlations for high-frequency, multi-scale appearance components including mesoscopic facial details.Source: §2 · p. 3
3. Initial Face Model Fitting
Initial fitting estimates a complete low-frequency face model and extracts visible high-frequency albedo for later analysis. The fitting objective combines image consistency, landmark alignment, and model regularization.
3. Initial Face Model Fitting
- The initial stage fits shape and low-frequency albedo with a PCA-based morphable face model, while estimating pose, camera parameters, and illumination from one unconstrained image.The resulting complete low-frequency map initializes texture synthesis, and visible-region high-frequency albedo feeds texture analysis.Source: §3 · pp. 3–4
- The fitting objective combines rendered-image consistency, 2D landmark alignment, and regularization toward the face-model distributions.The image-consistency, landmark, and regularization weights are 1, 10, and 2.5 × 10^-5, respectively.Source: §3 · pp. 3–4
- A semantic visibility mask restricts image consistency to valid face pixels, improving robustness to occlusion.The mask is computed from a semantic facial segmentation estimated with a two-stream deconvolution network [46].Source: §3 · p. 4
- The fitting objective is optimized with Gauss–Newton based on iteratively reweighted least squares across three image-pyramid levels.The reported convergence takes 30, 10, and 3 steps from coarsest to finest level.Source: §3 · p. 4
- The method removes estimated shading and uses automatic facial segmentation to extract partial high-frequency albedo from visible face regions.The extraction estimates illumination, surface normals, and partial face geometry using methods.Source: §3 · p. 4
4. Texture Analysis
The section represents mesoscopic facial detail with multi-scale CNN feature correlations and infers complete-face detail from a partial view. Database correlations are fitted under matched visibility, then used to define a complete detail target.
4. Texture Analysis
- The fitted coefficients produce complete-texture correlations as the target detail distribution, while processing on Y in YIQ preserves overall color.The correlations are derived from high-resolution texture maps.Source: §4 · p. 4
- Mid-layer feature correlations and later-layer feature responses characterize facial details sufficiently to infer a photorealistic complete texture from a partially visible face.Figure 3 indicates processing through a deep convolutional network.Source: §4 · p. 4, Figure 3 · p. 4
- The desired feature response is estimated from the complete low-frequency albedo because the final CNN layers encode low-frequency appearance.Source: §4 · p. 4
- Database textures are masked to the input's visibility before their correlations are recomputed, enabling comparison with correlations from the partial face.Non-visible pixels are assigned an intensity of 0.5.Source: §4 · p. 5
- Nonnegative blending weights summing to one combine database correlations to fit the correlations observed in the partial input.Figure 4 illustrates blends using different numbers of subjects.Source: Figure 4 · p. 4, §4 · p. 5
- Constraining the blended correlation matrix to the database matrices' convex hull can reduce artifacts when the observed input region is poor quality.Figure 5 illustrates detail preservation for low-quality and noisy input data.Source: §4 · p. 5, Figure 5 · p. 5
5. Texture Synthesis
Texture synthesis optimizes a complete albedo to match estimated high- and low-frequency targets. The method balances these targets and uses back-propagated gradients to update the input texture.
5. Texture Synthesis
- The complete albedo is optimized to match reconstructed correlations in high-frequency-preserving layers and feature responses in low-frequency-preserving layers.Source: §5 · p. 5
- α balances high- and low-frequency effects, and the experiments use α = 2000 to preserve details.The objective is non-convex, with gradients computed for optimization.Source: §5 · p. 5
- Gradients are back-propagated through the network to the input, which is optimized with L-BFGS and initialized from the estimated low-frequency albedo.The Gramian-matrix gradients can be computed analytically.Source: §5 · p. 5
6. Results
The method produces high-resolution facial textures with mesoscopic detail across varied unconstrained images, including challenging lighting and non-frontal views. Comparisons and user studies report richer detail than PCA-based fitting and visual comparability to Light Stage capture.
Results
- Across varied subjects and challenging images, the method produces high-resolution textures with pores, stubble, moles, and complex skin tones beyond the details captured by low-frequency PCA albedo.Renderings use the estimated shape and synthesized texture with Arnold and additional rendering components.Source: §6 · p. 6, Figure 10 · p. 7
Evaluation
- Across texture-synthesis evaluations using CaffeNet, VGG-16, and VGG-19, deeper architectures tend to produce fewer artifacts and higher-quality textures; using fewer than VGG-19’s five mid-layers makes synthesized textures blurrier.Source: §6 · p. 6
Results
- Reconstructions from different viewpoints remain consistent, including detailed freckles in extreme profiles.Figure 9 presents reconstructions from two viewpoints.Source: §6 · p. 6, Figure 9 · p. 6
Comparison
- Our method produces pores and stubble at higher resolution than visio-lization, while both methods produce higher-fidelity textures than linear PCA fitting.The comparison reports random noise patterns in visio-lization.Source: §6 · p. 6, Figure 11 · p. 8
User Study B
- In a crowdsourced comparison, our synthesized textures were visually comparable to Light Stage renderings, and non-technical viewers found photorealism difficult to judge.The study used three subjects and 100 turkers; 56% rated our renderings more realistic than Light Stage renderings.Source: §6 · p. 8, Figure 12 · p. 8
7. Discussion
The method infers high-fidelity albedo textures from one unconstrained image, with plausible appearances and realistic mesoscopic detail. Its detail placement and synthesis depend on limitations in control and initialization.
Discussion
- Mid-layer feature correlations capture high-frequency details and general appearance, while convex combinations help ensure consistent fine-scale details.The method produces textures from a single unconstrained image and reports renderings visually comparable to Light Stage capture.Source: §7 · p. 8
Limitations
- The multi-scale detail representation cannot control the exact appearance or placement of synthesized high-frequency details, so a mole may appear where none exists.Source: §7 · p. 8
- Non-convex final optimization requires good initialization; if PCA albedo estimation misses a goatee, the synthesized texture can lack facial hair.Source: §7 · p. 9
Future Work
- Future work includes high-level controls for specifying mesoscopic detail distributions and synthesizing fine-scale geometry such as wrinkles.Source: §7 · p. 9
Appendix I. Additional Results
Additional evaluations report detailed reconstructions from low-resolution and varied input images, and compare network architectures and synthesis alternatives. Convex combinations produce mesoscopic detail, while PatchMatch can fill unseen regions with repeating or semantically incorrect structures.
Additional Results
- AFW examples and other unconstrained inputs yield high-fidelity textures with pore-level details, including from mostly low-resolution, non-frontal images under challenging lighting.The appendix also shows final renderings from several views.Source: Appendix I · p. 9
Evaluation
- VGG-19 produces the best textures among the tested networks; CaffeNet yields blocky artifacts and VGG-16 is slightly noisy around boundaries.The networks were pre-trained for image classification.Source: Appendix I · p. 9, Figure 14 · p. 9
- From a 108 × 162-pixel input, the method reconstructs a tiny cheek mole alongside complex skin pigmentation.The input was downsampled from a 435 × 652-pixel Light Stage capture.Source: Appendix I · p. 9, Figure 15 · p. 9
Comparison
- Among the tested synthesis alternatives, only convex combinations generate mesoscopic texture details; PatchMatch instead produces repeating structures and semantically wrong fillings.PCA fitting and visio-lization yield lower-frequency textures and less similar faces than ground truth.Source: Appendix I · pp. 9–10, Figure 16 · p. 10, Figure 17 · p. 10
Appendix II. User Study Details
The appendix details two AMT studies: one compares reconstructed faces with ground truth and tests method variants, while the other compares renderings against Light Stage capture. The studies report viewers’ realism judgments and describe relevant sample and task conditions.
User Study A
- The paper reports that its user-study interfaces were deployed on Amazon Mechanical Turk (AMT), and Figure 18 identifies an AMT interface specifically for user study A.Source: Appendix II. User Study Details · p. 10, Figure 18 · p. 10
- The convex-combination variant outperformed the other methods overall, though variants of the proposed method had similar means and medians.Only 65.6% of real images were correctly marked as real, which the authors attribute partly to viewers knowing that half the images were real.Source: Appendix II. User Study Details · pp. 10–11
User Study B
- Study B used three subjects and 100 turkers to rank renderings; after excluding suspected outliers, 57% considered the method more photorealistic than Light Stage capture.The authors report that synthetic details may confuse viewers for subjects with smoother skin.Source: Appendix II. User Study Details · p. 11, Figure 19 · p. 11