Source-linked AI summary
AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"
Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, Stefanos Zafeiriou
TL;DR
Single-image reconstruction methods lacked high-resolution, photorealistic, render-ready faces from in-the-wild images because of limited training data and high-resolution methodology. AvatarMe combines a large facial capture dataset with successive texture and reflectance refinement, producing detailed render-ready reconstructions from arbitrary images while retaining identified limitations for some subjects and input conditions.
Problem
Existing methods lack high-resolution photorealistic 3D face reconstruction from in-the-wild images, partly because training data are scarce and robust very-high-resolution methodologies are limited.
Method
AvatarMe builds on 3D face reconstruction, uses captured facial geometry and reflectance from over 200 subjects, and learns diffuse and specular albedo and normal components.
Results
AvatarMe produces high-quality render-ready reconstructions from arbitrary images, including varied poses, occlusions, black-and-white images, and drawn portraits, while preserving subject identity.
Takeaways & Limitations
The method extends single-image reconstruction toward realistic rendering of detailed faces across unconstrained image types and environments.
Takeaways & Limitations
Performance is weaker for darker skin subjects, some high-frequency pore details are blurred, and reconstruction accuracy depends partly on input photograph quality.
Abstract
from arXiv · showhide
Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce high-resolution photorealistic 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this paper, we introduce AvatarMe, the first method that is able to reconstruct photorealistic 3D faces from a single "in-the-wild" image with an increasing level of detail. To achieve this, we capture a large dataset of facial shape and reflectance and build on a state-of-the-art 3D texture and shape reconstruction method and successively refine its results, while generating the per-pixel diffuse and specular components that are required for realistic rendering. As we demonstrate in a series of qualitative and quantitative experiments, AvatarMe outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image that, for the first time, bridges the uncanny valley.
1. Introduction
Existing methods reconstruct increasingly detailed 3D faces from unconstrained images, but often lack high-frequency, render-ready appearance. AvatarMe addresses this gap by combining high-quality synthesized textures with learned reflectance components and a large facial capture dataset.
- 3DMM methods represent facial shape and texture with PCA bases, limiting high-frequency detail and complex in-the-wild texture reconstruction.
- Deep encoder-decoder methods improve single-image reconstruction, but basic reflectance models, synthetic data, and colored-mesh convolutions limit highly detailed texture and shape.
- GANFIT improves texture and shape quality using high-resolution GAN UV maps and face-recognition features, but its outputs contain baked illumination and lack high-frequency normals and specular reflectance.
- Earlier render-ready approaches depend on low-quality or occluded input textures, nonrepresentative 3D models, or limited data, so they target digital avatars rather than high-quality in-the-wild reconstruction.
- AvatarMe proposes high-quality render-ready reconstruction from arbitrary images by refining GAN-based results with learned diffuse and specular albedo and normals.
2. Related Work
Prior work established high-quality facial reflectance capture and single-image reconstruction, while image-to-image translation enabled high-resolution appearance processing. AvatarMe combines these directions for render-ready facial reconstruction from unconstrained images.
- Facial reflectance capture: Active illumination methods capture separated diffuse and specular albedos and normals, with polarized systems providing detailed reflectance and unpolarized systems reducing capture requirements.
- Proposed pipeline: The proposed pipeline fits a 3DMM, upsamples and de-lights its texture, then infers diffuse and specular albedo and normals before transferring reflectance to a head model.
- Facial reflectance capture: Passive multiview capture has improved facial geometry and appearance estimation, but its acquired data remains lower quality than data from active illumination.
- Facial reflectance capture: AvatarMe uses two active-illumination multiview capture methods to acquire high-quality facial reflectance data for training.
- Image-to-image translation: Image-to-image translation maps inputs into designated target domains and is used here with pix2pixHD variations for de-lighting and extracting high-resolution reflectance maps.
- Single-image reconstruction: Single-image facial reconstruction evolved from PCA-based 3DMMs and CNN regressors toward high-frequency reflectance and displacement inference, but prior methods often remain limited in quality or scope.
3. Training Data
AvatarMe trains on high-resolution facial reflectance and geometry acquired with active illumination. The resulting RealFaceDB covers over 200 individuals, multiple characteristics, and seven expressions.
- Acquisition: The dataset captures pore-level facial reflectance using a polarized LED sphere with 168 lights and nine DSLR cameras.
- Acquisition: Unpolarized LED color-space analysis and multiview capture provide similar-quality unwrapped textures with less than half the captured data and a simpler setup.
- Dataset: RealFaceDB contains faces of over 200 individuals across different ages and characteristics, captured under seven expressions and registered to a standard topology.
- Dataset: The dataset is named RealFaceDB and is described as the largest dataset of this type, with plans for public release.
4. Method
AvatarMe begins with a low-resolution 3D reconstruction and progressively refines its geometry and texture into physically based reflectance components. It separately models diffuse and specular albedo and normals for realistic rendering.
- Reflectance representation: Given a single unconstrained face image, the method infers facial geometry, diffuse albedo, diffuse normals, specular albedo, and specular normals.
- Progressive refinement: The pipeline enhances the reconstructed texture with super-resolution, de-lights it to obtain high-resolution diffuse albedo, and infers the remaining three components from that albedo and base geometry.
- Initial estimation: A 3DMM reconstruction supplies the initial base shape and texture, while GANFIT provides realistic texture UV maps synthesized from a GAN.
- Reflectance representation: Diffuse normals are inferred to support reflection modeling as used in state-of-the-art specular-diffuse separation techniques.
- Progressive refinement: The method obtains shape normals from reconstructed geometry and uses them with enhanced texture to estimate physically based rendering components.
4.2. Super-resolution
AvatarMe super-resolves GANFIT’s texture from 576×384 to 4608×3072 before retopologizing and up-sampling it to 6144×4096.
- 576×384 to 4608×3072: RCAN increases the resolution of GANFIT’s UV maps before retopologization and up-sampling to 6144×4096.The super-resolution network maps 48×48 texture patches to 384×384 patches and is trained on acquired low-resolution textures.
4.3. Diffuse Albedo Extraction by De-lighting
AvatarMe removes baked illumination by simulating GANFIT-like lighting on captured facial data, then training a geometry-aware image-translation network to recover diffuse albedo.
- GANFIT textures contain sharp highlights, shadows, and baked environment illumination that prevent photorealistic rendering.
- The method estimates GANFIT’s point-light directions and environment map, then renders 200 captured subjects under varied simulated illumination.The rendered data retain accurate ground-truth albedo and normals.
- The simulated illuminated textures and corresponding diffuse albedo form a domain-adaptation dataset for training the de-lighting network.
- The network combines illuminated texture with object-space depth or normals because texture-only input causes inference artifacts.Texture channels and depth are normalized to [−1,1] and concatenated into a 4D input tensor.
- 512×512 overlapping patches augment the high-resolution training data and help avoid overfitting.The translation network predicts 3-channel diffuse albedo patches from illuminated texture and geometry.
4.4. Specular Albedo Extraction
AvatarMe predicts specular albedo from diffuse albedo rather than directly from baked-illumination texture, using patch-based translation to model facial reflection structure.
- Specular reflection can be represented using per-pixel specular intensity, whose spatial variation correlates with pores, wrinkles, and hair.
- Directly computing specular albedo from baked-illumination texture is unreliable because environment illumination and occlusion strongly bias the component.
- AvatarMe infers specular albedo from diffuse albedo with a patch-based image-to-image translation network trained on RealFaceDB.
- The network differentiates hair from skin and learns high-frequency variation caused by pore occlusion of specular reflection.
4.5. Specular Normals Extraction
AvatarMe extracts specular normals with a translation network that uses detailed diffuse-albedo structure together with shape-normal information to recover facial high-frequency detail.
- Specular normals contain sharp details such as wrinkles and pores, but their appearance depends on lighting and viewpoint.
- Diffuse albedo is preferred over reconstructed texture because highlights could be misinterpreted as facial features, while albedo retains pore and wrinkle structure.
- The network maps grayscale diffuse albedo concatenated with tangent-space shape normals to specular normals.
- Using shape normals with the detailed diffuse albedo enables generation of both mid- and high-frequency details.
4.6. Diffuse Normals Extraction
Diffuse normals are inferred from grayscale diffuse albedo together with object-space shape normals, then used to enhance reconstructed geometry with plausible detail.
- Diffuse normals capture how scars, wrinkles, and non-skin features alter otherwise uniform skin diffusion.
- Diffuse normals are predicted from the concatenation of grayscale diffuse albedo and object-space shape normals.The mapping produces diffuse normals at 512 × 512 × 3 resolution.
- The inferred normals refine reconstructed geometry by adding plausible surface features through tangent-space integration and displacement embossing.
5. Experiments
Experiments evaluate AvatarMe's staged reconstruction, high-resolution patch processing, rendering quality, robustness to lighting and pose, and limitations in data coverage and input quality.
- 5.1.1 Patch-Based Image-to-image translation: Figure 5 traces reconstruction from base geometry through super resolution and de-lighting to the final rendering.
- 5.1.1 Patch-Based Image-to-image translation: 512 × 512 overlapping patches address hardware limits for training on captured data exceeding 4K resolution.Patches use a stride of 256, creating partial overlap across each UV map.
- 5.1.1 Patch-Based Image-to-image translation: Shape normals are incorporated with texture inputs because texture-only translation cannot exploit geometric detail.
- 5.2. Evaluation: The method outperforms [6] and by a significant margin on PSNR and achieves the highest facial-identity match to input images.Identity comparisons use renderings with reconstructed geometry and reflectance, including eyes.
- 5.2. Evaluation: Qualitative comparisons report artifact-free reconstructions that handle extreme poses, sunglasses, black-and-white images, and drawn portraits.Patch-based inference is reported to produce higher-quality details than methods that infer whole-face maps at lower resolution.
- 5.2. Evaluation: Extracted normals and diffuse and specular albedos remain consistent across input illumination, while renderings support environment relighting with detailed reflections and subsurface scattering.
- 5.2. Evaluation: The method also infers complete head topology and textures using the Universal Head Model.
- 5.3. Limitations: The dataset lacks sufficient examples from certain ethnicities, and results are weaker for darker skin subjects; input quality also affects reconstruction accuracy.High-frequency pore details can be slightly blurred by alignment errors, while well-lit, higher-resolution photographs produce more accurate results.
6. Conclusion
AvatarMe combines high-quality facial capture with image-translation networks to produce rendering-ready reconstructions from varied in-the-wild images while preserving subject identity.
- AvatarMe estimates diffuse and specular albedos and normals from arbitrary in-the-wild face images using a dataset of 200 captured faces.The demonstrated inputs vary in pose and occlusion and include black-and-white images and drawn portraits.
- The resulting reconstructions exhibit high detail and realism while preserving the identity of subjects in the input photographs.