PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple View Neural Regression of a Facial Shape Model

Xiang Li

arXiv:2608.22655v1cs.CV

TL;DR

Creating high-quality 3D facial models is time-consuming and expensive. This work develops a neural approach using synthetic multi-view imagery and a 3D morphable model; reported precision drops by more than 50% near extreme viewing angles.

  • Problem

    Creating high-quality face models takes substantial time, effort, and expense.

    Source: Plain Language Abstract · p. 2

  • Method

    The work develops a neural approach to face-shape regression using multi-view imagery and a 3D Morphable Model.

    Source: §7.1 · p. 86

  • Results

    More than 50%: deep-neural-network precision rate drops approaching θ = 0◦ and θ = 180◦, despite almost 100% accuracy at the center view.

    Source: §6.4 · p. 75

  • Takeaways & Limitations

    Landmarking performance is lower with a wider field of view and improves as focal length increases.

    Source: §6.4 · p. 74

    The training dataset is small and narrow, containing only 48 faces.

    Source: §7.6 · p. 107

Abstract

from arXiv · show

Creating re-topologized 3D facial meshes is essential for high-quality facial animation but remains labor-intensive and time-consuming. This dissertation explores more efficient approaches for capturing production-ready facial meshes through: (1) the development of VarIS, a custom light sphere for capturing high-resolution stereo geometry and reflectance maps; (2) analysis of camera parameters affecting automatic 2D and 3D landmarking; (3) synthetic-data methods for training neural face regression; and (4) techniques for improving neural multi-view face-shape regression. While VarIS enables photorealistic face capture, its operational and processing costs motivate a more scalable approach. A deep learning framework is therefore proposed to directly predict re-topologized facial meshes from synthetic multiview images generated with Visage Craft, an in-house physically based rendering system using an Appearance 3D Morphable Model (A3DMM). The system produces standardized meshes ready for rigging and animation with minimal human supervision. Results show that incorporating accurate camera intrinsics and extrinsics improves landmark accuracy and geometric consistency, while 3D landmark regularization further improves reconstruction quality.

Plain Language Abstract

This dissertation seeks to make realistic 3D face models faster to create and less labor-intensive, combining a custom multiview capture device with an AI pipeline trained on simulated faces. It reports that accurate camera settings improve facial-feature matching and that the project offers a way to produce usable models with little manual work.

Plain Language Abstract

  • Creating high-quality 3D face models currently takes substantial time, effort, and expense.Source: Plain Language Abstract · p. 2
  • VarIS captures a person’s face from different angles to help build detailed 3D models, but processing its output still requires manual work.Source: Plain Language Abstract · p. 2
  • An AI pipeline predicts 3D face shapes directly from images, trained on computer-generated simulations of real faces created with Visage Craft.Source: Plain Language Abstract · p. 2
  • Accurate camera settings improve the AI system’s facial-feature matching, and the project offers a faster way to create clean, usable 3D face models with little manual work.Source: Plain Language Abstract · p. 2

Introduction

The dissertation addresses the labor and precision demands of high-quality facial analysis and production-ready mesh creation. It combines custom capture and synthetic-data tools with camera-parameter analysis and neural multi-view face-shape regression.

Introduction

  • Facial analysis detects faces, identifies landmarks, and aligns faces, while facial modeling aims to preserve anatomical accuracy and surface detail in expressive digital representations.These tasks support evaluation and visually convincing renderings.Source: Introduction · p. 11

1.1 Overview of Facial Analysis and Acquisition

  • Traditional pipelines combine detection, landmark tracking, 3D reconstruction, and retopology, but animation-ready meshes still require skilled human refinement.Real-world backgrounds, expressions, and occlusions complicate high-quality analysis, while neural methods depend on large labeled datasets often collected under controlled conditions.Source: §1.1 · p. 12

1.2 Motivation

  • The dissertation combines traditional graphics techniques and deep learning to develop scalable, production-oriented facial modeling tools and methods.It introduces VarIS for controlled high-fidelity capture, Visage Craft for physically based synthetic face-mesh generation, and a multi-view neural regression framework for animation-ready meshes.Source: §1.2 · p. 13
  • Precise camera intrinsics and extrinsics improve landmark localization and alignment consistency across views, supporting multi-view learning.The dissertation evaluates their effects on landmark localization and 3D synthesis quality.Source: §1.2 · p. 13

1.3 Contributions

  • VarIS captures high-resolution stereo geometry and reflectance maps, while the work studies camera-parameter effects on automatic 2D and 3D landmarking.The capture system supports programmable illumination and records diffuse, specular, and normal components.Source: §1.3 · p. 14
  • The contributions also include synthetic-data methods for neural face regression and multi-view face-shape techniques using visibility-point projection and landmark regularization.These methods target neural multi-view regression of face shape.Source: §1.3 · p. 14

Background Material

This background surveys facial reconstruction and representation, including methods for capturing facial geometry and appearance and the role of retopologized meshes in animation. It also outlines photorealistic rendering techniques, from projecting 3D models into camera images to mesh shading and lighting.

Background Material

  • Facial reconstruction combines recognition, 3D data acquisition, geometry modeling, retopologizing, and texturing, with each stage challenged by facial complexity and computational limits.The resulting methods have applications in film, virtual reality, games, medicine, forensic science, and anthropology.Source: Background Material · p. 15

2.1 Facial Analysis

  • Facial analysis detects and locates landmarks in images or video frames; point-to-point correspondences between keypoints across multiple image pairs can be aligned across images to construct a more complete, detailed 3D facial model.Source: §2.1 · p. 16

2.2 Face Modeling and Animation

  • Detailed geometry and reflectance remain longstanding goals [38], but traditional prosthetic molds are labor- and time-intensive, and 3D representations trade off detail against animation suitability.Polygonal meshes, implicit surfaces, point clouds, and volumetric representations offer different advantages for preserving detail, rendering, or animation.Source: §2.2 · p. 17

2.2.1 Representing a Face in 2D

  • The Eigenface method provided promising results, while ASM used manually annotated facial landmarks to train a statistical face model.Source: §2.2.1 · pp. 17–18

2.2.2 Representing a Face in 3D

  • 3D face geometry uses geometric models that preserve individual detail but struggle to represent general shape, or statistical models that provide retopologized representations of face distributions.Geometry methods recover 3D points by finding correspondences between surface points in paired images.Source: §2.2.2 · p. 18
  • Photometric methods estimate surface normals from reflectance changes under varying illumination, in contrast to methods using projected light patterns for correspondence.Source: §2.2.2 · p. 18

2.2.2.1 3D Face Representation

  • The 3D Morphable Model (3DMM), introduced by Blanz and Vetter, was trained on 200 textured 3D meshes, while fitting methods minimize energies between models and images.Acquiring large facial datasets for 3DMMs remains challenging.Source: §2.2.2.1 · p. 19

2.2.2.2 Geometry Methods for Facial Capturing

  • Active multi-view stereo uses projected illumination patterns for image correspondence, whereas passive stereo avoids illumination and can derive depth maps from dense epipolar correspondences.Passive multi-view images can support precise facial geometry estimation.Source: §2.2.2.2 · pp. 19–20

2.2.2.3 Image-Based Photogrammetry

  • Image-based photogrammetry supplies color information alongside geometry, but self-shadowing, environmental lighting, and occlusion can introduce texture artifacts.A light stage captures facial reflectance under novel lighting; polarized spherical illumination supports acquisition of high-resolution normals, diffuse albedos, and specular textures.Source: §2.2.2.3 · p. 20

2.2.3 Face Animation

  • Facial mesh production must balance high-resolution geometry against resource and rendering constraints, while quad-based retopology remains labor-intensive and can take several hours per person.Neutral faces alone do not capture the range of expressions needed for realistic facial rendering.Source: §2.2.3 · pp. 20–21

2.3 Photorealistic Facial Rendering

  • Photorealistic facial rendering aims to create photo-like images from a retopologized, animatable mesh and image sequences depicting desired expressions.Source: §2.3 · p. 22
  • Model-view-projection transforms place models in world and camera space, project them perspectively into clip coordinates, and use z-buffer testing to determine hidden surfaces.The model matrix encompasses translation, rotation, and scale.Source: §2.3 · p. 22
  • Retopologized quad meshes are tessellated into triangles; texture mapping applies albedo images, and barycentric coordinates interpolate triangle pixel colors.Source: §2.3 · p. 22
  • Early facial rendering back-projected camera RGB colors onto triangle vertices, while Gouraud smooth shading averages shared vertex normals to produce smooth lighting.Source: §2.3 · p. 22
  • Phong lighting combines ambient, diffuse, and specular components, whereas physically based rendering models light-material interactions using a BRDF and microfacet reflection [127].The microfacet model treats rough surfaces as small mirror-like facets and uses normal-distribution, geometry-attenuation, and Fresnel terms.Source: §2.3 · p. 23

Related Work · 3.1.1 Principal Component Analysis · 3.1.2 3D Morphable Model

These sections trace image-based 3D face reconstruction from analysis-by-synthesis and PCA to 3D Morphable Models (3DMMs), then describe PCA and how 3DMMs encode facial shape and appearance. They also note the growing use of deep networks to predict 3D meshes directly from photos.

Related Work

  • Analysis-by-synthesis estimates a 3D face by minimizing the energy between observed images and synthesized images, and 3DMMs are widely used for image-based reconstruction [43].Source: Related Work · p. 24
  • Deep convolutional networks have improved facial synthesis, with the chapter focusing on learning direct mappings from facial photos to 3D face meshes.Source: Related Work · p. 24

3.1 Appearance 3D Morphable Model

  • ASM and AAM used PCA to model and reconstruct shapes from 2D images; 3DMM extended this approach to reconstruct 3D facial shapes.Source: §3.1 · p. 24
  • A 3DMM is a PCA-based statistical parametric framework that represents facial geometry and texture with relatively few parameters and supports face synthesis and related tasks.Source: §3.1 · p. 25, §3.1.2 · p. 26

3.1.1 Principal Component Analysis

  • PCA identifies orthogonal vectors that reduce high-dimensional data to fewer components while retaining essential features and extracting primary variation.Source: §3.1.1 · p. 25
  • PCA eigenvectors span the representation space, and the first k vectors, ordered by eigenvalue, represent primary components used to approximate data points by projection.Source: §3.1.1 · p. 26

3.1.2 3D Morphable Model

  • 3DMM training meshes require consistent topology, with corresponding vertices sharing semantic meaning; shape vectors encode vertex coordinates and texture vectors encode RGB values.Source: §3.1.2 · pp. 26–27
  • PCA yields separate shape and texture eigenvector matrices, whose linear combinations synthesize new faces.Source: §3.1.2 · p. 27

3.1.3 PCA Applications · 3.2.1 Multiview Stereo Reconstruction

The section traces statistical face-shape models from landmark-based approaches to 3D appearance models, then explains Light Stage capture and multiview stereo reconstruction. It describes rectification and pixel matching as steps toward disparity-based depth recovery.

3.1.3 PCA Applications

  • ASM models deformable-face shape variation from landmark-point statistics, while AAM extends the framework to model both shape and appearance.Source: §3.1.3 · p. 27
  • Blanz and Vetter’s 3DMM represents facial geometry and appearance with low-dimensional PCA models trained on 3D scans; BFM and FLAME followed.BFM and FLAME improve expression accuracy, generalization across identities, and reconstructed-detail fidelity.Source: §3.1.3 · p. 27

3.2 Light Stage: Facial Light Field Acquisition

  • Light Stage uses individually controlled dome lights and surrounding cameras to capture subject geometry and reflectance, producing realistic appearance and texture maps.Source: §3.2 · p. 28

3.2.1 Multiview Stereo Reconstruction

  • Multiview stereo derives detailed 3D models from images captured at different viewpoints by matching corresponding pixels and triangulating their 3D positions.Source: §3.2.1 · pp. 28–29

3.2.1.1 Image Rectification

  • Image rectification makes epipolar lines horizontal and parallel, reducing correspondence search from arbitrary tilted lines to a one-dimensional horizontal search along each row.Source: §3.2.1.1 · p. 29

3.2.1.2 Pixel Pairs & Depth Map

  • Normalized Cross-Correlation compares N × N neighboring windows; matched horizontal coordinates yield disparity, which, with camera baseline and focal length, supports depth calculation.Source: §3.2.1.2 · pp. 31–32

3.2.2 Textures Acquisition

The Light Stage captures high-resolution geometry and facial reflectance maps by illuminating the subject from many directions with varied polarization and intensity. Gradient lighting and polarization separation enable recovery of surface normals and diffuse and specular appearance.

3.2.2 Textures Acquisition

  • The Light Stage captures high-resolution geometry and normal, diffuse, and specular reflectance maps under varied spherical illumination.Hundreds of LED sources surround the subject, and illumination direction, polarization, and intensity are systematically varied.Source: §3.2.2 · p. 32
  • Three spherical-gradient lighting conditions aligned with the global X, Y, and Z axes, alongside an ambient image, support per-pixel surface-normal recovery.The gradient components are computed as ratios of directional gradient images to the ambient image, producing a unit normal vector per pixel.Source: §3.2.2 · pp. 32–33
  • Parallel-polarized images retain diffuse and specular reflection, while cross-polarized images largely eliminate specular reflection to isolate diffuse appearance.Polarizers are mounted on both the lights and camera lenses, and two images are captured per lighting condition.Source: §3.2.2 · p. 33
  • Figure 3.7 presents normal, specular, and diffuse albedo maps computed from captured gradient-illumination data at a single viewpoint.The maps demonstrate the appearance outputs obtained through active illumination and polarization.Source: §3.2.2 · p. 34, Figure 3.7 · p. 34

3.2.3 Related Work · 3.3.1 Convolutional Neural Network

The related work traces facial capture from specialized illumination and multiview systems, while the CNN overview explains how convolution and downsampling make image processing more parameter-efficient. The discussion also outlines neural-network training and the progression from image features to compact representations, including for 3D volumes.

3.2.3 Related Work

  • Light Stage systems combine programmable spherical illumination and high-resolution cameras to capture facial reflectance and geometry, while Beeler et al. introduced passive single-shot multiview stereo for detailed geometry.Beeler’s system recovers facial geometry without active illumination.Source: §3.2.3 · p. 34
  • Beeler’s multiview work contributed to Disney’s Medusa capture system, and Ma et al. used gradient-based spherical illumination to extract facial normal maps, specular reflectance, and diffuse albedo.These methods are described as contributing to facial capture and reflectance acquisition.Source: §3.2.3 · p. 34

3.3 Deep Learning in Facial Synthesis

  • A 2018 survey observed growing interest in deep learning for performance and 3D alignment and construction, while backpropagation uses gradient descent to adjust model parameters.The section introduces neural-network training as iterative parameter updates.Source: §3.3 · pp. 35–36

3.3.1 Convolutional Neural Network

  • CNNs address conventional networks’ image-scale parameter burden: a 100 × 100 × 3 image requires 30,000 input-layer weights, motivating an image-focused architecture that reduces training parameters.The stated example illustrates the scalability constraint that motivates CNNs.Source: §3.3.1 · p. 37

3.3.1.1 Convolutional Layers

  • Convolutional filters produce feature maps that capture image patterns from edges to object structures, with deeper CNN layers progressing toward more complex features.Feature maps consolidate patterns, enabling shared learning parameters for each map.Source: §3.3.1.1 · pp. 37–38

3.3.1.2 Downsampling

  • Stride reduces feature-map dimensions, while pooling compresses spatial dimensions by selecting prominent activations; max pooling takes the highest value in each window.These operations reduce computational demands.Source: Figure 3.12 · p. 38, §3.3.1.2 · p. 39

3.3.1.3 Typical Convolutional Neural Network Architectures

  • Typical CNNs stack convolution and pooling layers to distill images into compact feature maps, then flatten those maps for fully connected layers or further classification or regression.The described architecture reduces spatial size while preserving essential feature information.Source: §3.3.1.3 · p. 40, Figure 3.14 · p. 41
  • CNNs also process 3D volumetric data with three-dimensional filters that recognize spatial patterns and structural shapes beyond standard 2D image analysis.The operation produces a smaller volume that retains essential structural information.Source: §3.3.1.3 · p. 40, Figure 3.15 · p. 41

3.3.2 Classic CNN Models

Classic CNNs progressively extract image features through downsampling, while later architectures address deeper-network training and preserve or reconstruct spatial information through skip connections and encoder-decoder designs.

Classic CNN Models

  • LeNet-5 supported document recognition, AlexNet improved image-classification accuracy, and VGG-16 [122] used 3 × 3 filters and max-pooling to enable deeper models.The passage describes these architectures as pioneering advances that helped establish contemporary deep-learning methods.Source: §3.3.2 · p. 41
  • ResNet [61] addressed vanishing-gradient instability as networks grew deeper by using residual connections to forward earlier-layer outputs and stabilize training.Source: §3.3.2 · p. 42
  • Autoencoders form an hourglass by compressing images into compact latent representations and upsampling them back to their original dimensions.Source: §3.3.2 · p. 42
  • U-Net preserves fine spatial information by passing encoder-stage feature maps directly to corresponding decoder stages through skip connections.Source: §3.3.2 · p. 42

3.3.3 Facial Alignment and Landmarking in Deep Learning · 3.3.4 Facial Synthesis in Deep Learning

The sections review deep-learning approaches to facial landmarking and synthesis, outlining methods for predicting 3D facial structure from images and limitations in data requirements, representation quality, and production readiness.

3.3.3 Facial Alignment and Landmarking in Deep Learning

  • Accurate facial landmarks identify key features such as the eyes, nose, lips, and facial contour, and are crucial in facial recognition and synthesis.Source: §3.3.3 · p. 42
  • Single-image 3D facial-geometry prediction is hindered by head-pose variation, occlusion, image resolution, and illumination, particularly for convolutional neural networks.Source: §3.3.3 · p. 43
  • Feng et al. regress 2D images to UV positional maps that represent full 3D facial geometry and provide dense correspondence among faces [47].Source: §3.3.3 · p. 43
  • 3DFAN combines 2D-to-3D alignment with stacked heatmaps to predict depth and 2D landmarks, while 3DSTN estimates camera projection and handles occluded regions.Source: §3.3.3 · p. 43

3.3.4 Facial Synthesis in Deep Learning

  • Lombardi et al. predict facial appearance and mesh, combine multi-view images into a view-independent average texture, and use an autoencoder to model nonlinear texture behavior [89].Source: §3.3.4 · pp. 43–44
  • The autoencoder approach improves reconstructed-face accuracy but requires millions of images of one subject captured in a controlled environment for training.Source: §3.3.4 · p. 44
  • A multiview-supervised 3D-volume representation avoids triangulated meshes, but voxel-grid resolution constrains rendering quality; warp fields indexing an RGBα volume produce more detailed rendering [90].Source: §3.3.4 · p. 44
  • ToFu and Tempeh fuse features from raw multiview stereo scans to estimate mesh points, but their noisy triangle meshes require wrapping before rigging and production [15].Source: §3.3.4 · p. 44

3D Mesh Acquisition

This section surveys facial mesh acquisition for production-ready animatable models, covering conventional capture and 2D facial-analysis models alongside deep-learning approaches such as direct retopologized-mesh prediction.

3D Mesh Acquisition

  • Automation is sought to produce animatable meshes compatible across applications while scaling to the large data volumes needed for sophisticated models.The section frames scalability and adaptability across use cases as central motivations.Source: 3D Mesh Acquisition · p. 45

4.1 History & Traditional Pipeline for 3D Mesh

  • The conventional pipeline combines light-stage capture, stereo geometry reconstruction, mesh alignment and registration, texture mapping, and rendering; retopology fits anatomical quad meshes to dense scans for animation.The topology follows facial muscles and anatomical landmarks to support convincing deformations.Source: §4.1 · p. 47, Figure 4.1 · p. 47
  • The conventional process is labor-intensive: subjects must remain still during scanning, and artists manually refine topology and textures, requiring technical skill and time.Its manual complexity also creates challenges for resource allocation.Source: §4.1 · p. 48

4.2 3D Mesh Acquisition Models

  • For 2D preprocessing, Viola-Jones detects faces, alignment compensates for pose and view variation, and ASM learns corresponding landmarks from pre-annotated images.The section identifies face detection, landmarking, alignment, geometry acquisition, and retopology as subtasks in producing a re-topologized mesh.Source: §4.2 · pp. 48–49
  • AAM extends landmark-based shape analysis with texture for a more comprehensive 2D face representation; these models localize, landmark, and reconstruct facial features in images.The section presents 2D face analysis as groundwork for 3D facial synthesis.Source: §4.2 · p. 49
  • Multiview stereo reconstructs facial geometry by capturing images from different viewpoints, matching features, acquiring depth maps, back-projecting point clouds, and converting them to polygonal meshes.The resulting meshes have reduced point density that is more suitable for rigging and animation.Source: §4.2 · p. 49
  • Production-ready facial capture requires geometry, diffuse texture, and normal texture; controlled programmable lighting can capture reflectance fields for realistic facial models.The lighting environment can vary color, intensity, and direction to reproduce real-world conditions.Source: §4.2 · pp. 49–50
  • Supervised neural methods require enormous labeled datasets, and facial acquisition methods rely on large datasets of real individuals or datasets constructed from statistical models.The cited discussion notes that real individuals are captured under controlled environments.Source: §4.2 · p. 50

4.3 Issue in Facial Mesh Acquisition

  • Deep learning can directly convert face photographs into cohesive 3D retopologized meshes, avoiding intermediate subprocesses and improving workflow efficiency.The section presents this as an end-to-end alternative to the traditionally labor-intensive production pipeline.Source: §4.3 · p. 51
  • VAE approaches decode latent representations either to rendered volumes [90], which have production-use limitations, or to customizable common retopologized meshes [89].The common mesh representation gives artists flexibility to customize facial geometry.Source: §4.3 · p. 51
  • Neural inference can predict within milliseconds [89], versus seconds per image for statistical approaches [26], enabling real-time feedback and live-video facial avatars.The passage attributes this speed to GPU parallel processing.Source: §4.3 · p. 52

Facial Data Acquisition Systems

This section presents systems for facial data acquisition and mesh production, including VarIS for controlled capture of facial geometry and reflectance and 3DMM tools for producing animatable meshes. It also describes a microfacet-based physically based rendering model and Visage Craft for generating synthetic multiview facial data with known camera parameters and consistent mesh topology.

Facial Data Acquisition Systems

  • Creating high-quality digital faces is complex and labor-intensive, involving detection, landmarking, geometry acquisition, alignment, correspondence, reflectance capture, and retopology, often with substantial human artistic input.High-fidelity capture also requires precise control of lighting, geometry, and material properties, as well as specialized setups and processing.Source: Facial Data Acquisition Systems · p. 53

5.1 Variable Illumination Sphere

  • VarIS is a multi-purpose Light Stage designed to acquire and process high-fidelity facial geometry and appearance data for graphics research and production.It captures detailed geometry and reflectance information under controlled illumination.Source: Facial Data Acquisition Systems · p. 53, §5.1 · p. 54

5.1.1 Hardware Setup

  • The VarIS hardware uses an 8-foot geodesic dome carrying 364 modular lighting units, each with three RGBW LED arrays and Fresnel lens collimation.The system uses distributed serial control and the ESTA DMX512A standard for lighting.Source: §5.1.1 · pp. 54–55

5.1.2 Software & Application

  • VarIS software provides a camera-control interface for focus, exposure, subject positioning, and material viewpoints, and supports gradient-illumination and polarization capture.Four sequential photographs—X-, Y-, and Z-axis gradients followed by uniform cross-polarized illumination—are used to derive normal, specular, and diffuse texture maps.Source: §5.1.2 · p. 56, Figure 5.3 · p. 56

5.2 Visage Craft for Synthetic Data

  • Public 3DMMs can lack the resolution or flexibility required for specialized applications, motivating a high-resolution Appearance 3D Morphable Model tailored to research and production needs.3DMMs represent facial variation and support shape and texture synthesis as well as reconstruction from partial data.Source: §5.2 · p. 57

5.2.1 Model Building Preparation

  • The A3DMM preparation began with 48 same-topology face meshes, each accompanied by diffuse, normal, and gloss maps.Meshes were rigidly aligned using Procrustes analysis without scaling, then PCA was applied separately to geometry and texture maps.Source: §5.2.1 · p. 57
  • The model-building workflow uses a microfacet-based BRDF combining diffuse and specular reflection, with the Cook-Torrance model for specular reflection.Fresnel reflectance is accounted for when computing the diffuse component.Source: §5.2.1 · pp. 57–58

5.2.2 Visage Craft

  • The Visage Craft GUI exposes the top five PCA components for mesh geometry and diffuse textures; varying sliders within ±3√λi covers approximately 98% of training-set variance.The sliders let users observe each component’s effect on facial shape and texture appearance.Source: §5.2.2 · p. 60
  • Visage Craft’s mesh model is adaptable for specialized research, and its Appearance 3D Morphable Model is constructed from 48 high-quality facial scans.It can synthesize random face samples within the standard-deviation range and be customized for different applications.Source: §5.2.2 · p. 61
  • Visage Craft generates synthetic multiview face images with virtual cameras whose intrinsic and extrinsic parameters are precisely known.The proprietary 3DMM enables cameras to be positioned anywhere; Figure 5.6 shows six stereo pairs with known camera parameters.Source: §5.2.2 · p. 61, Figure 5.6 · p. 61
  • Shared mesh topology establishes geometric correspondences that support facial landmark extraction, registration, and region masking.These correspondences also facilitate isolating and masking facial regions.Source: Figure 5.6 · p. 61, §5.2.2 · pp. 61–62

Importance of Camera Intrinsic

This section evaluates how camera focal length, position, and viewing angle affect facial landmarking, using controlled conditions, ground-truth measurements, and comparisons among landmark methods.

and Extrinsic Matrices

  • The study compares traditional and neural landmarking methods across focal lengths and viewing angles using photorealistic synthetic images with ground-truth landmarks.The images vary camera parameters to assess general performance and performance across focal length and viewing angle.Source: and Extrinsic Matrices · p. 63

6.1 Introduction

  • The review examines five notable methods because prior comparisons generally omitted lens-perspective effects and systematic evaluation across viewing angles.The study focuses on focal length and viewing angle, conditions that earlier datasets and comparisons often did not cover.Source: §6.1 · p. 64

6.2.1 Landmark Schemes

  • The evaluation chose the 68-point scheme from Multi-PIE, popularized by iBUG’s 300W, included seven landmarks (four eye corners, one nose tip, two mouth corners), and provides 3D ground truth for 84 participant faces.The 84 comprise 64 FACS-capture expressions from two individuals and 20 unique individuals; jawline points are evenly distributed along the chin, and eyebrow points are marked anatomically at the supraorbital ridge.Source: §6.2.1 · pp. 65–66

6.2.2 Evaluation Metrics

  • Performance is measured with RMSE and NRMSE, normalizing landmark error by head bounding-box width to accommodate perspective changes at side angles.The evaluation also reports the percentage of accepted landmarks based on whether normalized error falls below a threshold.Source: §6.2.2 · pp. 66–67

6.2.3 Camera Position & Focal Length

  • The camera setup samples 49 spherical-coordinate positions and six common focal lengths: 24mm, 28mm, 35mm, 50mm, 85mm, and 135mm.Polar angles span 45°–135° in 15° increments, and azimuthal angles span 180°–0° in 30° intervals. The focal-length range excludes extreme wide-angle and telephoto lenses.Source: §6.2.3 · pp. 68, 70
  • Short focal lengths widen the field of view and distort facial perspective, while longer focal lengths approach orthographic projection and better preserve relative landmark distances.To maintain a face at a similar image size across focal lengths, camera distance must change; effects may be more pronounced near frame edges, which were not tested.Source: §6.2.3 · p. 69

6.2.4 Face Landmark & Alignment Methods

  • The comparison tests Holistic and Patch Active Appearance Models alongside PRNet [47], 3D-FAN [20], and 3DSTN [11].The authors selected three recent high-performing deep-learning methods with available implementations, noting that the comparison could not include all recent methods.Source: §6.2.4 · p. 71
  • The evaluated deep-learning methods were trained on 300W-LP, except 3D-FAN, which used 230,000 + 300W-LP.The selection was based on reported performance, and the authors expected other recent methods to perform similarly on the same datasets.Source: §6.2.4 · p. 71

6.3 Procedure

  • Landmark RMSE and NRMSE measurements use manually annotated 3D ground-truth landmarks, projected into 2D with the same camera matrices used to render each view and focal length.Faces were rendered at 49 camera locations and six synthesized focal lengths, producing images for each face across all view-and-focal-length combinations.Source: §6.3 · pp. 72–73, Figure 6.6 · p. 72
  • The evaluation manually annotated 3D face captures, rasterized each face at 49 angles and 6 focal lengths, and compared predicted 2D landmarks using NRMSE.Source: §6.3 · p. 73

6.4 Results and Discussion

  • CNN methods generally outperform traditional statistical methods, but differing training data prevent declaring one method best overall.3D-FAN used an augmented dataset, while the other methods used publicly available pretrained networks.Source: §6.4 · p. 74, §6.5 · p. 76
  • Wider fields of view reduce landmarking performance, which improves as focal length increases before leveling off.The conclusion attributes the improvement to reduced projection distortion.Source: §6.4 · p. 74, §6.5 · p. 76
  • Landmarks near the face center are more accurate than facial-edge landmarks, especially around eye and lip corners, due to projective distortion.This pattern is consistent with the reported focal-length effects.Source: §6.4 · pp. 74–75
  • Landmark prediction is generally better on neutral faces than on FACS faces, whose greater facial expression increases prediction difficulty.Source: §6.4 · p. 75

6.5 Conclusion

  • Future evaluations and training data should account for camera and lens parameters, including controlled variation in viewing angles and focal length.The conclusion notes that metadata about capture lenses is often removed from large image datasets.Source: §6.5 · p. 78

Multiple View Neural Regression

The section develops a neural approach for predicting riggable, retopologized 3D facial meshes from synthetic multiview data using a 3D Morphable Model. It addresses mesh-creation labor and cross-view correspondence through a method that establishes correspondences and refines predictions in stages.

of Facial Shape Model

  • Creating production-ready retopologized face meshes requires substantial manual effort and time, motivating an automated approach.The introduction also notes that active capture often requires specialized hardware and passive multiview capture can be time-consuming.Source: of Facial Shape Model · p. 83, §7.1 · p. 84

7.1 Introduction

  • Single-view regression can degrade on non-frontal faces amid pose, occlusion, lighting, and depth ambiguity, while multiview inputs provide extra information for more accurate geometry inference.Learning across views still faces difficulty establishing accurate feature correspondences.Source: §7.1 · p. 85
  • The proposed pipeline learns retopologized 3D facial meshes from multiview images and investigates training neural regression with fully synthetic data.Its target is a production-ready, riggable face mesh.Source: of Facial Shape Model · p. 83, §7.1 · pp. 85–86
  • Projecting mesh points as preprocessing supplies supervised ground truth, while point samples in the loss establish one-to-one correspondences across views.The approach is described as adaptable to other neural-network architectures.Source: of Facial Shape Model · p. 83, §7.1 · p. 86
  • Visage Craft generates synthetic multiview facial data with camera parameters using an A3DMM built from a small collection of facial scans and physically based shading.The contribution describes the rendered data as photorealistic.Source: §7.1 · p. 86
  • Controllable synthetic data supports consistent retopologized meshes and lets datasets be customized to training requirements.The controlled data also allows manual landmarks and dense point-to-point correspondences to be created and integrated into training.Source: §7.1 · p. 86

7.2 Related Work

  • Single-view inputs lack depth, while multiview methods face cross-view correspondence challenges and volumetric approaches can produce noisy meshes requiring wrapping before rigging.The cited limitations motivate learning correspondences from synthetic multiview data and predicting retopologized meshes directly.Source: §7.2 · p. 88

7.3 Method

  • The network predicts feature maps and approximate landmarks, aligns a sampling grid using a rigid transformation, then refines each vertex with locally sampled features.The three stages are global stage one, global stage two, and final refinement.Source: §7.3 · pp. 88–89
  • Visage Craft generates photorealistic multiview training data using physically based shading and a scan-built Appearance 3D Morphable Model with known camera parameters.Because the training data is retopologized, it provides accurate landmark correspondences; visibility masks identify non-occluded regions for dense correspondences across views.Source: §7.3 · p. 89

7.3.1 Training Data

The section builds a synthetic multi-view training dataset to reduce reliance on costly, controlled capture of individual faces. It combines a PCA-based model of facial shape and appearance with controllable face and camera sampling and physically based rendering.

7.3.1 Training Data

  • The 3DMM integrates facial geometry with diffuse, normal, and gloss maps, using separate PCA analyses of 48 retopologized meshes and their corresponding maps.The resulting eigenvectors and eigenvalues are used to build the model and generate training data.Source: §7.3.1 · p. 90
  • Face images are rendered with Cook-Torrance BRDF shading, using Lambertian diffuse, GGX specular distribution, Schlick-Beckmann geometry shadowing, and Schlick Fresnel approximation.Source: §7.3.1 · p. 90
  • Visage Craft samples each principal component within ±3 standard deviations, covering approximately 98% of potential shape variations, and combines normal and uniform samples to include atypical geometries.The passage attributes improved generalization capabilities to augmenting the training set with rarely observed shapes.Source: §7.3.1 · pp. 90–91
  • Synthetic rendering provides precise camera control through six stereo-camera pairs, producing 12 viewpoints with known intrinsic and extrinsic matrices for each face.The generated dataset is partitioned into training and validation subsets.Source: §7.3.1 · pp. 90–91

7.3.2 Neural Network Architecture

The architecture adapts TEMPEH to regress retopologized facial meshes from multiview features, using head-shaped sampling, landmark-based alignment, visibility-aware feature fusion, and mesh refinement and regularization.

Neural Network Architecture

  • The three-stage pipeline extracts shared multiview features, predicts a rough mesh, realigns sampling using landmarks, then refines vertices with an 8-by-8 grid.A ResNet34-based U-Net produces per-image feature maps; aggregated sampled features feed a 3D U-Net for the initial mesh prediction.Source: §7.3.2 · p. 92

Sample Grid Generation

  • A dataset-derived nonuniform grid fits head proportions more closely than a cube, avoiding unused volume and improving sampling efficiency.Its dimensions are based on the bounding box encompassing the training samples.Source: §7.3.2 · p. 93

Rigid Transformation on Landmarks

  • Explicit landmarks from the initial mesh are aligned to ground-truth mesh landmarks to compute the rigid grid transformation, avoiding reliance on landmark detectors.The method selects 11 landmarks, including eye and mouth corners, lip midpoints, earlobe junctions, and the philtrum.Source: §7.3.2 · p. 94

View Feature Vectors and Fusion

  • Precomputed visibility masks weight feature aggregation across views; consistent retopology and known cameras enable per-view visibility to be computed with Z-buffering.The masks are generated for each viewpoint using meshes and camera matrices, avoiding additional visibility computations during training.Source: §7.3.2 · p. 95

Loss Functions

  • A landmark-based loss complements point-to-point mesh supervision, encouraging rigid alignment to ground-truth landmarks and improving reconstruction accuracy and alignment precision.The point loss is mean squared error between predicted and corresponding ground-truth mesh vertices.Source: §7.3.2 · p. 96

7.3.2 Neural Network Architecture

  • Training combines vertex-position and landmark-position losses with weights controlling their relative importance; landmark loss is disabled in Global Stage 1 and refinement, but enabled in Global Stage 2.The landmark-loss weight is 0 in Global Stage 1 and refinement, and 1 in Global Stage 2.Source: §7.3.2 · p. 97
  • The TEMPEH-inspired network uses a head-shaped grid to reduce wasted space and landmarks for grid alignment and geometric regularization.This replaces uniform cubic sampling and uses landmarks both to align the sampling grid with the facial region and within the loss function.Source: §7.3.2 · p. 97
  • Visibility masks and view-specific feature vectors come from the predicted coarse meshes, enabling more targeted refinement in later stages.Source: §7.3.2 · p. 97

7.3.3 Testing Data

This section evaluates synthetic-data training for facial mesh synthesis using Metahuman renderings of real scan identities, with attention to production readiness and the effects of bounding-box sampling and landmark-based methods. It also describes limits imposed by training-data quantity and diversity and frames synthetic training data as a lower-cost route to training neural face models.

7.3.3 Testing Data

  • The test set uses distinct 3D Scan Store facial scans converted into Metahuman models, rendered with Visage Craft camera settings and Scan Store maps.Additional facial details were applied in Metahuman Creator, and wrinkle maps were refined for realism.Source: §7.3.3 · p. 97, Figure 7.7 · p. 98
  • Because test meshes with matching topology are unavailable, 11 landmarks were manually annotated across 12 views and reconstructed by SVD for Global Stage 2 alignment.Only landmarks visible in each view were recorded; each view belongs to a stereo pair.Source: §7.3.3 · pp. 98–99

7.4 Implementation Details

  • Training uses 48 meshes to build an Appearance Morphable Model and generate 10,000 synthetic faces, split into 7,500 training and 2,500 validation meshes.Half the synthetic faces are sampled within ±3 standard deviations of a normal distribution and half uniformly; meshes contain 12,466 points.Source: §7.4 · p. 99

7.5 Evaluation

  • Predictions retain standardized one-to-one topology without manual cleanup, while landmark-estimated transforms center meshes and preserve real-world head dimensions.The sample grid is computed from the training-set bounding box.Source: §7.5 · p. 101
  • Visual comparisons show that the method’s output closely matches test facial structure and identity, with strong agreement against ground-truth meshes.The comparison presents the method’s output, ground truth, and TEMPEH result side by side.Source: §7.5 · p. 101, Figure 7.10 · p. 102
  • Among the tested neural architectures, TEMPEH produced the most stable performance and best overall results.The study’s primary objective is feasibility of production-ready retopologized meshes from synthetic training data, rather than direct competition with facial-synthesis networks.Source: §7.5 · p. 103
  • Facial-region reconstruction errors remain under 5 mm, although TEMPEH is more accurate in high-frequency areas such as wrinkles and ear contours.The reported lower-head discrepancies stem primarily from differences in the compared meshes.Source: §7.5 · p. 103
  • The bounding-box grid more closely matches the subject’s facial structure and ground truth, whereas the cubic grid produces visibly distorted, “chubby” geometry.The comparison used identical models at the Coarse 1 stage.Source: §7.5 · p. 105, Figure 7.12 · p. 105
  • Adding landmark weights to the point-synthesis loss can create dents and spikes around landmarks, so the final loss instead uses a separate landmark regularization term.The two strategies are weighting landmark points in the synthesis loss or adding a separate landmark loss term.Source: §7.5 · pp. 105–106
  • Landmark-based rigid alignment improves the predicted meshes’ positional correspondence with ground truth, indicating greater spatial consistency.The alignment uses landmarks from Global Stage 1 matched to corresponding ground-truth landmarks.Source: §7.5 · p. 105, Figure 7.13 · p. 106, Figure 7.14 · p. 106

7.6 Limitations

  • The training set’s 48 faces, with identical 12,466-vertex topology and 1024 × 1024 textures, limit variation learning and lead to overly smooth meshes that regress toward the mean.The small sample makes it difficult to learn the full range of real facial variation, including high-frequency details such as wrinkles and skin texture.Source: §7.6 · p. 107
  • Because only one or two of the 48 textures represent darker skin tones, the model poorly synthesizes faces with very light or very dark tones.The authors identify insufficient diversity in textures, skin tones, and facial features as the underlying limitation.Source: §7.6 · p. 107

7.7 Conclusion and Discussion

  • The pipeline uses a small set of publicly available retopologized meshes to build an Appearance 3D Morphable Model and generate large-scale synthetic training data, reducing budget requirements for smaller studios and research labs.This offers an alternative to pipelines requiring expensive capture equipment, substantial human involvement, and costly data acquisition.Source: §7.7 · pp. 107–108
  • Future work may improve Visage Craft by building a more comprehensive 3DMM: it currently incorporates only 48 faces, limiting representation of facial-structure variability, and additional facial scans could expand the dataset.Source: §7.7 · p. 108

Conclusions and Discussion

The dissertation bridges traditional facial capture and deep learning with a multi-view neural regression framework for producing high-fidelity, animation-ready facial meshes. It discusses the work’s contributions and identifies ways to better connect synthetic training data with real-world capture and expressive avatars.

8.1 Discussion of Dissertation Work

  • VarIS uses variable lighting and stereo reconstruction to produce high-quality facial meshes, while conventional capture and retopology workflows remain costly in time, effort, and resources.These bottlenecks motivate deep learning approaches that directly synthesize detailed facial meshes from images.Source: §8.1 · p. 109
  • Visage Craft addresses the lack of extensible, physically accurate 3DMM training data by supporting high-fidelity synthetic datasets and known camera, lighting, and mesh-correspondence parameters.The platform is presented as a foundation for training and validating neural networks.Source: §8.1 · p. 110
  • Experiments found that neglecting camera calibration can cause significant multi-view landmark prediction errors, underscoring the importance of accurate camera modeling for synthesis and geometric alignment.The study investigated both camera intrinsics and extrinsics.Source: §8.1 · p. 110
  • The three-stage multi-view neural regression framework produced topologically consistent, animation-ready meshes from synthetic imagery, supporting inference-driven alternatives to traditional mesh-creation pipelines.The dissertation reports improved production efficiency while preserving geometric fidelity and photorealism.Source: §8.1 · p. 110

8.2 Suggested Directions of Research

  • Integrating VarIS camera and lighting configurations into Visage Craft could align synthetic meshes with physical capture and enable a closed-loop path from controlled image acquisition to real-world deployment.The proposed pipeline would use VarIS capture and models trained on its synthetic analogs.Source: §8.2 · p. 111
  • Because Visage Craft currently generates static face meshes, future work could use Metahuman to create multi-view datasets with dynamic expressions, improving trained networks’ expressiveness and generalization.The dissertation identifies dynamic-expression data as important for advancing toward fully animatable avatars.Source: §8.2 · p. 111

Related papers

Loading 2608.22655v1…