Source-linked AI summary
Deep Appearance Models for Face Rendering
Stephen Lombardi, Jason Saragih, Tomas Simon, Yaser Sheikh
TL;DR
Real-time facial rendering requires realism despite costly, precision-dependent traditional graphics pipelines and difficult sensor correspondence. The paper learns a deep joint model of facial geometry and view-dependent appearance, then maps headset images into its latent facial states. It reports realistic renderings suitable for real-time VR, while acknowledging limited relighting and broader-body modeling challenges.
Problem
Traditional facial rendering depends on accurate geometry, shading components, and substantial computation, while VR headset sensors provide incomplete or differing-modality views that complicate correspondence.
Method
A conditional variational autoencoder jointly models facial geometry and view-dependent texture, while a shared latent representation maps headset images to facial states without manual correspondences.
Results
The model enables photo-realistic real-time facial rendering from multiview image data and captures subtle motion and complex reflectance behavior, with some artifacts such as mouth blurring.
Takeaways & Limitations
Dynamic textures with flat lighting and sparse geometry make the representation suitable for existing real-time rendering engines and interactive VR avatars.
Takeaways & Limitations
The current approach has limited ability to relight, and extending it to full-body avatars introduces clothing, limb articulation, and inter-person appearance challenges.
Abstract
from arXiv · showhide
We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).
1 INTRODUCTION
The paper replaces precision-dependent facial rendering with a deep, data-driven model that jointly represents geometry and view-dependent appearance. Its conditional latent representation and unsupervised image-to-state mapping target realistic, real-time VR avatars.
- Motivation: Traditional rendering requires accurate geometry and shading while remaining computationally expensive, making realistic dynamic facial rendering difficult in real time.Facial artifacts are especially consequential because humans are highly sensitive to errors in rendered faces.
- Deep Appearance Model: Inspired by Active Appearance Models, the method jointly models sparse 3D geometry and view-dependent texture with a deep conditional variational autoencoder.The decoder generates rendering elements while conditioning texture synthesis on viewing direction.
- Deep Appearance Model: View-dependent texture models nonlinear effects such as specularity and compensates for limitations in sparse or biased geometry.The approach therefore avoids requiring high-precision rendering elements.
- VR Driving: A shared latent representation maps headset sensor images to facial states despite differing viewpoints and modalities, without manually defined correspondences.This mapping can be evaluated in real time for interactive VR use.
- Contributions: The paper contributes a realistic deep appearance model and a semi-supervised technique for relating its latent codes to sensor images.The stated target settings include real-time interactive applications such as VR.
2 RELATED WORK
The related work spans statistical face models, deep image generators, image-based avatars, view-dependent appearance representations, and cross-domain facial-state regression. The paper combines these directions into a rendering-focused deep appearance model.
- Statistical Face Models: AAMs and 3D morphable models register faces through analysis-by-synthesis in a low-dimensional linear subspace of shape and appearance.The paper repurposes the generator component of such statistical models for rendering.
- Deep Face Models: Deep face models use GANs, VAEs, or semi-automatic disentanglement to represent facial attributes such as expression, pose, lighting, and shape.These methods primarily operate on image representations rather than the paper’s joint geometry-and-rendering formulation.
- Image-Based Avatars: Image-based avatar methods blend captured images or use geometric facial models to reproduce expression- and viewpoint-dependent appearance.The paper is designed to exploit larger quantities of high-quality data for facial depiction.
- View-Dependent Appearance: Light-field, image-based rendering, and Bidirectional Texture Function methods represent spatially or view-varying appearance using sampled or parameterized representations.These approaches motivate the paper’s treatment of facial appearance as view dependent.
- Cross-Domain Driving: Prior face-registration methods commonly assume complete facial views and explicit correspondences, whereas this work learns a direct mapping across differing sensor modalities.The paper also relates to regression from images to blendshape coefficients and unsupervised domain adaptation.
3 CAPTURING FACIAL DATA
The capture system combines synchronized multiview video, dense lighting, standardized facial performances, and tracked personalized meshes. These data provide the images, textures, and geometry used to train the appearance model.
- Capture Apparatus: The capture apparatus uses 40 synchronized machine-vision cameras recording 5120×3840 images at 30 frames per second from the frontal hemisphere.The cameras are positioned about one meter from the face.
- Training Data Preparation: The pipeline unwraps camera images into view-specific texture maps, averages them across cameras per frame, and feeds them with the mesh into a variational autoencoder.The decoder reconstructs mesh and texture conditioned on the output viewpoint.
- Capture Apparatus: The cameras use 50mm lenses that capture pore-level detail, with each pixel covering about 50µm on the face.
- Capture Apparatus: Two hundred directional LED point lights are arranged around the face to promote uniform illumination during capture.
- Performance Data: Each subject performs 122 predefined facial expressions and 50 phonetically balanced sentences to keep expression distributions consistent across identities.Semantic metadata from the recordings is not used in this work.
- Tracking and Preparation: Personalized blendshape models track the face through speech performances by matching captured images with dense 3D reconstructions.The resulting tracked facial mesh is paired with the original images as model input.
4 BUILDING A DATA-DRIVEN AVATAR
The system learns a joint, data-driven representation of facial geometry and view-dependent appearance from multiview capture. A conditional variational autoencoder decodes facial state and viewpoint into renderable geometry and texture for real-time avatar rendering.
- Model construction: The model jointly encodes facial geometry and appearance in a variational autoencoder learned from multiview facial capture.It combines 3D mesh vertices with texture information in a shared latent representation.
- Viewpoint conditioning: View-dependent texture synthesis models effects such as specularity, imperfect geometry, occlusion, and missing facial data.The decoder is conditioned on the viewer’s viewpoint to generate the appropriate texture for rendering.
- Latent facial state: The latent vector z_t provides a low-dimensional, data-driven representation of facial state, including eye gaze, mouth pose, and tongue expression.The encoder samples this latent representation and passes it to the decoder for reconstruction.
- Decoding and rendering: The decoder transforms facial codes and viewpoint vectors into reconstructed mesh vertices and textures that can be rendered with camera-pose information.The mesh is designed to remain independent of viewpoint, while texture depends on the conditioning viewpoint.
- Real-time rendering: 5 milliseconds is the reported decoder runtime on an NVIDIA GeForce GTX 1080 GPU, within the 11.1-millisecond target for real-time rendering.In VR, decoding twice creates one texture per eye and can improve depth impressions in regions poorly approximated by sparse geometry.
- Identity conditioning: The system produces person-specific avatars, while identity conditioning is proposed as a way to model multiple people with one encoder and decoder.The authors report that identity conditioning can yield an identity-independent facial-expression representation in the latent code.
5 DRIVING A DATA-DRIVEN AVATAR
The system drives person-specific avatars from headset images by translating headset observations into the rendering model’s latent space. It uses image-based rendering, modality adaptation, and latent optimization to produce real-time avatar geometry and texture.
- Video-driven Animation: Synthetic headset images are generated from multi-camera data by raycasting tracked geometry and projecting points into captured images.This produces headset-perspective images from the multi-camera capture system.
- Video-driven Animation: The method addresses lighting mismatch between headset and multi-camera images, which makes direct regression from synthetic images to rendering codes unlikely to generalize to real headset images.The alternative trains on both real and synthetic headset images and relies on a common latent representation.
- Video-driven Animation: A shared VAE encodes real and synthetic headset images, while a binary decoder condition separates modalities so the latent code need not contain modality-specific information.A linear transformation maps the tracking code to the rendering code using correspondence available for synthetic headset images.
- Video-driven Animation: The authors observe that expression semantics tend to be preserved when decoding between modalities, although this correspondence is not guaranteed.They attribute this observation to closely aligned image distributions and shared encoder features.
- Video-driven Animation: The pipeline encodes a headset frame into tracking code y, maps y to rendering code z, and decodes avatar geometry and texture for rendering.Only latent code z needs to be transmitted for networked social interaction.
- Artist Controls: For artist controls, latent code z is optimized under geometric vertex constraints with Gaussian-prior regularization, while joint geometry-appearance modeling produces plausible appearance.The decoded mesh is compared with desired constrained vertex locations, using λP = 0.01.
6 RESULTS
The evaluation examines rendering quality under geometric coarseness, viewpoint sparsity, and architecture choices, then tests semantic preservation across synthetic and real headset modalities and the complete HMD-to-avatar pipeline.
- Results: The experiments evaluate geometric coarseness, viewpoint sparsity, and architecture choice on avatar quality.They also examine tracking VAE semantic preservation between synthetic and real modalities.
- Results: The complete pipeline encodes HMD images, decodes realistic avatars, and renders them.This follows the modality-preservation experiments described for the tracking VAE.
6.1 Qualitative Results
Qualitative results show that the model captures subtle facial motion and complex reflectance, while its multi-identity model is demonstrated on data from eight people.
- Qualitative Results: The renderings capture subtle motion and complex reflectance behavior by directly modeling facial appearance.Some artifacts remain, including blurring inside the mouth, but they are described as less distracting than other facial-rendering artifacts.
- Qualitative Results: The multi-identity model is trained on data from 8 different people using a network conditioned on a latent identity vector.The authors report that it currently cannot generate plausible interpolations between people.
6.2 Effect of Geometric and Viewpoint Coarseness
The experiments show that jointly modeling geometry and texture can compensate for mesh-tracking bias, but generalization degrades as geometry becomes coarser and viewpoints become sparser. Accurate geometry supports novel-view rendering with fewer training viewpoints, whereas coarse geometry produces artifacts or viewpoint replication.
- Geometric and viewpoint coarseness: Joint geometry-texture modeling can correct some bias in mesh tracking by altering view-specific texture maps.The texture network has limited ability to resolve geometric errors, motivating experiments with progressively coarser geometry and sparser viewpoints.
- Geometric and viewpoint coarseness: Image-space MSE decreases in quality as geometry becomes more coarse, because the texture network must fill larger gaps between the input and true geometry.When geometry fits well, the model performs well even with few training viewpoints; coarse geometry requires more viewpoints for generalization.
- Geometric and viewpoint coarseness: Accurate tracked geometry generalizes to unseen viewpoints with few training views, while ellipsoid and billboard geometry generalize poorly.Qualitative failures include artifacts and simply replicating the closest training viewpoint.
- Qualitative results: The rendered results achieve high realism on held-out frames, but mouth blurring remains because tracked geometry inside the mouth is unavailable.The authors report that these artifacts are less bothersome than those in traditional real-time human face renderings.
6.3 Architecture Search
The architecture experiments compare viewpoint-conditioning and normalization choices, alongside linear, bilinear, and nonlinear latent representations. The results favor transformed single-layer viewpoint conditioning, weight normalization with per-channel-and-pixel biases, and the proposed compact nonlinear model.
- Conditioning methods: Single early-layer viewpoint conditioning outperforms conditioning all decoder layers on validation-view image-space MSE.Transforming the view vector through a fully connected layer is also beneficial.
- Normalization methods: Weight normalization far outperforms no normalization and batch normalization, while per-channel-and-pixel biases are important for accuracy.The architecture evaluation identifies the combination of weight normalization and per-channel-and-pixel biases as producing the highest quality avatar.
- Latent representation: The nonlinear model has advantages over linear and bilinear models in reconstruction performance and parameter compactness.The proposed model uses a deep nonlinear latent subspace that increases expressiveness while reducing the number of parameters.
- Latent representation: The proposed model uses 110MB, compared with 806MB for the linear model and 1.6GB for the bilinear model.These parameter sizes are reported alongside image-space mean-squared-error comparisons in Table 1.
6.4 Image-based vs. Texture/Geometry-based Loss
The paper reformulates appearance modeling so training can use image-based reconstruction error through a semidifferentiable renderer. Adding image loss improves validation MSE, while retaining geometry loss is necessary for stable training and structured geometry.
- Image-based loss: The autoencoder reformulation makes the loss a function of the reconstructed image, aligning training with the image-based evaluation metric.It also permits refinement of potentially erroneous tracked geometry.
- Image-based loss: The semidifferentiable rendering layer rasterizes triangle indices, computes texel coordinates, samples the texture map, and supports backpropagation.The image-based loss is optimized over rendered pixels using camera parameters, mesh vertices, and texture sampling.
- Loss comparison: Removing the geometry term makes training unstable and causes the geometry to lose its structure.The authors attribute this to insufficient long-range signal in image gradients.
- Loss comparison: Texture + geometry + image loss produces lower validation-view MSE than texture + geometry loss.The comparison evaluates image-based loss on eight held-out validation viewpoints.
6.5 Video-driven Animation
The video-driven pipeline uses a tracking VAE to represent facial state across sensor modalities and drive a person-specific avatar in real time. Cross-modal decoding preserves semantics, while live results capture expressions and subtle speech motion.
- Cross-modal facial-state representation: The tracking VAE learns a common representation of facial state across two modalities, within an entirely person-specific pipeline.The system encodes with one modality and decodes with the other at inference time.
- Cross-modal facial-state representation: Cross-modal translation preserves facial semantics despite training encoders and decoders on matching modalities.The authors associate this behavior with the VAE prior encouraging a unit Gaussian latent distribution.
- Live animation: The real-time avatar pipeline works for facial expressions and subtle motion during speech.The model encodes facial state into a joint geometry-and-appearance representation that can model changes in blood flow, complex materials, and incorrect geometry estimates.
7 DISCUSSION
The method unifies multiview facial appearance capture, view-dependent rendering, and deep networks for real-time photo-realistic rendering. Its current scope includes person-specific models, limited relighting, and remaining artifacts in difficult regions.
- The method unifies Active Appearance Models, view-dependent rendering, and deep networks to capture, encode, and render human facial appearance in real time.
- Directly predicting shaded appearance textures enables rendering without explicitly predicting physically inspired lighting-model parameters, but limits relighting ability.
- Accurate appearance prediction depends on tracked geometry closely matching the face surface, although complex regions such as eyes and teeth can still be modeled.
- The current approach is limited to person-specific trackers and rendering models; supporting arbitrary people requires semantic facial correspondence across individuals.
- Extending the approach to full-body avatars introduces challenges involving clothing dynamics, limb articulation, and interactions between individuals.