Source-linked AI summary
Generating 3D faces using Convolutional Mesh Autoencoders
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, Michael J. Black
TL;DR
Existing linear 3D face representations do not capture extreme nonlinear expressions. CoMA learns a nonlinear hierarchical representation with spectral mesh convolutions and mesh sampling operations, using 20,466 expressive meshes for training. It reports 50% lower reconstruction error than PCA while using up to 75% fewer parameters.
Problem
Linear and higher-order tensor 3D face representations cannot capture extreme deformations and nonlinear facial expressions.
Method
CoMA uses spectral convolutions on mesh surfaces with mesh downsampling and upsampling operations to learn hierarchical multiscale face representations.
Results
50% lower reconstruction error than PCA and up to 75% fewer parameters are reported for CoMA.
Takeaways & Limitations
CoMA provides a compact nonlinear representation for expressive 3D faces and can replace FLAME’s expression space with lower reconstruction error.
Takeaways & Limitations
CoMA requires all meshes to have dense vertex correspondence and shared topology.
Abstract
from arXiv · showhide
Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and non-linear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We also show that, replacing the expression space of an existing state-of-the-art face model with our autoencoder, achieves a lower reconstruction error. Our data, model and code are available at http://github.com/anuragranj/coma
1 Introduction
Existing 3D face models rely mainly on linear representations that miss extreme nonlinear expressions, while mesh CNNs face resolution, sampling, and data limitations. CoMA addresses these issues with hierarchical mesh convolutions and sampling, supported by a dataset of 20,466 expressive meshes.
- Motivation: Linear and tensor-based 3D face models do not capture the nonlinear deformations caused by extreme facial expressions.Such expressions are important for realistic 3D faces.
- Motivation: Volumetric convolutions are memory-intensive and limited to low resolution, whereas mesh convolutions can process high-resolution 3D structures more efficiently.Generalizing CNN operations and hierarchical pooling to meshes remains nontrivial.
- Approach: CoMA combines spectral mesh convolutions with novel downsampling and upsampling operations that preserve mesh topology across multiple scales.The resulting hierarchical autoencoder represents nonlinear shape and expression variation on the mesh surface.
- Dataset: 20,466 high-resolution meshes capture complex, asymmetric extreme expressions from 12 subjects, addressing limited training data for expressive 3D faces.The expressions involve significant deformation in facial tissue.
- Results: CoMA reports 50% lower reconstruction error than PCA while using up to 75% fewer parameters than linear PCA models.The model also replaces FLAME’s expression space and supports sampling diverse facial meshes from a Gaussian distribution.
2 Related work
Prior face representations predominantly use linear spaces, while earlier mesh and graph CNNs generally lack a complete dimensionality-reduction structure. CoMA combines mesh convolutions with mesh downsampling and upsampling to form an autoencoder for complex 3D faces.
- Face Representations: The Basel Face Model and related systems represent neutral face shape primarily with PCA-based linear models.Other approaches use linear blendshapes, multiple PCA spaces, or expression residual models.
- Face Representations: Hierarchical multiscale linear models capture localized facial details, but localized sampling can make plausible global face shapes difficult to obtain.Brunton et al.’s approach also lacks parameters shared across the entire domain.
- Convolutional Networks: Volumetric face representations require memory, whereas CoMA operates directly on meshes with significantly fewer parameters.Mesh-space processing avoids the memory constraints affecting volumetric convolution methods.
- Convolutional Networks: Earlier mesh and graph CNN methods generalize convolutions but do not provide mesh dimensionality reduction or a complete mesh autoencoder structure.These methods include intrinsic mesh convolutions, spectral graph CNNs, and planar embeddings.
- Convolutional Networks: CoMA uses truncated Chebyshev polynomials for mesh convolutions together with mesh downsampling and upsampling layers.This combination forms an autoencoder for highly complex 3D faces and achieves state-of-the-art 3D face modeling results.
3 Mesh Operators
CoMA defines mesh convolutions spectrally and adds hierarchical sampling operators that preserve neighborhood context while changing mesh resolution.
- Mesh representation: A facial mesh is represented by vertices and edges, with adjacency matrix A encoding vertex connections.The mesh is F = (V, A), where vertices lie in 3D Euclidean space.
- Fast spectral convolutions: Spectral convolution applies Chebyshev-polynomial filters to the graph Laplacian, avoiding dense Fourier-basis multiplication.The filter uses the scaled Laplacian and recursively computed Chebyshev polynomials.
- Fast spectral convolutions: Each convolution maps Fin input features to Fout output features using trainable Chebyshev-coefficient vectors for every feature pair.The input face mesh has three features corresponding to 3D vertex positions.
- Hierarchical sampling: Hierarchical mesh representations let shallow kernels capture local context while deeper kernels capture global context.The model introduces sampling operators to support this multiscale structure.
- Hierarchical sampling: Downsampling contracts vertex pairs using quadric-error approximations, while upsampling restores discarded vertices through stored barycentric coordinates.Retained vertices are transformed convolutionally before discarded vertices are mapped back onto the down-sampled surface.
- Hierarchical sampling: The autoencoder alternates mesh-resolution changes with convolutional processing, using red arrows for downsampling and blue arrows for upsampling.The figure identifies the direction of the two sampling operations in the network.
4 Mesh Autoencoder
The CoMA autoencoder encodes facial meshes into a compact latent vector and decodes them through interleaved convolution and upsampling layers. Latent-space variation produces diverse face meshes around the mean face.
- Encoder: The encoder uses four Chebyshev convolutional filters with K = 6, interleaved with downsampling layers that reduce vertex counts approximately fourfold.A final fully connected layer maps the mesh to an 8-dimensional latent vector.
- Decoder: The decoder maps the latent vector to an intermediate representation, then uses four convolutional and upsampling layers to reconstruct a 5023 × 3 mesh.Each upsampling layer increases the number of vertices approximately fourfold.
- Training: The network is trained with stochastic gradient descent using an L1 vertex-reconstruction loss and L1 weight regularization.Training uses momentum 0.9, weight decay 5e-4, and Chebyshev filtering with K = 6.
5 Experiments
The experiments evaluate CoMA on extreme facial-expression meshes through latent-space sampling, reconstruction, interpolation, extrapolation, and integration with FLAME. CoMA produces diverse samples with variational loss and improves reconstruction over PCA and FLAME-related baselines while requiring fewer parameters.
- Sampling the Latent Space: Variational loss enforces a multivariate Gaussian latent space, enabling more diverse Gaussian-noise samples than training without that loss.With w_kld = 0.001, the KL term imposes a unit Gaussian prior; without it, Gaussian sampling produces limited diversity.
- Comparison with PCA Spaces: CoMA reconstructs expressive faces with 50% lower error than PCA while using about 75% fewer parameters.The comparison uses an 8-dimensional latent space for both models.
- Comparison with PCA Spaces: 72.6% of vertices fall within 1 mm for CoMA versus 47.3% for PCA in interpolation, and 63.8% versus 45% in extrapolation.Extrapolation tests expressions completely excluded from training and also compare CoMA with FLAME.
- DeepFLAME: Replacing FLAME’s PCA expression space with CoMA yields lower median reconstruction error at every tested latent dimension and lower mean error up to 12 variables.The resulting DeepFLAME model is described as more compact because it captures more shape variation with fewer latent variables.
- Discussion: CoMA’s graph-convolutional mesh representation is evaluated for reconstruction and can generalize to other mesh-processing applications, but requires dense vertex correspondence.All meshes must share the same topology; learning directly from raw scans or images is left for future work.
6 Conclusion
CoMA represents varying 3D face shape and expression hierarchically across multiple scales, improving reconstruction while reducing model size. Its variational form also synthesizes expressive faces by sampling the latent space.
- CoMA uses mesh sampling, fast graph convolutions, and nonlinear activations to capture global and local facial variations across multiple scales.
- 75% fewer model parameters accompany CoMA's superior performance over state-of-the-art models in 3D face reconstruction applications.
- 50% better interpolation performance than PCA and better generalization to completely unseen facial expressions demonstrate CoMA's reconstruction advantages.
- In a variational setting, CoMA synthesizes new expressive faces by sampling its latent space.
S. Details of the Dataset
The dataset contains extreme facial-expression sequences from 12 subjects, with expressions selected to produce substantial, uncorrelated facial-tissue deformation. The section also includes sequence-length information and dataset samples.
- The dataset contains sequences from 12 subjects across different age groups, each performing 12 extreme facial expressions.
- The expressions were selected to cause substantial facial-tissue deformation and avoid correlations between expressions.
- Table 6 lists the number of frames in each expression sequence.
- Figure 8 presents samples from the dataset.