Source-linked AI summary
Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, Stefanos Zafeiriou
TL;DR
Existing 3D shape-generation methods often rely on intermediate representations, while linear morphable models have limited representational power for fixed-topology deformable shapes. The paper introduces spiral convolutions and Neural 3D Morphable Models that operate directly on meshes, achieving state-of-the-art reconstruction results and demonstrating mesh generation capabilities.
Problem
Representation learning and generation for deformable 3D shapes remain limited, with prior nonlinear methods often using intermediate representations and linear morphable models constrained by eigenvector spans.
Method
The paper builds hierarchical mesh autoencoders and a mesh Wasserstein GAN from spiral convolutions that impose consistent neighbor orderings and encode fixed mesh connectivity.
Results
The Neural 3D Morphable Model achieves state-of-the-art mesh reconstruction results across multiple face and body datasets and supports latent-space arithmetic and novel facial-identity synthesis.
Takeaways & Limitations
Spiral convolutions provide an anisotropic, topology-aware, lightweight building block for direct mesh representation learning and generation on fixed-topology deformable shapes.
Takeaways & Limitations
The framework is developed for fixed-topology meshes, while extending it to general graphs and arbitrary-topology 3D shapes remains future work.
Abstract
from arXiv · showhide
Generative models for 3D geometric data arise in many important applications in 3D computer vision and graphics. In this paper, we focus on 3D deformable shapes that share a common topological structure, such as human faces and bodies. Morphable Models and their variants, despite their linear formulation, have been widely used for shape representation, while most of the recently proposed nonlinear approaches resort to intermediate representations, such as 3D voxel grids or 2D views. In this work, we introduce a novel graph convolutional operator, acting directly on the 3D mesh, that explicitly models the inductive bias of the fixed underlying graph. This is achieved by enforcing consistent local orderings of the vertices of the graph, through the spiral operator, thus breaking the permutation invariance property that is adopted by all the prior work on Graph Neural Networks. Our operator comes by construction with desirable properties (anisotropic, topology-aware, lightweight, easy-to-optimise), and by using it as a building block for traditional deep generative architectures, we demonstrate state-of-the-art results on a variety of 3D shape datasets compared to the linear Morphable Model and other graph convolutional operators.
1. Introduction
Geometric deep learning seeks direct operations for non-Euclidean 3D data, but representation learning and generation for deformable meshes remain limited. The paper proposes Neural 3D Morphable Models using spiral convolutions that encode fixed mesh connectivity through consistent local vertex orderings.
- Geometric deep learning extends convolutional architectures to non-Euclidean data, where meaningful mesh and point-cloud operations remain a central challenge.
- Representation learning and shape synthesis are important for downstream 3D tasks and applications including reconstruction, animation, virtual reality, and training-data generation.
- Earlier generative approaches commonly use point clouds, voxels, or flat-domain mappings, introducing computational costs, nonsmooth representations, or preprocessing and postprocessing requirements.
- The paper proposes a representation-learning and generative framework for fixed-topology meshes.
- Spiral convolution orders each vertex’s neighbors consistently, mapping them one-to-one to learnable filter parameters and explicitly encoding fixed graph connectivity.
- Neural 3D Morphable Models use spiral convolution in hierarchical mesh autoencoders and achieve state-of-the-art reconstruction results across face and body datasets against linear and graph-convolutional baselines.
2. Related Work
Related work for 3D shape generation includes volumetric, point-cloud, image-based, implicit-surface, and statistical morphable-model approaches. These methods trade off computational cost, smoothness, preprocessing, inference speed, or representational simplicity.
- Volumetric CNNs generate arbitrary shapes from voxel grids but have high computational complexity and produce coarse, redundant representations.
- Point clouds offer compact, lightweight representations but lack an underlying smooth structure for realistic, high-quality geometry generation.
- Multi-view and flat-domain image methods can be computationally demanding, require preprocessing and postprocessing, and may produce undesirable artifacts.
- Implicit-surface methods can yield accurate results but require dense 3D-space sampling followed by marching cubes, resulting in slow inference.
- For deformable shapes with fixed topology, statistical morphable models remain popular because of their simplicity, with PCA-based 3D Morphable Models serving as a facial baseline.
3. Spiral Convolutional Networks
Spiral convolution imposes consistent local vertex orderings on fixed-topology meshes, enabling anisotropic filters that directly process mesh features. Used in hierarchical autoencoders, this supports nonlinear mesh morphable models with multiscale representations and fewer parameters.
- Spiral Convolution: Fixed-topology meshes permit locally ordered neighbours, enabling graph convolution with a one-to-one mapping between neighbours and filter weights.The operator treats each ordered neighbour as a single patch element.
- Spiral Convolution: Spiral trajectories define the local ordering by traversing mesh rings, with direction and starting vertex fixed using the underlying geometry.Spirals are truncated or zero-padded to a fixed length L.
- Spiral Convolution: Consistent ordering across meshes avoids random neighbourhood rotations that would otherwise force the network toward rotational invariance and reduce capacity.The compared recurrent spiral formulation also has higher computational complexity and is harder to optimise.
- Spiral Convolution: Unlike spectral filters that are locally rotationally invariant, spiral filters are anisotropic by construction and can use one-hop neighbourhoods.The paper contrasts this with the larger receptive fields used by the compared mesh autoencoder.
- Neural 3D Morphable Models: Neural 3D Morphable Models are fully differentiable nonlinear mesh autoencoders that learn hierarchical, multiscale shape representations from mesh connectivity.The architecture uses convolutional layers with small receptive fields and pooling or unpooling, reducing parameters and avoiding distributional assumptions.
4. Evaluation
The evaluation compares spiral convolutions with alternative graph operators and representation-learning baselines through ablations, quantitative reconstruction tests, and generative assessments.
- 4. Evaluation: The experiments compare spiral convolutions with ChebNet, soft-attention graph neural networks, and alternative ordering strategies using matched autoencoder architectures.The study also evaluates representation quality against linear 3DMM and COMA, plus qualitative vector arithmetic and an intrinsic GAN.
- 4. Evaluation: The evaluation measures reconstruction generalisation, parameter efficiency, latent compactness, and the ability to generate novel high-resolution realistic examples.
- 4. Evaluation: The experiments use normalised deformations from the mean shape as mesh signals to encourage stationarity and facilitate optimisation.
4.1. Datasets
The study evaluates mesh representation learning on facial-expression, facial-identity, and human-body datasets, using reconstruction generalisation as the quantitative metric.
- 4.1. Datasets: DFAUST contains 40K+ scans of 10 identities performing actions such as leg and arm raises and jumps, with specified train, validation, and test splits.
- 4.1. Datasets: Generalisation is the average per-sample, per-vertex Euclidean reconstruction distance in millimetres for novel shapes from the training distribution.
4.2. Implementation Details
The implementation uses hierarchical spiral-convolutional mesh autoencoders with repeated downsampling and upsampling, while Figure 4 and Figure 5 frame quantitative baseline and filter comparisons.
- 4.2. Implementation Details: The simple Neural3DMM encoder stacks four spiral-convolution stages with factor-4 downsampling before a fully connected latent layer.
- 4.2. Implementation Details: The decoder mirrors this hierarchy with a fully connected expansion, four factor-4 upsampling stages, and a final three-channel spiral convolution.
- 4.2. Implementation Details: Mein3D adds an extra convolution and downsampling/upsampling layer because of its higher vertex count, while the larger model increases the parameter space.
- 4.2. Implementation Details: Figure 4 evaluates Neural3DMM against baselines by generalisation and parameter count, while Figure 5 compares spiral and ChebNet spectral filters.
4.3. Ablation Studies
The ablations test spiral filters against spectral and soft-attention operators and examine how consistent neighbour ordering affects performance.
- 4.3.1 Isotropic vs Anisotropic Convolutions: Spiral autoencoders consistently outperform spectral autoencoders across latent dimensions on both datasets.The spiral model uses added parameters more effectively, especially at latent dimensions 16, 32, 64, and 128; on COMA, the spectral model flattens between 64 and 128.
- 4.3.2 Spiral vs Attention based Convolutions: With similar parameter counts, Neural3DMM outperforms soft-attention counterparts; against over-parameterised networks, it either wins or performs slightly worse.The authors report that spiral operators are lightweight alternatives that retain performance and allow fast per-mesh inference measurement.
- 4.3.3 Comparison to Lim et al. [25]: The ordering experiment compares random orientations per mesh and epoch, random orientations per epoch, fixed per-mesh orientations, and fixed ordering.
- 4.3.3 Comparison to Lim et al. [25]: 28% improved performance is achieved over the LSTM-based approach of [25], supporting the benefit of passing corresponding points through the same transformations.
4.4. Neural 3D Morphable models
Neural3DMM is evaluated against PCA, 3D Morphable Models, COMA, and other mesh autoencoders for reconstruction and latent-space operations. The experiments also demonstrate interpolation, extrapolation, and shape analogies using latent representations.
- Quantitative results: The reconstruction-error visualization compares ground truth with PCA, COMA, and Neural3DMM using per-vertex Euclidean error colour coding.The figure places ground truth in the top row and the three reconstructions beneath it.
- Quantitative results: Neural3DMM achieves smaller generalisation errors than the compared methods across the tested latent-space dimensions.The comparison includes PCA, the 3D Morphable Model, COMA, ChebNet, and multiple Neural3DMM variants.
- Quantitative results: Localised filters reconstruct smaller shape regions effectively, whereas PCA performs a more global reconstruction.For COMA and DFAUST, hierarchical intrinsic architectures outperform PCA at small latent sizes; PCA can place body parts incorrectly when prioritising overall body shape.
- Latent-space operations: Latent interpolation generates intermediate encodings along the line between two encoded test samples.The construction uses z = az1 + (1 − a)z2 with a between 0 and 1.
- Latent-space operations: Latent extrapolation samples beyond the segment between two encodings to exaggerate a characteristic, using neutral expression or pose as one endpoint.The procedure is demonstrated for expressions in COMA and poses in DFAUST.
- Latent-space operations: Shape analogies transfer a characteristic between meshes by solving A:B::C:D in the latent space and decoding the resulting representation.The relation is constructed from differences between encoded mesh pairs.
4.5. GAN evaluation
The intrinsic 3D GAN generates facial identities that appear realistic and span a substantial portion of the dataset’s variation. Compared with the 3DMM, it produces finer facial detail that makes samples harder to distinguish from real scans.
- Generated identities: Generated faces span a large proportion of the dataset’s real distribution across ethnicity, gender, and age.Figure 10 presents sampled identities from the latent distribution of the trained generator.
- Generated identities: The model produces fine facial details that make generated samples hard to distinguish from real 3D scans.The comparison is with the 3DMM, which produces smooth surfaces but can be easier to identify as artificial.
5. Conclusion
The paper concludes that spiral convolutions support an efficient representation-learning and generative framework for fixed-topology 3D deformable shapes. It reports strong mesh reconstruction and latent-space generation results, while identifying extensions to broader topologies and domains as future work.
- Framework: The framework uses spiral convolutions to encode the inductive bias of fixed-topology 3D deformable shapes.The conclusion characterizes the operator as a mesh convolutional building block for representation learning and generation.
- Results: The mesh autoencoder achieves state-of-the-art mesh reconstruction with reduced computational complexity compared with prior graph convolutional operators.The conclusion presents both representational power and computational efficiency as properties demonstrated by the framework.
- Generation: The models demonstrate generation through vector-space arithmetic and synthesis of novel facial identities.These capabilities are presented alongside the reconstruction results.
- Future work: Future work targets general graphs, arbitrary-topology 3D shapes, and domains with an implicit ordering of primitives, including point clouds.These extensions define the stated scope beyond fixed-topology meshes.