Source-linked AI summary
Variational Autoencoders for Deforming 3D Mesh Models
Qingyang Tan, Lin Gao, Yu-Kun Lai, Shihong Xia
TL;DR
Deforming 3D meshes are difficult to embed and synthesize because they involve diverse shapes and large-scale nonlinear deformations. The paper introduces mesh VAE, using a mesh-based rotation-invariant representation and an adjustable latent prior, and reports competitive performance across several shape applications, outperforming state-of-the-art methods.
Problem
Analyzing and synthesizing deforming 3D meshes is challenging because diverse shapes can contain large-scale nonlinear deformations, while acquiring high-quality models is time-consuming.
Method
Mesh VAE combines a variational autoencoder with a rotation-invariant mesh feature representation, and its extended model adjusts latent-variable significance through the prior distribution.
Results
The framework produces competitive results for shape generation, interpolation, embedding, and exploration, outperforming state-of-the-art methods.
Takeaways & Limitations
Mesh VAE learns a reasonable representation of deformable shape collections and can generate plausible novel models, including condition-specific models.
Abstract
from arXiv · showhide
3D geometric contents are becoming increasingly popular. In this paper, we study the problem of analyzing deforming 3D meshes using deep neural networks. Deforming 3D meshes are flexible to represent 3D animation sequences as well as collections of objects of the same category, allowing diverse shapes with large-scale non-linear deformations. We propose a novel framework which we call mesh variational autoencoders (mesh VAE), to explore the probabilistic latent space of 3D surfaces. The framework is easy to train, and requires very few training examples. We also propose an extended model which allows flexibly adjusting the significance of different latent variables by altering the prior distribution. Extensive experiments demonstrate that our general framework is able to learn a reasonable representation for a collection of deformable shapes, and produce competitive results for a variety of applications, including shape generation, shape interpolation, shape space embedding and shape exploration, outperforming state-of-the-art methods.
1. Introduction
The paper targets analysis and synthesis of deforming 3D meshes, whose large-scale nonlinear deformations make meaningful embedding difficult. It introduces mesh VAE, a neural generative framework with an adjustable latent prior for varied shape-analysis applications.
- Motivation: Deforming meshes represent animations and same-category shape collections, but their complex nonlinear deformations challenge meaningful embedding.High-quality 3D models are also time-consuming to acquire because multiple scans are often needed to address occlusion.
- Proposed framework: Mesh VAE explores the latent space of deforming 3D shapes and generates models absent from the original dataset.It trains on collections of 3D shapes with the same connectivity, including datasets such as MPI FAUST.
- Proposed framework: The framework combines the RIMD surface representation, a variational autoencoder, a fully connected network, and an MSE reconstruction loss.This design supports meshes with arbitrary connectivity within the same-connectivity training collection.
- Extensions and applications: An extended model adjusts latent-variable significance through the prior distribution, encouraging selected dimensions to represent important dataset differences.The framework reports competitive results across shape generation, interpolation, embedding, and exploration, outperforming state-of-the-art methods.
2. Related Work
Related work develops representations and learning-based synthesis methods for 3D shapes, but many approaches rely on restricted topology, regularized representations, or specialized object structure. The paper positions its method for deformable, unsegmented shape collections and nonlinear embedding.
- Shape representation and interpolation: RIMD is translation- and rotation-invariant and is presented as suitable for data-driven analysis of shape collections with consistent connectivity.The paper uses RIMD rather than raw vertex positions, which are not translation- or rotation-invariant.
- Shape representation and interpolation: Data-driven interpolation can use example-shape knowledge, while the paper's embedding application is nonlinear rather than PCA-based linear embedding.The paper reports comparable or better interpolation results than state-of-the-art data-driven methods later in the paper.
- Deep learning and synthesis: Prior 3D CNN methods analyze shapes through multi-view, panoramic, volumetric, or geometry-image representations.These approaches adapt image-oriented learning strategies to 3D shape analysis and synthesis.
- Deep learning and synthesis: Graph-based intrinsic learning targets vertex-level correspondence, whereas this paper addresses shape-level collections and tasks such as shape generation.Other neural synthesis methods target man-made, part-segmented objects and cannot directly handle unsegmented or deformed shapes.
- Deep learning and synthesis: Geometry-image methods preserve surface details but require nonunique parameterizations with distortions, especially for complex topology.The parameterization challenge limits direct suitability for shapes with high-genus topology.
3. Feature Representation
The paper represents deformations with the rotation-invariant mesh difference feature, built from local rotation differences and scaling/shear information. Matrix logarithms make the representation linearly combinable and support efficient mesh reconstruction.
- RIMD construction: RIMD is computed for M models with n one-to-one-corresponding vertices, using a reference model and deformed models.The choice of base mesh is later reported not to affect the results.
- RIMD construction: The deformation gradient is decomposed as T_i = R_iS_i, separating rotation from rotation-invariant scaling and shear.Cotangent weights help avoid discretization bias from the underlying smooth surface to the mesh.
- RIMD construction: Rotation differences between adjacent vertices cancel global rotation, and RIMD collects their logarithms together with each vertex's scaling/shear matrix.This combines edge-based rotation information with vertex-based deformation information.
- Reconstruction: The matrix logarithm makes RIMD features linearly combinable, while a generated RIMD feature can be converted back to a mesh by optimizing the reconstruction energy.The representation is reported to have mean reconstruction error of 10^-4 with no visual difference from ground-truth shapes.
4. Mesh VAE
Mesh VAE combines RIMD mesh features with variational autoencoding to represent, reconstruct, generate, and explore deforming 3D shapes. Its extensions condition synthesis on labels and adjust latent-variable priors to support low-dimensional, controllable embeddings.
- Feature preprocessing: The model preprocesses features to fit the tanh decoder range, using uniform normalization to [−a, a] with a = 0.9 and ϵ = 10^-6.The bounded output is intended to avoid the difficulty of training with an excessively broad activation range while reducing tanh gradient saturation.
- Basic mesh VAE: Mesh VAE uses probabilistic encoders and decoders over preprocessed RIMD mesh features, with a default 128-dimensional latent space.For meshes with 5000 faces, the typical input feature dimension is K = 60012.
- Basic mesh VAE: Its loss combines mean square reconstruction error with KL divergence between the posterior and prior latent distributions.The reconstruction term is used instead of a probabilistic reconstruction formulation for the high-dimensional mesh features.
- Network structure: A fully-connected encoder and decoder handle irregular spatial relationships because RIMD features have dataset-dependent neighboring structures.Internal layers use batch normalization and Leaky ReLU activations.
- Conditional mesh VAE: Conditional mesh VAE incorporates labels into both encoder and decoder inputs and uses conditional prior and posterior distributions for controlled shape synthesis.The conditional loss replaces q(z|ef) and p(z) with q(z|ef, c) and p(z|c).
- Extended model with improved low-dimensional embedding: The extended model changes the latent prior variances so selected dimensions can be emphasized for low-dimensional embedding and exploration.Small σ_object components encourage corresponding latent variables to capture dominant dataset changes, with behavior similar to PCA but nonlinear.
5. Experiments
Experiments show that mesh VAE is trainable on small deformable-shape datasets, reconstructs shapes effectively with RIMD features, and supports generation, interpolation, embedding, and latent-space exploration.
- Framework Evaluation: 128 latent dimensions improve reconstruction quality, while lower dimensions lose information and higher dimensions add little benefit and may overfit.The comparison uses held-out models and per-vertex position errors.
- Framework Evaluation: The alternative probabilistic reconstruction loss fails to converge even after three times the training epochs used for MSE loss.
- Framework Evaluation: 59%−94% lower reconstruction loss than 3D-coordinate baselines and 27%−72% lower loss than aligned-coordinate baselines demonstrate the effectiveness of RIMD features.The method remains effectively trainable on datasets typically containing 70-150 models.
- Robustness: Changing chicken-wing mesh density or base mesh choice changes per-vertex reconstruction errors by less than 10−4, with no visual difference.The tested datasets contain 6002, 6890, and 8002 points.
- Generation: The framework generates plausible models outside the training data, while nearest-neighbor comparisons test whether generated shapes are merely memorized.Experiments include SCAPE, Swing, face, and hand datasets.
- Interpolation: Latent-space interpolation avoids self-intersections and produces reasonable motion sequences, including interpolations across body shapes, poses, newly generated models, and non-human models.The comparison includes direct RIMD interpolation and a data-driven method.
- Embedding: The learned embedding separates body shapes while keeping similarly posed models close, outperforming PCA, NPE, and t-SNE qualitatively on diverse motion sequences.PCA is sparse and misses pose variation in the 2D embedding, while NPE cannot discriminate body shapes.
- Synthesis-based Exploration: Adjustable latent deviations let users explore dominant shape deformations first and finer variations later, including height and supporting-leg changes.The first two dimensions are explored over [−2, 2], followed by conditional browsing of the third and fourth dimensions.
6. Conclusions
The paper introduces mesh VAEs using a mesh-based rotation-invariant representation to analyze deformable shape collections and generate novel shapes. Experiments report high-quality models and performance exceeding state-of-the-art methods, while the approach remains limited to homogeneous meshes.
- Mesh VAEs combine a variational autoencoder with a mesh-based rotation-invariant feature representation.
- The framework supports shape-collection analysis and generation of novel deformable shapes with rich details.
- Experiments show competitive performance across various applications, outperforming state-of-the-art methods.
- The model can process only homogeneous meshes; handling shapes with different topologies remains future work.