Source-linked AI summary
SDM-NET: Deep Generative Network for Structured Deformable Mesh
Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, Hao Zhang
TL;DR
Existing 3D shape representations and generative models struggle to combine fine mesh detail, flexible topology, and coherent global structure. SDM-NET addresses this with a two-level VAE over deformable parts and their organization, and experiments report improved shape generation and interpolation with structured, detailed meshes.
Problem
Current mesh generative models are limited by genus-zero topology, shared connectivity, or unstructured patches, despite meshes' advantages for compact and detailed shape representation.
Method
SDM-NET uses a two-level VAE in which PartVAE models deformable part geometry and SP-VAE jointly learns part structure and geometry.
Results
SDM-NET outperforms state-of-the-art methods for shape generation and interpolation while achieving flexible topology and fine geometric details.
Takeaways & Limitations
Its structured deformable meshes support plausible structure-changing interpolation, mesh editing, and reuse in assembly-based modeling.
Takeaways & Limitations
Parts are constrained to genus-zero box topology, so a part containing a hole cannot be represented exactly or retain that hole.
Abstract
from arXiv · showhide
We introduce SDM-NET, a deep generative neural network which produces structured deformable meshes. Specifically, the network is trained to generate a spatial arrangement of closed, deformable mesh parts, which respect the global part structure of a shape collection, e.g., chairs, airplanes, etc. Our key observation is that while the overall structure of a 3D shape can be complex, the shape can usually be decomposed into a set of parts, each homeomorphic to a box, and the finer-scale geometry of the part can be recovered by deforming the box. The architecture of SDM-NET is that of a two-level variational autoencoder (VAE). At the part level, a PartVAE learns a deformable model of part geometries. At the structural level, we train a Structured Parts VAE (SP-VAE), which jointly learns the part structure of a shape collection and the part geometries, ensuring a coherence between global shape structure and surface details. Through extensive experiments and comparisons with the state-of-the-art deep generative models of shapes, we demonstrate the superiority of SDM-NET in generating meshes with visual quality, flexible topology, and meaningful structures, which benefit shape interpolation and other subsequently modeling tasks.
1 INTRODUCTION
SDM-NET addresses the difficulty of generating high-quality meshes with flexible topology and coherent structure by representing shapes as deformable parts and jointly learning their geometry and organization. Its two-level VAE generates structured meshes that support plausible reconstruction, interpolation, and downstream modeling.
- Triangle meshes preserve compactness, fine surface details, controllability, and visual quality, but their irregular tessellations hinder conventional deep-learning operations.
- Existing mesh generative models are limited by genus-zero topology, shared connectivity, or seams and incoherence from unstructured patches.
- SDM-NET decomposes complex shapes into box-homeomorphic parts whose fine geometry is recovered through deformation.
- Its two-level VAE uses PartVAE for part geometry and SP-VAE to jointly learn global part structure and local geometry.
- SDM-NET generates closed deformable mesh parts arranged according to global relations such as symmetry and support, with a varying number of parts up to a maximum.
- Experiments report quality meshes, flexible structure, fine details, plausible structure-changing interpolation, and support for mesh editing and related modeling tasks.
2 RELATED WORK
Related work explores voxel, image, point, mesh, patch, implicit, probabilistic, and neural representations for 3D shape modeling. SDM-NET differs by jointly encoding structure and deformable mesh geometry while supporting arbitrary topologies and support-aware arrangements.
- Voxel grids: Voxel representations are CNN-friendly but impose high memory costs at high resolution and limit geometric detail through voxel or octree resolution.
- Multi-view images: Multi-view images are regular and efficient, but omit full 3D information and may fail to recover geometric details without reconstruction processing.
- Point clouds: Point clouds flexibly represent scanner data, but their irregular structure creates challenges for deep learning.
- Meshes and multi-chart representations: Template-based mesh deformation captures fine details but requires shared connectivity, while multi-chart methods remain constrained by topology or produce seams and disconnected patches.
- Implicit representations: Implicit representations support diverse shape tasks including autoencoding, generation, interpolation, completion, and single-view reconstruction.
- Shape structures: Earlier structure-generation methods use probabilistic models or separate structure and geometry networks, whereas SDM-NET jointly encodes both and produces deformable mesh parts.
- Shape structures: Compared with GRASS, SDM-NET handles support information, requires consistent semantic segmentation rather than symmetry hierarchies, and outputs deformable meshes instead of voxel-filled bounding boxes.
3 METHODOLOGY
SDM-NET represents shapes as structured sets of deformable boxes and uses a two-level VAE to jointly model part geometry and global structure. Support and symmetry relations organize parts, while refinement optimization improves plausibility and physical validity.
- 3 METHODOLOGY: The method represents each labeled shape as a structured set of deformable bounding boxes, allowing coarse boxes to capture part structure while deformation recovers geometry details.Each part is modeled from an axis-aligned bounding box and a common unit-cube template.
- 3.1 Encoding of a Shape Part: Each part code combines existence, support, symmetry, position, and PartVAE geometry information into a representation vector.The code includes relations indicating which parts support or are supported by the current part, together with symmetry and geometric deformation information.
- 3.2 PartVAE for Encoding Part Geometry: PartVAE encodes fixed-connectivity, genus-zero part meshes as deformations of template boxes and learns a separate model for each part type.Its input describes local vertex-neighborhood deformation using rotation axes, rotation angles, and scaling factors, then reconstructs the deformation representation.
- 3.3 Supporting Structure Analysis: Support and symmetry relations provide the structural representation, with detected support relations converting part adjacency into a directed support graph.The support encoding accommodates multiple parts supporting one part and one part being supported by multiple parts.
- 3.4 SP-VAE for Structured Deformable Mesh Encoding: SP-VAE jointly encodes the layout, structural relations, and PartVAE geometry codes so generated part geometry remains coherent with global structure and compatible styles.The concatenated part representation is encoded in a consistent order, and the Gaussian latent space supports sampling and interpolation.
- 3.5 Shape Generation and Refinement: Refinement optimization enforces symmetry, equal-length, stable-support, and support constraints to produce more plausible and physically valid generated shapes.The optimization addresses typical structural artifacts while balancing its objective terms with α fixed to 10 in the experiments.
4 DATASET AND NETWORK IMPLEMENTATION
SDM-NET is implemented with separate PartVAE and SP-VAE components trained on consistently segmented shape collections. Its experiments evaluate reconstruction under varying hyper-parameters, while comparisons show detailed outputs and efficient generation.
- Data: The dataset combines pre-aligned ShapeNet Core V2 and ModelNet meshes, but ModelNet lacks semantic segmentation.Some source models also lack sufficiently detailed segmentation for support-structure descriptions.
- Data: SDM-NET requires consistently segmented inputs, registers each part from its bounding box by non-rigid deformation, and supports substantial geometric variation within part types.Test-shape segmentation can be obtained with supervised methods such as PointNet++.
- Network: The network contains PartVAE for per-part deformation encoding and SP-VAE for jointly encoding global structure and part geometry.PartVAE is trained separately for each part type, while SP-VAE encodes the combined part representation.
- Network: SP-VAE uses a concatenated all-part representation with fully connected layers of dimensions 1024, 512, and 256, followed by a 128-dimensional latent space.Leaky ReLU is used as the activation function.
- Evaluation: The experiments use a 75% training and 25% test split for tables, and average bidirectional Chamfer reconstruction errors across 10 evaluations.The lowest reported SP-VAE error uses λ1 = 1.0 and λ2 = 0.5.
- Efficiency: Generating one shape takes about 36 milliseconds, while structure optimization takes about 100 milliseconds.Training both VAEs takes about 300 minutes for a typical category.
5 RESULTS AND EVALUATION
The evaluation reports high-quality reconstruction, generation, interpolation, and editing using SDM-NET’s structured representation. Results attribute stable, coherent outputs to joint structure–geometry modeling and support constraints, while identifying topology and category limitations.
- Shape Reconstruction: SDM-NET reconstructs finer-detailed geometry than PSG, AtlasNet, and Adaptive O-CNN, whose outputs exhibit coarseness or patch artifacts.Adaptive O-CNN recovers planar chair regions but not curved regions such as the chair-back top.
- Shape Reconstruction: SDM-NET achieves the best performance on nearly all reported JSD, Coverage, and MMD metrics.MMD and Coverage are evaluated with both Chamfer Distance and Earth Mover’s Distance.
- Shape Generation: Compared with GRASS, SDM-NET produces physically stable and well-connected results, avoiding structural noise such as disconnections and asymmetries.The paper attributes GRASS’s noise partly to inconsistent automatically generated symmetry hierarchies for new inputs.
- Shape Generation: SDM-NET generates novel monitor shapes by combining structural and part information from SP-VAE with deformation information from PartVAE.The monitor examples use training data from ModelNet.
- Shape Interpolation: Latent-space interpolation produces smoother and more realistic transitions than AtlasNet, including natural gradual disappearance of chair armrests.Jointly training geometry and structure learns their implicit joint distribution.
- Shape Interpolation: SDM-NET interpolates naturally between shapes with different topologies and substantial geometric changes.These results are attributed to its flexible structure representation.
- Ablation Studies: Support constraints yield physically more stable results, while decoupling structure and geometry produces inconsistent part styles.Joint encoding makes generated part geometry consistent with structural interpretations such as sofas.
- Ablation Studies: Training separate PartVAEs for each part type captures unique geometric features better than a single PartVAE trained across all categories.The single-PartVAE alternative leads to poor geometric reconstruction.
6 CONCLUSIONS AND FUTURE WORK
SDM-NET represents 3D shapes as structured deformable meshes using a two-level VAE, achieving flexible topology and fine geometric details. The authors also identify limitations and future extensions involving reconstruction, deformation, category coverage, and storage efficiency.
- Conclusions: SDM-NET represents shapes as deformable boxes and uses a two-level VAE for local part geometry and global structure.The PartVAE encodes individual part geometry, while the structural VAE encodes the structure and geometries of all parts.
- Conclusions: The representation achieves flexible topology and fine geometric details while outperforming state-of-the-art methods for shape generation and interpolation.
- Future Work: The method could be extended to reconstruct 3D shapes from images by conditioning the SP-VAE on learned image part segmentations.
- Limitations: Current failures include parts with holes, complex chair legs, and grid backs because every part must have fixed genus-zero-box topology.
- Limitations: Fixed high-resolution primitives preserve geometric detail but produce large storage requirements, motivating adaptive resolutions for parts with different geometric richness.
APPENDIX: SUPPORT RELATIONSHIP FORMULATION.
The support relationship formulation enforces nested projected bounding boxes for parts i and j. Binary variables and a large constant encode which box contains the other, allowing the constraint to be solved as an integer program.
- Support Relationship Constraint: Projected bounding boxes of parts i and j must satisfy either ˜bi ⊆ ˜bj or ˜bj ⊆ ˜bi.
- Optimization: The nesting requirement is formulated as an integer programming problem and solved efficiently during optimization.
- Integer Formulation: A large positive constant M makes the constraint conditional on binary variables δi,j.M is chosen larger than any possible shape coordinate.
- Integer Formulation: The formulation requires at most one of δi,j^1 and δi,j^2 to equal 1, so at least one variable is 0.
- Containment Cases: When δi,j^1 = 0, the equations enforce ˜bi ⊆ ˜bj; when δi,j^2 = 0, they enforce ˜bj ⊆ ˜bi.