Source-linked AI summary
GRASS: Generative Recursive Autoencoders for Shape Structures
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, Leonidas Guibas
TL;DR
GRASS addresses how to represent and generate 3D shapes whose parts have hierarchical structure, including adjacency and symmetry. It uses a recursive autoencoder and generative models to synthesize structures and part geometries, producing meaningful hierarchies, compact codes, and shapes with substantial topological and geometric variation.
Problem
Existing generative modeling successes for images and speech raise the question of how to represent 3D shapes, whose canonical representation remains unsettled.
Method
GRASS recursively encodes and decodes oriented bounding-box structures with connectivity and symmetry, then uses generative models for box hierarchies and volumetric part geometries.
Results
The network learns meaningful structural hierarchies and compact codes, while supporting shape synthesis and interpolation with significant topology and geometry variations.
Takeaways & Limitations
Gross structure can support shape recognition, retrieval, classification, synthesis, and interpolation even without fine-grained geometry or supervision.
Takeaways & Limitations
Structure and fine-geometry synthesis are decoupled, and sampling is confined to a local patch near exemplar codes rather than arbitrary extrapolation.
Abstract
from arXiv · showhide
We introduce a novel neural network architecture for encoding and synthesis of 3D shapes, particularly their structures. Our key insight is that 3D shapes are effectively characterized by their hierarchical organization of parts, which reflects fundamental intra-shape relationships such as adjacency and symmetry. We develop a recursive neural net (RvNN) based autoencoder to map a flat, unlabeled, arbitrary part layout to a compact code. The code effectively captures hierarchical structures of man-made 3D objects of varying structural complexities despite being fixed-dimensional: an associated decoder maps a code back to a full hierarchy. The learned bidirectional mapping is further tuned using an adversarial setup to yield a generative model of plausible structures, from which novel structures can be sampled. Finally, our structure synthesis framework is augmented by a second trained module that produces fine-grained part geometry, conditioned on global and local structural context, leading to a full generative pipeline for 3D shapes. We demonstrate that without supervision, our network learns meaningful structural hierarchies adhering to perceptual grouping principles, produces compact codes which enable applications such as shape classification and partial matching, and supports shape synthesis and interpolation with significant variations in topology and geometry.
1 Introduction
GRASS addresses the difficulty of generating structured 3D shapes by representing part layouts as recursively organized symmetry hierarchies with fixed-length root codes. An RvNN autoencoder, adversarial code-space model, and geometry generator together support structure synthesis, interpolation, and analysis.
- 3D shapes combine irregular representations with hierarchical structure and substantial within-class variation, challenging image- and speech-inspired neural-network assumptions.
- GRASS recursively contracts shape hierarchies into fixed-length root codes, unifying structures with different sizes and combinatorial organizations.
- The unsupervised RvNN autoencoder groups OBBs by connectivity or symmetry while jointly encoding their geometry and structure.
- A GAN learns a class-specific manifold of root codes, which are sampled and decoded into new OBB arrangements before a second model generates detailed part geometry.
- The architecture extends recursive neural networks with generative decoding and multiple merging operations for assembly, reflectional symmetry, and rotational or translational symmetries.
- Without supervision, the network learns perceptually meaningful hierarchies and compact codes supporting classification, partial matching, synthesis, and interpolation with topology and geometry variation.
2 Related work
Prior 3D shape models capture statistical variation through templates, part-based models, voxel grids, views, surfaces, or graph networks, but remain limited in representing complex structural layouts or generating them directly.
- Statistical shape methods model continuous deformation or discrete part variation using templates, grammars, Bayesian networks, and modular representations.
- Many deep 3D models adapt image-based methods to voxel grids, projected views, or non-Euclidean surfaces, with emphasis on shape analysis and classification.
- Existing modular and surface-based generative approaches remain limited in the variety of layouts they can represent.
- GAN-based voxel decoders support topology-varying morphs, whereas GRASS targets complex hierarchical part-layout variation beyond low-resolution grids and independently of voxel resolution.
- Graph neural networks operate on arbitrary graph domains, but the cited approaches do not provide generative models and struggle to enforce high-level graph organization.
3 Overview
GRASS uses a three-stage pipeline that encodes OBB-based shape structures into fixed-length root codes, learns a manifold of plausible structures, and synthesizes detailed part geometry.
- GRASS represents shape structures as hierarchies of oriented bounding boxes whose fixed-length codes capture geometry and grouping mechanisms.
- Stage 1: Recursive autoencoder: A recursive autoencoder collapses arbitrarily arranged OBB layouts into fixed-length root codes and decodes them to reconstruct the boxes.
- Stage 2: Learning manifold of plausible structures: A GAN learns a low-dimensional manifold of root codes so sampled codes can be projected into plausible structures.
- Stage 3: Part geometry synthesis: A final model maps synthesized boxes and their structural context to voxelized part geometries.
- Chaining structure generation with part geometry synthesis yields the full GRASS pipeline for recursive 3D shape synthesis.
4 Recursive model of shape structure
The recursive model learns symmetry hierarchies from unlabeled part layouts by combining adjacency and symmetry-specific encoders and decoders, then infers hierarchies through reconstruction error.
- Criteria for recursive merging: The model recursively merges adjacent parts or symmetry groups, reflecting perceptual hierarchies based on component proximity and symmetry.
- GRASS adapts recursive autoencoders from binary trees to hierarchical shape structures with multiple merge types, non-binary symmetry groups, and probabilistic generation.
- Criteria for recursive merging: It represents part geometry with oriented bounding boxes and distinguishes adjacency, translational, rotational, and reflective symmetry criteria.
- Autoencoder model: BOXENC and BOXDEC translate between 12D box parameters and internal codes, while ADJENC/ADJDEC and SYMENC/SYMDEC encode and decode their respective grouping operations.
- Autoencoder model: NODECLSFR predicts whether a decoded node is an adjacent pair, symmetry group, or leaf, selecting the corresponding decoder during testing.
- Training and testing: Training minimizes leaf-box reconstruction error, whereas testing uses greedy local search to choose merge sequences with the lowest reconstruction error.
- Training and testing: For a swivel chair, grouping a wheel and spike before applying 5-fold rotational symmetry produces lower reconstruction error than applying separate rotational symmetries first.
5 Learning manifold of plausible structures
The paper extends a recursive autoencoder into a generative model by learning a plausible manifold of root codes and decoding sampled codes into shape structures. Adversarial training is stabilized with structural priors, constrained sampling, and separate geometric and structural tuning.
- Motivation: The autoencoder alone reconstructs layouts from root codes but does not ensure that arbitrary random codes decode to plausible structures.This motivates learning a distribution over statistically plausible root codes.
- Applications: The learned manifold also supports interpolation between shape codes for shape morphing.This capability is presented as an application of the learned structure distribution.
- GAN architecture: The GAN generator reuses and fine-tunes autoencoder modules to map random codes to hierarchical structures judged plausible by a discriminator.The discriminator distinguishes training structures from generated ones, while the generator is trained to fool it.
- Training strategies: Training uses a structure prior, separate geometric and structural losses, and constrained latent-code sampling to improve convergence.These strategies address implausible hierarchies, weak backpropagation pathways, and different decoder roles.
- Latent distribution: The latent distribution samples codes from Gaussians whose parameters are predicted from encoded training shapes, with reconstruction and KL losses added to GAN training.The combined objective uses α1 = 10^-2 for reconstruction and α2 = 10 for KL divergence.
- Generated structures: The resulting generator maps standard-normal vectors through f_l to potential root codes, whose tuned decoders produce structures on the plausible manifold.A separate geometry module then supplies fine-grained part geometry for complete shape generation.
6 Part geometry synthesis
The final generation stage synthesizes fine-grained part geometry conditioned on each part’s bounding box and structural context. It uses fixed-dimensional contextual features and a learned low-dimensional geometry manifold before reconstructing a meshed shape.
- Geometry model: The geometry module conditions fine-grained part synthesis on the part bounding box and layout.Its two components encode contextual part features and learn a manifold of plausible part geometries.
- Structure-aware features: SARF concatenates recursive codes from a part, its immediate parent, and the root to form a fixed-dimensional structural feature.This approximates the variable-length path context while preserving local and global hierarchy information.
- Geometry synthesis: The geometry network maps each SARF feature to a 32×32×32 voxel grid through a learned low-dimensional manifold of plausible parts.The manifold avoids directly learning an unconstrained 8000-dimensional output mapping.
- Full pipeline: Given a generated part structure, the system synthesizes per-box volumes, embeds them into a global volume, and reconstructs the final meshed model.This is the pipeline illustrated from structure to volumetric geometry to mesh.
7 Results and evaluation
Experiments evaluate hierarchy learning, code utility, and generative synthesis on segmented 3D shapes. The model learns mostly consistent perceptual groupings, achieves 96.1% average subclass classification accuracy, retrieves matching subparts, and supports topology-changing interpolation.
- Evaluation setup: The evaluation tests symmetry-hierarchy quality, classification, partial matching, and the generative capability of the VAE-GAN.The dataset contains 1,000 models across chairs, bikes, aeroplanes, excavators, and candelabra.
- Hierarchy learning: The RvNN correctly parses six of seven handcrafted grouping arrangements, with the G2 example preferring translational over reflectional symmetry.The arrangements test seven precedence rules originally designed to model perceptual grouping.
- Hierarchy consistency: Average hierarchy consistency is 0.81 on training shapes and 0.72 on test shapes across chair and candelabra categories.The reported values indicate consistent inferred hierarchies across different shapes in these categories.
- Classification: 96.1% average subclass classification accuracy is obtained across four shape classes using compact hierarchy-derived codes.The experiment covers airplane, chair, bike, and candle subclasses.
- Classification: The method outperforms two state-of-the-art descriptors in classification, although it uses prior unlabeled part segmentation while the baselines do not.The method uses oriented bounding-box parameters rather than fine-grained part geometry.
- Generation and matching: Subtree codes retrieve matching parts in partial structure queries, while the generative pipeline synthesizes shapes and interpolates between topologically and geometrically different examples.Interpolation maintains symmetry constraints and handles smooth topological changes in layouts and individual parts.
8 Discussion, limitation, and future work
The authors describe GRASS as an initial structure-aware generative model, while identifying limitations in decoupled synthesis, code interpretability, and coverage of the generative structure space.
- Limitations and future work: GRASS decouples structure synthesis from fine-geometry synthesis, motivating future integration of the two processes.The authors report cleaner part structures and more regularized geometries than voxel-field methods, but identify the separation as unsatisfying.
- Limitations and future work: The learned fixed-length code combines structural and geometric information, but its internal mixing mechanisms and representation of arbitrary-depth hierarchies remain unclear.The authors propose visualizing the code to improve separation between structure-related and low-level geometric components.
- Limitations and future work: The VAE-GAN setup limits synthesis and interpolation to a local region near or between exemplar codes rather than enabling arbitrary extrapolation.The paper also leaves open whether richly varying shape collections form a low-dimensional generative structure manifold.
- Observed capability: Linear interpolation between root codes can produce plausible morphs between shapes with significantly different topologies.This demonstrates variation in topology despite the broader limitation on extrapolating through the generative structure space.
Appendix: Precedence rules for symmetry hierarchy
The appendix specifies precedence rules that deterministically order symmetry grouping and assembly operations when constructing a symmetry hierarchy.
- Grouping and assembly: Symmetry grouping generally precedes assembly, except when equivalent grouping symmetries require assembly first.This establishes the primary ordering between grouping and connectivity operations.
- Symmetry grouping: Higher-order symmetry cliques are grouped before lower-order cliques when both remain available.The rule applies when cliques of order greater than two are present in the contraction graph.
- Symmetry grouping: For only order-2 cliques, reflectional symmetry is grouped before rotational and translational symmetries.This rule resolves precedence among the listed symmetry types when no higher-order clique remains.
- Symmetry grouping: When symmetry type and order do not resolve precedence, the closer symmetry ensemble is grouped first.This proximity rule addresses ties such as rotational versus translational symmetries of equal order.
- Assembly: Symmetry-preserving assembly precedes symmetry-breaking assembly, with geometric connectivity strength resolving remaining ties.The assembly rules therefore prioritize preservation before geometric connectivity.