Source-linked AI summary
Learning Shape Abstractions by Assembling Volumetric Primitives
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik
TL;DR
The paper addresses how to explain complex 3D shapes with simpler, interpretable structures without primitive annotations. It trains CNNs unsupervised to assemble shapes from volumetric cuboids, jointly learning shared structure across a dataset. The resulting representations support consistent parsing, interpretable shape similarity, image-based prediction, and shape manipulation.
Problem
Complex 3D shapes are high-dimensional, while existing visual understanding commonly relies on human-labeled objects or parts rather than simpler underlying structures.
Method
An unsupervised CNN predicts up to M primitive parts and composes them into a target 3D object, using distance-field and surface-sampling losses.
Results
The learned primitives capture coarse, structurally varying shapes consistently across chairs, aeroplanes, and animals, while parsing reaches 89.0% mean accuracy.
Takeaways & Limitations
The framework discovers descriptive, consistent, interpretable shape abstractions that can support parsing, shape similarity, perception, and manipulation.
Takeaways & Limitations
The reported parsing comparison is not exactly comparable because the cited prior work used a preliminary Shape COSEG dataset version unavailable to the authors.
Abstract
from arXiv · showhide
We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives. In addition to generating simple and geometrically interpretable explanations of 3D objects, our framework also allows us to automatically discover and exploit consistent structure in the data. We demonstrate that using our method allows predicting shape representations which can be leveraged for obtaining a consistent parsing across the instances of a shape collection and constructing an interpretable shape similarity measure. We also examine applications for image-based prediction as well as shape manipulation.
1. Introduction
The paper revisits explaining complex 3D objects with volumetric primitives, using unsupervised learning and CNNs to discover shared structure across datasets. It produces parsimonious, interpretable representations that support several shape-understanding applications.
- Motivation: Volumetric primitives offer a compact way to describe complex objects using relatively few generalized cylinders and parameters.This idea dates to generalized cylinders and was motivated by parsimony and single-image shape inference.
- Motivation: Learning-based visual understanding exploits repeating structure, but complex structures are usually modeled through human-labeled objects or parts.The paper argues for explaining complex structures through simpler underlying structures instead.
- Approach: The method trains CNNs with unsupervised learning to assemble arbitrary 3D objects from rigidly transformed cuboids.Jointly explaining the full dataset allows common 3D patterns to be learned directly from data.
- Contribution: Primitive-based explanations are parsimonious despite meshes and voxel occupancies being complex and high-dimensional.The learned representation is applied to part discovery, image-based abstraction, and shape manipulation.
2. Related Work
Prior work emphasized compact 3D representations and exploited compositionality for parsing, reconstruction, and manipulation. This paper distinguishes its approach through data-driven learning and consistent indexed representations that enable efficient inference on novel shapes.
- 3D Representation and Reconstruction: Classical 3D modeling approaches emphasized compact representations, while mainstream reconstruction methods typically used higher-dimensional representations.Modern primitive-based efforts included Lego pieces and qualitative 3D blocks.
- Comparison with Classical Methods: Unlike per-instance classical optimization, this learning-based approach produces a consistent indexed set of primitives and enables efficient test-time inference for novel shapes.The consistent ordering supports applications examined later in the paper.
- Parsing Objects, Scenes and 3D Shapes: Data-driven compositionality has supported unsupervised discovery, co-segmentation, scene parsing, novel scene generation, and 3D shape analysis.In 3D shapes, compositionality has been used for representation, parsing, and manipulation.
- Deep Generative Models: Deep generative models can learn visual data distributions, but latent spaces may lack interpretability, while some factorized methods rely on supervision.These limitations motivate approaches that discover interpretable structure directly from data.
3. Learning Object Assembly
The framework learns a CNN to assemble 3D objects from transformed volumetric primitives, using shape agreement with the target for unsupervised training. It separates primitive shape from spatial transformation and supports variable numbers of parts.
- Core formulation: A CNN predicts up to M primitive parts and their parameters, which are composed into an assembled target shape without primitive annotations.Training uses agreement between the assembled shape and the target object as supervision.
- Primitive representation: Each primitive is represented by canonical shape z and transformation parameters q and t, separating part geometry from its spatial placement.This factorization supports independent reasoning about shape and transformation variation.
- Loss function: The loss combines Coverage Loss and Consistency Loss to penalize missing target regions and predicted geometry extending beyond the target.Distance fields and sampled primitive-surface points make these comparisons differentiable.
- Loss function: Composed-shape distance fields are computed as the pointwise minimum of the distance fields of the constituent primitives.For cuboids, the primitive distance field is computed from their three-dimensional extents.
- Loss function: Rigid transformations evaluate a transformed primitive’s distance field by mapping a query point back to the canonical object frame.The inverse transformation uses rotation and translation parameters and remains differentiable.
- Variable number of primitives: The variable-part extension predicts each primitive’s existence probability p_m, allowing instances to use different numbers of primitives within a maximum of M.A Bernoulli existence variable determines whether each predicted primitive is included in an assembly.
4. Experiments
Experiments on chairs, aeroplanes, and animals evaluate the framework’s data, training procedure, predictions, and tradeoff between parsimony and reconstruction accuracy.
- Dataset: Experiments use ShapeNet airplane and chair categories with thousands of aligned, fixed-scale meshes, plus around 100 manually downloaded four-legged animal models.
- Network Architecture and Training: The CNN encodes 32 ∗32 ∗32 occupancy grids and predicts transformed cuboid primitive parameters, trained so assembled shapes match target objects.
- Implementation Details: The two-stage procedure first fixes high primitive-existence probabilities, then learns them with a parsimony reward to reduce noisy early gradients and encourage simpler assemblies.
- Results and Analysis: Predictions capture coarse structure consistently across chairs, aeroplanes, and animals while representing structural variations, including rarer configurations.
- Results and Analysis: The loss initially decreases rapidly, then rises when primitive probabilities become learnable because some primitives disappear; fewer-primitive reconstructions trade accuracy for parsimony.
- Results and Analysis: Post-processing removes predicted parts that significantly overlap with others, producing progressively more parsimonious representations during training.
5. Applications
The learned primitive representation supports consistent part parsing, interpretable and controllable shape similarity, image-based abstraction, and shape manipulation across objects.
- 5.1. Unsupervised Parsing and Correspondence: The same primitive can explain corresponding parts across a category, enabling consistent point labeling for unsupervised parsing.Points are assigned to the primitive with the lowest distance field, producing category-wide correspondences.
- 5.1. Unsupervised Parsing and Correspondence: 89.0% mean accuracy is obtained on Shape COSEG, compared with 78.6% and 84.8% for the initial and refined parsings of.The evaluation assigns each predicted partition a ground-truth label despite differing numbers of clusters.
- 5. Applications: The learned parameter set succinctly captures the geometry of each underlying object.The trained CNN maps every 3D shape to primitive parameters {(z_m, q_m, t_m)}.
- 5.2. Interpretable Shape Similarity: The primitive embedding separates fine-level chair structure better than voxel IoU and supports similarity measures focused on selected parts or properties.Using chair back and seat primitives ignores handles, while chair-back orientation produces a 1D manifold ordered by tilt.
- 5.3. Image-Based Abstraction: A model trained from volume inputs can be distilled into an image-based predictor that outputs the underlying shape abstraction.The image network is trained on rendered volume-image pairs to match the volume model’s primitive predictions.
- 5.4. Shape Manipulation: Primitive abstractions provide a skeleton for manipulating meshes by assigning local primitive-frame coordinates and transforming the corresponding primitives.The method demonstrates deformation toward a target configuration and supports edits such as lengthening legs or tilting a back.
6. Conclusion
The paper demonstrates data-driven 3D object explanation using cuboidal primitives as a simple setting for parsimonious visual descriptions. It positions this framework as an initial step toward richer parametrized primitive catalogues for understanding 3D structure.
- 6. Conclusion: The framework uses unsupervised, data-driven learning to explain visual information with 3D volumetric entities rather than 2D pixel patches.It draws inspiration from generalized cylinders and geons while letting the data discover the representation.
- 6. Conclusion: The demonstrated setting explains objects from cuboidal primitives and serves as an initial step toward parsimonious descriptions of visual input.The conclusion suggests extending the approach with a wider catalogue of basic parametrized primitives.
A1. Consistency Loss via Differentiable Surface Sampling
The consistency loss is made differentiable by sampling primitive surfaces through a reparameterization and weighting samples according to cuboid-face areas.
- The consistency loss samples points on each primitive surface and penalizes their squared distance to the ground-truth object.
- Surface points are generated by sampling an untransformed primitive and then applying its predicted rotation and translation.
- Cuboid dimensions parameterize the primitive, while reparameterization separates random sampling from these dimensions for gradient computation.
- Sampling each face equally and assigning area-proportional importance weights reproduces uniform cuboid-surface sampling.
- Because the importance weights are differentiable with respect to primitive dimensions, the consistency loss is differentiable with respect to predicted parameters.
A2. Gradient computation for primitive existence probabilities pm
Primitive existence probabilities are differentiated using sampled categorical choices, a parsimony reward for absent primitives, and a running-loss baseline for variance reduction.
- The method represents each primitive’s available choices with probabilities, including nonexistence and cuboid existence.
- The categorical choice can be extended beyond nonexistence and cuboid existence, for example by adding a cylinder option.
- The sampled primitive choice incurs a loss l, while absent primitives receive an external parsimony reward r.
- A running average of incurred losses is subtracted as baseline b to reduce variance in the estimated gradients.
A3. Architecture and Initialization
The network uses a five-layer 3D CNN followed by fully connected layers, with initialization that activates many small primitives before learning organizes them.
- The architecture has five strided 3D convolution layers followed by two 100-unit fully connected layers and a final primitive-parameter prediction layer.
- The primitive count M is set above twice the expected number of parts, using M=20 for chairs and smaller values for planes and animals.
- Primitive dimensions are initialized as small cubes with existence probability pm = 0.9, while other parameters and weights are random.
- Training is reported to discover consistent solutions despite random initialization because of commonalities in the data.
A4. Visualization
The paper visualizes primitive representations, their projected labels, Shape COSEG parsing, and several shape-embedding constructions.
- Shape COSEG: Shape COSEG visualizations compare dataset meshes, inferred primitive representations, unsupervised mesh parsing, ground-truth labels, and assigned predicted labels.
- Shape Embedding: Embedding visualizations use voxel IoU, squared Euclidean distance in primitive space, selected chair primitives, or chair-back orientation.