Source-linked AI summary
3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks
Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, Derek Hoiem
TL;DR
The paper addresses representing and reconstructing 3D objects from limited sensor data when shapes contain variable numbers of parts. It proposes 3D-PRNN, which sequentially generates primitive-based shapes and uses Gaussian-field fitting to create training data. The method achieves comparable accuracy to voxel-based reconstruction while using fewer representation degrees of freedom.
Problem
Predicting useful 3D geometry from noisy or partial observations is difficult because complex shapes require compact representations with variable numbers of parameterized components.
Method
3D-PRNN is a recurrent generative model that predicts context-sensitive sequences of oriented cuboid primitives, trained using Gaussian-field and energy-minimization fits from 3D meshes.
Results
3D-PRNN achieves comparable accuracy to voxel-based reconstruction methods despite having fewer representation degrees of freedom, while rotation-axis constraints significantly boost performance.
Takeaways & Limitations
Primitive-based generation provides a compact representation for reconstructing, synthesizing, and segmenting 3D shapes from partial observations.
Abstract
from arXiv · showhide
The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data. Inspired by the nature of human perception of 3D shapes as a collection of simple parts, we explore such an abstract shape representation based on primitives. Given a single depth image of an object, we present 3D-PRNN, a generative recurrent neural network that synthesizes multiple plausible shapes composed of a set of primitives. Our generative model encodes symmetry characteristics of common man-made objects, preserves long-range structural coherence, and describes objects of varying complexity with a compact representation. We also propose a method based on Gaussian Fields to generate a large scale dataset of primitive-based shape representations to train our network. We evaluate our approach on a wide range of examples and show that it outperforms nearest-neighbor based shape retrieval methods and is on-par with voxel-based generative models while using a significantly reduced parameter space.
1. Introduction
3D-PRNN represents 3D objects as compact sequences of primitives predicted recurrently from partial observations. The paper also introduces Gaussian-field fitting to create training representations and reports performance comparable to voxel methods despite fewer degrees of freedom.
- Motivation: 3D-PRNN addresses the challenge of predicting useful 3D geometry from noisy or partial observations by representing objects with oriented cuboid primitives.The representation is more compact than a 32x32x32 voxel grid and supports reasoning about object structure.
- Method: The model encodes an implicit shape representation and sequentially generates primitives using a recurrent neural network.Each generated primitive approximates part of the shape, enabling objects with varying numbers of components.
- Method: The recurrent generator combines Long Short-Term Memory networks with a Mixture Density Network to predict context-sensitive primitive sequences.Predictions are conditioned on the depth feature and previously sampled primitives.
- Results: The authors report comparable accuracy to voxel-based reconstruction methods despite using fewer representation degrees of freedom.Encoding symmetry and rotation-axis constraints significantly boosts performance.
- Training data: Gaussian Fields and energy minimization fit primitives to point clouds, producing primitive-based training data from unannotated 3D meshes.The resulting representations provide training samples for 3D reconstruction.
2. Related Work
The paper situates primitive-based modeling between longstanding component-based shape representations and modern voxel or retrieval methods. It uses recurrent generation to produce parameterized 3D primitives while explicitly encoding geometric constraints such as symmetry and rotation.
- Primitive-based shape modeling: Primitive-based representations model shapes as collections of components and have been used in psychology, computer vision, computer-aided design, and shape completion.The paper references Biederman’s geons and primitive-based scene sketches and completion methods.
- Existing reconstruction methods: Modern 3D reconstruction approaches have mainly emphasized retrieval from repositories or voxelized volumetric representations from RGB or depth images.Prior work also considers part deformation, symmetry, and skeleton-based reconstruction.
- Representation trade-off: Compared with mesh- or voxel-based reconstruction, aggregations of primitives offer lower computational and storage cost.The paper presents this as a benefit of its representation.
- RNN generative models: 3D-PRNN adapts recurrent generation ideas to sequentially produce parameterized oriented cuboids rather than text, handwriting strokes, pixels, or voxel structures.Its RNN combines explicit geometric constraints with sequential primitive prediction.
- Geometric constraints: The network separately predicts rotation-axis decisions and rotation amounts, which improves results over directly predicting rotation values.This design encodes symmetry and rotation structure in the recurrent model.
3. Fitting Primitives from Point Clouds
The paper fits rectangular primitives to point clouds through sequential energy minimization, using negative-space penalties, symmetry, and refinement to obtain compact shape decompositions.
- Primitive fitting: Rectangular cuboids provide a plausible abstraction for man-made objects and support parsing shapes with varying complexity into primitive sets.The method generates primitive-based reconstruction data from point clouds for training 3D-PRNN.
- Primitive fitting: Each primitive is parameterized by scale, translation, and Euler-angle rotation, while the goal is to find a set that best fits the input point cloud.The parameter vector is x = [sx, sy, sz, tx, ty, tz, θx, θy, θz].
- Energy minimization: The fitting energy combines Gaussian Force Fields and TSDF-based terms, with distant point pairs penalized less to accommodate partial matching.The resulting continuously differentiable energy is convex in a large neighborhood of the parameters.
- Energy minimization: A negative shape samples non-occupied space inside the object bounding box, encouraging primitives to fit tightly rather than merely covering observed points.Its energy contribution is weighted relative to the positive-shape fitting term by α.
- Sequential optimization: Sequential fitting randomly initializes 10 primitives, optimizes each candidate, selects the best one, removes its fitted points, and repeats until the point cloud is covered.The implementation alternates optimization of scale and translation with rotation updates.
- Refinement and symmetry: Symmetry duplicates a fitted primitive across a detected global symmetry plane, while coarse-to-fine fitting and post-refinement reduce accumulated errors.The method uses σ = 2 for fast convergence, σ = 0.5 for detailed fitting, and refines after every three primitives.
4. 3D-PRNN: 3D Primitive Recurrent Neural Networks
3D-PRNN encodes a depth image and recurrently generates a variable-length sequence of primitive configurations. It models parameter uncertainty with mixture distributions and improves rotation prediction through axis-aware constraints.
- Motivation: 3D-PRNN addresses the complex, multimodal distribution and variable primitive counts involved in generating primitive-based 3D shapes.It can generate novel shapes randomly or condition generation on a single depth map.
- Network architecture: Given a depth image, the network sequentially predicts each primitive’s dimensions, translation, rotation, and a binary end-of-generation signal.The stopping signal allows the generated sequence to have variable length.
- Network architecture: The depth encoder produces a 1×32 feature vector, while a three-layer LSTM recurrent generator combines that feature with the previously predicted primitive.The recurrent output parametrizes the distribution for the next primitive.
- Primitive distribution: Each axis parameter pair is modeled as a conditional mixture of K Gaussians, with mixture weights, means, standard deviations, and correlations predicted from the recurrent output.The experiments set K = 20, and sampling includes a binary stopping variable.
- Training objective: The mixture density network maximizes the log likelihood of ground-truth primitive parameters at each time step rather than relying on sequential sampling during training.The paper reports that this stepwise supervised training works well.
- Geometric constraints: Jointly predicting rotation axis and rotation value improves rotation prediction and alleviates overall primitive-distribution modeling.The axis and value use separate fully connected predictors with sigmoid and Tanh outputs, respectively.
- Evaluation: The validation comparison averages mixture-density and rotation-MSE losses by sequence length, with rotation values normalized and potentially reaching around 13 versus MSE below 1.The caption attributes better estimation performance to enforcing loss on the rotation axis.
- Shape synthesis: Randomly sampled outputs demonstrate that 3D-PRNN synthesizes varied primitive-based shapes without context input, with colors marking prediction order.The first recurrent feature is initialized from a randomly sampled primitive configuration.
5. Experiments and Discussions
Experiments evaluate 3D-PRNN for synthetic and real single-depth-view reconstruction, shape synthesis, and segmentation. The model generates plausible class-representative shapes, matches voxel methods in IoU with fewer representation degrees of freedom, and has limitations on difficult real-data classes.
- Evaluation setup: 3D-PRNN is quantitatively evaluated on synthetic depth maps and real NYU Depth V2 depth maps.
- Evaluation setup: The ModelNet training set contains 889 chairs, 392 tables, and 200 nightstands, with one network trained jointly across classes.
- Model operation: The model predicts primitives sequentially until a stop signal and uses nearest-neighbor initialization from the encoded depth feature at test time.
- Shape synthesis: Random sampling produces representative shapes across multiple classes and sometimes hybrids from multiple classes.
- Quantitative reconstruction: 3D-PRNN achieves comparable accuracy to voxel-based reconstruction despite using fewer representation degrees of freedom, while rotation-axis constraints significantly improve performance.
- Real-data reconstruction: On real NYU Depth V2 data, nightstands perform worse because limited examples and occlusions can make them resemble tables or chairs.
- Shape segmentation: Primitive-based reconstructions also support shape segmentation because their parts follow meaningful configurations.
- Conclusion: The conclusion reports comparable reconstruction results with compact parametric models and identifies cylinders, spheres, joints, and spatial relationships as future directions.
A. Vp in primitive fitting energy
The primitive fitting energy uses volumetric sampling and differentiable optimization to fit primitives to shapes. Its derivatives with respect to translation, scale, and rotation support robust numerical minimization.
- Vp is defined as a primitive’s volume divided by its number of sampled points m.
- The primitive volume is sxsysz, and experiments fix the sample count at m = 7 × 7 × 7.
- The differentiable fitting energy is optimized with robust L-BFGS techniques.
- Analytic derivatives are provided for translation T, scale S, and rotation θ in the primitive parameter set x.
C. Evaluation on primitive fitting
Primitive fitting is evaluated on a labeled ShapeNet chair dataset using triangle-face labeling accuracy. Predicted primitive assignments are projected onto sampled shape points and compared with ground-truth part labels.
- The evaluation uses 704 test-split chairs with labels for seat, back, arm, and leg parts.
- Face labeling accuracy measures the fraction of triangle faces assigned the correct part labels.
- For each voxelized predicted shape, N points are sampled, assigned to the nearest predicted primitive, and projected to face labels.
D. LSTMs sequential prediction order
The recurrent generator predicts primitive parameters in a fixed sequential order, emitting size, translation, rotation, and stopping information. Sampling choices affect stability because atypical samples can accumulate prediction error.
- Sequential parameter order: The generator predicts primitive size parameters sequentially, with each sampled instance drawn from the distribution f(yt).
- Sequential parameter order: For each primitive, time steps encode width, length, and height together with x-, y-, and z-axis translations.
- Termination: Generation stops when the End of Generation signal is predicted.
- Rotation prediction: Rotation values and binary rotation-axis indicators are predicted alongside the main primitive sequence.
- Sampling stability: An unexpected sample far from the distribution mean can cause accumulated error in subsequent predictions.
- Sampling stability: Synthetic testing samples from the two most probable mixture components, whereas real-data testing samples randomly across all components to produce successive reasonable shapes.
F.1. Synthetic data
Synthetic-depth experiments show qualitative reconstructions from 3D-PRNN alongside multiple sampled outputs and the ground-truth primitive representation.
- The figure presents the input synthetic depth map and the most probable 3D-PRNN reconstruction.
- It also shows three successive random sampling results from the model.
- Each reconstruction is compared with the ground-truth primitive representation.
F.2. Real data
Real-depth experiments show qualitative reconstructions from 3D-PRNN alongside multiple sampled outputs and the ground-truth primitive representation.
- The figure presents the input real depth map and the most probable 3D-PRNN reconstruction.
- It also shows two successive random sampling results from the model.
- Each reconstruction is compared with the ground-truth primitive representation.
G. Application: shape segmentation
The paper applies primitive-based reconstructions to semantic shape segmentation, evaluating 3D-PRNN on COSEG chairs with oriented-box ground truth and held-out testing.
- Application: shape segmentation: Primitive reconstructions align with semantic part configurations, enabling their use for segmenting fully observed input shapes.
- Evaluation setup: COSEG evaluation uses tightest oriented boxes for labeled segments as ground-truth primitives, with ordering determined by box-center height.
- Evaluation setup: The evaluation trains on 85% of the data and tests on the remaining 15% of unseen shapes.
- Training: Fine-tuning after pre-training on ModelNet chairs increases segmentation accuracy by 5%.
- Results: When 3D-PRNN predicts the correct number of primitives, it outperforms Kim et al.'s template-based segmentation method.