Source-linked AI summary
Learning to Infer and Execute 3D Shape Programs
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu
TL;DR
3D shape understanding often captures geometry without higher-level relationships such as symmetry and repetition. This paper introduces executable shape programs and learns to infer them from unlabeled shapes, achieving accurate reconstructions and generalization across categories and from images.
Problem
Existing 3D shape representations and sensing methods capture low-level geometry but less effectively represent higher-level structure such as symmetry and repetition.
Method
The method combines a DSL for geometric parts and structural statements with a neural program synthesizer and executor trained through self-supervised reconstruction.
Results
Experiments show accurate shape-program inference on complex ShapeNet shapes, generalization beyond training categories, and more accurate, physically plausible reconstructions from color images.
Takeaways & Limitations
Executable shape programs provide a representation that captures object geometry together with semantic and structural relationships across parts.
Takeaways & Limitations
The current DSL focuses on furniture semantics, which can limit generalization beyond the training superclass.
Abstract
from arXiv · showhide
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.
1 INTRODUCTION
The paper argues that 3D shape understanding should capture structural regularities such as symmetry and repetition, not only low-level geometry. It introduces shape programs and learns to infer and execute them without program annotations.
- Motivation: Human perception recognizes both geometric parts and higher-level regularities such as bilateral symmetry and repeated, equally spaced legs.These structural relationships support visual recognition and reasoning.
- Motivation: Existing voxel, point-cloud, mesh, and primitive-based methods capture geometry but fall short of representing strong shape priors and within-shape regularity.The paper specifically highlights sharp edges, smooth surfaces, symmetry, and repetition.
- Approach: The proposed shape-program DSL combines geometric and semantic primitives with loop statements that enforce higher-level structural priors.This representation is intended to capture both object parts and relationships among them.
- Approach: Because real shapes lack program annotations, a learned executor enables self-supervised inference by reconstructing unlabeled shapes from predicted programs.The model is initially bootstrapped before learning through reconstruction.
- Results: Experiments show accurate program inference for complex ShapeNet shapes and more accurate, physically plausible reconstructions when the model is extended to RGB images.The image-based extension integrates with an image-to-shape reconstruction module.
2 RELATED WORK
Related work spans inverse procedural graphics, geometric and CAD program inference, interactive procedural modeling, neural program synthesis, and neural program execution. This paper differs by recovering 3D graphics programs end to end from voxel inputs while modeling semantically meaningful parts and relationships.
- Inverse procedural graphics: Inverse procedural graphics systems often rely on large shape-component libraries or sparse solution spaces to make program inference tractable.The cited work frames program inference from images or declarative specifications.
- 3D shape representations: Recent 3D methods use geometric primitives or CAD programs, while this paper emphasizes semantically meaningful parts and their relationships.The distinction extends primitive-based representations toward structured 3D programs.
- Interactive procedural modeling: Interactive modeling work learns to instantiate procedural primitives with human assistance, whereas this approach instantiates multiple primitives without user assistance.The comparison concerns the degree of human involvement during inference.
- Program synthesis: Neural program synthesis has recovered 2D graphics programs and accelerated low-level cue inference, while this paper applies an end-to-end network to 3D voxel inputs.The paper positions its method alongside symbolic and neural program-synthesis approaches.
- Program execution: Neural Program Interpreters decompose execution traces into primitive operations and predict subsequent actions from state and prior actions.This provides context for learning to execute structured programs.
3 3D SHAPE PROGRAMS
The 3D shape-program representation uses a DSL whose Draw statements describe semantic parts and whose For statements express repeated transformations. Programs are generated sequentially through block- and step-level prediction to reconstruct voxelized input shapes.
- DSL: The DSL defines programs as variable-length sequences of Draw or For statements, which are tokenized for neural prediction.Draw specifies a primitive; For contains a repeatedly executed subprogram and its parameters.
- DSL: Shape primitives encode geometry such as position and orientation together with semantic roles such as top, back, or leg.Semantic attributes associate parts with meanings without changing their geometry.
- DSL: For statements encode high-level regularities such as symmetric legs or regularly spaced bars and can recursively contain subprograms.This supports repeated and transformed part arrangements.
- Program generation: Two LSTMs generate the program hierarchically: the Block LSTM produces block features, and the Step LSTM predicts each block's internal program.The generator uses rendered previous blocks and the raw shape as inputs to a 3D ConvNet before recurrent prediction.
- Task definition: The inference task is to predict a program whose execution reconstructs the input voxelized shape at 32 × 32 × 32 resolution.The program is evaluated by executing it against the input shape.
4 INFERRING AND EXECUTING 3D SHAPE PROGRAMS
Shape Programs combines a program generator with a neural executor to infer structured programs and reconstruct 3D shapes through differentiable execution. The generator uses block- and step-level recurrent modeling, while guided adaptation trains it on unlabeled shapes by back-propagating reconstruction errors.
- Model overview: The model generates primitive programs from an input 3D shape and executes them to produce a corresponding reconstructed shape.The executor enables self-supervised learning by comparing generated reconstructions with input shapes.
- Program generator: Program generation uses blocks for individual parts or compound symmetry structures, reflecting a part-based decomposition of shapes.Blocks may describe single drawing statements or loops that repeatedly translate or rotate parts.
- Program generator: Two orthogonal LSTMs model programs hierarchically: a Block LSTM connects blocks, while a Step LSTM generates tokens within each block.Rendered previous blocks and the raw shape are combined and processed by a 3D ConvNet before block-level prediction.
- Neural program executor: The executor encodes program steps with an LSTM and decodes their features into a 3D shape using a 3D DeconvNet.This learned executor provides a differentiable alternative to explicitly executing symbolic commands.
- Guided adaptation: Block-level shapes are combined by max-pooling, allowing variable-length programs and gradient propagation through vacant tokens.The resulting reconstruction is compared with the input, and voxel reconstruction errors update the program generator during guided adaptation.
5 EXPERIMENTS
Experiments evaluate program generation, execution, guided adaptation, structural validity, generalization, and image-based shape completion. Across these settings, the model reconstructs complex shapes, adapts to unseen categories, and produces more stable, connected, or perceptually preferred results.
- Experimental setup: The model was evaluated on synthetic data, ShapeNet, and Pix3D using a single pretrained model subsequently adapted with neural program execution.The ShapeNet and Pix3D adaptations use reconstruction guidance without program-level supervision.
- Synthetic dataset: More than 99.9% of synthetic programs were accurately predicted, with rendered shapes reaching an average IoU of 0.991.The synthetic training set contained 100,000 chairs and 100,000 tables, while evaluation used 5,000 shapes.
- Program executor: The neural program executor achieved IoU 0.93 for single drawing statements and 0.88 for compound statements against ground-truth shapes.It was trained on 500,000 synthetic block-program and shape pairs and tested on 30,000 pairs.
- Guided adaptation on ShapeNet: On ShapeNet, guided adaptation improved reconstruction metrics, while the model without adaptation outperformed Tulsiani et al. and CSGNet by capturing symmetry and translation regularities.The evaluation used IoU, Chamfer distance, and Earth Mover’s distance; the nearest-neighbor baseline did not generalize well to test shapes.
- Qualitative adaptation: Guided adaptation produced more meaningful programs and reasonably reconstructed noisy chair and table inputs by adding or deleting translations, rotations, and templates.The changes included modeling armrests, four legs, horizontal bars, rectangle layers, and previously unseen table templates.
- Stability and connectivity: The model produced significantly more stable and connected shapes, and guided adaptation further improved connectivity and overall stability-related results.Stability is defined using the center of mass and ground contacts; connectivity requires all voxels to form one connected component.
- Generalization: For unseen beds, sofas, cabinets, and benches, guided adaptation increased IoU from 0.23 to 0.37, 0.30 to 0.60, 0.25 to 0.48, and 0.18 to 0.42, respectively.These categories were absent during pre-training and were adapted using unlabeled shapes.
- Shape completion and smoothing: On Pix3D, stacking the model on MarrNet did not change IoU, but 78.9% of AMT responses preferred the model’s more visually appealing and human-perceptible results.The program-based stage smooths and completes fragmentary shapes reconstructed from 2D images.
6 DISCUSSION
The model combines neural program synthesis and execution to explain shapes as programs, while exposing interpretable latent geometric controls. Its DSL supports within-superclass generalization but may constrain transfer across broader domains.
- The model combines a neural program synthesizer with a neural executor to infer shape programs and reconstruct shapes.
- The model explains ShapeNet shapes, generalizes beyond training categories, and can infer programs from color images through an image-to-shape extension.
- The executor’s 64-dimensional intermediate representation contains interpretable dimensions for primitive height, radius, and repetition count.
- Design of the DSL: The furniture-focused DSL supports semantic correspondences and within-superclass generalization but may limit transfer to other superclasses or domains.
- Structure search vs. amortized inference: 5 ms is required to infer a shape program with a Titan X GPU using amortized inference rather than structure search.
A.1 DEFINED PROGRAMS
The furniture DSL uses category-specific and shared semantic parts, with associated parameters listed for the model’s defined programs.
- Some shape semantics are category-specific, such as “ChairBeam,” while others are shared across categories, such as “Leg” and “Top”.
- The defined programs encode furniture parts through semantic labels and their associated parameters.
- Table 5 lists the semantics, shapes, and associated parameters used by the model.
A.2 ARCHITECTURE DETAILS
The architecture uses convolutional and recurrent modules for program generation and execution, with a differentiable executor connecting continuous inferred programs to voxel reconstruction.
- Program Generator: The program generator uses an eight-layer 3D ConvNet followed by block and step LSTMs.
- Figure A1 contrasts synthetic table templates with raw and reconstructed ShapeNet tables.
- Program Executor: The program executor uses an LSTM to aggregate block-level programs and a 3D DeConvNet to reconstruct voxels.
- End-to-end differentiability: The neural executor jointly approximates program discretization and execution, making the overall pipeline end-to-end differentiable.
A.3 SYNTHETIC TEMPLATES V.S. SHAPENET
The synthetic dataset provides simple table templates, whereas ShapeNet contains substantially more complex tables; this complexity gap affects reconstruction performance.
- The synthetic dataset contains 10 simple table templates designed to provide minimal guidance to the network.
- ShapeNet tables are significantly more complex than the sampled synthetic templates.
- The disparity in complexity explains the dramatic drop in IoU when the model is tested directly on ShapeNet.
A.4 ADDITIONAL RESULTS
Additional results compare synthetic pretraining with guided adaptation on ShapeNet across several object categories. Guided adaptation improves program inference and reconstruction, while reducing mispredictions when transferring beyond the synthetic training categories.
- Cross-category results: Figures A2–A8 compare generated shapes and programs before and after guided adaptation across chairs, tables, benches, couches, cabinets, and beds.Chairs and tables were seen during synthetic pretraining; benches, couches, cabinets, and beds represent additional categories shown in the figures.
- Adaptation effects: For categories beyond the synthetic training set, direct transfer can misinterpret shapes as tables or chairs.The pretrained network was trained jointly only on synthetic table and chair objects.
- Adaptation effects: Even for chairs and tables, guided adaptation improves program inference and shape reconstruction on ShapeNet.The improvement reflects the greater complexity of ShapeNet relative to the synthetic data.
- Adaptation effects: Guided adaptation dramatically reduces these cross-category mispredictions.The reduction is reported after adapting the pretrained model on ShapeNet data.