PaperlayerView exact revision on arXiv

Source-linked AI summary

ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis

R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie

arXiv:2009.08026v1cs.GRcs.CVcs.LG

TL;DR

3D shape generation seeks both the broad variability of deep models and the quality and editability of procedural representations. ShapeAssembly addresses this gap by training a hierarchical sequence VAE to generate executable, parameterized shape programs. The resulting shapes are more plausible and physically valid than those of other methods, with smoother latent interpolations and useful point-cloud fitting, subject to program-extraction and connectivity limitations.

  • Problem

    Existing procedural models are difficult to author and limited in variation, while deep generative models can produce artifacts and representations that are difficult to edit.

    Source: §1 · p. 2

  • Method

    ShapeAssembly defines a differentiable DSL for hierarchical cuboid assemblies and trains a hierarchical sequence VAE on extracted programs to generate new executable shape programs.

    Source: §1 · p. 2, Defining a DSL for connected,… · p. 4

  • Results

    Generated shapes are more plausible and physically valid than those from other methods, while the learned latent space is better structured with smoother interpolations and supports improved point-cloud program fitting.

    Source: §1 · p. 3, §7.1.1 · p. 9

  • Takeaways & Limitations

    Generating procedural programs combines neural models’ variability with procedural representations’ interpretability, editability, and physically valid construction.

    Source: §1 · p. 2

    Training excludes programs whose orientations cannot be specified solely through part-to-part attachments and shapes with more than 12 cuboid declarations, limiting learned structural complexity.

    Source: Limitations · p. 15

Abstract

from arXiv · show

Manually authoring 3D shapes is difficult and time consuming; generative models of 3D shapes offer compelling alternatives. Procedural representations are one such possibility: they offer high-quality and editable results but are difficult to author and often produce outputs with limited diversity. On the other extreme are deep generative models: given enough data, they can learn to generate any class of shape but their outputs have artifacts and the representation is not editable. In this paper, we take a step towards achieving the best of both worlds for novel 3D shape synthesis. We propose ShapeAssembly, a domain-specific "assembly-language" for 3D shape structures. ShapeAssembly programs construct shapes by declaring cuboid part proxies and attaching them to one another, in a hierarchical and symmetrical fashion. Its functions are parameterized with free variables, so that one program structure is able to capture a family of related shapes. We show how to extract ShapeAssembly programs from existing shape structures in the PartNet dataset. Then we train a deep generative model, a hierarchical sequence VAE, that learns to write novel ShapeAssembly programs. The program captures the subset of variability that is interpretable and editable. The deep model captures correlations across shape collections that are hard to express procedurally. We evaluate our approach by comparing shapes output by our generated programs to those from other recent shape structure synthesis models. We find that our generated shapes are more plausible and physically-valid than those of other methods. Additionally, we assess the latent spaces of these models, and find that ours is better structured and produces smoother interpolations. As an application, we use our generative model and differentiable program interpreter to infer and fit shape programs to unstructured geometry, such as point clouds.

1 INTRODUCTION

ShapeAssembly combines procedural shape programs with deep generative modeling to produce editable, variable 3D shape structures. The approach generates plausible and physically valid shapes while supporting smoother latent interpolation and point-cloud fitting.

Motivation

  • The pipeline addresses the trade-off between procedural editability and deep models’ broad variability by learning neural models that generate procedural programs.Procedural models are difficult to author and limited in variation, whereas deep models can produce artifacts and inscrutable representations.Source: §1 · p. 2

Method

  • ShapeAssembly represents hierarchical shape structures with cuboids, attachments, symmetry, parameterized dimensions, and a differentiable interpreter.Its program parameters can describe families of related shapes while retaining interpretable, editable structure.Source: §1 · p. 2
  • A hierarchical sequence VAE learns to write ShapeAssembly programs from extracted dataset programs, while semantic constraints prevent invalid outputs.Each hierarchy node uses a recurrent language model to generate program text and choose subroutine expansions.Source: §1 · p. 2

Results

  • Generated shapes are more plausible and physically valid than those from other methods, with better-structured latent spaces and smoother geometric and structural interpolations.The evaluation compares plausibility, diversity, complexity, and physical validity across recent shape-structure generators.Source: §1 · pp. 2–3

Application

  • The method also supports fitting generated programs to unstructured point clouds, with differentiable interpreter optimization magnifying the performance advantage.This application combines program decoding with parameter optimization against unstructured geometry.Source: §1 · p. 3

Contributions

  • The paper contributes the ShapeAssembly language and its differentiable interpreter.These components provide the procedural representation and differentiable execution used by the generative pipeline.Source: §1 · p. 3

2 RELATED WORK

ShapeAssembly differs from prior 3D shape generators and procedural systems by generating procedural programs rather than single shapes or fixed procedural models. It combines stochastic neural program generation with deterministic, editable shape construction.

Deep Generative Models of 3D Shapes.

Procedural and Inverse Procedural Modeling.

  • Procedural modeling systems commonly use grammars, whereas ShapeAssembly imperatively declares and connects cuboid proxy parts.Its representation is designed for connected part assemblies rather than recursive string rewriting or spatial splitting.Source: Procedural and Inverse Procedural… · p. 3
  • Prior inverse procedural methods infer interpretable stochastic programs from examples, while ShapeAssembly uses deterministic shape programs and a stochastic neural network to generate them.This separates the representation of each shape from the learned generation process.Source: Procedural and Inverse Procedural… · p. 3

Visual Program Induction.

  • ShapeAssembly addresses a different problem from visual program induction by training a generative model to create novel shape programs from scratch.Program extraction from existing shapes is used as a subroutine to create training data.Source: Visual Program Induction · p. 3
  • The paper claims the first deep generative model that produces novel shape programs, each representing a parametric family of related 3D shapes.This pipeline combines neural generation with procedural shape modeling.Source: Visual Program Induction · p. 3

3 APPROACH

The approach converts hierarchical 3D part graphs into ShapeAssembly programs and trains a generative model to produce new executable programs. The pipeline combines a connected hierarchical DSL, canonical program extraction, and neural program generation.

Input.

  • The pipeline starts from hierarchical 3D part graphs whose nodes are parts, attachment edges encode physical connections, parent-child edges encode hierarchy, and leaves use cuboid proxies.These graphs provide the structural input from which programs are extracted.Source: Input · p. 3

Defining a DSL for connected, hierarchical shapes.

  • ShapeAssembly is a DSL that represents shapes by declaring parts and attaching them, extending connected cuboid assemblies to hierarchical structures.The language is built around the part-connection structure of the input graphs.Source: Defining a DSL for connected,… · p. 4

Creating a dataset of shape-program pairs.

  • Programs are extracted through data cleaning and geometric analysis, then canonicalized using ordering and filtering steps to form shape-program pairs.This procedure supplies standardized training programs from the input dataset.Source: Creating a dataset of shape-program… · p. 4

Learning to generate programs.

  • The pipeline trains a deep generative model on extracted programs so sampled programs can be executed to synthesize novel shape structures.The generated programs are the final learned representation rather than direct geometry outputs.Source: Learning to generate programs · p. 4, Figure 2 · p. 4
  • A hierarchical ShapeAssembly program can execute either as a flat assembly or with subprograms expanding selected parts into detailed components.The Chair example illustrates hierarchical expansion through Base and Back subprograms.Source: Figure 3 · p. 4

Defining a DSL for connected, hierarchical shapes.

  • The interpreter incrementally instantiates cuboids at the origin and moves them through attachment, while macros such as reflect create additional cuboids and attachments.This imperative execution model constructs the shape command by command.Source: Figure 4 · p. 5

4 AN ASSEMBLY LANGUAGE FOR SHAPES

ShapeAssembly is a domain-specific language for constructing hierarchical shape structures from connected cuboid parts. Its imperative attachment operations and symmetry macros produce differentiable, editable assemblies, including nested sub-programs.

4 AN ASSEMBLY LANGUAGE FOR SHAPES

  • ShapeAssembly represents shape structures using cuboid part proxies connected through attachment operations.Cuboids are the language’s sole data type for shape structure synthesis; surface geometry can be generated later using other techniques.Source: §4 · p. 5, Table 1 · p. 5
  • Attachment operations position and orient cuboids, while reflect, translate, and squeeze macros encode higher-level spatial relationships.Macros expand into multiple cuboid and attachment commands during execution.Source: §4 · p. 5, Table 1 · p. 5
  • Its four program blocks declare a bounding volume, cuboids, attachments, and symmetry transformations.The BBlock, CBlock, ABlock, and SBlock organize geometry declarations, connections, and repeated instances.Source: §4 · p. 5

Semantics.

  • Imperative execution immediately changes the constructed geometry, and the resulting geometry is differentiable with respect to continuous program parameters.Gradients can be computed with respect to values such as cuboid dimensions and attachment-point locations.Source: Semantics · pp. 5–6
  • Hierarchical shapes are represented by treating selected non-leaf cuboids as bounding boxes for nested ShapeAssembly programs.This allows cuboids to expand into sub-programs within the hierarchy.Source: Handling hierarchy · p. 6

5 TURNING SHAPES INTO TRAINING PROGRAMS

The authors convert hierarchical part graphs into canonical, validated ShapeAssembly programs through regularization, geometric parsing, ordering, and filtering. The pipeline identifies attachments and symmetry while constraining execution order and program complexity.

5 TURNING SHAPES INTO TRAINING PROGRAMS

  • The extraction pipeline converts each hierarchical part graph into program information, candidate programs, and a validated ShapeAssembly program.The process includes data regularization, cuboid-parameter recording, attachment detection, symmetry identification, ordering, and validity checks.Source: §5 · p. 6, §5.1 · p. 6

Regularization.

  • Regularization locally flattens hierarchies and shortens intersecting leaf parts to create more regular surface-to-surface connections.These preprocessing steps make physically related leaf parts more often siblings and reduce interpenetration.Source: Regularization · p. 6

Cuboids.

  • Attachments are located from dense cuboid-surface point-cloud intersections, while symmetry groups retain cuboids sharing axis-aligned reflectional or translational relationships and connectivity.Squeeze relationships replace compatible attachments, and symmetry groups are represented by transforms applied to a single cuboid.Source: Attachment · p. 6, Symmetry · p. 6

5.2 Creating Candidate Programs

  • Canonical ordering sorts cuboid and symmetry lines semantically, then selects grounded attachment orders in which each new cuboid connects to an already grounded cuboid.Geometric fit and specified semantic tie-breakers resolve remaining ordering ambiguities.Source: §5.2 · pp. 6–7

5.3 Validating Programs

  • Programs are retained only if they reconstruct the target geometry, obey connectivity and bounding-volume semantics, and contain no more than 12 leaf cuboid instantiations.Reconstruction is checked with a point-cloud F-score threshold of 75; symmetry expansion can still produce more than 12 executed leaf cuboids.Source: §5.3 · p. 7, Reconstruction · p. 7, Semantics · p. 7

6 LEARNING TO GENERATE PROGRAMS

The generative model is a hierarchical sequence VAE that encodes ShapeAssembly programs into latent representations and recursively decodes new hierarchical programs. Training combines line-level sequence reconstruction with semantic validity enforcement during generation.

6.1 Model Architecture

  • The hierarchical sequence VAE encodes ShapeAssembly sub-programs from leaves to root and recursively decodes hierarchical programs from latent codes.Separate μ and σ vectors parameterize the VAE bottleneck.Source: §6.1 · p. 7, Figure 6 · p. 8
  • Within each hierarchy node, a GRU predicts successive grammar lines, while the line decoder uses bounding-volume information to output a 63-dimensional command representation.The representation includes command type, cuboid attributes, indices, attachment coordinates, squeeze parameters, and symmetry parameters.Source: §6.1 · p. 7

Hierarchical decoding.

  • After each Cuboid prediction, a child module decides whether to expand it into a sub-program and produces the child latent code when expansion occurs.This explicitly supports recursive generation of hierarchical ShapeAssembly programs.Source: Hierarchical decoding · p. 7

6.2 Learning Procedure

  • Training uses teacher-forced sequence prediction with type-specific reconstruction losses and a weighted KL term in the standard VAE objective.Cross-entropy handles one-hot outputs, L1 loss handles continuous values, and irrelevant fields are excluded for each target line.Source: §6.2 · pp. 7–8

Enforcing semantically-valid output

  • Semantic validity checks use ShapeAssembly execution constraints to repair invalid predicted lines through backtracking or reject them when repair fails.Unconditional generation rejects unrepaired samples, while interpolation omits unrepaired lines; the checks operate on program semantics rather than grammar syntax alone.Source: Enforcing semantically-valid output · p. 8

7 RESULTS AND EVALUATION

ShapeAssembly generates novel hierarchical programs that produce plausible, editable, diverse, and physically valid 3D structures. Across synthesis, editability, variability, clustering, surface detail, interpolation, and point-cloud reconstruction, the evaluations support the benefits of program-based generation.

7.1 Novel Shape Synthesis

  • ShapeAssembly generates diverse, structurally complex shapes without copying training geometry or program structure, while maintaining realistic part attachments.Compared with StructureNet and 3D-PRNN, direct attachment operations reduce floating or spatially jumbled parts.Source: §7.1 · pp. 8–9, Figure 7 · p. 9, Figure 9 · p. 11

7.1.1 Analysis of Shape Quality.

  • Across 1000 unconditional samples, ShapeAssembly largely outperforms StructureNet and 3D-PRNN on realism, stability, rootedness, and distributional quality, with StructureNet strongest mainly on rootedness.3D-PRNN has similar FD and % fool scores to StructureNet but markedly worse rootedness and stability.Source: §7.1.1 · p. 9, Table 2 · p. 10

7.1.2 Analysis of Editability.

  • ShapeAssembly programs are more compact and use more macro functions than parsed StructureNet programs, while 3D-PRNN’s compactness often reflects simpler shapes rather than useful editability.The editability comparison measures program length and the fraction of lines using macro types.Source: §7.1.2 · p. 10, Table 3 · p. 12

7.1.4 Program Clustering.

  • Program clustering shows that one ShapeAssembly structure can represent related shapes through continuous parameter changes, with stronger clustering at intermediate hierarchy nodes than across complete programs.This supports using hierarchical programs to share local structure without requiring entire object hierarchies to match.Source: §7.1.4 · p. 12, Figure 10 · p. 13

7.1.3 Analysis of Variability.

  • ShapeAssembly achieves the best or tied-best coverage and variety across categories, avoids apparent overfitting, and generates 65% of Chair, 85% of Table, and 53% of Storage programs with unseen structures.Comparison methods’ extra leaf parts often appear as colocated or overlapping cuboids rather than greater structural complexity.Source: §7.1.3 · pp. 10, 12, Table 4 · p. 12

7.1.5 Synthesizing Surface Geometry.

  • A point-cloud encoder and decoder add fine-grained surface detail, including rounded legs, back slats, and curved chair backs, beyond cuboid proxies.The proof-of-concept conversion introduces geometric detail but also some artifacts.Source: §7.1.5 · pp. 12–13, Figure 11 · p. 13

7.2 Latent Space Interpolation

  • ShapeAssembly produces smoother geometric and program interpolations than StructureNet, combining continuous geometric changes with discrete structural transitions.The comparison evaluates random interpolation sequences using Chamfer distance and tokenized program edit distance; 3D-PRNN lacks a latent variable space for this test.Source: §7.2 · p. 13, Table 6 · p. 13, Figure 12 · p. 14

7.3 Synthesis from Unstructured Geometry

  • For point-cloud reconstruction, ShapeAssembly outperforms StructureNet in accuracy and physical validity, while differentiable program optimization provides the strongest accuracy–validity trade-off.Program-based structural regularization also reduces the tendency to lose semantic parts such as small legs.Source: §7.3 · pp. 13, 15, Table 7 · p. 15

8 CONCLUSION AND FUTURE WORK

The paper combines neural generation with procedural ShapeAssembly programs to synthesize editable, physically plausible 3D structures, while identifying limits in extraction coverage, connectivity guarantees, and shape variability. Future work targets richer procedural representations, tighter surface integration, multimodal synthesis, and physical behavior.

8 CONCLUSION AND FUTURE WORK

  • The hybrid approach produces more plausible and physically-valid shapes, a better-structured latent space, and more compact, editable programs than direct geometry-generation alternatives.The system also supports differentiable parameter optimization for fitting programs to unstructured point clouds.Source: §8 · p. 15, Figure 13 · p. 16

Limitations.

  • Training-program extraction does not succeed for every dataset shape; programs requiring parts attached to “floating” points in space are not added to training data.Programs with more than 12 total Cuboid declarations are also discarded, limiting synthesis of the most complex shape structures in the dataset.Source: Limitations · p. 15
  • The representation trades training regularity and generated-shape quality against variability, so higher variability may reduce synthesis quality despite ShapeAssembly programs capturing parametrically related shapes.The authors identify this regularity–variability tradeoff as a central limitation and motivation for future language extensions.Source: Limitations · p. 15
  • ShapeAssembly strongly biases programs toward physically connected shapes but does not guarantee leaf-to-leaf connectivity, especially for non-axis-aligned structures with loose intermediate bounding cuboids.The paper proposes investigating mechanisms that guarantee leaf connectivity.Source: Limitations · p. 15

Future work.

  • Future work includes closer joint structure-surface generation, non-cuboid atomic proxies, parameter constraints, synthesis from additional modalities, and kinematic or dynamic information.The proposed directions aim to represent more variability procedurally and support richer shape manipulation and embodied-AI applications.Source: Future work · p. 16

A SEMANTICS OF THE ATTACH COMMAND

The attach interpreter greedily executes each declared attachment using fixed-function differentiable operations designed to minimize changes to existing cuboid shapes. Its behavior varies with prior attachments, alignment, and whether exact constraints are solvable.

A SEMANTICS OF THE ATTACH COMMAND

  • The interpreter immediately executes attachments greedily rather than solving a globally optimal constrained cuboid configuration.The procedure is designed to remain predictable and induce few changes to current cuboid shapes.Source: A SEMANTICS OF THE ATTACH COMMAND · p. 18

No prior attachments.

  • With no prior attachments, the interpreter translates c1, or scales and rotates it using n/k, so its attachment points satisfy the target connection.The scale axis is chosen by how quickly scaling reduces n/k, where n and k are distances between existing and target attachment points.Source: One prior attachment · p. 18, No prior attachments · p. 18

One prior attachment.

  • With one prior attachment, the interpreter finds an exact solution when possible and otherwise approximately satisfies the new constraint instead of failing.Three point constraints on a cuboid can be overconstrained.Source: Two or more prior attachments · p. 18

Two or more prior attachments.

  • With two or more prior attachments, the interpreter rotates around colinear attachment axes and conditionally scales along a face normal when the target direction is within τ = 25 degrees.Aligned cuboids retain orientation and are minimally grown to preserve specified connectivity rather than guaranteeing exact attachment-point colocation.Source: Two or more prior attachments · p. 18, Aligned Cuboids · p. 18

B SEMANTICS OF SHAPEASSEMBLY MACRO FUNCTIONS

ShapeAssembly macro semantics expand compact structural commands into attachment and cuboid operations, while extraction uses geometric analysis and preprocessing to create regularized programs from part graphs. The extraction pipeline detects connections, shortens parts, and organizes hierarchy before canonicalization.

B SEMANTICS OF SHAPEASSEMBLY MACRO FUNCTIONS

  • The squeeze macro expands into two attach commands that place one cuboid between two others at corresponding face coordinates.For example, squeeze(cn1,cn2,cn3,left,.1,.4) attaches cn1 to cn2 and cn3 at the specified normalized positions.Source: Squeeze · p. 19

Squeeze.

  • Hierarchy rearrangements empirically produce cleaner, more regularized training data for the generative model.The procedure collapses selected intermediate nodes into leaf nodes and discards their children for category-specific structures.Source: Squeeze · p. 19

Attachment Point Detection.

  • Program extraction detects cuboid connections by comparing sampled point clouds, then samples denser points in intersections to select face-to-face or mean attachment locations.The same procedure records connections to the top or bottom of the bounding volume using normalized y-coordinate ranges.Source: Attachment Point Detection · p. 19

Reflect.

  • Reflect duplicates a cuboid and transforms prior attachment points across an axis, while translate creates repeated cuboids and translates their corresponding attachment points.Both macros preserve the source attachment coordinates while transforming target locations in the relevant bounding-volume frame.Source: Reflect · p. 19, Translate · p. 19

C PROGRAM EXTRACTION PROCEDURE

Symmetry Detection.

Symmetry detection groups cuboids or connected components when their connectivity, transformations, and attachment points are symmetric, while preserving those connectivity relationships. The parser finds valid programs for 46% of Chairs, 65% of Tables, and 58% of Storage shapes.

Symmetry Detection.

  • Symmetry groups require shared connectivity, reflectional or translational symmetry about a parent-volume axis, and correspondingly symmetric outgoing attachment points.Cuboids or attachment points qualify when transforming one produces a sufficiently close parameterization of the other.Source: Symmetry Detection · p. 19
  • The parser forms symmetry groups over connected components before individual cuboids when needed, abstracting those components with bounding volumes to preserve structure and symmetry.This captures additional spatial symmetries while maintaining the relationship between symmetry and part connectivity.Source: Symmetry Detection · p. 19
  • Valid ShapeAssembly programs are found for 46% of Chairs, 65% of Tables, and 58% of Storage shapes in PartNet.Source: Symmetry Detection · p. 19

D DECODER SEMANTIC VALIDITY CHECKS

The decoder enforces semantic validity conditions during program generation and uses backtracking or rejection to handle invalid outputs.

D DECODER SEMANTIC VALIDITY CHECKS

  • The reported fake-classification percentage averages predictions over the final 50 epochs to reduce fluctuation.Source: D DECODER SEMANTIC VALIDITY CHECKS · p. 20
  • Semantic checks constrain coordinates, dimensions, attachments, grounding, symmetry, command ordering, and containment within the bounding volume.Invalid command predictions can trigger backtracking to select a semantically valid alternative.Source: D DECODER SEMANTIC VALIDITY CHECKS · p. 20
  • 10%–20% of unconditional samples are rejected when bad continuous parameters and structural predictions cannot be repaired.During interpolation, invalid lines are omitted instead of rejecting the entire sample.Source: D DECODER SEMANTIC VALIDITY CHECKS · p. 20

E SHAPE QUALITY METRICS

Shape quality is assessed with rootedness, physical stability, and realism metrics that test connectivity, response to perturbations, and similarity to training data.

E SHAPE QUALITY METRICS

  • Rootedness measures whether every part connects to the ground through paths whose gaps do not exceed 2% of the shape’s bounding-box diagonal.Source: E SHAPE QUALITY METRICS · p. 20
  • Stability converts shapes into rigid bodies and applies gravity, mass-proportional vertical forces, and random perturbations before checking component height changes.A shape is unstable when any connected component changes resting height by more than 10%.Source: E SHAPE QUALITY METRICS · p. 20
  • Realism is the percentage of test-set shapes classified as generated by a binary PointNet trained to distinguish generated shapes from training examples.Source: E SHAPE QUALITY METRICS · p. 20

Related papers

Loading 2009.08026v1…