StructureNet: Hierarchical Graph Networks for 3D Shape Generation

1 INTRODUCTION

• The model addresses shape diversity spanning continuous part deformations and discrete changes such as adding or removing armrests, legs, bases, or backs.The intended latent space is continuous while representing varied shape constituents and compositions.Source: §1 · p. 1
• Unlike GRASS, whose canonical binary-tree search becomes difficult for large shape families, StructureNet is designed to train robustly on complex datasets such as PartNet [Mo et al. 2019].GRASS requires binarization of n-ary hierarchies to obtain consistent binary representations.Source: §1 · p. 2
• StructureNet directly encodes hierarchical n-ary graphs with variable child counts and horizontal sibling relationships, rather than binarizing object structures.This representation captures both part hierarchy and inter-part relationships such as adjacency and symmetry.Source: §1 · p. 2
• Order-invariant sibling processing, assignment-based decoding, and graph message passing simplify learning while preserving structural and geometric variation.Working directly with n-ary graphs avoids data variation introduced by binarization.Source: §1 · p. 2
• StructureNet supports raw-input abstraction, novel shape generation, structure-aware interpolation, and part-level object manipulation.These applications include projecting images, point clouds, and partial scans into a learned structural latent space.Source: §1 · p. 2

2 RELATED WORK

• Part-based methods identify parts or relations, but are often restricted in the complexity and variety of layouts they can handle.Hierarchical representations address complexity, while StructureNet additionally augments the hierarchy with horizontal relations.Source: §2 · p. 3
• Existing 3D generative models commonly focus on low-level geometry, whereas StructureNet models object structure jointly with geometry.Earlier approaches use volumetric, point-cloud, depth-map, octree, mesh, or string representations without overall object structure in generation.Source: §2 · pp. 3–4
• Flat part arrangements and ordered vectors struggle with large topological differences or complex hierarchical structures, motivating StructureNet’s n-ary graph representation.Several prior methods either require dense correspondences, model flat parts, or do not freely generate structured shape programs.Source: §2 · p. 3
• Compared with GRASS [Li et al. 2017], StructureNet avoids arbitrary node ordering by directly handling n-ary graphs and explicitly modeling relationships between parts.The paper links this design to robust training on much larger datasets and compares interpolation behavior in Figure 3.Source: Figure 3 · p. 3, §2 · p. 4
• Its graph encoding combines recursive neural networks with graph convolutions to process hierarchical shape graphs specialized to geometry and structure.This differs from generic graph-generation methods that are not specialized to the shape domain.Source: §2 · p. 4

3 OVERVIEW

• Shapes are represented as n-ary trees of parts or assemblies, with sibling edges encoding geometric relationships such as adjacency and symmetry.This representation matches recurring part hierarchies within shape categories while retaining geometric relations.Source: §3 · p. 4
• A variational autoencoder maps the combined shape structure and geometry into a latent space using recursive encoder-decoder networks with graph convolutions at each hierarchy level.The graph convolutions process relationships among parts during hierarchical encoding and decoding.Source: §3 · p. 4
• The learned latent representation supports shape generation, interpolation, retrieval, editing, and structure discovery from unannotated images, point clouds, and raw scans.These applications use a shared representation of both shape structure and geometry.Source: §3 · p. 4

4 A HIERARCHY OF GRAPHS FOR SHAPE STRUCTURE

• Each shape is represented by part geometries P and two superimposed graphs: hierarchical decomposition H and geometric relationships R.Part geometry can be encoded as oriented bounding boxes or point clouds, with semantic labels shared across a category.Source: §4 · p. 4
• The hierarchy begins at a root and recursively decomposes parts into children, allowing any number of children and unbalanced paths to leaves.This supports variable-depth, n-ary part structures rather than fixed-width or balanced trees.Source: §4 · p. 5
• The model restricts explicit relationship edges to siblings because all-pairs modeling would require Θ(N^2) relationships, while important relations often occur among siblings.Relations between other parts are treated as less significant or indirectly implied through the hierarchy.Source: §4 · p. 5
• Sibling relationships encode adjacency, reflective symmetry, rotational symmetry, and translational symmetry as additional undirected typed edges.A sibling pair may carry multiple relationship types, forming an n-ary graph at each hierarchy level.Source: Figure 4 · p. 4, §4 · p. 5

5 HIERARCHICAL GRAPH NETWORKS

• StructureNet maps hierarchical n-ary graphs to latent vectors through recursive geometry and graph encoders, then decodes the hierarchy top-down.Leaf geometry is encoded first; intermediate nodes use graph message passing, and the root feature is the complete-shape code.Source: §5.1 · p. 5, §5 · p. 5

5.1 Encoder

• The geometry encoder supports bounding boxes and point clouds, using specialized networks including PointNet for normalized point-cloud geometry.Point-cloud center and scale are encoded separately from normalized points before producing the part feature.Source: §5.1 · p. 5
• Graph message passing incorporates edge types and aggregates neighboring part features, with max-pooling and iteration concatenation producing fixed-length graph features.The experiments use two message-passing iterations for each graph.Source: §5.1 · pp. 5–6

5.2 Decoder

• The decoder reconstructs geometry for every hierarchy node, providing intermediate reconstruction supervision; point-cloud geometry uses a pretrained part autoencoder for greater training stability.Bounding boxes use an MLP, while point clouds are decoded into normalized points plus center and scale.Source: §5.2 · p. 6
• The graph decoder predicts a fixed maximum of 10 child parts and all pairwise edges, discarding predictions below existence thresholds before refining features through message passing.It simultaneously decodes parts and relations, then predicts semantic labels and leaf status; leaf prediction helps stop recursion and improves early convergence.Source: §5.2 · pp. 6–7

5.3 Training and Losses

• Training uses a reconstruction loss, a structure-consistency loss, and VAE regularization, with structure consistency enforcing agreement between reconstructed geometry and relationship edges.The relationship constraints extend from related parent parts to their entire subtrees, including symmetry and adjacency.Source: §5.3 · pp. 7–8
• Order-invariant training matches parts by linear assignment within child graphs rather than reconstructing sibling order, using geometry-based correspondence.The authors report that the order-invariant network produced superior performance empirically.Source: §5.3 · p. 7
• The shape distance combines geometry, normal, part-existence, edge-existence, semantic, and leaf losses, with box geometry compared through boundary-point Chamfer distance.The normal loss addresses orientation insensitivity for small bounding boxes, while unmatched-part geometry is trained toward zero.Source: §5.3 · pp. 7–8

6 EXPERIMENTS

• Experiments use filtered PartNet cabinets, chairs, and tables represented by hierarchical parts, bounding boxes or point clouds, semantic labels, and symmetry or adjacency relationships.The retained datasets contain 4871 chairs, 5099 tables, and 862 cabinets, split 7:1:2 into training, validation, and test sets.Source: §6 · p. 9

6.1 Shape Reconstruction

• 0.061 geometry error shows StructureNet outperforming GRASS Original (0.103), GRASS PartNet (0.082), and StructureNet without edges (0.065) on chair reconstruction.The comparison uses a reduced subset of 4031 chairs because GRASS failed on the remaining 840; direct n-ary encoding and relationship edges improve reconstruction.Source: §6.1 · p. 10

6.2 Shape Generation

• StructureNet generates diverse, functionally plausible shapes with varied geometry, hierarchies, semantics, symmetry, and adjacency, while differing from nearest training examples in both geometry and structure.Relative quality and coverage both improve over GRASS when comparing 1000 generated shapes.Source: §6.2 · p. 11, Table 2 · p. 11

6.3 Shape Interpolation

• Interpolations produce smooth, valid structural transitions, with parts being simplified, added, replaced, or removed through small intuitive steps.Compared with GRASS and holistic interpolation, StructureNet changes structure more gradually and preserves cleaner, functionally valid per-part geometry.Source: §6.3 · pp. 11, 13

6.4 Shape Abstraction

• Image and point-cloud encoders map inputs into StructureNet’s latent space, enabling recovery of full structured shapes with accurate overall forms and details, including plausible matches from incomplete real-world scans.The image encoder uses rendered training views, while the point-cloud encoder is PointNet++; ScanNet scans may differ statistically and omit large regions.Source: §6.4 · pp. 13–15

6.5 Shape Editing

• Latent-space optimization enables structure-preserving edits by satisfying target box transformations while remaining close to the original shape and consistent with part relationships.The objective combines latent proximity, edit satisfaction through squared Chamfer distance, and relationship consistency to maintain plausibility.Source: §6.5 · p. 15
• Editing latent-space shapes preserves plausibility and similar structure while suggesting structural changes lie farther apart than geometric changes.Results adjust other parts as needed, with only minor structural exceptions such as added bars between chair legs.Source: §6.5 · p. 16
• Point-cloud editing closely follows edited boxes but produces artifacts at hard segment boundaries.The method assigns points to nearest bounding boxes and applies each box transformation; soft assignments or segment refinement could reduce artifacts.Source: §6.5 · p. 16

6.6 Limitations and Failure Cases

• StructureNet inherits dataset sampling bias, may generate detached or asymmetric parts, and cannot encode more than 10 children under any part.The sibling-count restriction has quadratic memory cost in np, while current memory use is approximately 1–2 GB.Source: §6.6 · p. 16
• Noise primarily disrupts discrete structure, causing missing, duplicate, or detached parts, while pretraining point-cloud networks improves robustness but loses fine-grained geometry.Observed point-cloud failures also include asymmetric parts and fuzzy generation.Source: §6.6 · p. 16, Figure 16 · p. 16

• StructureNet jointly encodes part geometry and adjacency or symmetry relations in hierarchical n-ary graphs through a recursive encoder-decoder VAE.The design predicts part and relationship presence, is invariant to sibling ordering, and uses geometry-structure consistency losses.Source: §7 · p. 16
• The learned latent space supports projecting partial scans, point clouds, or images, editing individual parts, hierarchy-preserving interpolation, and sampling novel shape variations.Source: §7 · p. 16
• Future work includes user feedback for un-annotated structure estimation, finer-grained part geometry, point-level correspondences for appearance transfer, and extensions from objects to scenes.Source: §7 · p. 16

• Top-3 image and point-cloud retrievals are qualitatively good matches to a query shape in the joint multi-modal latent space.Figure 17 compares full-shape queries with part-based queries using chair backrests and bases.Source: A JOINT EMBEDDING OF SHAPES AND IMAGES · p. 18, Figure 17 · p. 19
• Different modalities align in the latent space, where nearby samples represent similar shapes and smoothly transition between chair structures.Sofa chairs and chairs with multiple vertical backrest bars form modality-consistent clusters.Source: A JOINT EMBEDDING OF SHAPES AND IMAGES · p. 18
• Part-based retrieval yields meaningful chair-base and backrest representations even though the part latent spaces are not explicitly regularized for smoothness.Source: A JOINT EMBEDDING OF SHAPES AND IMAGES · p. 18

• Removing relationship edges significantly increases geometry reconstruction error, highlighting the contribution of graph relations.Source: B ABLATION STUDY · p. 19
• Normal reconstruction and structure consistency losses lower geometry or consistency errors but slightly increase hierarchy error by shifting capacity toward part geometry.Source: B ABLATION STUDY · p. 19
• Decoder message passing significantly lowers hierarchy error and reduces the hierarchy cost introduced by the normal and structure consistency losses.Source: B ABLATION STUDY · p. 19

C IMPLEMENTATION

• C IMPLEMENTATION: Hierarchical graph-network subnetworks use ReLU MLPs, batch size 32, and per-shape loss computation before batch aggregation.Source: C IMPLEMENTATION · p. 19
• C IMPLEMENTATION: Table 3 compares the full method against variants removing message passing, normal reconstruction loss, structure consistency loss, or relationship edges.The table evaluates how these components affect geometry, consistency, and hierarchy reconstruction.Source: Table 3 · p. 19
• C IMPLEMENTATION: Bounding-box networks typically train in 1–2 days, whereas point-cloud networks require 2–4 days on the stated hardware, with approximately 1–2 GB memory use.Source: C IMPLEMENTATION · p. 19

D SEMANTIC HIERARCHIES

• D SEMANTIC HIERARCHIES: PartNet semantic hierarchies are provided for chairs, tables, and storage furniture, with labels mapped to visualization colors.Source: D SEMANTIC HIERARCHIES · p. 19

E MORE OBJECT CATEGORIES

• Vases and trash cans provide additional shape-generation and interpolation results, while Figure 24 presents a training attempt on a severely under-sampled dataset.Source: E MORE OBJECT CATEGORIES · p. 19

F ADDITIONAL GENERATED SHAPES

• Additional StructureNet VAE results demonstrate generation for both box-shapes and point cloud shapes.Source: F ADDITIONAL GENERATED SHAPES · p. 19

G ADDITIONAL SHAPE INTERPOLATIONS

• Additional interpolation results cover both box-shapes and point cloud shapes.Source: G ADDITIONAL SHAPE INTERPOLATIONS · p. 19

H ADDITIONAL SHAPE ABSTRACTIONS

• The multimodal latent space jointly embeds images, point clouds, and shapes, with modalities sampled across a two-dimensional visualization.Source: Figure 18 · p. 20
• Semantic hierarchies are illustrated for chairs, tables, and storage furniture using OR-node and AND-node annotations.Source: Figure 19 · p. 21, Figure 20 · p. 22, Figure 21 · p. 23
• Vase and trash-can figures show generation and smooth interpolation of both structure and part geometry.The vase and trash-can datasets contain 505 and 83 samples, while beds contain 54 samples and are not large enough for reliable realism discrimination.Source: Figure 22 · p. 24, Figure 23 · p. 24, Figure 24 · p. 25
• Further supplementary figures provide additional box-shape generation, point-cloud generation, shape interpolation, and shape abstraction results.Source: Figure 25 · p. 26, Figure 26 · p. 27, Figure 27 · p. 28