PaperlayerView exact revision on arXiv

Source-linked AI summary

PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction

Sinisa Stekovic, Arslan Artykov, Stefan Ainetter, Mattia D'Urso, Friedrich Fraundorfer

arXiv:2404.10620v2cs.CVcs.LG

TL;DR

Shape programs offer interpretable, editable, compact object representations, but recovering their continuous and discrete parameters from images remains challenging. PyTorchGeoNodes turns Blender shape programs into differentiable PyTorch code and combines it with genetic search; the authors evaluate reconstruction on synthetic and real scenes, manually annotating 176 objects for parameter evaluation.

  • Problem

    Shape-program parameter recovery is challenging because programs mix continuous and discrete values, and existing datasets lack annotations for those parameters.

    Source: §1 · pp. 1, 3

  • Method

    PyTorchGeoNodes generates differentiable PyTorch code from Blender shape programs and uses genetic algorithms to optimize continuous and discrete parameters.

    Source: §1 · p. 2

  • Results

    The authors evaluate on synthetic and real scenes, manually annotating 176 objects for parameter evaluation; on synthetic scenes, genetic search performs extremely well and adding gradient descent has an advantage.

    Source: §1 · pp. 2–3, §1.2 · p. 12

  • Takeaways & Limitations

    The modular tool can be extended to other 3D scene-understanding uses, including procedural Gaussian splatting.

    Source: §1 · p. 2

    Shape programs must be designed by humans for different categories, which can limit expressiveness in wild settings.

    Source: §6 · p. 8

Abstract

from arXiv · show

We propose PyTorchGeoNodes, a differentiable module for reconstructing 3D objects and their parameters from images using interpretable shape programs. Unlike traditional CAD model retrieval, shape programs allow reasoning about semantic parameters, editing, and a low memory footprint. Despite their potential, shape programs for 3D scene understanding have been largely overlooked. Our key contribution is enabling gradient-based optimization by parsing shape programs, or more precisely procedural models designed in Blender, into efficient PyTorch code. While there are many possible applications of our PyTochGeoNodes, we show that a combination of PyTorchGeoNodes with genetic algorithm is a method of choice to optimize both discrete and continuous shape program parameters for 3D reconstruction and understanding of 3D object parameters. Our modular framework can be further integrated with other reconstruction algorithms, and we demonstrate one such integration to enable procedural Gaussian splatting. Our experiments on the ScanNet dataset show that our method achieves accurate reconstructions while enabling, until now, unseen level of 3D scene understanding.

1. Introduction

Shape programs offer interpretable, editable, compact object representations, but recovering their parameters is challenging because they mix continuous and discrete values. PyTorchGeoNodes makes Blender shape programs differentiable and combines gradient optimization with genetic search for reconstruction from images.

1. Introduction

  • Shape programs generate objects from semantic and geometric parameters, and their outputs are interpretable, editable, and small in memory.They can represent properties such as object dimensions and the presence of chair armrests or backrests.Source: §1 · p. 1
  • Recovering a program from an image is challenging because its parameters can be continuous, such as width, or discrete, such as shelf count.Changing the number of shelves from five to six is one example of a discrete change.Source: §1 · p. 1
  • The compiler converts Blender shape programs into PyTorch code differentiable with respect to continuous parameters, allowing integration with 3D reconstruction algorithms.The framework also provides a method to jointly optimize continuous and discrete parameters from RGB-D scans.Source: §1 · p. 2
  • Genetic-algorithm-based search jointly optimizes continuous and discrete shape-program parameters from RGB-D scans.The compiler supports continuous optimization, while the genetic algorithm addresses the discrete-parameter challenge.Source: §1 · p. 2
  • Procedural Gaussians add fine detail and appearance while constraining Gaussian splats to the procedural model and tying together instances on cloned primitives.When one repeated part is visible, its cloned counterpart can help recover an occluded part.Source: §1 · p. 2
  • The method is evaluated on synthetic and real scenes, with 176 objects manually annotated because existing datasets lack shape-program parameter values.The evaluation includes reconstruction of both continuous and discrete parameters.Source: §1 · pp. 2–3

2. Related Work

Related work covers CAD-model retrieval and procedural shape programs. Retrieval is limited by database coverage, while shape programs offer parameterized representations but can struggle with complex geometry.

2.1. 3D Model Retrieval

  • CAD-model retrieval selects database models using input 3D data or images, through classification or similarity comparisons between rendered and real inputs.Direct classification does not scale well to large databases, and retrieval is constrained by the database's expressiveness.Source: §2.1 · p. 3
  • Retrieval databases may lack modified or articulated target objects, while mesh refinement can introduce artifacts and category-unrepresentative shapes.The cited example is a work chair whose seat height, armrest, or back support differs from the database default.Source: §2.1 · p. 3

2.2. Shape Programs

  • Shape programs use draw statements for primitives and for statements for repeated subprograms; prior work [35] showed they can be inferred from input 3D shapes.Basic geometric primitives can make complex geometry difficult to model and produce shapes unrepresentative of the target category [35].Source: §2.2 · p. 3
  • GeoCode [27] uses hand-designed Blender geometry-node programs to map category-specific 3D shapes to human-interpretable parameters, with nodes for primitives, transformations, replication, and control flow.Its parameters can be predicted from input point clouds or sketches [27].Source: §2.2 · p. 3

3. PyTorchGeoNodes

PyTorchGeoNodes translates Blender Geometry Nodes shape programs into differentiable PyTorch code for fitting parameterized 3D shapes. Its node implementations cover parameter inputs, arithmetic and control flow, geometry creation, transformations, instancing, and mesh output.

3. PyTorchGeoNodes

  • The cabinet example transforms width, height, and board parameters through arithmetic and geometry operations to create a cuboid and instanced dividing boards.The illustrated shape program is part of a larger cabinet model.Source: Figure 2 · p. 4
  • The framework currently handles input, math, switch, combine, primitive, transform, mesh-line, points-on-instances, join-geometry, and output-geometry nodes.Inputs may be floating-point, integer, or boolean values; node behavior can depend on continuous, integer, or boolean parameters.Source: §3 · p. 4
  • A parser maps Blender geometry-node functionality to corresponding PyTorch or PyTorch3D nodes, enabling shape parameters to be fitted to an input scene.The resulting code supports derivative computation for differentiable optimization.Source: §3 · p. 4

4. Search Algorithm

The search objective fits generated shapes to RGB-D-derived point clouds while accounting for occlusion and floor contact. A genetic algorithm searches discrete and continuous parameters, with gradient descent refining continuous values; the framework also extends shape programs with Gaussian primitives for detailed appearance fitting.

4. Search Algorithm

  • The search objective combines point-cloud fit, occlusion penalties, and floor alignment, without penalizing generated points hidden from the input views.The target point cloud is reconstructed from depth maps and object masks; the occlusion grid is sampled along camera-to-surface rays.Source: §4 · p. 5
  • Because shape parameters include discrete values and the objective has no exploitable special form, genetic search is combined with gradient descent for continuous-parameter refinement.The search also includes object pose, with translation initialized from an estimated object center and rotation constrained to the vertical axis.Source: §4 · p. 5

4.1. Shape Parameter Search

  • The genetic algorithm iterates crossover, mutation, and selection, retaining the best-performing individuals to form the next population.Crossover mixes pairs of parent parameters, while mutation alters offspring parameters to explore further.Source: §4.1 · p. 5
  • Discrete mutations sample valid values, while continuous mutations use discretized candidates or Gaussian perturbations before selection by the negative objective score.During each selection phase, gradient-based optimization further refines continuous parameters; the best individual is returned after the specified iterations.Source: §4.1 · pp. 5–6

4.2. Integrating Gaussians Splats

  • Gaussian primitives augment procedural shape programs to capture fine object details while retaining the advantages of the shape-program representation.Mesh primitives are subsampled into triangles that initialize trainable Gaussian means, scales, and quaternions; colors use the closest scan point and opacities are fixed high.Source: §4.2 · p. 6
  • After estimating shape parameters, the method optimizes Gaussians with a render-and-compare loss that includes image and depth consistency, shape consistency, and regularization.Regularization keeps Gaussian mean and scale offsets small and colors on the same primitive close; the authors report greater robustness from estimating shape parameters first.Source: §4.2 · p. 6

5. Validation

The section evaluates furniture shape-program reconstruction on ScanNet and compares genetic search with supervised and coordinate-descent baselines.

5. Validation

  • Validation uses Blender programs for sofas (18 parameters), chairs (20), and tables (10), with 176 manually annotated ScanNet objects across these categories.ScanNet scenes contain partially visible, highly occluded furniture and incomplete depth maps.Source: §5 · pp. 6–7

5.1. Evaluation Criteria

  • Parameter evaluation compares reconstructed values with ground truth using discrete classification accuracy and continuous absolute difference; object masks generate dense point clouds for the objective.ScanNet objects were manually annotated with shape parameters.Source: §5.1 · p. 7, §5.3 · p. 7

5.2. Search Baseline Details

  • GeoCode predicts parameters from point clouds after synthetic training, while coordinate descent cycles through per-parameter updates and gradient-refines candidates.GeoCode is trained on 10,000 generated parameter-and-rotation instances; coordinate descent loops over parameters three times.Source: §5.2 · p. 7

5.3. Validation on the ScanNet Dataset

  • Figure 5 shows chair, table, and sofa parameter recovery, including semantic parts and measurements, aligned against sampled target-scene views.Examples include chair-leg configurations, table dimensions and shelves, and sofa arms, backrests, and L-element measurements.Source: Figure 5 · p. 7
  • The genetic algorithm significantly outperforms other baselines and performs well across metrics except for very small details; recovered models are visually comparable to ground truth.GeoCode performs worst due to the domain gap, while coordinate descent is prone to local minima.Source: Table 1 · p. 8, §5.3 · p. 8

6. Conclusion

The conclusion presents Gaussian integration as an example of PyTorchGeoNodes’ modularity and identifies extensions needed for broader procedural modeling.

6. Conclusion

  • PyTorchGeoNodes can integrate other 3D representations; Figure 6 shows Gaussian integration alongside reconstructed-shape overlays and ground-truth comparison.The conclusion also points to differentiable rendering as a basis for self-supervised methods that create shape programs from images.Source: §6 · p. 8, Figure 6 · p. 8
  • Future object-level extensions include support for deformable and articulated objects and integrated textures.These are presented as areas still needing development.Source: §6 · p. 8
  • Scene-level procedural models or programs are another proposed extension, for reasoning about object relationships and scene physics.The conclusion notes such programs could be learned or designed in Blender.Source: §6 · p. 8

Reconstruction – Supplementary

The supplementary material lists further evaluation and framework details, and includes videos demonstrating reconstruction progress and Gaussian-splat integration.

Reconstruction – Supplementary

1. Additional Validations

Additional evaluations compare the genetic search and Gaussian-splat integration with alternatives, and show further uses of recovered shape programs. The results also identify objective-function and data-visibility limits for some categories and details.

1.1. Additional Validation on ScanNet

  • On ScanNet, the genetic algorithm outperforms alternative baselines, while deep-learning inference struggles with domain gap and partial point clouds and coordinate descent can fail from poor initialization.The objective also handles small details such as sofa legs poorly.Source: §1.1 · p. 11
  • Qualitative ScanNet examples recover varied sofa, chair, and table parameters, while cabinet board counts and bookshelf reconstruction remain difficult under occlusion or segmentation noise.The cabinet and bookshelf limitations are also reported for state-of-the-art methods.Source: §1.1 · p. 11, Figure 1 · p. 11, Figure 5 · p. 13
  • Recovered computational graphs can segment semantic parts in partial point clouds by assigning points to procedural base primitives using chamfer distance.Figure 2 illustrates this application.Source: §1.1 · p. 11, Figure 2 · p. 12

1.2. Validation on Synthetic Data

  • On synthetic complete point clouds, the genetic algorithm performs extremely well, gradient-based refinement improves results, and it outperforms coordinate descent while performing comparably to GeoCode.The authors identify the performance boost over ScanNet as an indication that novel objective terms could further improve reconstruction.Source: §1.2 · p. 12, Table 4 · p. 15, Table 5 · p. 16

1.3. Validation of Gaussian Splats Integration

  • Gaussian-splat metrics improve after optimization, and procedural constraints yield higher-quality reconstructions than vanilla splatting under occlusion, limited views, and noisy observations.The evaluation uses 20 training and 10 validation views per scene, with 10 selected samples and only 30 views per scene for optimization.Source: Figure 4 · p. 12, §1.3 · p. 12

2. PyTorchGeoNodes – Implementation Details

PyTorchGeoNodes represents shape programs as computational graphs whose nodes pass geometry and parameters through connected sockets. Topological ordering and caching make graph evaluation more efficient.

2.1. Computational Graph

  • Graph nodes are PyTorch modules with unique identifiers, and edges connect nodes through input and output sockets.Nodes may take constants or outputs of other nodes and can have multiple input and output sockets.Source: §2.1 · p. 17
  • Forward passes use a hash map to retrieve node outputs, parse named shape parameters at input nodes, and collect output geometries while allowing gradients to flow.Experiments use one output node, although the implementation is not limited to one.Source: §2.1 · p. 17

2.2. Efficient Implementation

  • Topological sorting lets inference evaluate nodes in dependency order without recursive calls, and caching avoids recomputing input-independent nodes.The framework addresses the rising runtime of graphs with 100–200 nodes.Source: §2.2 · p. 17

3. Search Algorithm – Implementation Details

The search implementation discretizes continuous parameter values and uses an empirically selected genetic-algorithm configuration with periodic gradient-based refinement.

Discretizing continuous parameters

  • Continuous parameters are discretized in linear steps of 0.2, while object rotations use 90° steps.The 0.2 step is used for parameters represented in meters or as aspect ratios.Source: §3 · p. 17

Search details

  • The genetic search runs 50 generations, starts with 500 random individuals, retains 64, and generates 128 offspring per iteration through parent-pair crossover.Mutation probability decays from 0.9 to 0.1, with continuous-value noise σ = 0.05.Source: §3 · p. 17
  • Every five iterations, 20% of offspring receive Adam refinement to convergence at learning rate 0.01.Source: §3 · p. 17

4. Integration of Gaussian Nodes – Implementation Details

Gaussian nodes are integrated with procedural meshes so their parameters remain tied to mesh geometry. The implementation optimizes selected Gaussian and mesh offsets, while gradient-based shape-program optimization can target individual regions.

4. Integration of Gaussian Nodes – Implementation Details

  • Gaussian nodes extend primitive-mesh generation to produce Gaussian parameters fitted to the corresponding meshes, keeping object geometry and splats consistent.The design is inspired by SuGaR.Source: §4 · p. 17
  • Gradient descent on shape-program parameters can scale individual parameters independently to target specific geometry while preserving compactness.The example optimizes continuous sofa parameters from an initial estimate using a 3D geometric loss.Source: Figure 8 · p. 18
  • The integration derives Gaussian means from mesh triangles, freezes thickness at 10^-3, and optimizes scale and in-plane rotation offsets, colors, opacity, and mesh-vertex offsets.Means use four points per face in barycentric coordinates; colors initialize from nearest point-cloud points and opacity from 0.7.Source: §4 · p. 18
  • Gaussian parameters are trained for 1,000 Adam iterations with separate learning rates for mesh offsets, scales, colors, and opacity.The rates are 10^-4, 5×10^-3, 10^-2, and 10^-2, respectively.Source: §4 · p. 18

5. Shape Program Designs for Validation

The validation shape programs represent furniture through named geometric parameters and feature controls. The designs specify bounded dimensions and structural options for chairs, sofas, tables, and cabinets.

5. Shape Program Designs for Validation

  • The chair program has 23 parameters—19 continuous and 4 boolean—for seat, legs, supports, back, and arm geometry.Examples include seat height in [0.3, 0.9] meters and a boolean selecting four legs or one central star-shaped leg.Source: §5 · pp. 18–19
  • The sofa program has 19 parameters—13 continuous and 6 boolean—for base dimensions, legs, arms, back, and an optional L-shaped extension.The L extension’s dimensions are controlled separately, and a boolean determines whether it is on the left or right.Source: §5 · p. 19
  • The table program has 15 parameters for dimensions and top shape, with controls for leg configuration, boards, cabinet legs, and leg placement.Its leg-type parameter selects one central leg or four side legs; separate controls adjust leg scale and offset.Source: §5 · p. 19
  • The cabinet program has 12 parameters for dimensions and board thickness, plus controls for back, legs, drawers, and 2–5 dividing boards.Leg dimensions and dividing-board thickness are also parameterized.Source: §5 · p. 19

Related papers

Loading 2404.10620v2…