Source-linked AI summary
CSGNet: Neural Shape Parser for Constructive Solid Geometry
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji
TL;DR
Existing shape-generation methods use low-level representations, while bottom-up CSG parsing faces large primitive-combination searches and ordered operations. CSGNet uses a recurrent top-down parser to generate compact CSG programs for 2D and 3D shapes, combining supervised learning, reinforcement learning, rendering-based rewards, and search. The authors report effective cross-domain parsing, stronger and faster shape detection than state-of-the-art detectors, and learning on novel datasets without ground-truth programs.
Problem
Existing methods often represent shapes as pixels, voxels, or points, while CSG parsing must infer variable primitive sets and operation orders.
Method
CSGNet uses a recurrent top-down network to generate CSG programs, with rendering-based rewards and policy gradients for datasets without program annotations.
Results
The parser is reported to work across 2D and 3D domains, learn on novel datasets without ground-truth programs, and outperform and run faster than state-of-the-art detection approaches.
Takeaways & Limitations
The work demonstrates automatic generation of compact, interpretable modeling programs for logos, 2D shapes, and 3D CAD shapes.
Takeaways & Limitations
The evaluated 2D images and 3D shapes may be relatively simple, and extending the approach to longer programs with larger parameter spaces remains future work.
Abstract
from arXiv · showhide
We present a neural architecture that takes as input a 2D or 3D shape and outputs a program that generates the shape. The instructions in our program are based on constructive solid geometry principles, i.e., a set of boolean operations on shape primitives defined recursively. Bottom-up techniques for this shape parsing task rely on primitive detection and are inherently slow since the search space over possible primitive combinations is large. In contrast, our model uses a recurrent neural network that parses the input shape in a top-down manner, which is significantly faster and yields a compact and easy-to-interpret sequence of modeling instructions. Our model is also more effective as a shape detector compared to existing state-of-the-art detection techniques. We finally demonstrate that our network can be trained on novel datasets without ground-truth program annotations through policy gradient techniques.
1. Introduction
CSGNet parses 2D and 3D shapes into compact constructive solid geometry programs composed of primitives and boolean operations. Its top-down recurrent design addresses variable program structure and operation ordering while supporting supervised and reinforcement-learning training.
- Motivation: CSG modeling represents shapes with higher-level primitives and operations rather than collections of pixels, voxels, or points.Examples include parametric curves, basic shapes, boolean operations, deformations, and extrusions.
- Program representation: The parser produces compact programs whose primitives are recursively combined with boolean operations to generate 2D or voxelized 3D shapes.Figure 1 illustrates the input shape, induced program, parse tree, and executed result.
- Challenges: The parser must handle variable numbers of primitives and operations, with operation order affecting the resulting shape.Swapping two operations can produce an entirely different object.
- Approach: A memory-enabled network generates CSG programs for target 2D images or 3D shapes.The authors train on automatically generated 2D and 3D programs, then use policy gradients to adapt to new domains without program annotations.
- Contributions: CSGNet is reported as efficient and effective across 2D and 3D domains, including learning on novel datasets without ground-truth programs.The authors also report improved speed and effectiveness for shape detection relative to bottom-up approaches.
2. Related Work
Prior work includes neural program induction, inverse graphics, primitive fitting, and grammar-based parsing. CSGNet instead predicts ordered visual programs of modeling operations, combining neural decoding with execution and optimization to address the larger output space.
- Neural program induction: Neural program induction commonly uses recurrent networks to predict high-level programs from exemplar inputs.Applications include arithmetic, logical, and semantic parsing tasks.
- Visual reasoning: Visual reasoning systems can execute image-derived programs, whereas CSGNet generates shape-modeling programs that match a target image.Its output is generative rather than a program for compositional reasoning alone.
- Vision-as-inverse-graphics: Inverse-graphics methods generate and fit scene or object hypotheses to image data, including procedural-model parameters.Related approaches use probabilistic inference, CNNs, or proposal-based object detection.
- Distinction from inverse graphics: CSGNet parses images or 3D shapes into ordered sequences of operations on primitives rather than collections of objects and their parameters.The authors address the larger output space with supervised pretraining, reinforcement learning, reward design, and parameter post-optimization.
- Neural primitive fitting: Neural primitive-fitting methods predict primitives, but prior approaches may use a fixed number or focus on boxes, while CSGNet derives programs of primitives and operations.The supplied passage introduces this comparison but truncates its final contrast.
- Architecture: The CSGNet architecture combines a CNN encoder, recurrent decoder, and rendering engine to predict and execute visual programs.Figure 2 depicts predicted primitives P1–P4 and intermediate boolean-operation outputs E1–E2.
- Efficiency: Neural program generation evaluates in a fraction of a second, contrasting with an alternative optimization process that can take hours.The comparison concerns test-time program creation versus computationally expensive solving.
- Bottom-up parsing: Bottom-up grammar-based parsing often depends on shallow grammars or accurate primitive proposals, while CSG primitive detection is difficult because boolean operations substantially change shapes.This motivates top-down parsing in the CSG setting.
3. Designing a Neural Shape Parser
The neural shape parser encodes an input shape and recurrently decodes a grammatical CSG program, which is rendered and evaluated by reconstruction-based rewards. Training combines supervised program likelihood with policy gradients when target programs are unavailable, plus beam search and parameter refinement.
- Parser architecture: An encoder maps input image I to features Φ(I), while a recurrent decoder produces a probability distribution over grammatical instruction sequences.The parser is expressed as π(I) = Θ ◦ Φ(I).
- Program grammar: CSG programs use grammar rules that combine primitives such as spheres, cubes, and cylinders with boolean modeling operations.Instructions can be represented in postfix notation, such as SHAPE1SHAPE2OP1SHAPE3OP2.
- Learning objective: The parser is trained to minimize reconstruction error between the executed program’s shape and the target while learning to generate valid grammatical programs.Invalid programs are excluded through the reward design.
- Supervised learning: With target programs, supervised learning maximizes the log-likelihood of the ground-truth instruction sequence.The RNN predicts a categorical distribution over instructions at each time step.
- Policy-gradient learning: Without target programs, policy gradients optimize expected rewards because discrete program outputs and nondifferentiable execution prevent direct gradient-based minimization.Sampled programs are executed, compared with target shapes, and rewarded according to visual similarity.
- REINFORCE: REINFORCE approximates the large program-space expectation with Monte Carlo samples, using sampled instruction sequences and execution-based rewards.A baseline reduces sampling variance without changing estimator bias.
- Reward design: Chamfer distance measures visual similarity between edge point sets, and its normalized value is shaped into the reward.The implementation uses a maximum program length T = 13 and f(x) = (1 − x)^γ with γ = 20.
- Decoding: Greedy decoding selects the most likely instruction at each step, whereas beam search retains the k-best partial sequences.Experiments report results for varying beam sizes.
4. Experiments
CSGNet is evaluated on synthetic 2D and 3D shapes, web-mined CAD shapes, logos, and primitive detection. Across these settings, the parser benefits from beam search, reinforcement-learning adaptation, and visually guided refinement, while performance can degrade for long or unfamiliar programs.
- Datasets and evaluation: The experiments cover synthetic 2D and 3D CSG programs, 2D CAD shapes without ground-truth programs, web logos, and primitive detection.The network is evaluated both for inferring complete programs and for inferring primitives as an object detector.
- Datasets and evaluation: Synthetic 2D programs use square, circle, and triangle primitives combined by intersect, union, and subtract operations on a 64×64 canvas.Primitive locations and sizes are discretized, and programs contain varying numbers of primitives sampled from a uniform distribution over primitive and operation types.
- Datasets and evaluation: The 3D setting uses sphere, cube, and cylinder primitives with the same three boolean operations, rendered from 64×64×64 voxel grids into high-resolution polygon meshes.The 3D-CSGNet combines a 3D-CNN with a GRU and outputs CSG primitives and operations rather than voxels.
- Inferring programs: On synthetic 2D shapes, CSGNet outperforms the Nearest Neighbor baseline, and increasing beam size consistently improves Chamfer distance.The comparison uses Chamfer Distance in pixels across decoding beam sizes.
- Inferring programs: On 2D CAD shapes, reinforcement-learning fine-tuning improves results beyond the supervised and Nearest Neighbor approaches, while visually guided refinement produces the smallest Chamfer Distance.Direct reinforcement-learning training alone performs poorly, supporting the reported two-stage supervised-learning and RL procedure.
- Generalization: The method parses logos into primitives well overall, but performance can degrade for long programs or shapes unlike the available primitives.In 3D, the induced programs are described as promising and can reconstruct voxelized surfaces as high-resolution surfaces.
- Primitive detection: For primitive detection, the parser outperforms Faster R-CNN on squares and triangles, gives slightly better circle results with larger beams, and runs considerably faster on the same GPU.Detection scores are estimated from the fraction of sampled beam-search programs containing each primitive.
5. Conclusion
The paper demonstrates automatic generation of CSG modeling programs from visual content across multiple domains, while identifying simplicity and program-length challenges that motivate future work.
- The method generates programs for logos, 2D binary shapes, and 3D CAD shapes, and supports 2D shape primitive detection through analysis-by-synthesis.
- The authors report competitive or better results than state-of-the-art object detectors despite the relative simplicity of the parsed shapes.
- Good program generation required memory-enabled networks, supervised and reinforcement-learning strategies, beam search, and local state-space exploration.
- Future work includes longer programs, larger operation-parameter spaces, more sophisticated rewards, and combining bottom-up proposals with top-down parsing.
6. Supplementary
The supplementary material details dataset construction, CSG execution, network architectures, and qualitative evaluations for 2D and 3D shape parsing.
- The supplementary material covers synthetic 2D and 3D dataset creation, neural architectures, and additional qualitative test results.
- Synthetic datasets: Synthetic 2D programs have lengths from 3 to 13 and restrict primitives and operations to avoid invalid or spurious shapes.
- Synthetic datasets: The 3D dataset contains 64 × 64 × 64 voxel-grid shapes paired with programs of up to seven instructions.
- CSG execution: The CSG engine executes primitives with a stack and applies union, intersection, or subtraction to the top two canvases.
- 2D architecture: The 2D architecture encodes 64 × 64 images with a CNN and decodes programs using a GRU over 396 primitives, three operations, and STOP.
- 3D architecture: The 3D architecture encodes 64 × 64 × 64 voxel grids and predicts distributions over 6631 primitives, three operations, and STOP.
- Qualitative evaluation: Qualitative evaluations compare top-10 beam-search results with nearest neighbors on synthetic 2D data and show full-model outputs on 2D CAD images.