PaperlayerView exact revision on arXiv

Source-linked AI summary

PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions

R. Kenny Jones, Homer Walke, Daniel Ritchie

arXiv:2011.13045v4cs.CVcs.GRcs.LG

TL;DR

Inferring programs from shapes is important for editing and reverse engineering, but paired shape-program data is often unavailable. The paper groups pseudo-label and approximate-distribution training under PLAD and introduces latent execution self-training. Across multiple 2D and 3D domains, PLAD outperforms policy gradient reinforcement learning, and combined PLAD updates outperform individual methods.

  • Problem

    Shape-program inference matters for editing and reverse engineering, but real shapes often lack their generating programs, limiting access to paired training data.

    Source: §1 · p. 1

  • Method

    PLAD groups maximum-likelihood fine-tuning methods that form shape-program pairs through pseudo-labels or approximate distributions, including latent execution self-training.

    Source: §1 · p. 2

  • Results

    Across 2D CSG, 3D CSG, and ShapeAssembly, PLAD achieves better reconstruction accuracy and faster convergence than policy gradient reinforcement learning, while combined PLAD updates outperform any individual technique.

    Source: §5 · p. 8

  • Takeaways & Limitations

    PLAD provides a framework for fine-tuning shape-program inference with paired data formed from pseudo-labels or approximate distributions.

    Source: §1 · p. 2

    The work focuses on reconstruction quality; good program structure for editing can only be injected through synthetic pretraining, where the required knowledge must be expressed procedurally.

    Source: §5 · p. 8

Abstract

from arXiv · show

Inferring programs which generate 2D and 3D shapes is important for reverse engineering, editing, and more. Training models to perform this task is complicated because paired (shape, program) data is not readily available for many domains, making exact supervised learning infeasible. However, it is possible to get paired data by compromising the accuracy of either the assigned program labels or the shape distribution. Wake-sleep methods use samples from a generative model of shape programs to approximate the distribution of real shapes. In self-training, shapes are passed through a recognition model, which predicts programs that are treated as "pseudo-labels" for those shapes. Related to these approaches, we introduce a novel self-training variant unique to program inference, where program pseudo-labels are paired with their executed output shapes, avoiding label mismatch at the cost of an approximate shape distribution. We propose to group these regimes under a single conceptual framework, where training is performed with maximum likelihood updates sourced from either Pseudo-Labels or an Approximate Distribution (PLAD). We evaluate these techniques on multiple 2D and 3D shape program inference domains. Compared with policy gradient reinforcement learning, we show that PLAD techniques infer more accurate shape programs and converge significantly faster. Finally, we propose to combine updates from different PLAD methods within the training of a single model, and find that this approach outperforms any individual technique.

1. Introduction

Shape program inference seeks a program that generates an observed shape, but paired shape-program data is often unavailable. The paper groups fine-tuning approaches that form such pairs through pseudo-labels or approximate distributions under PLAD, and introduces latent execution self-training.

1. Introduction

  • Shape program inference is challenging because programs are high-dimensional structured outputs with discrete and continuous components, while real shapes often lack their generating programs.Paired shape-program data would make learning the inference model tractable.Source: §1 · p. 1
  • Synthetic program outputs can differ distributionally from real shapes, while policy gradient reinforcement learning is often unstable and slow to converge.The paper describes both as complications in fine-tuning inference models toward the target shape distribution.Source: §1 · p. 1
  • PLAD groups fine-tuning regimes that create shape-program pairs using either pseudo-labels or an approximate shape distribution, then train with maximum likelihood updates.The methods treat the program executor as a black box.Source: §1 · pp. 1–2
  • Latent execution self-training (LEST) executes programs inferred for target shapes, ensuring each generated shape matches its program while potentially shifting the shape distribution from the target.The executor makes the inferred program a correct label for its executed output shape.Source: §1 · p. 2
  • Across 2D CSG, 3D CSG, and ShapeAssembly, PLAD methods achieve better reconstruction performance and require significantly less computation time than policy gradient reinforcement learning.Combining updates from multiple PLAD methods also outperforms any individual method.Source: §1 · p. 2
  • The paper presents PLAD as a framework for related self-supervised shape-program inference techniques and identifies LEST as a new method within it.Its experiments also compare PLAD fine-tuning with policy gradient reinforcement learning and combine PLAD methods.Source: §1 · p. 2

2. Related Work

Prior work fine-tunes shape-program inference models using reinforcement learning, differentiable executors, generative models, and self-training. These approaches differ in executor requirements and in the compromises involved in constructing training pairs.

2. Related Work

  • Prior shape-program inference work commonly pretrains on synthetic shape-program pairs, then fine-tunes toward a target distribution; Table 1 compares fine-tuning methods by model requirements, executor treatment, shape-distribution match, and gradient properties.The table caption identifies its last three rows as PLAD methods.Source: §2 · p. 2, Table 1 · p. 3
  • Policy gradient reinforcement learning treats the inference model as a policy and the executor as a black box, but high-variance gradients make it unstable and slow to converge.The paper contrasts it with PLAD methods, which also treat the executor as a black box.Source: §2 · p. 2
  • Differentiable-executor approaches require a known differentiable executor or a learned proxy, while shape programs often include discrete choices and learned proxies introduce errors.PLAD does not require a differentiable executor.Source: §2 · pp. 2–3
  • Generative-model approaches include variational autoencoders, which require a differentiable executor for gradient training, and wake-sleep, which alternates training on samples produced by the other model.Wake-sleep is presented as an alternative when the executor is not differentiable.Source: §2 · p. 3
  • Self-training can degrade when too many pseudo-labels are incorrect, and the paper says it had not been applied to visual program inference to its authors’ knowledge.In PLAD, pseudo-labels can change during fine-tuning as the model finds programs that better explain input shapes.Source: §2 · p. 3

3. Method

PLAD fine-tunes a pretrained shape-program inference model toward a target shape distribution through repeated program inference, pair construction, and maximum-likelihood training. Its methods differ in how they construct training pairs, and can also be mixed during training.

3. Method

  • PLAD alternates program inference for S*, construction of shape-program pairs, and maximum-likelihood updates to specialize p(z|x) to S*.The framework assumes a shape dataset from S*, an inference model, and a program executor; improvements across iterations can make the pairs better match S* statistics.Source: §3 · p. 3

3.1. Wake-Sleep (X, Z) Construction

  • Wake-sleep trains a generative program model on current best programs and their executed shapes, then samples programs to create training pairs with valid labels but an approximate shape distribution.The implementation uses a VAE; mismatch is expected to shrink as the best programs better approximate S*, assuming the generative model has enough capacity.Source: §3.1 · p. 4

3.2. Self-Training (X, Z) Construction

  • Self-training pairs target shapes S* with current best programs, preserving the target shape distribution but producing biased updates when pseudo-labels do not recreate their paired shapes.The authors report strong reconstruction performance when X is a good approximation to S*.Source: §3.2 · p. 4

3.3. LEST (X, Z) Construction

  • LEST pairs each current best program with its executed shape, making labels correct for paired shapes while risking distributional mismatch with S*.The mismatch decreases as the best programs better approximate S*.Source: §3.3 · p. 5

3.4. Inferring Programs with p(z|x)

  • Across rounds, P_BEST retains a program for each training shape, updating it to the inferred program whose execution has the highest similarity to that shape.The search uses beam search as a speed-performance trade-off, and the similarity metric varies by domain.Source: §3.4 · p. 5

3.5. Training p(z|x) with multiple PLAD methods

  • A single inference model can be fine-tuned with multiple PLAD methods by sampling each training batch from one method’s separately constructed pair distribution.The authors describe combinations such as LEST and self-training.Source: §3.5 · p. 5

4. Results

The experiments compare shape-program fine-tuning across 2D CSG, 3D CSG, and ShapeAssembly, including reconstruction performance, search time, training-set size, and convergence speed. Combined PLAD methods achieve the strongest reconstruction results, while PLAD methods converge faster than policy-gradient reinforcement learning.

4.1. Shape Program Domains

  • CSG combines parametric primitives with Boolean operations, while ShapeAssembly builds manufactured-object structures from cuboid parts and assembly operators.The experiments cover 2D CSG, 3D CSG, and ShapeAssembly.Source: §4.1 · p. 5

4.2. Experimental Design

  • Methods fine-tune a shared synthetically pretrained model using RL, wake-sleep, self-training, LEST, or combinations on held-out shape datasets.The study uses CAD datasets and domain-specific encoder-decoder architectures; 2D CSG is evaluated by Chamfer distance, and 3D CSG and ShapeAssembly by IoU.Source: §4.2 · pp. 5–6

4.3. Reconstruction Accuracy

  • LEST+ST+WS achieves the best test-set performance in every domain, while ST and LEST outperform RL across all studied domains.The 2D CSG result also surpasses CSGNet and CSGNetStack in the black-box-executor setting.Source: Table 2 · p. 6, §4.3 · p. 6
  • Qualitative comparisons cover inferred test-shape programs from 2D CSG, 3D CSG, and ShapeAssembly; combined PLAD reconstructions better reflect input shapes.The reported qualitative trend reinforces the quantitative comparisons.Source: §4.3 · pp. 6, 8, Figure 3 · p. 7

4.4. Inner-loop Search Time

  • More time for inner-loop search improves test performance, and combined PLAD methods outperform individual methods across the tested beam sizes.The search budget controls a trade-off between convergence speed and test-set reconstruction performance.Source: §4.4 · p. 8

4.5. Number of Training Shapes from S∗

  • At 1,000 training shapes, LEST+ST and LEST+ST+WS generalize better than RL trained on 10,000; all methods improve with more training shapes.The training shapes are sampled from S∗.Source: §4.5 · p. 8

4.6. Convergence Speed

  • 36 hours let RL reach converged test-set CD 1.097; LEST matched it in 1.1 hours and LEST+ST in 0.85 hours (42x faster).LEST reached the matched performance 32x faster.Source: §4.6 · p. 8

5. Conclusion

The paper presents PLAD and LEST for fine-tuning shape program inference, reporting better reconstruction accuracy and faster convergence than policy-gradient RL. It also identifies limits in distributional generalization and program structure quality.

5. Conclusion

  • Across 2D CSG, 3D CSG, and ShapeAssembly, PLAD methods achieve better reconstruction accuracy and converge faster than policy-gradient RL; combining methods outperforms any individual technique.The paper groups fine-tuning methods using Pseudo-Labels and Approximate Distributions under PLAD, and introduces LEST, which executes inferred programs to create an approximate shape distribution.Source: §5 · p. 8
  • Because PLAD training pairs approximate the statistics of S*, the fine-tuned model may generalize less well to shapes outside that distribution.Training a general-purpose inference model for all shapes expressible under the grammar is identified as future work.Source: §5 · p. 8
  • For editing tasks, reconstruction quality alone is insufficient: producing programs with good structure matters too.The paper says such structural knowledge currently enters only through synthetic pretraining and must be expressed procedurally, which may be harder to elicit than declarative knowledge.Source: §5 · p. 8
  • The authors suggest PLAD may apply beyond evaluated shape domains when an executor exists and its concrete output can be encoded by a neural network.They give graphics shader programs and audio synthesizers as possible examples.Source: §5 · p. 8

A. Details of Domain Grammars

The appendix specifies the grammars used for 2D CSG, 3D CSG, and ShapeAssembly, including their primitives, operations, and parameterization choices.

2D CSG

  • The 2D CSG grammar uses intersect, union, and subtract operations with square, circle, and triangle primitives parameterized by L and R tuples.The grammar follows CSGNet.Source: A. Details of Domain Grammars · p. 10

3D CSG

  • The 3D CSG grammar uses cuboids and ellipsoids with six parameters per primitive controlling position and scale, each at 32-bin voxel-grid granularity.This avoids the overly discretized primitive spacing and positioning of the CSGNet 3D grammar.Source: A. Details of Domain Grammars · p. 10

ShapeAssembly

  • ShapeAssembly builds 3D structures by instantiating cuboids, attaching parts, and applying squeeze, reflect, or translate macro operators.Its continuous parameters are discretized to match the CSG experiments.Source: A. Details of Domain Grammars · p. 10

B. Details of Synthetic Pretraining

Synthetic pretraining uses domain-specific program-generation procedures for 3D CSG and ShapeAssembly, while 2D CSG uses CSGNet's released pretrained weights.

2D CSG

3D CSG

  • For 3D CSG pretraining, programs combine 2–12 sampled primitives in a Boolean tree, with operation validity constrained by primitive overlaps.The procedure samples 2,000,000 training programs and 1,000 validation programs; after 40 epochs, reconstruction IoU reaches 90 on both sets.Source: B. Details of Synthetic Pretraining · p. 11

ShapeAssembly

  • ShapeAssembly pretraining samples 2–8 primitive blocks, applies attachment and optional symmetry operations, and rejects samples whose cuboids uniquely occupy fewer than 8 voxels.It uses 2,000,000 training programs and 1,000 validation programs; after 26 epochs, reconstruction IoU reaches 70 on both sets.Source: B. Details of Synthetic Pretraining · p. 11

C. Experiment Hyperparameters

The experiments use 3D CNN encoders and Transformer decoders for 3D tasks, with separate 2D CSG settings. Training uses validation-based stopping, and Table 3 contrasts two P_BEST update policies.

3D Experiments

  • The 3D encoder maps a 32 × 32 × 32 voxel grid to an 8-token embedding, which the Transformer decoder attends to alongside up to 100 sequence tokens.The encoder uses four convolutional layers; the decoder has 8 layers, 16 heads, and hidden dimension 256.Source: C. Experiment Hyperparameters · pp. 11–12

Training settings

  • Training uses Adam at learning rate 0.0005 except RL, which uses SGD at 0.01; fine-tuning batches are 100 for PLAD and 4 for RL.Validation-based early stopping uses patience 100 epochs between PLAD rounds and 10 within a round; final reconstruction is evaluated with beam size 10.Source: C. Experiment Hyperparameters · p. 12

2D Experiments

  • For 2D CSG, the experiments follow CSGNet's architecture, use dropout 0.2 and batch size 100, and train PLAD with Adam at learning rate 0.001.The patience is 1,000 epochs for deciding whether to introduce rounds and 10 epochs within each round; the cited stopping criterion uses a 0.005 CD improvement threshold.Source: C. Experiment Hyperparameters · p. 12

P_BEST updates

  • Table 3 compares clearing P_BEST between rounds with retaining each input shape's best program across rounds.The all-time policy retains the best program across multiple rounds, whereas the per-round policy clears the data structure between rounds.Source: Table 3 · p. 12

D. P Best Update mode

Maintaining each shape’s best program across rounds yields better converged reconstruction accuracy than resetting the best program each round.

D. P Best Update mode

  • Maintaining P_BEST entries across rounds gives every fine-tuning strategy better converged reconstruction accuracy than resetting them each round.The comparison is reported for 2D CSG.Source: D. P Best Update mode · p. 12

E. Failure to generalize beyond S∗

PLAD fine-tuning specializes the model toward the target shape distribution, but can reduce accuracy on out-of-distribution 2D icons. The paper also notes potential environmental and intellectual-property implications.

E. Failure to generalize beyond S∗

F. Potential Societal Impacts

  • Fine-tuning the neural networks requires substantial electricity, while shape-program inference may enable reverse engineering of protected intellectual property.The paper notes potential environmental impact and possible implications for copyright enforcement.Source: F. Potential Societal Impacts · p. 13

G. Additional Qualitative Results

The paper presents additional qualitative comparisons of fine-tuning methods across 2D CSG, 3D CSG, and ShapeAssembly.

G. Additional Qualitative Results

SP WS RL ST LEST LEST+ST LEST+ST+WS Target

Qualitative examples are provided for 2D CSG, 3D CSG, and ShapeAssembly.

Related papers

Loading 2011.13045v4…