Learning to Edit Visual Programs with Self-Supervision

1 Introduction

• Program authoring commonly involves trial, execution, debugging, and revision, motivating an approach that can edit intermediate programs rather than rely only on one-shot generation.Visual program induction seeks a program whose execution recreates a target visual input.Source: §1 · p. 1
• Prior learning-based approaches include supervised methods for annotated domains and unsupervised or self-supervised methods for domains lacking program annotations.The paper frames one-shot inference as a common limitation of prior approaches.Source: §1 · p. 1
• The edit network uses a complete input program, its executed state, and a visual target to propose a local change toward the target.Unlike one-shot prediction, it reasons over the complete program and its execution when choosing how to modify it.Source: §1 · p. 2
• Without ground-truth target programs, paired edit-training examples are built from start and end programs and domain-aware edits, jointly finetuning the edit and one-shot networks.The one-shot model initializes editing, while improved edit-based inference supplies better bootstrapped training data for the one-shot model.Source: §1 · p. 2
• Across multiple visual programming domains, the joint edit-and-one-shot approach improves reconstruction over one-shot models under equal inference-time budgets.The gap widens with more test-time search and with less training data.Source: §1 · p. 2

Visual Program Induction

• For annotated domains, visual program induction methods can use ground-truth program supervision; many visual domains of interest lack such annotations.Source: §2 · p. 2
• For unannotated visual domains, prior methods include reinforcement learning, differentiable execution, proxy executors, and bootstrapping, while still operating in a one-shot paradigm.SIRI uses analytical rewriting operations; this work instead learns edits rather than relying on fixed domain-specific operations.Source: §2 · p. 2
• Unlike approaches that reason over partial executions to guide autoregressive synthesis, this edit network predicts local changes to a complete program.Source: §2 · p. 2

Program Repair

• Program-repair methods have addressed annotated domains, whereas this method targets more complex visual programs without program annotations and predicts one local edit per step.The cited approaches include Karel synthesis with execution and debugging and latent-space fixes for RobustFill programs.Source: §2 · p. 3
• Early LLM program-repair work uses prior program versions and input/output mismatches, but the passage says LLMs have not yet written complex visual programs.For general synthesis, performance gains from code-editing LLMs are described as not definitive.Source: §2 · p. 3

3 Method

• The method frames unsupervised visual program induction as finding a DSL program whose execution resembles a target, using a domain-specific reconstruction metric.It replaces predicting a complete program from scratch with predicting a local edit to an input program.Source: §3 · p. 3

3.1 Edit Network Design

• The edit network receives an input program, its execution, and a visual target, then predicts an operation type, token location, and any required parameters.Its constrained DSL-aware edit set is local and designed to promote syntactically valid modified programs.Source: §3.1 · p. 4

3.2 Learning Paradigm

• Training pairs are created by analytically finding edits between start and end programs, then used with teacher forcing to train edit predictions.For unannotated targets, bootstrapped finetuning integrates this edit training into PLAD; the edit network is trained on transformations from sampled start programs to generated programs, with synthetic pretraining also described.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 Inference Algorithm

• Inference initializes a program population from the one-shot model, applies sampled edits over rounds, and resamples according to reconstruction ranking.The joint inference loop is also used during finetuning to populate the best-program set, linking improved search with subsequent bootstrapped training data.Source: §3.3 · pp. 5–6
• Across visual programming domains, the joint edit-and-one-shot paradigm outperforms one-shot-only inference on test-set reconstruction.Source: Table 1 · p. 6

4.1 Experimental Design

• The evaluation compares jointly finetuned OS+Edit with OS Only, using a sampling-based search baseline and matched inference time across Layout, 2D CSG, and 3D CSG.Reconstruction is measured with cIoU for Layout, edge-based Chamfer distance for 2D CSG, and IoU for 3D CSG.Source: §4.1 · pp. 6–7

4.2 Reconstruction Accuracy

• OS+Edit consistently improves reconstruction over OS Only across three domains under equal inference time; on 2D CSG, CD is 0.111 versus UCSG-Net’s 0.320, SIRI’s 0.260, and ROAP’s 0.210.The paper cautions that direct comparisons with related methods are not absolute because their DSLs, architectures, objectives, and inference procedures differ.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3 Search Time

• As test-time search time increases, the reconstruction gap between OS+Edit and OS Only widens on 2D CSG.The gap is present from the first round, partly because the OS+Edit one-shot network had better P BEST entries during finetuning; Figure 5 shows the population evolving toward the target.Source: §4.3 · p. 7, Figure 4 · p. 8, Figure 5 · p. 9

4.4 Training with limited data

• With just 100 training shapes, OS+Edit matches OS Only trained with 10 times more data on 2D CSG.The authors attribute the data efficiency to edit learning being more local and allowing many edit-operation training examples to be sourced from each visual datum.Source: §4.4 · p. 8

4.5 Method Ablations

• Predicting local edit operations outperforms predicting either the next program or the final program, and is faster enough to allow more inference rounds.The final-program alternative is described as an essentially one-shot model with extra conditioning but the same target sequences.Source: §4.5 · p. 8
• The default training-pair construction outperforms corruption-based pairing because findEdits better matches the program distribution available at inference time.The corruption variant samples inverse edits from end programs, which is difficult to align with the inference-time start-program distribution.Source: §4.5 · p. 8
• Using all three training phases yields the best results, while edit mutations can substantially improve a starting population when the one-shot model is not finetuned.Without one-shot finetuning, cIoU rises from 0.88 to 0.972 (0.092 improvement); the default variant starts at 0.925 (an improvement of .055).Source: §4.5 · p. 9
• The default inference algorithm outperforms applying only the highest-likelihood edit or initializing with random programs, though edits still help in both alternatives.Naive OS improves by 0.022 from first to last round; Rand+Edit starts at 0.302 cIoU, and the paper reports better reconstruction from combining edits with one-shot initial guesses.Source: §4.5 · p. 9

5 Discussion

• Jointly finetuning an edit network and a one-shot model improves reconstruction over one-shot inference alone, especially with more search time or less training data.The discussion presents the method’s ablation study as support for its design.Source: §5 · p. 9
• The method requires training an additional network, increasing pretraining and finetuning time, and its full benefit uses more costly search budgets.The authors note it may suit workflows that can afford more program-search time and identify speed-ups as useful future work.Source: §5 · p. 9
• The formulation requires a domain-aware findEdits operation; replacing it with more general program-difference algorithms remains future work.The authors report their implementation generalizes across a range of visual-programming domains.Source: §5 · p. 10

A Appendix Overview

• The appendices cover additional experiments, domain and experimental-design details, editing operations, and program-corruption experiments; Figure 6 shows 3D CSG challenge-task reconstructions.Source: A Appendix Overview · p. 13, Figure 6 · p. 13

B.1 Performance on more challenging tasks

• The authors hypothesize that local edits can generalize to complex or out-of-distribution targets by comparing a current program’s execution with the target and proposing goal-directed fixes.They motivate the test by asking whether the performance gap might shrink on harder problems, but report that the joint approach instead outperforms one-shot alone.Source: §B.1 · p. 13
• Challenge tests use unseen Layout concepts such as butterflies and snowmen, and unseen ShapeNet categories for 3D CSG rather than the original finetuning categories.The 3D CSG test categories include airplanes, knives, lamps, laptops, motorbikes, mugs, pistols, skateboards, rifles, and vessels; the original categories were chairs, tables, benches, and couches.Source: §B.1 · p. 13
• Across 192 Layout and 100 3D CSG challenge tasks, the joint one-shot/edit approach outperforms one-shot alone, including on targets further outside the training distribution.Figure 7 shows more accurate Layout reconstructions for the approach (column 5) than one-shot alone (column 4), alongside targets in column 6.Source: §B.1 · p. 14, Figure 7 · p. 14

B.2 Comparison to large vision-language models

• In qualitative Layout comparisons, the proposed method outperforms GPT-4V tested zero-shot, with an in-content example, or with a one-shot program as input.The authors caution that these results do not show large vision-language models will never be useful, but suggest the task is not easily solved by currently available frontier models.Source: §B.2 · p. 14, Figure 7 · p. 14

B.3 Method Ablations on 2D CSG domain

• A pretrained edit network that remains frozen during joint finetuning still helps find better inference solutions and supplies better training data for the one-shot network.System performance is maximized when the edit network is also updated during finetuning.Source: §B.3 · p. 15
• 0.111 Chamfer distance is achieved by the default 2D CSG approach, compared with 0.230 when the one-shot network is not finetuned.The no-one-shot-finetuning condition performs strongly on Layout, improving cIoU from 0.88 to 0.972, versus 0.925 to 0.980 for the default approach.Source: §B.3 · p. 15

C Domain Details

• The Layout DSL places colored primitives on a 2D canvas and combines them; union layers shapes, while symmetry, color, move, and scale operations transform a shape.A primitive command creates a shape from a parameter-type argument alone.Source: C Domain Details · p. 16
• The 2D CSG DSL unions positive sub-expressions, unions negative sub-expressions, then subtracts the latter group from the former.Its combinators are union, difference, and intersection; move, scale, rotate, and reflect are transformations, and primitives create shapes from parameters.Source: C Domain Details · p. 16
• The 3D CSG DSL generalizes the 2D version with three-dimensional move and scale parameters, three-axis rotation, and cuboid, sphere, or cylinder primitives.The grammar retains union, difference, intersection, reflection, and positive/negative expression blocks.Source: C Domain Details · p. 17
• Models use the same synthetic-program sampling logic, with rejection criteria such as excluding execution errors and outputs outside the canvas.The authors report that this logic is used for all evaluated models.Source: C Domain Details · p. 17

D Experimental Design Details

• The 2D domains use a CNN that converts 64×64 inputs into 16 visual tokens, while the 3D CNN produces eight 256-dimensional tokens from voxel grids.The 2D CSG input has one occupancy channel and Layout has RGB channels; the 3D model processes voxel grids of size 32^3.Source: D Experimental Design Details · p. 17
• Transformer networks use eight layers, 16 heads, hidden dimension 256, and maximum program lengths of 128, 164, and 256 tokens for Layout, 2D CSG, and 3D CSG.Maximum edit lengths are 32, 32, and 48, respectively; edit prediction heads are three-layer MLPs.Source: D Experimental Design Details · p. 17
• Training uses Adam with a 1e-4 learning rate, synthetic pretraining until validation convergence, and finetuning for up to six days or until convergence.Pretraining iteration counts and batch sizes differ by model and domain; finetuning uses up to 40, 40, and 30 bootstrap rounds for Layout, 2D CSG, and 3D CSG.Source: D Experimental Design Details · pp. 17–18
• Test-time search uses equal inference-time budgets for the edit-based and one-shot-only modes: 5, 10, and 60 seconds per shape across the three domains.Population sizes and rounds are Layout (32, 32), 2D CSG (32, 32), and 3D CSG (80, 25) for the edit-based method.Source: D Experimental Design Details · p. 18

E.1 Local Edit Operations

• The edit network selects among seven local operations: modify parameters or transforms, add or remove transforms, and modify, add, or remove combinators.The DSL is assumed to be functional, with valid functions returning shape-typed values; transformations consume one shape argument and combinators consume multiple.Source: §E.1 · p. 18
• Edits are local in program structure, though adding or removing combinators can alter whole expression trees and produce dramatic changes in execution.Changing a CSG boolean operation from difference to union is one example of a local edit with a potentially large execution-space effect.Source: §E.1 · pp. 18–19

E.2 findEdits Algorithm

• findEdits analytically derives a set of operations to transform a start program into an end program, sourcing training data for the edit network.The algorithm aims for a minimal-cost edit set and evaluates transformations by program semantics, including equivalent executions rather than syntax alone.Source: §E.2 · p. 19
• The algorithm computes sub-expression match costs and uses the Hungarian matching algorithm to select a valid overall matching.Its matching formulation first estimates the edit operations needed to reproduce one sub-expression’s visual output from another.Source: §E.2 · pp. 19–20
• For sub-expressions, findEdits recursively compares transforms and parameters, recurses through paired combinators, and adds or removes expression trees when only one program has a combinator.Order-dependent transforms may require insertion, removal, or modification, with dependencies tracked for later edits.Source: §E.2 · p. 19
• For whole programs, the method matches sub-expressions while respecting domain-specific ordering, including positive-to-positive and negative-to-negative CSG matches.Layout keeps sub-expression order fixed but permits matches to empty expressions, corresponding to removal or addition edits.Source: §E.2 · pp. 19–20

• The edit set may have ordering dependencies, but otherwise its operations can be ordered arbitrarily.Source: §E.3 · p. 20
• For n independent edits, the procedure considers subsets of i prior edits for each i from 0 to n−1, subsampling five previous edit sets to avoid exponential blow-up.Source: §E.3 · p. 20
• For each sampled prior-edit set and eligible next edit, it creates a training triplet from the partially edited program, target visual, and target edit.Source: §E.3 · p. 20

• The framework could apply to other visual-program and program-synthesis DSLs that meet the stated criteria, including DSLs expressible as a Context Free Grammar.Source: §E.4 · p. 20
• Transform-function edits locally delete, insert, or modify without changing branching factor, while combinator-function edits delete or insert and can change it.Source: §E.4 · p. 20
• The framework and training scheme are not tied to this edit set and could use other local edits if an analogous findEdits algorithm can be designed.Source: §E.4 · p. 20

• The domain-specific process starts from an end program and randomly applies inverse edits, keeping intermediate programs valid; it ignores the start program.Source: F Program Corruption · p. 20
• The corruption variant is less performant than the default, yet still provides useful edit-network training data.Source: F Program Corruption · p. 20
• Corruption can substitute when program-difference scripts are hard to find, but designing a process that keeps edit distributions aligned with the one-shot model is described as non-trivial.The default findEdits approach derives edits from start/end program pairs and its edit distribution evolves with the finetuned one-shot model.Source: F Program Corruption · p. 21