Source-linked AI summary
Learning to Infer Graphics Programs from Hand-Drawn Images
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, Joshua B. Tenenbaum
TL;DR
The paper asks how to infer useful, human-readable graphics programs from hand-drawn images. It combines neural specification inference with program synthesis, using learned proposals and search to recover structured programs. The resulting system parses synthetic and real drawings, with the combined neural-network-plus-SMC approach generalizing beyond its training scene complexity.
Problem
The paper addresses the problem of learning high-level, human-readable graphics programs from perceptual input rather than only recovering low-level drawing commands.
Method
The system uses a convolutional network to infer drawing-command specifications, then applies stochastic search and constraint-based program synthesis to recover structured graphics programs.
Results
The combined neural-network-plus-SMC system generalizes from scenes with ≤12 training objects to scenes with many more objects, unlike either component alone.
Takeaways & Limitations
Graphics programs provide a representation that captures higher-level visual structure beyond a specification’s scene contents.
Takeaways & Limitations
Training assumes a canonical ordering of drawing commands because rendering ignores command order and maps multiple specifications to the same image.
Abstract
from arXiv · showhide
We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of \LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input.
1 Introduction
The paper frames visual understanding as learning high-level graphics programs from hand-drawn images, separating primitive extraction from program synthesis. This factoring supports programs that represent structure such as symmetry, repetition, reuse, and nested loops.
- The model aims to learn high-level graphics programs from simple 2D drawings, capturing symmetry, repetition, and reused structure.
- Pixels-to-program inference separates identifying primitive objects from recognizing higher-level visual concepts governing their arrangement.
- The generative model executes a latent program into drawing commands, renders them into an image, and treats the commands as an intermediate specification.
- Stochastic search infers specifications from images, while constraint-based synthesis recovers programs containing structures such as symmetries, loops, and conditionals.
- Neural inference and Bias–Optimal Search amortize the costs of specification inference and program synthesis, respectively.
2 Neural architecture for inferring specs
The neural architecture predicts drawing-command specifications from images and combines learned proposals with Sequential Monte Carlo for robust inference. Noise-augmented training extends the approach toward real hand drawings, while experiments show that the combination generalizes beyond training scene complexity.
- The network predicts a specification one drawing command at a time, conditioning on the target image and commands drawn so far.It encodes the 256 × 256 target and a rendering of prior commands as a two-channel image before predicting the next command.
- The model factorizes the next-command distribution over command tokens and emits STOP when the specification explains the image.
- Sequential Monte Carlo uses neural outputs as proposals and scores candidate specifications with a rendering-based likelihood, allowing recovery from incorrect predictions.
- Only the neural-network-plus-SMC system generalizes from training scenes with ≤12 objects to substantially more complex scenes.The comparison includes neural beam search, SMC alone, and a one-shot LSTM baseline; models are compared at approximately equal runtime.
- Training data uses a canonical ordering of drawing commands because rendering ignores their ordering, making the spec-to-image mapping many-to-one.
- Generalizing to real hand drawings: Training adds rendering noise that mimics hand-drawing variation and learns a surrogate likelihood based on differences between drawing-command specifications.
Appendix A.3 details the architecture and training of Llearned.
On 100 real hand-drawn figures, the neurally guided SMC sampler recovered exact ground-truth specs for 63% of drawings, while additional samples often produced closer matches.
- 63% of the drawings had a Top-1 sample exactly matching the ground-truth spec.The evaluation used 103 samples per drawing.
- With larger sample sets, the model produced specs closer to the ground-truth annotation.Figure 8 compares sets of 1, 5, and 100 samples using intersection over union.
3 Synthesizing graphics programs from specs
The synthesizer searches a graphics-program DSL for minimum-cost programs consistent with a spec, then uses a learned policy to prioritize tractable and promising program subspaces. Bias-optimal parallel search preserves coverage while allocating more computation to policy-favored spaces, though general synthesis can be computationally expensive.
- 3 Synthesizing graphics programs from specs: The DSL represents graphics programs with loops, conditionals, reflections, variables, and affine transformations.These constructs encode prior knowledge about typical graphics programs.
- 3 Synthesizing graphics programs from specs: The synthesizer seeks the minimum-cost program that satisfies the spec, with cost based on statement count and numerical-constant usage.The program prior is proportional to exp(−cost(p)), while consistency with the spec determines likelihood.
- 3 Synthesizing graphics programs from specs: The synthesizer requires a finite program-space bound and guarantees global optimality, but provides no runtime guarantee.The syntax-tree depth bound was set to 3, and synthesis times ranged from minutes to hours.
- 3.1 Learning a search policy for synthesizing programs: The search policy πθ(σ|S) maps a spec to a distribution over program subspaces, balancing tractability against the chance of containing a concise explanation.Each σ is a restricted subset of the DSL, and the family of subspaces covers every DSL program.
- 3.1 Learning a search policy for synthesizing programs: Bias-optimal search runs program searches in parallel and allocates compute to each subspace in proportion to πθ(σ|S).This strategy explores the entire program space while emphasizing regions judged promising by the learned policy.
- 3.1 Learning a search policy for synthesizing programs: The learned policy minimizes expected bias-optimal synthesis time on a training corpus, but unrestricted Sketch synthesis timed out for 27% of drawings after one hour.The search policy was evaluated against Sketch, DC, End-to-End, and Oracle alternatives; End-to-End failed to find valid programs 94% of the time.
4 Applications of graphics program synthesis
Program synthesis improves candidate drawing specifications by favoring concise, general programs, enabling error correction, program-based similarity, and figure extrapolation.
- 4.1 Correcting errors made by the neural network: It can favor precise alignment, repeated parts, loops, and reflectional symmetry because these structures often yield shorter or more general programs.This allows synthesis to correct some errors in the neural network’s proposed specifications.
- 4.1 Correcting errors made by the neural network: The synthesizer evaluates candidate specs using a learned program prior whose features include program size and loop counts.The prior is log-linear and is estimated from annotated graphics-program synthesis problems.
- 4.1 Correcting errors made by the neural network: 63% to 67%: learning the program prior modestly improves Top-1 accuracy after synthesizing programs for the neural network’s top 10 specs.The best possible improvement from considering those top 10 specs is 70%.
- 4.2 Measuring similarity between drawings: Program features support similarity measures based on high-level structures such as loops, reflectional symmetry, and grid-like organization.These measures can distinguish abstract geometric similarity from pixel-level similarity.
- 4.3 Extrapolating figures: Source programs enable coherent image editing, including extrapolating figures by increasing the number of loop executions.The system demonstrates extrapolations from provided repetitive drawings.
5 Related work
The work connects program induction, deep learning, and sketch-based graphics systems while targeting high-level programs from noisy visual input.
- Program Induction: The program-search approach draws on Levin search and OOPS, and learns likely program components while modeling tractability versus success probability.DeepCoder also predicts likely program components, but this work emphasizes the tractability–success trade-off.
- Deep Learning: The neural architecture combines object decomposition ideas from Attend-Infer-Repeat with training and stochastic inference from Neurally Guided Procedural Modeling.Unlike IM2LATEX’s equation derendering, the goal is recovering a high-level program from noisy input.
- Hand-drawn sketches: The method complements Sketch-n-Sketch by potentially supplying inferred programs as starting points for further bidirectional editing.Related graphics systems also convert sketches into procedural or parametric representations.
6 Contributions
The paper presents graphics-program inference from hand-drawn images and demonstrates parsing and extrapolation, positioning visual-program inference as a research direction.
- 6 Contributions: The system infers LaTeX-style graphics programs from hand-drawn images using deep neural networks, stochastic search, and program synthesis.It parses drawings into symbolic specifications before inferring structured programs.
- 6 Contributions: The authors evaluate parsing on novel images and demonstrate extrapolation from provided drawings.They identify visual-program inference as a promising direction in machine perception.
A.1.1 Convolutional network
The convolutional network uses a two-layer architecture with convolutions, ReLU nonlinearities, and max pooling to process 2 × 256 × 256 image volumes.
- A.1.1 Convolutional network: The network receives two 256 × 256 images as a 2 × 256 × 256 volume and processes them through two convolutional layers separated by ReLU nonlinearities and max pooling.The input consists of the target image and another image representation used by the model.
- A.1.1 Convolutional network: Layer 2 uses 10 8 × 8 convolutions, followed by 4 × 4 pooling with stride 4.This completes the described convolutional feature extractor.
A.1.2 Autoregressive decoding of drawing commands
The model autoregressively predicts drawing-command tokens from image features, using attention conditioned on previously predicted tokens, then combines neural proposals with synthesis and search for robustness.
- Autoregressive decoding: Image features are used to predict the first drawing-command token from circle, rectangle, line, or STOP.The first-token predictor is logistic regression.
- Autoregressive decoding: Subsequent tokens are predicted autoregressively with attention conditioned on the previously generated command tokens.Spatial Transformer Networks focus prediction on relevant image regions, such as conditioning a circle’s y coordinate on its command identity and x coordinate.
- Autoregressive decoding: Each drawing-primitive token has its own learned MLP, with a 32-hidden-node sigmoid network for line coordinates and logistic regression for other tokens.The token-specific networks specialize prediction by primitive and coordinate type.
- Evaluation: On scenes with more objects than training examples, only the neural-network-plus-SMC combination passed the generalization test; direct image-to-program baselines failed on sophisticated scenes.The direct no-spec baseline received substantially more synthetic training data yet failed for most test cases.