Source-linked AI summary
Synthesizing Programs for Images using Reinforced Adversarial Learning
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, Oriol Vinyals
TL;DR
SPIRAL addresses the challenge of scaling inverse graphics without labels or hand-crafted image objectives by training an adversarial reinforcement-learning agent to generate visual programs executed by a renderer. The approach scales to real-world and procedural datasets, and discriminator-based rewards improve optimization over direct pixel-error rewards.
Problem
Existing deep generative and renderer-based methods can rely on weak decoder biases, hand-crafted objectives, labels, or inference procedures that do not scale readily to richer datasets.
Method
SPIRAL trains an adversarial reinforcement-learning agent whose policy generates commands or visual programs that a renderer executes to produce images.
Results
SPIRAL scales visual program synthesis without labels across MNIST, Omniglot, celebrity faces, and synthetic 3D scenes, while discriminator rewards significantly outperform direct pixel-error optimization.
Takeaways & Limitations
Using a Wasserstein discriminator’s output as the reward with asynchronous reinforcement learning provides a potential scaling path for visual program synthesis.
Takeaways & Limitations
The current entropy-based exploration strategy and stroke parameterization are limited, motivating more sophisticated search algorithms and alternative action representations.
Abstract
from arXiv · showhide
Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level details and represent images as high-level programs. Current methods that combine deep learning and renderers are limited by hand-crafted likelihood or distance functions, a need for large amounts of supervision, or difficulties in scaling their inference algorithms to richer datasets. To mitigate these issues, we present SPIRAL, an adversarially trained agent that generates a program which is executed by a graphics engine to interpret and sample images. The goal of this agent is to fool a discriminator network that distinguishes between real and rendered data, trained with a distributed reinforcement learning setup without any supervision. A surprising finding is that using the discriminator's output as a reward signal is the key to allow the agent to make meaningful progress at matching the desired output rendering. To the best of our knowledge, this is the first demonstration of an end-to-end, unsupervised and adversarial inverse graphics agent on challenging real world (MNIST, Omniglot, CelebA) and synthetic 3D datasets.
1. Introduction
SPIRAL addresses the challenge of recovering structured visual programs from images by combining graphics engines with adversarially trained reinforcement learning. It generates programs for conditional interpretation or unconditional sampling, without paired supervision, and shows that discriminator rewards improve optimization.
- Motivation: Inverse graphics is difficult because renderers require long sequences of discrete commands, while black-box simulators are nondifferentiable.These constraints make matching rendered images to real data an optimization challenge.
- Approach: SPIRAL uses an adversarially trained agent to generate visual programs executed by a graphics engine for image interpretation and generation.The agent supports both conditional generation from data and unconditional generation.
- Approach: The agent is trained with distributed reinforcement learning, receives reward from fooling a discriminator, and requires no extra supervision.The discriminator distinguishes rendered images from real images.
- Contributions: SPIRAL’s architecture is agnostic to visual-program semantics and domain, enabling application across different program spaces.The contribution is framed as an agent that operates in the space of visual programs rather than a single handcrafted representation.
- Applications: The model discovers pen strokes for MNIST and Omniglot, brush strokes for celebrity faces, and scene descriptions for reconstructed 3D images.These experiments span real-world and procedural datasets.
- Result: Using a discriminator’s output as the reinforcement-learning reward significantly outperforms directly optimizing pixel error for optimizing rendering pixel error.The paper identifies discriminator reward as a key ingredient for meaningful progress.
2. Related Work
SPIRAL builds on inverse graphics, visual program induction, neural image generation, and adversarial learning while reducing reliance on handcrafted supervision. Its design targets broader domain coverage than prior structured-image methods.
- Inverse graphics: Prior inverse-graphics research explored simulator inversion and structured object-attribute de-rendering for images and videos.Related approaches include differentiable inverse graphics and constructive solid geometry for binary images.
- Motor programs: Earlier MNIST motor-program work used prototype programs, corresponding observations, and noise-based training examples rather than automatically learned curricula.SPIRAL is contrasted with this manually initialized training procedure.
- Visual program induction: Visual program induction on Omniglot required parses from a handcrafted algorithm and was not demonstrated beyond handwritten characters.Other work inferred LaTeX programs for diagrams or generated sketches with sequence-to-sequence learning.
- Neural generation: Neural image-generation models produce impressive images but generally lack structured representations of those images.This motivates combining generative modeling with program-like representations.
- Adversarial learning: SPIRAL applies adversarial training to visual program synthesis, extending ideas previously used for generative modeling and domain adaptation.The related literature includes GANs and image-domain transfer methods.
- Positioning: SPIRAL uses minimal hand-crafting, requires no paired programs and images, and is presented as applicable across a wide range of domains.This positioning directly distinguishes it from more supervised or narrowly demonstrated approaches.
3. The SPIRAL Agent
SPIRAL combines a recurrent policy with an external renderer and adversarial training to recover image-generating programs without aligned supervision. Because the generator is non-differentiable, the policy is optimized with reinforcement learning using discriminator-based rewards, with distributed workers supporting training.
- 3.1. Overview: SPIRAL models image generation as recovering an action distribution whose rendered outputs approximate the target data distribution.A recurrent policy samples commands, and the renderer transforms those commands into images.
- 3.2. Objectives: Adversarial training makes the generator confuse a discriminator distinguishing real data from generated renders, without requiring aligned examples.The discriminator is trained on samples from the data and generated distributions, while the generator seeks to make those distributions closer.
- 3.2. Objectives: The generator uses a recurrent policy to sample command sequences that an external rendering simulator converts into images.The policy predicts distributions over commands at each time step, while the renderer produces samples from the generated distribution.
- 3.2. Objectives: Because the generator is non-differentiable, SPIRAL optimizes its policy through advantage actor-critic reinforcement learning rather than naive gradient descent.The policy objective uses expected returns, with a value-function approximation and Monte Carlo return estimates.
- 3.2. Objectives: Discriminator output is used as the terminal reward, while optional intermediate rewards can bias the search using renderings or the commands that produced them.The terminal reward is zero before the final step and equals D(R(a1, a2, . . . , aN)) at the final step.
- 3.4. Distributed Learning: The distributed training pipeline separates actors, policy learning, and discriminator learning, using a replay buffer to decouple discriminator updates from policy updates.Actors generate trajectories, the policy learner updates π, and the discriminator learner trains on real examples and final generated renders.
4. Experiments
SPIRAL is evaluated across handwritten characters, celebrity faces, and synthetic 3D scenes using renderer-specific action spaces. Discriminator-based rewards consistently support more effective reconstruction than direct ℓ2 optimization, including out-of-domain parsing and near-perfect scene reconstructions.
- Datasets: The evaluation spans MNIST, OMNIGLOT, CELEBA, and procedural MUJOCO SCENES, with images rescaled to 64 × 64 for shared network architectures.MNIST contains handwritten digits; OMNIGLOT adds variability and structural complexity; CELEBA contains over 200,000 varied celebrity headshots; MUJOCO SCENES contains 50,000 rendered RGB images.
- Environments: In libmypaint, the agent generates sequences of strokes by selecting control points, endpoints, pressure, brush size, color, and action type.Locations use a 32 × 32 grid, pressure has 10 levels, and each color component has 20 bins.
- Environments: For MUJOCO SCENES, each action specifies an object type, location, size, and color, while allowing skipped moves or edits to the latest object.The environment renders scenes containing simple 3D primitives scattered around a square platform.
- MNIST: On MNIST, discriminator-based conditional training accelerates learning and achieves lower final ℓ2 error than fixed ℓ2 rewards.Without auxiliary rewards, ℓ2-based runs failed to learn reasonable reconstructions.
- OMNIGLOT: On OMNIGLOT, discriminator rewards yield convincing conditional reconstructions, whereas ℓ2-based training fails to learn sufficiently well because exploration is inadequate.The trained agent also reconstructs previously unseen line drawings well, though it performs slightly better on the OMNIGLOT test set.
5. Discussion
SPIRAL scales adversarial inverse graphics to real-world and combinatorial datasets using black-box renderers, while revealing limitations in current exploration and action parameterizations.
- Discriminator training produces significantly lower ℓ2 reconstruction distances than directly minimizing pixel error across datasets.
- SPIRAL reliably processes every MUJOCO SCENES holdout image in one pass, whereas blocked Metropolis-Hastings failed after thousands of evaluations.
- Using a Wasserstein discriminator reward with asynchronous reinforcement learning provides a scaling path for visual program synthesis.
- Future work should improve the entropy-based exploration strategy and action-space parameterizations for richer visual scenes.
- SPIRAL can reconstruct 3D scenes by emitting MuJoCo commands, inferring object counts, locations, sizes, and colors.
- The paper positions inverse simulation and program synthesis as applicable beyond vision, including speech, music, and scientific simulators.
A. Optimal D for Conditional Generation
For conditional generation with a point-mass target distribution, ℓ2 distance is an optimal discriminator, but discriminator solutions are non-unique and can provide different learning signals in practice.
- When the data distribution is a Dirac delta, the Wasserstein formulation admits ℓ2 distance as an optimal discriminator.
- The ℓ2 solution is not unique, so alternative 1-Lipschitz functions can achieve the same Wasserstein objective value.
- Because ℓ2 is constant on points equidistant from the target, it expresses no preference among potentially semantically different candidates.
- Functions with non-zero local slope can shift search toward more promising subspaces, unlike a flat ℓ2 signal near equidistant points.
- In practice, dynamically trained discriminators and fixed ℓ2 distances are not equivalent because the exact dual is not optimized to convergence.
B. Network Architectures
SPIRAL uses a recurrent policy that encodes the current canvas and previous action, then autoregressively samples each component of the next action tuple.
- The policy network combines canvas observations and the previous action through convolutions, ResNet blocks, a fully connected layer, and an LSTM.
- An autoregressive decoder samples each next action component from categorical distributions conditioned on recurrent hidden representations.
- Scalar and spatial action components use separate decoder branches, with spatial components processed by ResNet and transpose-convolution layers.
- After sampling each component, the decoder embeds it into a 16-dimensional code and updates the hidden vector until the tuple is complete.
- The discriminator uses a conventional architecture similar to DCGAN.
C. Training Details
Training combines Adam-optimized discriminator learning with population-based exploration for the reinforcement-learning generator, using distributed actors and replay-buffer sampling.
- The discriminator uses Adam with learning rate 10^-4 and β1 = 0.5.
- Population-based exploration tunes the policy learner’s entropy-loss coefficient and learning rate across 12 training instances.
- Each training instance runs 64 CPU actor jobs and two GPU jobs, one for policy learning and one for discriminator learning.
- Both learners use batch size 64, while generated data is sampled uniformly from a replay buffer holding 20 batches.
- Figure 11 contrasts the image data, pixel-space ℓ2 distance to the target, and discriminator score in a toy circle-location experiment.
- The policy and autoregressive decoder architectures are documented separately for single-step processing and action-tuple generation.