Source-linked AI summary
Learning to Paint With Model-based Deep Reinforcement Learning
Zhewei Huang, Wen Heng, Shuchang Zhou
TL;DR
The paper asks how machines can paint complex images with sequential strokes despite the difficulty of planning stroke decompositions without human tracking data. It uses a differentiable neural renderer within model-based DRL to control stroke parameters and plan over long sequences. Experiments show good performance across simple and real-world target images, including portraits and texture-rich natural scenes.
Problem
Teaching machines to paint complex, texture-rich real-world images requires ordered stroke decomposition and long-term planning, while ground-truth stroke decompositions are hard to define.
Method
The paper combines model-based DRL with a differentiable neural renderer, learning sequential stroke actions that control position, shape, color, and transparency.
Results
The learned agent handles multiple target-image types and paints real-world images such as human portraits and texture-rich natural scenes.
Takeaways & Limitations
The approach can decompose complicated scenes into hundreds or even thousands of strokes and generate vivid paintings without requiring human-painter experience or stroke-tracking data.
Abstract
from arXiv · showhide
We show how to teach machines to paint like human painters, who can use a small number of strokes to create fantastic paintings. By employing a neural renderer in model-based Deep Reinforcement Learning (DRL), our agents learn to determine the position and color of each stroke and make long-term plans to decompose texture-rich images into strokes. Experiments demonstrate that excellent visual effects can be achieved using hundreds of strokes. The training process does not require the experience of human painters or stroke tracking data. The code is available at https://github.com/hzwer/ICCV2019-LearningToPaint.
1. Introduction
The paper frames machine painting as a difficult sequential planning problem and proposes model-based DRL with differentiable rendering to decompose complex images into strokes. Experiments report strong results across simple and real-world image types.
- Approach: The agent models painting as sequential decision-making, with actions controlling each stroke’s position, shape, color, and transparency.The policy maps canvas-and-target states to deterministic stroke actions over multiple steps.
- Challenges: Machine painting requires parsing the target, tracking canvas status, and planning an ordered sequence of future strokes without ground-truth decompositions.Texture-rich images may require hundreds of strokes, making long-horizon planning especially difficult.
- Contributions: The paper’s contributions include decomposing target images into hundreds of sequential strokes and recreating paintings on canvas.The study also motivates painting agents as a basis for painting-assistant tools.
- Approach: The differentiable neural renderer supports efficient painting and multiple stroke designs, including Bézier curves, triangles, and circles.It enables end-to-end training of a model-based DRL agent and detailed feedback from rendered images.
- Results: The proposed agent handles handwritten digits, streetview house numbers, human portraits, and natural scene images.The reported evaluation spans both simple structured images and texture-rich real-world scenes.
2. Related work
Related work applies stroke-based rendering and reinforcement learning to image decomposition, but prior systems have limitations on detail, color generalization, or task complexity.
- Stroke-based rendering: Stroke-based rendering recreates images by placing discrete drawing elements such as paint strokes or stipples on a canvas.Earlier approaches often use greedy search, user interaction, grids, or manually controlled stroke parameters.
- Reinforcement-learning approaches: Recent reinforcement-learning methods target stroke decomposition, including adversarial DRL, differentiable-renderer agents, DQN doodling, and sequential sketch generation.The cited systems address different painting-related tasks and representations.
- Prior limitations: SPIRAL fails to recover human-portrait details, while StrokeNet fails to generalize to color images.These limitations motivate methods targeting richer textures and more complex real-world images.
3. Painting Agent
The painting agent treats image recreation as sequential decision-making, selecting compatible strokes while optimizing long-term rewards. Model-based DDPG uses a differentiable neural renderer to model transitions and rewards, while action bundles accelerate training by predicting multiple strokes together.
- 3.2. The Model: The agent decomposes a target image into an ordered stroke sequence, conditioning each action on the current canvas, target image, and step number.The state is st = (C_t, I, t), and each action controls stroke position, shape, color, and transparency.
- 3.2. The Model: The reward measures improvement in the canvas loss between consecutive states, and the agent maximizes discounted cumulative future rewards over the episode.The loss is defined using a discriminator score in this work.
- 3.3. Learning: Model-based DDPG uses a neural renderer to explicitly model the environment, giving the actor access to differentiable transition and reward information for efficient policy improvement.The renderer implements st+1 = trans(st, at), while the actor maximizes immediate reward plus the value of the resulting state.
- 3.3.1 Model-based DDPG: Unlike original DDPG, model-based DDPG trains a value function on the next state rather than a critic that includes the current action's reward.This changes the actor objective to combine the current reward with the value of the differentiably rendered next state.
- 3.3.2 Action Bundle: Action Bundle predicts k strokes per step and renders them in order, encouraging exploration of stroke combinations while reducing computation.The authors report that k = 5 significantly improves performance and learning speed, with the discount factor changed from γ to γ^k.
4. Stroked-based Renderer
The neural stroke renderer converts stroke parameters into rendered images and supports differentiable, efficient simulation of varied stroke designs. Its differentiability enables end-to-end model-based training for painting agents.
- Renderer design: Differentiable rendering enables end-to-end training, while GPU execution improves efficiency over many hand-crafted simulators.The renderer can generate flexible stroke styles and runs efficiently on GPUs.
- Renderer design: The renderer maps stroke parameters to rendered stroke images using a neural network trained on randomly generated computer-graphics samples.The model-based transition dynamics and reward function are differentiable.
- Stroke design: Stroke representations include position, shape, color, and transparency, with quadratic Bézier curves used to model brush-like strokes.The curve is controlled by three points, endpoint thickness and transparency, and RGB color.
- Stroke design: The stroke action is represented by control-point coordinates, endpoint thickness and transparency, and RGB color.The quadratic Bézier curve is defined over 0 ≤ t ≤ 1.
- Renderer design: Changing the final rendering layer allows the same neural-renderer architecture to implement different stroke designs.The renderer can therefore support curves or geometries such as triangles and circles.
5. Experiments
The experiments evaluate the painting agent across four image datasets, including handwritten digits, street-view digits, celebrity faces, and natural scenes. The authors report strong performance across varied real-world image types.
- Experimental datasets: Experiments use MNIST, SVHN, CelebA, and ImageNet to evaluate painting across diverse image types.The datasets span handwritten digits, street-view house numbers, celebrity faces, and natural scenes.
5.1. Datasets
The datasets range from small grayscale digit images to large, diverse collections of faces and natural scenes. ImageNet additionally supports testing generalization beyond the training images.
- Datasets: MNIST contains 70,000 28 × 28 grayscale handwritten-digit images, split into 60,000 training and 10,000 testing examples.
- Datasets: SVHN contains 600,000 color street-view digit images at 32 × 32 pixels, with 200,000 randomly sampled for the experiments.
- Datasets: CelebA provides approximately 200,000 celebrity face images, using the officially provided center-cropped images.
- Datasets: ImageNet contains 1.2 million natural-scene images across 1,000 categories, with 200,000 sampled for training and a separate test set for generalization.The task aims to paint images beyond those seen during training.
5.2. Training
Training uses resized 128×128 images, mini-batch optimization, and a single GPU, with separate training schedules for each dataset. The CelebA ablations compare painting distance, algorithm choice, stroke count, and action-bundle size.
- Training setup: All images are resized to 128×128 pixels before training, and painting 200 strokes takes about 2.1 seconds on a 2.2GHz Intel Core i7 CPU.An NVIDIA 2080Ti provides a reported 9.5× acceleration.
- Ablation evaluation: Figure 8 reports testing ℓ2-distance between CelebA paintings and target images for the ablation studies.
- Ablation evaluation: The CelebA ablations evaluate algorithm choice, stroke count, action-bundle size, and reward design through testing loss curves.These settings are represented in the supplied Figure 8 description and training passages.
- Training setup: Training uses Adam with mini-batches of 96 on a single GPU, with schedules ranging from 2 × 10^4 to 2 × 10^5 mini-batches across datasets.ImageNet and CelebA require about 40 hours, SVHN 20 hours, and MNIST two hours.
- Training setup: The critic, actor, and discriminator are updated in turn, using a replay buffer containing the latest 800 episodes.
5.3. Results
The agents reproduce simple images exactly and handle increasingly complex datasets with hundreds of strokes, while retaining overall structure despite losing some detail. Against SPIRAL on CelebA, the method achieves a substantially smaller ℓ2 distance.
- Testing ℓ2-distance curves are reported for agents trained on different datasets.
- MNIST agents using five strokes and SVHN agents using 40 strokes perfectly reproduce their target images.
- 200-stroke CelebA paintings are quite similar to target portraits but lose a certain level of detail.
- 400-stroke ImageNet paintings preserve object and background outlines and colors while losing some textures.
- 3x smaller ℓ2 distance is achieved by the method than by SPIRAL on the comparable CelebA evaluation.The comparison uses a 20-stroke agent with opaque strokes.
5.4. Ablation Studies
Ablations show that model-based DDPG, WGAN rewards, more strokes, five-stroke action bundles, and multiple stroke designs each support painting quality or flexibility. The effects include lower distances, richer textures, detail reconstruction, and varied artistic styles.
- Model-based vs. Model-free DDPG: Model-based DDPG achieves 5× smaller ℓ2 distance than DDPG with PatchQ and 20× smaller distance than original DDPG.DDPG with PatchQ still outperforms original DDPG.
- Rewards: WGAN rewards produce richer, more vivid textures and a lower testing ℓ2 loss than directly using ℓ2 rewards.The authors therefore report WGAN distance as a better metric for differences between paintings and real-world images than ℓ2 distance.
- Stroke Number and Action Bundle: Larger stroke numbers improve painting quality, and more strokes help reconstruct details in texture-rich paintings.Agents were trained with 100, 200, 400, and 1000 strokes.
- Stroke Number and Action Bundle: Five actions per Action Bundle achieve the best performance among the tested settings.The authors conjecture that bundling strokes supports longer-term plans while increasing each decision’s difficulty.
- Stroke Designs: Alternative renderers for straight strokes, circles, and triangles produce distinct artistic effects while still resembling target images.Restricting stroke transparency also yields ink-painting and oil-painting effects.
6. Conclusion
The paper trains DRL agents to decompose target images into ordered stroke sequences and combines this process with a differentiable neural renderer. The resulting agents generate vivid paintings with hundreds or thousands of strokes across multiple image types.
- The agents decompose target images into ordered stroke sequences in a way that mimics human painting processes on canvases.
- A differentiable neural renderer enables model-based DRL algorithms to improve recreated-image quality.
- Agents predict hundreds or even thousands of strokes and achieve good performance on portraits and texture-rich natural scenes.
7. Appendix
The appendix documents network architectures, training hyperparameters, and the inputs used by the discriminator, renderer, actor, and critic. The diagrams identify common fully connected and convolutional components.
- The appendix identifies FC as fully connected and Conv as convolutional layers, with intermediate ReLU activations omitted from the diagrams.
- Tables 1 and 2 list hyperparameters for DDPG and discriminator training.
- Figures 11–14 show the architectures of the discriminator, neural renderer, actor, and critic.The actor and critic take the canvas, target image, and step number; the discriminator takes two images.