PaperlayerView exact revision on arXiv

Source-linked AI summary

Stylized Neural Painting

Zhengxia Zou, Tianyang Shi, Shuang Qiu, Yi Yuan, Zhenwei Shi

arXiv:2011.08114v1cs.CV

TL;DR

Existing image translation methods formulate artwork generation pixel by pixel, unlike the sequential, brush-based process of painting. This paper searches for vector stroke parameters using a differentiable neural renderer and optimal-transport perspective, and reports realistic paintings with fidelity in global appearance and local textures.

  • Problem

    Prior image-to-image translation methods formulate translation as pixel-wise prediction, unlike painting's sequential brush-based creation process.

    Source: §1 · p. 2

  • Method

    The method sequentially searches vector stroke parameters against a reference image using a differentiable neural renderer and an optimal-transport perspective on image similarity.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Tests on varied real-world images report vivid paintings with realism and artistic sense in both global visual appearance and local texture fidelity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method produces vector-format paintings with controllable styles and can be jointly optimized with neural style transfer.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

This paper proposes an image-to-painting translation method that generates vivid and realistic painting artworks with controllable styles. Different from previous image-to-image translation methods that formulate the translation as pixel-wise prediction, we deal with such an artistic creation process in a vectorized environment and produce a sequence of physically meaningful stroke parameters that can be further used for rendering. Since a typical vector render is not differentiable, we design a novel neural renderer which imitates the behavior of the vector renderer and then frame the stroke prediction as a parameter searching process that maximizes the similarity between the input and the rendering output. We explored the zero-gradient problem on parameter searching and propose to solve this problem from an optimal transportation perspective. We also show that previous neural renderers have a parameter coupling problem and we re-design the rendering network with a rasterization network and a shading network that better handles the disentanglement of shape and color. Experiments show that the paintings generated by our method have a high degree of fidelity in both global appearance and local textures. Our method can be also jointly optimized with neural style transfer that further transfers visual style from other images. Our code and animated results are available at \url{https://jiupinjia.github.io/neuralpainter/}.

1. Introduction

The paper treats painting as a sequential, brush-based process rather than pixel-wise image translation. Stylized Neural Painter searches for vector stroke parameters, addresses optimization and rendering challenges, and generates paintings with realistic global appearance and local textures.

1. Introduction

  • Unlike pixel-wise translation, painting proceeds sequentially with brushes, from abstract to concrete and from macro to detail.The passage notes that learning professional painting skills requires practice and domain expertise, and realistic works can take skilled painters hours or days.Source: §1 · p. 2
  • Stylized Neural Painter generates physically meaningful vector strokes sequentially, supports controllable styles and arbitrary output resolutions, and can be jointly optimized with neural style transfer.Styles include oil painting, watercolor ink, marker pen, and tape art.Source: §1 · p. 2
  • Self-supervised stroke-parameter search uses optimal transportation to address zero gradients, while a rasterization-and-shading renderer better disentangles shape and color.The transportation view treats moving a stroke between locations as a process whose effort should be minimized.Source: §1 · p. 2
  • Tests across portraits, animals, scenery, objects, art photography, and cartoons report vivid paintings with realism and artistic sense in global appearance and local textures.Source: §1 · p. 2

2. Related Work

Prior image translation and style-transfer methods create artwork pixel by pixel, while stroke-based painting methods generate sequences of strokes. Related work also covers differentiable rendering and neural rendering, which connects graphics rendering with neural networks.

2. Related Work

  • GAN-based translation and neural style transfer have been applied to artwork creation, but these methods generate paintings pixel by pixel.Source: §2 · p. 2
  • Differentiable rendering permits derivatives from rendered output to input parameters despite rasterization, and neural rendering bridges graphic rendering and deep networks.Source: §2 · pp. 2–3
  • Figure 2 depicts stroke-by-stroke rendering from an empty canvas with soft blending, and gradient descent seeks stroke parameters that minimize loss.Black arrows indicate forward propagation and red arrows indicate gradient back-propagation.Source: Figure 2 · p. 3
  • Stroke-based painting approaches include greedy search, recurrent networks, reinforcement learning, and adversarial training; Learning to Paint [36] uses RL, whereas this method uses parameter search and adds stroke-style transfer, optimal transport, and a redesigned renderer.Source: §2 · p. 3

3. Methodology

Stylized Neural Painter builds a painting by sequentially rendering and blending vectorized strokes, then searching their parameters to match a reference image. Its differentiable pipeline combines a dual-pathway renderer, optimal-transport similarity, and optional neural style transfer.

3.1. Overview

  • The pipeline draws strokes sequentially from an empty canvas, softly blends each foreground and alpha matte, then searches stroke parameters to make the final canvas resemble a reference.The stroke sequence is recursively mapped to the rendered canvas, and the parameters are optimized in a self-supervised manner.Source: §3.1 · p. 3

3.2. Disentangle neural rendering

  • The dual-pathway renderer separates rasterized stroke silhouettes from foreground color, addressing shape–color coupling in complex settings such as color transitions and stroke textures.The shading network uses color and shape parameters, while the rasterization network ignores color and generates the stroke silhouette; the silhouette masks the color map and is rescaled by input transparency to form the alpha matte.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Pixel similarity and zero-gradient problem

  • When a rendered stroke and target do not overlap, pixel loss can remain constant as the stroke moves, yielding zero gradients that cannot guide its update.The paper illustrates this with a square stroke moving toward a target location.Source: §3.3 · p. 4, Figure 4 · p. 4
  • The transportation loss makes stroke movement responsive to physical distance, and the illustrated stroke converges to its target where pixel loss fails.The pixel-loss failure is attributed to zero gradients in stroke position and scale.Source: §3.3 · pp. 4–5, Figure 5 · p. 4

3.4. Optimal transport for stroke searching

  • The method uses a differentiable, entropically regularized Sinkhorn distance to measure the minimum transportation effort between rendered and reference pixel distributions.It combines this optimal-transport loss with pixel ℓ1 loss for stroke-parameter searching.Source: §3.4 · p. 5

3.5. Joint optimization with neural style transfer

  • The similarity objective can include a style loss, enabling joint optimization of stroke parameters and global visual style.Following Gatys et al., the style loss compares Gram matrices of VGG-19 features from the 2nd, 4th, and 7th convolutional layers.Source: §3.5 · p. 5

3.6. Implementation Details

  • Progressive rendering searches first on a 128×128 canvas, then on overlapping blocks while incrementally adding strokes and updating the active set together.Each block receives 20 × N gradient-descent steps, where N is the number of strokes in that block.Source: §3.6 · p. 6

4. Experimental Analysis

The experiments show that the method produces stylized paintings and style-transfer results, compares favorably with prior stroke-based methods, and benefits from its transportation loss and dual-pathway renderer.

4.1. Stylized painting generation

  • Paintings capture global appearance early and progress from macro to detail; style transfer can change color and texture while preserving content.Figure 9 distinguishes transfers of both color and texture from color-only transfers.Source: §4.1 · pp. 6–7, Figure 9 · p. 7

4.2. Comparison with other methods

  • Compared with Learning-to-Paint [11] and SPIRAL [5], the method produces more vivid paintings with clearer brush textures, while those methods tend to look blurred.The comparison uses 400 strokes for this method and Learning-to-Paint, and 20 for SPIRAL.Source: §4.2 · p. 7, Figure 10 · p. 7

4.3. Controlled Experiments

  • Transportation loss helps recover image details, especially when strokes start outside the target region; pixel ℓ1 loss alone can yield poor stroke-parameter convergence.The visual experiment compares pixel ℓ1 loss with pixel ℓ1 plus transportation loss on a flower and MNIST digits.Source: §4.3 · p. 7, Figure 12 · p. 8
  • Across brush settings, the neural renderer exceeds alternatives by 2.5–5.0 PSNR and converges faster, while producing better color fidelity and high-frequency details.The comparison covers watercolor and oil-paint rendering; the authors report similar performance for marker-pen and tape-art renderers.Source: §4.3 · p. 8, Figure 13 · p. 8, Figure 14 · p. 8
  • Removing either the rasterization network or the shading network causes a noticeable drop in mean PSNR for the foreground and alpha matte.The ablation compares each component separately with the full renderer.Source: §4.3 · p. 8, Table 1 · p. 8

5. Conclusion

The conclusion presents painting as differentiable stroke rendering and parameter search, with optimal transport for similarity and a dual-pathway renderer for color and shape. It reports realistic vector artworks with controllable styles and says controlled experiments support the design.

5. Conclusion

  • The method produces realistic vector-format artworks with controllable styles by optimizing stroke parameters against a sequentially rendered canvas.The conclusion frames this as a stroke parameter-searching approach using differentiable rendering.Source: §5 · p. 8
  • Optimal transportation provides the image-similarity perspective, while a dual-pathway neural renderer addresses color-and-shape disentanglement.Source: §5 · p. 8
  • Controlled experiments are reported as supporting the proposed design.Source: §5 · p. 8

6. Appendix

The appendix details the renderer configurations and brush-specific stroke parameters, then illustrates vector-format results rendered at high resolution.

6.1. Detailed configurations of our neural renderer

  • The appendix gives separate shading- and rasterization-network configurations and explains the notation for layer dimensions and output sizes.A layer is described by filter count, filter size, and stride; outputs are formatted as height × width × channel.Source: §6.1 · p. 11, Table 2 · p. 11, Table 3 · p. 11

6.2. Stroke parameterization

  • Stroke parameterization covers oil-painting, marker-pen, watercolor-ink, and color-tape brushes, with brush-specific geometry, color, and transparency choices.Marker and watercolor trajectories use quadratic Bezier curves; oil paint uses fixed opacity and a texture map held constant during parameter search.Source: §6.2 · p. 11

6.3. High resolution results

Loading 2011.08114v1…