PaperlayerView exact revision on arXiv

Source-linked AI summary

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction

Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, Hao Wang

arXiv:2108.03798v2cs.CV

TL;DR

Stroke-based neural painting faces challenges from unstable RL training and time-consuming iterative stroke optimization. Paint Transformer predicts stroke sets in parallel with a feed-forward Transformer and self-trains on synthesized strokes. The paper reports state-of-the-art visual quality with high efficiency, while noting its current straight-line, uniform-color strokes as a boundary for future work.

  • Problem

    Sequence-based painting methods can have long training times or unstable agents, while iterative stroke optimization is time-consuming.

    Source: §1 · p. 2

  • Method

    Paint Transformer predicts multiple stroke parameters as a set with a feed-forward Transformer and trains from randomly synthesized strokes without an off-the-shelf dataset.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Experiments report state-of-the-art visual quality while maintaining high efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method offers a better trade-off between artistic abstraction and realism than state-of-the-art methods while maintaining high efficiency.

    Source: §5 · p. 8

    The current approach uses straight-line strokes with uniform colors; future work could explore more complex strokes and cross-patch context.

    Source: §5 · p. 8

Abstract

from arXiv · show

Neural painting refers to the procedure of producing a series of strokes for a given image and non-photo-realistically recreating it using neural networks. While reinforcement learning (RL) based agents can generate a stroke sequence step by step for this task, it is not easy to train a stable RL agent. On the other hand, stroke optimization methods search for a set of stroke parameters iteratively in a large search space; such low efficiency significantly limits their prevalence and practicality. Different from previous methods, in this paper, we formulate the task as a set prediction problem and propose a novel Transformer-based framework, dubbed Paint Transformer, to predict the parameters of a stroke set with a feed forward network. This way, our model can generate a set of strokes in parallel and obtain the final painting of size 512 * 512 in near real time. More importantly, since there is no dataset available for training the Paint Transformer, we devise a self-training pipeline such that it can be trained without any off-the-shelf dataset while still achieving excellent generalization capability. Experiments demonstrate that our method achieves better painting performance than previous ones with cheaper training and inference costs. Codes and models are available.

1. Introduction

Stroke-based painting seeks to imitate how people create images, but existing sequence-generation and stroke-search approaches leave efficiency and effectiveness challenges. Paint Transformer predicts stroke sets with a feed-forward network and uses self-training without an off-the-shelf dataset; experiments report state-of-the-art visual quality with high efficiency.

1. Introduction

  • Existing sequence-based methods can be slow to train or unstable, while iterative stroke optimization is extremely time-consuming.These efficiency and effectiveness gaps motivate a feed-forward alternative.Source: §1 · p. 2
  • Paint Transformer reframes neural painting as feed-forward stroke set prediction rather than stroke-sequence generation or optimization-based stroke search.A Transformer predicts multiple stroke parameters in parallel; predictions are rendered on the canvas over K coarse-to-fine scales.Source: §1 · p. 2
  • A self-training pipeline trains the predictor from synthesized strokes without an off-the-shelf dataset.Random foreground strokes are rendered onto a synthesized background canvas to create training targets.Source: §1 · p. 2
  • Experiments report state-of-the-art visual quality while maintaining high efficiency.The paper positions the method as an efficient approach to stroke-based painting.Source: §1 · p. 2

2. Related Works

Earlier stroke-based painting methods use heuristic or iterative search, recurrent networks, or reinforcement learning, with reported cost and training challenges. Paint Transformer draws on end-to-end set prediction in object detection, while adding stroke-specific design choices.

2.1. Stroke Based Painting

  • Stroke-painting approaches include heuristics, greedy search, recurrent networks, reinforcement learning, and iterative stroke optimization.The surveyed methods target sketches, ink painting, or more photo-realistic painting.Source: §2.1 · p. 2
  • Iterative stroke optimization can produce satisfactory artistic effects, but its computational cost limits applicability.This motivates seeking a better trade-off between painting performance and efficiency through feed-forward stroke set prediction.Source: §2.1 · p. 3

2.2. Object Detection

  • DETR performs end-to-end set prediction without post-processing, inspiring Paint Transformer’s stroke prediction framework.Unlike DETR, Paint Transformer adds binary neurons to decide whether strokes should be kept and takes both the current canvas and target image as input.Source: §2.2 · p. 3

3. Methods

Paint Transformer predicts parallel stroke sets from the target and current canvas, then renders them progressively. It trains the predictor on synthesized strokes and combines image-level and stroke-level losses.

3.1. Overall Framework

  • Random foreground and background stroke sets create training canvases and targets, allowing predictor training without an off-the-shelf dataset.The predictor is optimized at both stroke and pixel levels; the renderer is differentiable and has no trainable parameters.Source: §3.1 · p. 3

3.2. Stroke Definition and Renderer

  • A geometric, differentiable renderer transforms straight-line stroke shape and color parameters and composites rendered strokes onto the input canvas.A stroke uses center coordinates, height, width, rotation, and RGB parameters.Source: §3.2 · pp. 3–4

3.3. Stroke Predictor

  • The predictor encodes target and canvas patches with separate convolutional networks, then uses Transformer queries and fully connected branches to predict stroke parameters and confidence.A binary confidence decision determines whether a predicted stroke is plotted; sigmoid gradients are used in backpropagation.Source: §3.3 · p. 4

3.4. Loss Function

  • Training combines pixel reconstruction with stroke-level matching using parameter L1, Wasserstein, and confidence losses, with Hungarian assignment matching predictions to targets.The Wasserstein distance models rotated rectangular strokes as 2-D Gaussian distributions; stroke order is handled at image level rather than in the stroke-level loss.Source: §3.4 · pp. 4–5

3.5. Inference

  • Inference proceeds coarse-to-fine across K scales, processing non-overlapping patches independently through the predictor and renderer before combining them.The scale count is chosen so patch size P times 2^K covers the larger image dimension; K = 4 is the default setting in the comparison figure.Source: §3.5 · p. 5, Figure 5 · p. 6

4. Experiments

The experiments compare Paint Transformer with prior methods, examine its training losses, and test its implementation and extensions. Results report qualitative and efficiency advantages, distinct failure modes when losses are removed, and transfer to other brushes and stylized content.

4.1. Implementing Details

  • Training used 32 × 32 input patches with eight target strokes per patch; the model trained for 30,000 iterations in fewer than four hours on one Nvidia RTX 2080 Ti.For inference, results use 512 × 512 resolution with K = 4 unless specified otherwise.Source: §4.1 · p. 7

4.2. Comparison with State-of-the-Art Methods

  • Paint Transformer produces more appealing results than Optim and more vivid, clear-brush results than RL [10]; it runs significantly faster than Optim and slightly faster than RL [10].The efficiency comparison covers inference and training; the authors also report training the predictor in a few hours.Source: Table 2 · p. 7, §4.2 · pp. 7–8

4.3. Ablation study

  • Removing any optimization term reduces performance; omitting pixel, parameter L1, Wasserstein, or confidence loss respectively harms stroke placement and color, shape, scale handling, or stroke selection.The reported failure patterns include dirty textures, repeated stroke patterns, diminished large vivid strokes, and excessive small strokes covering the image.Source: §4.3 · p. 8

4.4. Extension of Paint Transformer

  • A predictor trained with an oil-painting brush also produces appealing results with rectangle and circle brushes, while style-transfer methods yield paintings with diverse colors and textures.The stylized paintings use LapStyle or AdaAttN on stylized content images.Source: §4.4 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

5. Conclusion and Future Works

Paint Transformer reframes neural painting as feed-forward stroke-set prediction and trains without a manually collected dataset. The authors report a better trade-off between artistic abstraction and realism than state-of-the-art methods while maintaining high efficiency, and identify richer strokes and cross-patch context as future directions.

5. Conclusion and Future Works

  • The framework predicts multiple stroke parameters with a feed-forward Transformer and uses self-training without a manually collected dataset.Source: §5 · p. 8
  • The authors report a better trade-off between artistic abstraction and realism than state-of-the-art methods while maintaining high efficiency.Source: §5 · p. 8
  • Future work includes strokes with varied shapes or color patterns and cross-patch context for long, narrow areas.The current strokes are straight lines with uniform colors.Source: §5 · p. 8

A. More Training Details

Training synthesizes background canvases and foreground strokes to teach the predictor to add strokes at progressively finer scales. The authors connect this procedure to successful generalization from synthesized data to real-world images.

A. More Training Details

  • Training begins with 8 background strokes on a 64 × 64 canvas, then samples 8 foreground strokes for each of four 32 × 32 blocks.Source: A. More Training Details · p. 11
  • The predictor learns to add foreground strokes to a background canvas, encouraging coarse-to-fine painting.Source: A. More Training Details · p. 11
  • The authors state that this training and inference setup lets the predictor generalize from randomly synthesized data to real-world images.Source: A. More Training Details · p. 11

B. More Inference Results

The appendix presents additional inference results, including outputs at 1024 × 1024 resolution. It also points readers to an animated painting process in the attachment or code page.

B. More Inference Results

Loading 2108.03798v2…