Source-linked AI summary
Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
Yongcheng Jing, Yang Liu, Yezhou Yang, Zunlei Feng, Yizhou Yu, Dacheng Tao, Mingli Song
TL;DR
Fast Style Transfer makes stylization faster, but flexible stroke-size control remains a challenge. The paper introduces a single-model network and training and runtime strategies for controlling stroke size. It reports continuous and spatial stroke control and identifies automatic spatial control as an open issue.
Problem
Fast Style Transfer lacks flexible stroke-size control: prior solutions cannot provide continuous changes or distinct sizes across image regions without a quality or efficiency trade-off.
Source: §1 · p. 2Method
A StrokePyramid network uses branches with different receptive fields, shared encoding and stroke decoding, plus training and runtime strategies for stroke-size control.
Source: §1 · p. 2, §4.2 · p. 6, §4.5 · p. 8Results
The authors report that one model achieves continuous and spatial stroke-size control without trading off quality and efficiency.
Source: §1 · p. 3, §6 · p. 14Takeaways & Limitations
The approach offers practitioners a tool to inject their artistic preferences into style-transfer results for production software and entertainment.
Source: §6 · p. 14Automatic spatial stroke-size control remains future work, and the relations among representations of different scales of the same style image remain unclear.
Source: §6 · pp. 14–15
Abstract
from arXiv · showhide
The Fast Style Transfer methods have been recently proposed to transfer a photograph to an artistic style in real-time. This task involves controlling the stroke size in the stylized results, which remains an open challenge. In this paper, we present a stroke controllable style transfer network that can achieve continuous and spatial stroke size control. By analyzing the factors that influence the stroke size, we propose to explicitly account for the receptive field and the style image scales. We propose a StrokePyramid module to endow the network with adaptive receptive fields, and two training strategies to achieve faster convergence and augment new stroke sizes upon a trained model respectively. By combining the proposed runtime control strategies, our network can achieve continuous changes in stroke sizes and produce distinct stroke sizes in different spatial regions within the same output image.
1 Introduction
Fast Style Transfer speeds image stylization, but controlling stroke size remains difficult. The paper proposes a single-model approach for continuous and spatial stroke control.
1 Introduction
- Fast Style Transfer was proposed to address the slow, iterative optimization required by Gatys et al.’s style transfer algorithm.Source: §1 · p. 1
- Figure 1 shows stylized results with different stroke sizes, all produced in real time by one model.Source: Figure 1 · p. 2
- Fast Style Transfer approaches include Per-Style-Per-Model, Multiple-Style-Per-Model, and Arbitrary-Style-Per-Model methods.They use one generator per style, one generator for multiple styles, or one model for arbitrary styles, respectively.Source: §1 · p. 2
- Multiple models for different stroke sizes are time- and space-consuming, while input resizing can hurt stylization quality.Neither solution achieves continuous control or distinct stroke sizes in different image regions without a quality or efficiency trade-off.Source: §1 · p. 2
- The proposed network incorporates multiple stroke sizes into one model and supports continuous and spatial control.It combines StrokePyramid, two training strategies, and two runtime control strategies.Source: §1 · pp. 2–3
2 Related Work
Related work covers control of perceptual factors in style transfer and methods for regulating neural-network receptive fields. The paper distinguishes its stroke control from prior work on brush strokes and receptive fields.
2 Related Work
- Few prior efforts address stroke-size control in Fast Style Transfer; studies color and spatial control, while adds semantic content for meaningful spatial control.Wang et al. target large brush strokes in high-resolution images, but one pre-trained model still produces one stroke size per input image.Source: §2 · p. 3
- Receptive-field regulation includes dilated convolution, deformable convolution, and a learning-based feature-map method.Dilated convolution is also called atrous convolution in.Source: §2 · p. 3
- Figure 2 compares results for learning the same large strokes with large versus small receptive fields.Source: Figure 2 · p. 4
3 Pre-analysis
The pre-analysis identifies style-image scale and the generative network’s receptive field as factors for stroke-size control. It explains how each can affect strokes and why the loss-network receptive field is not addressed here.
3 Pre-analysis
- Stroke size is defined as the average scale of an image’s composed stroke textons, its fundamental geometric micro-structures.Source: §3 · p. 4
- Style-image scale and receptive fields in both the loss and generative networks can influence stroke size.The analysis generally considers style-image scale and generative-network receptive field, but not the loss-network receptive field because style images are usually not high-resolution.Source: §3 · pp. 4–5
- Scale-sensitive style statistics let networks trained on different style-image scales produce different stroke sizes for the same content image.Scaling a style image very high, such as to 3000 × 3000 pixels, is infeasible because the fixed VGG loss-network receptive field cannot distinguish sufficiently large from larger stroke textons in a small region.Source: §3 · p. 4
- A larger generative-network receptive field produces larger strokes for some styles, and larger strokes require receptive fields that capture global stroke configuration.Smaller strokes require learning local features; when the receptive field is smaller than a stroke texton, the network learns only part of it in each region.Source: §3 · pp. 4–5
- Figure 3 indicates that equal-sized output regions have almost the same stroke sizes for small and large input images.Source: Figure 3 · p. 5
4 Proposed Approach
The approach targets a single fast style-transfer generator that can produce multiple stroke sizes for the same content image. It combines adaptive receptive fields, scale-aware style losses, progressive and incremental training, and runtime controls.
4.2 Network Architecture
- StrokePyramid uses progressively larger receptive fields across stroke branches; a shared pre-encoder extracts content and style features, and a stroke decoder produces the stylized output.A test-time signal selects the desired stroke size, and fractional gating weights support continuous control.Source: §4.2 · pp. 6–7, Figure 4 · p. 6
4.3 Loss Function
- The objective combines VGG-feature semantic preservation, scale-sensitive Gram-style matching via resized style images, and total-variation smoothness regularization.Different style-image scales provide representations for different stroke sizes.Source: §4.3 · pp. 7–8
4.4 Training Strategies
- Progressive training learns stroke branches in sequence, while incremental training adds a new branch layer and updates it with other network components fixed.The progressive strategy preserves previously learned sizes; the incremental strategy is intended to reach convergence quickly.Source: §4.4 · p. 8
4.5 Runtime Control Strategies
- Runtime feature-space interpolation between trained stroke sizes produces arbitrary intermediate sizes without training on many style-image scales.The method interpolates branch feature maps by adjusting gating functions before decoding.Source: §4.5 · pp. 8–9
- For spatial control, masked content is processed by corresponding stroke branches and the stylized results are combined, allowing different regions to use different stroke sizes.Masks can be manually labeled or generated using a pretrained semantic segmentation network.Source: §4.5 · p. 9
- Against proposed alternatives, feature-space interpolation gives perceptually superior results, while StrokePyramid outputs are more consistent in stroke orientation and configuration.The comparison also reports that the method outperforms ASPM in quality and stylization speed.Source: §4.5 · p. 9, Figure 5 · p. 10
5 Experiment
The experiments compare stroke-control results, training strategies, loss values, and runtime and model size. They report continuous and spatial control with one model, alongside faster training and competitive loss minimization.
5.1 Implementation Details
- Training uses MS-COCO images cropped and resized to 512 × 512 pixels, with three initial stroke sizes at style-image scales 256, 512, and 768.The setup uses Adam and a pre-trained VGG-19 loss network.Source: §5.1 · p. 10
5.2 Qualitative Evaluation
- Stroke orientations and configurations are more consistent across stroke sizes than in, and the model is 0.99 MB versus 32.2 MB for.The authors also describe the visual quality as comparable to while retaining the flexibility of their second possible solution.Source: §5.2 · p. 10, Figure 6 · p. 11
- The model produces continuous and spatial stroke-size control, with finer strokes and details than and mixed stroke sizes within one output.The spatial-control results are reported as enhancing contrast and making AI-created art closer to human-created art.Source: §5.2 · p. 11, Figure 7 · p. 12, Figure 9 · p. 13
5.3 Quantitative Evaluation
- Incrementally learning a stroke significantly speeds training versus training from scratch while achieving comparable stylization quality.Progressively learning multiple stroke sizes also converges relatively faster than learning each size individually; the authors attribute this to knowledge sharing between branches.Source: §5.3 · p. 12, Figure 10 · p. 13
- Average style loss is similar to, while average content loss is slightly lower, indicating comparable or slightly better loss minimization.Values average over 100 content images and 5 style images, with each style's losses averaged across three stroke sizes.Source: §5.3 · p. 13, Figure 11 · p. 14
- Generating a 1024×1024 image takes an average of 0.09 seconds on an NVIDIA Quadro M6000, and the model occupies 0.99 MB on disk.Training one model with three stroke sizes takes about 2 hours on the same GPU.Source: §5.3 · p. 14
6 Discussion and Conclusion
The paper presents stroke-size control as a tool for style-transfer practitioners and a first step toward adaptive receptive fields beyond style transfer. It identifies automatic spatial control and cross-scale style representations as areas needing further work.
6 Discussion and Conclusion
- The method offers practitioners a way to inject artistic preferences into style-transfer results for production software and entertainment.The authors also say the work takes a step toward adaptive receptive fields, whose significance is primarily validated in style transfer.Source: §6 · p. 14
- Automatic spatial stroke-size control remains an open issue; semantic segmentation is proposed to determine sizes for different regions.The authors say this could make spatial control more efficient and user-friendly.Source: §6 · p. 14
- The relation between style representations at different scales remains unclear, while transforming between them is identified as key to more flexible stroke-size control.Source: §6 · pp. 14–15