PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention-aware Multi-stroke Style Transfer

Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, Jun Wang

arXiv:1901.05127v1cs.CV

TL;DR

Existing neural style-transfer methods do not coordinate spatial attention between content and stylized images or render varied detail through different brush strokes. The paper proposes AAMS, combining self-attention, multi-scale style swap, and attention-guided fusion. Experiments report favorable stylization and visual consistency, with comparable multi-stroke results and controlled runtime.

  • Problem

    Existing methods lack coordinated spatial attention between content and stylized images and cannot render diverse detail levels through different brush strokes.

  • Method

    AAMS uses a self-attention autoencoder, multi-scale style swap, and attention-guided fusion to integrate distinct stroke patterns across output regions.

  • Results

    The method generates favorable stylized results with visual consistency and comparable multi-stroke performance against state-of-the-art methods.

  • Takeaways & Limitations

    AAMS preserves attention consistency while providing multi-stroke fusion control and automatic spatial stroke-size control in one output.

Abstract

from arXiv · show

Neural style transfer has drawn considerable attention from both academic and industrial field. Although visual effect and efficiency have been significantly improved, existing methods are unable to coordinate spatial distribution of visual attention between the content image and stylized image, or render diverse level of detail via different brush strokes. In this paper, we tackle these limitations by developing an attention-aware multi-stroke style transfer model. We first propose to assemble self-attention mechanism into a style-agnostic reconstruction autoencoder framework, from which the attention map of a content image can be derived. By performing multi-scale style swap on content features and style features, we produce multiple feature maps reflecting different stroke patterns. A flexible fusion strategy is further presented to incorporate the salient characteristics from the attention map, which allows integrating multiple stroke patterns into different spatial regions of the output image harmoniously. We demonstrate the effectiveness of our method, as well as generate comparable stylized images with multiple stroke patterns against the state-of-the-art methods.

1. Introduction

The paper identifies failures in coordinating attention across content and stylized images and proposes attention-aware multi-stroke transfer to address them. Its design combines self-attention, multi-scale style swapping, and attention-guided fusion.

  • Background: Style transfer enables image recomposition in the style of other images and supports art creation and image editing.Neural style transfer separates and recombines content and style representations of arbitrary images.
  • Motivation: Existing style-transfer methods can introduce insufficient stylization, unexpected patterns, or distorted attention regions when rendering diverse content regions indiscriminately.These issues are associated with Style-Swap, AdaIN, StrokePyramid, WCT, and Avatar-Net.
  • Proposed approach: AAMS introduces self-attention into an autoencoder to capture critical characteristics and long-range relations within content images.Reconstruction training produces an attention map that highlights salient content parts and supports attention consistency across long-range features.
  • Proposed approach: Multi-scale style swap produces multiple feature maps that reflect different stroke patterns and avoids the limitation of a fixed receptive field.The approach links different receptive fields with distinct stroke sizes.
  • Proposed approach: The fusion strategy combines attention information with multiple stroke patterns so distinct strokes can be integrated into different output regions harmoniously.This supports attention consistency between the content and stylized images.

2. Related Work

Related work develops arbitrary-style transfer through feature exchange or statistical alignment, but these approaches tend to produce uniform stroke patterns without attention-aware guarantees. The paper positions self-attention as a way to capture dependencies across image regions.

  • Neural Style Transfer: Arbitrary-Style-Per-Model methods transfer arbitrary styles through one model by combining image reconstruction with feature-statistics fusion.Representative methods use patch swapping, adaptive instance normalization, whitening and coloring transforms, or patch-based style decoration.
  • Neural Style Transfer: Existing methods either exchange closest feature patches locally or transfer feature statistics globally, which tends to produce uniform stroke patterns without attention-aware guarantees.The limitation concerns spatially varying visual attention during stylization.
  • Self-Attention: Self-attention models can capture multi-level dependencies across image regions, and this paper adapts self-attention to introduce a residual feature map for salient content characteristics.The adaptation is used within the paper’s style-transfer framework.

3. Proposed Approach

The proposed approach couples self-attention, multi-scale style swap, and attention-guided multi-stroke fusion in a bottleneck framework for one-pass arbitrary style transfer. Self-attention identifies salient regions and long-range dependencies, while clustered attention weights combine stroke features spatially.

  • Overall Framework: Three bottleneck modules—self-attention, multi-scale style swap, and multi-stroke fusion—cooperate to generate stylized images in one feed-forward pass.The framework also uses skip connections to adapt multiple synthetic-feature levels with style features.
  • Self-Attention Autoencoder: The self-attention autoencoder learns dependencies between distant image regions and adds a residual feature that highlights salient content regions during reconstruction.The encoder maps the input into H × W × C features, while the residual is appended before decoding.
  • Self-Attention Autoencoder: A sparse loss encourages the self-attention map to focus on small regions, alongside perceptual, pixel reconstruction, and total-variation losses.The perceptual and pixel reconstruction losses use normalized Euclidean distance, while total variation encourages spatial smoothness.
  • Multi-scale Style Swap: Multi-scale style swap varies the activation scale of whitened style features while keeping patch size fixed, producing swapped features with different stroke sizes.The scale coefficient β_k controls different stroke sizes, and the parallel style-swap process operates between whitened content and style features.
  • Multi-stroke Fusion: Absolute-value and Gaussian filtering normalize the attention map to [0, 1], enlarging salient-region influence while preserving correlations among distant regions.The Gaussian-kernel variance controls the proportion of salient regions represented in the content image.
  • Multi-stroke Fusion: The fusion module adds the whitened content feature as a fine-grained stroke, yielding K+1 features whose spatial blending is guided by clustered attention intensities.K-means forms K+1 attention clusters, and distances from cluster centers determine each stroke size’s contribution through softmax weighting.

4. Experiments

Experiments evaluate AAMS against prior arbitrary-style transfer methods, isolate its attention and multi-stroke components, and assess attention consistency, controllability, and runtime.

  • Qualitative Evaluation: AAMS produces visually plausible stylized results while preserving salient-region focus, including eyes, candles, houses, and farmers.The attention map supports seamless synthesis among multiple stroke sizes and maintains spatial visual-attention consistency.
  • Ablation Studies: Self-attention improves visual effects by emphasizing salient regions with fine-grained strokes and integrating local style patterns without sacrificing holistic perception.This comparison uses AAMS with self-attention against AAMS(-SA), which removes self-attention and uses average fusion.
  • Ablation Studies: Removing multi-scale style swap restricts transfer to a single stroke pattern, whereas AAMS integrates multiple stroke patterns into one stylized image.Stroke size is controlled through the scaling coefficient β; larger sizes produce coarser patterns.
  • User Studies: User studies collect 600 votes from 30 subjects, and AAMS receives the most votes for both stylization and attention consistency.The evaluation compares AAMS with AdaIN, WCT, and Avatar-Net using 10 content images and 15 style images.
  • Quantitative Evaluation: Attention consistency is dramatically improved across all saliency metrics, while three-stroke transfers take 0.80 seconds at 256 × 256 and 0.94 seconds at 512 × 512.Runtime is averaged over 400 transfers on a 12G Tesla M40 GPU, with controllable increases as stroke numbers grow.
  • Runtime Control: The method supports multi-stroke fusion control through the number of stroke sizes and level-of-detail adjustment via the smoothing factor γ.Its attention map also enables automatic spatial stroke-size control, assigning fine, middle, and coarse patterns to regions of differing saliency.

5. Conclusion

The paper concludes that its attention-aware multi-stroke model preserves attention consistency while providing controllable and automatic stroke-size selection for arbitrary-style transfer.

  • 5. Conclusion: The model combines self-attention, multi-scale style swap, and attention-guided fusion to integrate distinct stroke patterns across image regions.Experiments report favorable stylization effects and visual consistency with the content image.

6.1. Implementation Details

The self-attention autoencoder augments an encoder-decoder with a bottleneck self-attention module, using VGG-19-based encoding and symmetric decoding.

  • 6.1. Implementation Details: The self-attention autoencoder is formed by inserting a self-attention module into the bottleneck of an encoder-decoder framework.The architecture is designed to capture critical image characteristics and long-range region relations.
  • 6.1. Implementation Details: The encoder uses early VGG-19 layers, while the decoder mirrors the encoder with average pooling and nearest up-sampling replacing max pooling and pooling layers.Reflection padding is used in convolutional layers to avoid border artifacts.
  • 6.1. Implementation Details: Skip connections enhance style using adaptive instance normalization between corresponding encoder and decoder levels.Style features from multiple VGG-19 levels are combined with reconstruction features.
  • 6.1. Implementation Details: The attention module forms a residual feature by multiplying the feature map f_x with its self-attention map A_x to capture saliency characteristics.This residual construction differs from adding the self-attention output directly back to the input feature map.

6.2. Experiments and Results

The experiments examine skip connections and present additional stylization results using the proposed method. The supplementary results use three stroke scenarios.

  • Skip connections improve style fidelity by matching features across multiple levels, which captures low-level characteristics such as color saturation.
  • Additional stylization results for the proposed method are visualized in Figures 15 and 16.
  • The supplementary style-transfer results use three stroke scenarios under the default setting.

6.3. Multi-stroke Fusion Control

The method controls multi-stroke stylization by generating feature maps at multiple scales and fusing them according to content attention. Smoothing controls how distinctly each stroke pattern occupies its corresponding attention region.

  • Multi-scale stroke control: Unlike weighted interpolation, changing the scale coefficient generates continuous and discriminative stylized patterns.
  • Fusion procedure: The fusion procedure combines K+1 stroke feature maps with the content attention map by assigning finer strokes to higher attention values.The integrated feature map is computed as a weighted sum of the stroke feature maps.
  • Fusion control: Increasing the smoothing factor γ makes each stroke contribute more within its corresponding attention area, producing more discriminative stroke patterns.
Loading 1901.05127v1…