Source-linked AI summary
Arbitrary Style Transfer with Style-Attentional Networks
Dae Young Park, Kwang Hee Lee
TL;DR
Arbitrary style transfer struggles to balance content structure with global and local style patterns while remaining efficient and flexible. The paper introduces SANets with a learned soft-attention mechanism and identity loss to rearrange style features according to content semantics. Experiments report real-time, high-quality stylized images preferred over other state-of-the-art arbitrary style-transfer methods.
Problem
Existing arbitrary style-transfer methods struggle to balance content structure, global and local style patterns, and efficiency.
Method
The paper uses SANets and decoders that learn semantic correlations by spatially rearranging style features according to content features, with identity loss preserving content structure.
Results
Experimental results demonstrate that the method synthesizes images preferred over other state-of-the-art arbitrary style-transfer algorithms.
Takeaways & Limitations
The approach synthesizes high-quality stylized images in real time while balancing global and local style patterns and preserving content structure.
Takeaways & Limitations
The cited prior Gatys method is flexible for arbitrary content and style images but prohibitively slow because of iterative optimization.
Abstract
from arXiv · showhide
Arbitrary style transfer aims to synthesize a content image with the style of an image to create a third image that has never been seen before. Recent arbitrary style transfer algorithms find it challenging to balance the content structure and the style patterns. Moreover, simultaneously maintaining the global and local style patterns is difficult due to the patch-based mechanism. In this paper, we introduce a novel style-attentional network (SANet) that efficiently and flexibly integrates the local style patterns according to the semantic spatial distribution of the content image. A new identity loss function and multi-level feature embeddings enable our SANet and decoder to preserve the content structure as much as possible while enriching the style patterns. Experimental results demonstrate that our algorithm synthesizes stylized images in real-time that are higher in quality than those produced by the state-of-the-art algorithms.
1. Introduction
Arbitrary style transfer must combine global and local style patterns while preserving content structure, but existing methods face trade-offs among flexibility, quality, and efficiency. The paper proposes SANets with identity loss to synthesize high-quality stylized images in real time while preserving content structure.
- Arbitrary style transfer synthesizes global and local style patterns from a style image over a content image while maintaining its original structure.
- Existing approaches trade off flexibility, visual quality, and efficiency: iterative optimization is slow, while feedforward methods may support fixed styles or insufficient quality.
- SANet learns semantic correlations between content and style features by spatially rearranging style features according to content features.
- The proposed SANet uses a learned similarity kernel and soft attention to represent each content position as a weighted sum of similar style features.
- Identity loss trains the network to restore the original image from the same image pair, helping preserve content structure without losing style richness.
- The method flexibly matches semantically nearest style features onto content features and combines conventional style reconstruction loss with identity loss.
- 18–24 frames per second (fps) at 512 pixels is reported for synthesizing high-quality stylized images while balancing global and local style patterns and preserving content structure.
2. Related Work
Prior arbitrary style-transfer methods adjust feature statistics or use patch-based decoration, but face computational costs or content–style trade-offs. The paper addresses these issues with SANets and identity loss, while relating SANet to self-attention.
- Arbitrary style transfer seeks to achieve generalization, quality, and efficiency simultaneously, but existing methods present trade-offs among them.
- AdaIN matches content and style feature means and variances, whereas WCT uses whitening and coloring with covariance but becomes computationally expensive for high-dimensional features.
- The paper targets the content–style trade-off by using SANets and identity loss to represent global and local style patterns while maintaining content structure.
- SANet modifies self-attention by learning mappings between content and style features through weighted responses in an embedding space.
3. Method
The method combines an encoder–decoder with two SANets that embed style features according to content–style correspondences. Multi-level features and identity-aware training aim to preserve content structure while representing global and local style patterns.
- Network Architecture: The feedforward network uses a pre-trained VGG-19 encoder, two SANets, and a symmetric decoder to synthesize stylized images.Content and style images are encoded before SANet-based feature embedding and decoder reconstruction.
- SANet for Style Feature Embedding: SANet maps correspondences between content and style feature maps, producing style embeddings through learned relationships between their positions.Normalized feature maps are transformed into learned spaces, and the resulting response combines style features according to content–style affinity.
- Training Objectives: The style loss uses AdaIN-style feature statistics across five encoder layers, which the authors report as more satisfactory than Gram matrix loss.The evaluated layers are Relu 1_1 through Relu 5_1 with equal weights.
- Training Objectives: The identity loss trains the model on identical content or style inputs to maintain content structure while retaining style characteristics.Its formulation uses outputs reconstructed from two same content images or two same style images, with specified identity-loss weights.
4. Experimental Results
Experiments show that SANet preserves content structure while combining global and local style patterns, with strong qualitative preference and real-time performance. Ablations and runtime controls further demonstrate the roles of identity loss, multi-level features, and feature interpolation.
- Qualitative comparison: SANet preserves semantic style boundaries by transferring different style patterns to different content regions.For example, sky and buildings receive distinct styles instead of ambiguous boundaries.
- Qualitative comparison: The method captures multi-scale patterns while preserving content details that competing methods sometimes distort or lose.Reported patterns include color distribution, brush strokes, and rough textures.
- User study: User-study results favored the proposed method over the four comparison methods.The study collected 2,400 votes from 80 users across 980 synthesized images.
- Efficiency: 59 fps and 18 fps were achieved by the multi-scale models on 256- and 512-pixel images, respectively.Runtime measurements included style encoding.
- Ablation studies: Combining content-style loss with identity loss maintains content structure while enriching style patterns.Increasing content loss alone preserves structure but causes style characteristics to disappear, while distortion remains unavoidable.
- Ablation studies: Multi-level feature embedding combines complementary behavior: Relu 4_1 preserves global statistics and structure, while Relu 5_1 adds local patterns but can distort content.Two SANets are integrated to enrich style patterns across levels.
5. Conclusions
The paper concludes that SANet-based style transfer is effective and efficient, using learnable style decoration and identity loss to preserve content while enriching style. Experiments report preference over other state-of-the-art arbitrary style-transfer methods.
- Conclusions: SANet and decoders flexibly decorate style features through learning while preserving content structure and enriching local and global patterns.The approach uses conventional style reconstruction loss together with identity loss.
- Conclusions: Experimental results show that the proposed method is preferred over other state-of-the-art arbitrary style-transfer algorithms.
- Conclusions: The research received support from MCST and KOCCA through the 2019 Culture Technology Research and Development Program.