Source-linked AI summary
Arbitrary Style Transfer via Multi-Adaptation Network
Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, Changsheng Xu
TL;DR
Arbitrary style transfer must combine vivid style rendering with detailed content preservation, a balance existing methods do not efficiently achieve. The paper introduces a multi-adaptation encoder-decoder with specialized self- and co-adaptation modules and a disentanglement loss. Experiments show that it considers global content structure and local style patterns to generate stylized results, while style interpolation supports weighted combinations of multiple styles.
Problem
Arbitrary style transfer must preserve detailed content structures while rendering rich style patterns, but existing methods cannot efficiently balance these goals.
Method
A multi-adaptation encoder-decoder uses position-wise content attention, channel-wise style attention, content-guided style rearrangement, and disentanglement loss.
Results
Experiments show that the network considers global content structure and local style patterns, achieves a trade-off between structure preservation and style rendering, and produces stylized results with unified styles and salient content structures.
Takeaways & Limitations
The model supports flexible style transfer, including merging multiple style images into one result with adjustable style weights.
Takeaways & Limitations
Future work aims to develop style-image selection based on global semantic similarity between content and style for additional practical applications.
Abstract
from arXiv · showhide
Arbitrary style transfer is a significant topic with research value and application prospect. A desired style transfer, given a content image and referenced style painting, would render the content image with the color tone and vivid stroke patterns of the style painting while synchronously maintaining the detailed content structure information. Style transfer approaches would initially learn content and style representations of the content and style references and then generate the stylized images guided by these representations. In this paper, we propose the multi-adaptation network which involves two self-adaptation (SA) modules and one co-adaptation (CA) module: the SA modules adaptively disentangle the content and style representations, i.e., content SA module uses position-wise self-attention to enhance content representation and style SA module uses channel-wise self-attention to enhance style representation; the CA module rearranges the distribution of style representation based on content representation distribution by calculating the local similarity between the disentangled content and style features in a non-local fashion. Moreover, a new disentanglement loss function enables our network to extract main style patterns and exact content structures to adapt to various input images, respectively. Various qualitative and quantitative experiments demonstrate that the proposed multi-adaptation network leads to better results than the state-of-the-art style transfer methods.
1 INTRODUCTION
Arbitrary style transfer seeks to render vivid artistic patterns while preserving detailed content structure. The paper proposes a multi-adaptation network and disentanglement loss to improve this balance.
- Artistic style transfer renders natural images with artistic patterns while maintaining their content structures.
- Existing arbitrary style transfer methods align global statistics or swap patches, but can produce disappointing quality or transfer few patterns when content and style structures differ greatly.
- The Monet example preserves mountain and house structures while transferring painting strokes and color theme into the result.
- The multi-adaptation network uses content and style self-adaptation modules plus co-adaptation to disentangle representations and adjust style distribution to content distribution.
- The proposed disentanglement loss makes content features consistent across styles and style features consistent across contents, supporting extraction of directed content and style information.
- Experiments report preservation of detailed content structures and transfer of rich reference style patterns, while also examining convolutional receptive fields in co-adaptation.
2 RELATED WORK
Prior style-transfer work progressed from optimization-based and feed-forward methods to arbitrary-style approaches, but balancing structure preservation with style rendering remained difficult. The paper addresses this through specialized disentanglement modules within an encoder-decoder network.
- Style Transfer: CNN-based style transfer evolved from VGG feature optimization toward faster feed-forward networks and broader generation quality.
- Style Transfer: Arbitrary-style methods use global statistic alignment or patch swapping, yet they struggle to balance content structure preservation and style pattern rendering.
- Feature Disentanglement: GAN-based image-to-image translation can generate high-quality artistic works but has limited adaptation to arbitrary style transfer in unseen domains.
- Proposed Method: The network uses an encoder-decoder architecture with a multi-adaptation module that disentangles style and content, then rearranges style distribution according to content distribution.
- Feature Disentanglement: The content and style self-adaptation modules are designed specifically around content structure and style texture rather than relying on similarly structured encoders.
3 METHODOLOGY
The method uses an encoder-decoder with multi-adaptation modules to separate content and style, adapt style distributions to content, and generate stylized features. Three loss types further constrain perceptual similarity, identity mapping, and disentanglement.
- Network Architecture: A pretrained VGG19 encoder extracts content and style feature maps, while a mirrored decoder produces the generated image from stylized features.The network minimizes three types of loss during training.
- Multi-adaptation Module: Two independent self-adaptation modules disentangle content and style representations before co-adaptation combines them.Content and style features are transformed separately, then the co-adaptation module generates stylized features.
- Position-wise Content Self-adaptation Module: The position-wise content module uses whitened features and spatial attention to capture long-range information and enhance semantic content structure.The enhanced content feature is obtained through matrix multiplication and element-wise addition.
- Channel-wise Style Self-adaptation Module: The channel-wise style module uses channel attention on unwhitened style features to enhance texture and stroke patterns.Its adjusted style feature map is produced through matrix multiplication and element-wise addition.
- Co-adaptation Module: The co-adaptation module computes local content-style correlations non-locally, rearranges the style distribution according to content, and merges the results into stylized features.The correlation map compares content and style positions before generating rearranged style features.
- Loss Function: Training combines content and style perceptual losses, identity loss, and disentanglement loss to constrain generated outputs and separate representation factors.The disentanglement objective makes content features consistent across styles and style features consistent across contents.
4 EXPERIMENTS
Experiments compare the proposed method with prior arbitrary style transfer systems, assess disentanglement loss and receptive-field choices, and demonstrate controllable style-content trade-offs and style interpolation. User studies, classification results, and visual comparisons report stronger overall performance, while ablations identify effects of the loss and receptive-field size.
- Comparison with Prior Work: The evaluation compares the method with AdaIN, WCT, SANet, and AAMS using visual comparisons, user studies, and classification accuracy.The user study uses 20 style images, 15 content images, 30 participants, and 1,500 votes per question.
- Comparison with Prior Work: Users gave the proposed method the majority of votes for content, style, and overall visual effect, with overall performance exceeding every comparison method.Compared with AdaIN, style was clearly better and content preservation comparable; compared with WCT, content was clearly better and style patterns comparable; compared with AAMS, both content and style were clearly better.
- Comparison with Prior Work: The method achieves relatively high style and content classification accuracy, indicating a trade-off between style rendering and content preservation.SANet has the highest style classification accuracy but too-low content classification accuracy for attractive results.
- Ablation Study: Disentanglement loss produces unified style patterns across results and more visible, salient content structures.The ablation highlights extraction of key style patterns such as purple feathers while preserving content structure.
- Ablation Study: Larger receptive fields transfer more local style patterns but highly distort global content structures.Using conv5_1 or 3 × 3 kernels increases local pattern transfer, including circles and feathers, while distorting content structure.
- Applications: Style strength is controllable through α: α = 0 returns the original content image, whereas α = 1 returns the fully stylized image.The method also merges multiple style images into one result by changing their weights.
5 CONCLUSIONS AND FUTURE WORK
The multi-adaptation network disentangles global content and style representations, adjusts style distribution using long-range local similarity, and uses a disentanglement loss to balance content preservation with style rendering. Future work targets recommending suitable style images for given content images.
- The network disentangles global content and style representations and adjusts style distribution to content distribution using long-range local similarity.
- Its disentanglement loss constrains style features to remain independent from target content features.
- Adequate experiments report generated results that consider global content structure and local style patterns while achieving a trade-off between structure preservation and style rendering.
- Future work will develop style-image selection based on global semantic similarity between content and style for additional practical applications.