Source-linked AI summary
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer
Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, Errui Ding
TL;DR
Arbitrary style transfer remains limited by insufficient local feature modeling and reliance on global statistics or deep features. The paper introduces AdaAttN, which performs per-point attentive normalization using shallow and deep features, adds local feature loss, and extends the approach to video. Experiments report high-quality stylization for both images and videos, with the method achieving state-of-the-art arbitrary image/video style transfer.
Problem
Existing arbitrary style-transfer methods inadequately model local feature statistics and shallow low-level patterns, producing unnatural local distortions.
Method
AdaAttN computes attention from shallow and deep content and style features, aligns content with per-point attention-weighted style statistics, and uses local feature loss.
Results
Experiments demonstrate high-quality stylization results for both images and videos and state-of-the-art arbitrary image/video style transfer.
Takeaways & Limitations
With slight modifications, AdaAttN is ready for video style transfer and can produce stable, appealing results.
Abstract
from arXiv · showhide
Fast arbitrary neural style transfer has attracted widespread attention from academic, industrial and art communities due to its flexibility in enabling various applications. Existing solutions either attentively fuse deep style feature into deep content feature without considering feature distributions, or adaptively normalize deep content feature according to the style such that their global statistics are matched. Although effective, leaving shallow feature unexplored and without locally considering feature statistics, they are prone to unnatural output with unpleasing local distortions. To alleviate this problem, in this paper, we propose a novel attention and normalization module, named Adaptive Attention Normalization (AdaAttN), to adaptively perform attentive normalization on per-point basis. Specifically, spatial attention score is learnt from both shallow and deep features of content and style images. Then per-point weighted statistics are calculated by regarding a style feature point as a distribution of attention-weighted output of all style feature points. Finally, the content feature is normalized so that they demonstrate the same local feature statistics as the calculated per-point weighted style feature statistics. Besides, a novel local feature loss is derived based on AdaAttN to enhance local visual quality. We also extend AdaAttN to be ready for video style transfer with slight modifications. Experiments demonstrate that our method achieves state-of-the-art arbitrary image/video style transfer. Codes and models are available.
1. Introduction
Arbitrary style transfer offers flexible one-pass stylization but struggles to model local patterns and low-level details. AdaAttN addresses this with per-point attentive normalization using shallow and deep features, alongside a local feature loss and video extension.
- Arbitrary-style-per-model methods accept any style image and produce stylized results in a single forward pass after training.
- Global feature statistics and uniform transformations can dismiss local details and point-wise patterns, degrading local stylization.
- Current attention methods commonly rely on deep features and fuse re-weighted style features into content features, leaving low-level patterns underused.
- AdaAttN computes attention from shallow and deep content and style features, then aligns content statistics with per-point attention-weighted style mean and variance maps.
- Local feature loss regularizes generated local features to improve arbitrary style-transfer quality.
- Cosine-distance attention and image-wise similarity loss extend AdaAttN to video style transfer with stable and appealing results.
2. Related Works
Related work spans global and local transformations, attention-based correspondence, and methods designed to reduce temporal inconsistency in video stylization.
- Arbitrary Style Transfer: Global transformation methods modify feature representations globally, including whitening-coloring and adaptive mean-standard-deviation transfer.
- Arbitrary Style Transfer: Patch-based methods transfer styles through similarities between content and style patches, while multi-scale methods combine patch matching with adaptive normalization.
- Arbitrary Style Transfer: Attention-based arbitrary style transfer models correspondence among local style and content features to capture fine-grained relationships.
- Video Style Transfer: Video style transfer commonly adds optical-flow consistency constraints because independently stylized frame sequences can flicker from temporal inconsistency.
3. Methods
AdaAttN combines shallow and deep features to compute attention-weighted local style statistics, then normalizes content features per point for feature-distribution alignment. The framework adds local feature supervision and video-specific modifications for stable stylization.
- Adaptive Attention Normalization: AdaAttN computes attention from shallow and deep content and style features, then derives per-point weighted style statistics for normalization.Its three steps are attention-map computation, weighted mean and standard-variance estimation, and per-point content normalization.
- Adaptive Attention Normalization: Unlike AdaIN's global statistics and SANet's direct feature fusion, AdaAttN transfers attention-weighted feature distributions for local alignment.Setting every attention weight to 1/(HsWs) reduces AdaAttN to AdaIN.
- Loss Function: The overall objective combines global style loss with local feature loss, which supervises consistency between generated features and parameter-free AdaAttN transformations.The local feature loss is intended to improve local stylization compared with conventional content loss.
- Extension for Video Style Transfer: For video transfer, cosine-similarity attention and a cross-image similarity loss are introduced to stabilize local statistics and inter-frame relationships.The cosine-based attention produces a flatter distribution than Softmax, while the cross-image loss regularizes relevant contents across two frames.
4. Experiments
Experiments evaluate AdaAttN for image and video style transfer, including qualitative comparisons, loss ablations, style flexibility, and temporal consistency. Results show improved stylization, content preservation, and video stability, while supporting interpolation and multi-style transfer.
- Implementing Details: The model is trained on MS-COCO content images and WikiArt style images, with inference applicable to images of any resolution.Training uses random 256 × 256 crops from 512 × 512 images, while video inference uses 512 × 256 resolution.
- Image Style Transfer: AdaAttN qualitatively balances style transfer and content structure preservation better than compared arbitrary style transfer methods.It adaptively transfers style patterns to content-image locations through per-point attentive normalization.
- Image Style Transfer: 2000 votes from 100 users evaluated content preservation, stylization degree, and overall preference across synthesized image pairs.The study used 20 randomly sampled content-style pairs and presented results side-by-side in random order.
- Loss Ablation: Replacing the local feature loss with vanilla L2 content loss produces visibly worse results than the full model.The ablation supports the contribution of the proposed local feature loss to visual quality.
- Video Style Transfer: Cosine-similarity attention improves temporal consistency, and the modified AdaAttN is more stable than SANet, Linear, and MCCNet.The comparison uses optical-flow error, for which smaller values indicate better temporal consistency; means are calculated using 20 styles.
- Style Flexibility: AdaAttN supports style interpolation by averaging style statistics and multi-style transfer by concatenating multiple style images at runtime.The combined statistics or concatenated style image is processed for decoding through the AdaAttN pipeline.
5. Conclusion
AdaAttN transfers feature statistics with per-point attention using multi-level features, supports video style transfer, and produces high-quality image and video stylization.
- AdaAttN transfers style statistics through per-point attention-weighted mean and variance modulation.Attention weights use style and content features spanning low- to high-level representations.
- Slight modifications make the model applicable to video style transfer.
- Experiments report high-quality stylization results for both images and videos.
A.1. Decoder
The decoder combines outputs from three AdaAttN modules and progressively transforms and upsamples them in a structure roughly symmetrical to the VGG encoder.
- The decoder receives AdaAttN outputs from the ReLU-3_1, ReLU-4_1, and ReLU-5_1 layers.
- ReLU-5_1 features are upsampled and added element-wise to ReLU-4_1 features before a learnable 3 × 3 convolution block.
- The remaining decoder follows a structure symmetrical with the VGG encoder, with doubled ReLU-3_1 input channels for the additional AdaAttN output.
A.2. AdaAttN
The implementation defines AdaAttN with convolutional projections and a forward pass that computes attention-weighted style means and variances for feature normalization.
- The implementation provides an AdaAttN PyTorch module with time and space complexities matching SANet.
- AdaAttN initializes convolutional projections for query-key features and value features.
- The forward pass accepts content and style features at deep and shallow levels.
- Softmax attention produces weighted style means and variances, which modulate normalized content features before returning the AdaAttN output.
B.1. Image Style Transfer
The image-transfer experiments examine user-controlled regional stylization and ablations isolating shallow features and feature distribution alignment.
- User Control: Users can transfer features between selected content and style regions by manipulating the AdaAttN attention map.Regions can be selected by clicking points or outlining closed borders.
- More Ablation: The ablation study compares AdaAttN, AdaAttN without shallow features, SANet with shallow features, and SANet.
- More Ablation: Shallow features and feature distribution alignment each reduce dirty textures, while their combination gives the best stylization with the least distortion.
- Pair-wise Combination between Content and Style: Pair-wise combinations of 8 content images and 6 style images evaluate robustness across different contents and styles.
B.2. Video Style Transfer
The section supplements video style-transfer evaluation with optical-flow errors for 20 styles and additional qualitative video examples.
- Quantitative Results: Optical-flow errors for all 20 styles used in video stylization are reported in Table 4 as a supplement to Table 2.The styles come from AdaIN’s official codebase.
- Qualitative Results: Figure 13 provides additional snapshots of video stylization results, with full videos available in the attachments.The section also refers to full animations accompanying the examples.
- Figure 12 presents more image style-transfer results.