Source-linked AI summary
Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
Lu Sheng, Ziyi Lin, Jing Shao, Xiaogang Wang
TL;DR
Arbitrary-style transfer must balance generalization, visual quality, and efficiency. Avatar-Net uses a style decorator and multi-scale hourglass network to produce plausible stylizations in one feed-forward pass, with reported effectiveness and efficiency across applications.
Problem
Existing style-transfer methods struggle to jointly support arbitrary styles, high visual quality, and efficient execution.
Method
Avatar-Net decorates content features with semantically aligned style patterns and uses multi-scale style adaptation in an hourglass network.
Results
Experiments demonstrate superior visual quality and economical processing cost for arbitrary-style, multi-scale transfer in one feed-forward pass.
Takeaways & Limitations
The method supports fast multi-scale stylization across styles, including global hues, local strokes, sketches, multiple-style integration, and video stylization.
Takeaways & Limitations
Overly restrictive patch similarity can miss style patterns when content and style feature domains differ substantially.
Abstract
from arXiv · showhide
Zero-shot artistic style transfer is an important image synthesis problem aiming at transferring arbitrary style into content images. However, the trade-off between the generalization and efficiency in existing methods impedes a high quality zero-shot style transfer in real-time. In this paper, we resolve this dilemma and propose an efficient yet effective Avatar-Net that enables visually plausible multi-scale transfer for arbitrary style. The key ingredient of our method is a style decorator that makes up the content features by semantically aligned style features from an arbitrary style image, which does not only holistically match their feature distributions but also preserve detailed style patterns in the decorated features. By embedding this module into an image reconstruction network that fuses multi-scale style abstractions, the Avatar-Net renders multi-scale stylization for any style image in one feed-forward pass. We demonstrate the state-of-the-art effectiveness and efficiency of the proposed method in generating high-quality stylized images, with a series of applications include multiple style integration, video stylization and etc.
1. Introduction
The introduction frames zero-shot style transfer as a quality–efficiency and generalization dilemma, then presents Avatar-Net’s patch-based style decorator and multi-scale architecture as the proposed solution. The method aims to preserve semantic content while matching holistic style distributions and retrieving detailed style patterns in one feed-forward pass.
- Motivation: Existing style transfer methods face a dilemma between generalization and quality versus efficiency.CNN feature statistics enabled iterative optimization and feed-forward approaches, but their trade-offs hamper practical availability.
- Limitations: Prior techniques may over-distort content with unconstrained patterns or fail to retrieve complete style patterns across large content–style domain gaps.Feed-forward methods supervised by external style signals also rely on perceptual loss, which is described as unstable and compromising.
- Proposed method: Avatar-Net introduces a patch-based style decorator that decorates content features with style characteristics while keeping content semantically perceptible.The decorator transfers content features toward semantically nearest style features and minimizes discrepancy between their holistic feature distributions.
- Proposed method: A new hourglass network with multi-scale style adaptation enables visually plausible multi-scale transfer for arbitrary styles in one feed-forward pass.The architecture fuses multi-scale style abstractions for efficient stylization of arbitrary style images.
2. Related Work
Prior style-transfer methods evolved from texture-based rendering to feature-level optimization and Markovian feature assembling, with variants targeting different scenarios. Avatar-Net’s zero-shot approach preserves detailed style patterns, parses complete style features across domain gaps, and performs multi-scale adaptation in one feed-forward pass.
- Classical Style Transfer: Style transfer is related to texture synthesis and exploits local statistics for efficient cross-view dense correspondences and texture quilting.These methods produce appealing stylized images.
- Optimization-Based Methods: Neural style transfer formulates style as multi-level feature correlations and balances content similarity with style affinity through an iterative optimization problem.This feature-level objective is also termed perceptual loss.
- Optimization-Based Methods: Variants of the optimization framework address photorealistic rendering, semantically composite transfer, and temporal coherence, while Markovian feature assembling changes global style statistics.These variants adapt the framework to different scenarios and requirements.
- Zero-Shot Style Transfer: Avatar-Net’s style decorator reserves detailed style patterns beyond parameterized feature statistics, parses complete style features despite domain gaps, and enables multi-scale adaptation in one feed-forward pass.It improves AdaIN and WCT through pattern preservation, outperforms Style-Swap through complete style-feature parsing, and surpasses AdaIN and WCT for multi-scale transfer because AdaIN needs a style-oriented decoder while WCT requires recursive passes.
3. Style Transfer via Feature Manipulation
This section formulates style transfer as feature manipulation and introduces a style decorator that combines distribution alignment with semantically matched style patterns. The module uses normalized patch matching and reassembling, followed by reconstruction into the style-feature domain.
- Feature Transfer Objective: Feature transfer maps content features zc into the style domain to preserve content spatial distribution while adopting style texture characteristics.The transferred feature is defined as zcs = F(zc; zs), with holistic domain adaptation and semantic linking as the central design challenge.
- Limitations of Patch Swapping: Existing patch swapping preserves concrete style patterns but remains biased toward content features and captures only overlapping style patterns when feature domains differ.Because matched patches follow local content variations, styles such as spiral sky patterns may not propagate effectively.
- Style Decorator: The style decorator whitens content and style features, matches normalized content elements to nearest normalized style elements, and reassembles the corresponding style patterns.Whitening uses kernels derived from feature covariance matrices; normalized cross-correlation implements matching through convolutional operations and binary correspondence scores.
- Style Decorator: The decorator seeks maximal distribution alignment while making detailed style patterns semantically perceptible, enabling effective and complete correspondences between normalized content and style features.Normalization allows each content element to find a suitable style correspondence while potentially retrieving every style element.
- Design Choices: AdaIN provides a faster whitening and coloring alternative than ZCA, while larger patch sizes produce more global and blocky stylization patterns.AdaIN preserves slightly more content patterns but is considered sufficient for real-time zero-shot style transfer; patch size controls presented style scale.
4. Multi-scale Zero-shot Style Transfer
Avatar-Net performs multi-scale zero-shot style transfer in one feed-forward pass by progressively fusing style information into decoded features. Its hourglass encoder-decoder uses style adaptation and fusion modules to combine bottleneck and multi-scale style features efficiently.
- Architecture: Avatar-Net uses a hourglass network with multi-scale style adaptation modules that progressively fuse encoded styles into corresponding decoded features.This enables multi-scale style transfer in one feed-forward pass.
- Architecture: The main branch follows an encoder-decoder architecture, with an encoder that progressively extracts intermediate features and produces a bottleneck feature, and a decoder that reconstructs the output.Decoded features are updated by fusing them with corresponding encoded features through style adaptive feature fusion.
- Style Fusion: Style Fusion is analogous to AdaIN and uses channel-wise multiplication and channel-wise standard deviation to adapt decoded stylized features to encoded style features.Although it may not optimally match second-order statistics like ZCA, decoder training moves the module toward ZCA while retaining economical computational complexity.
- Transfer Procedure: Given content and arbitrary style images, Avatar-Net encodes both to bottleneck features, decorates the content bottleneck with style, and decodes it using progressively applied multi-scale style fusion.The style image also bypasses the encoder to provide multi-scale encoded style features, which guide fusion from the deepest level toward the shallowest.
5. Experimental Results and Discussions
Avatar-Net produces semantically aligned, multi-scale stylizations with coherent content structure and detailed style patterns across varied styles. It also supports efficient one-pass transfer, controllable style strength, style interpolation, and temporally coherent video stylization.
- Qualitative Evaluations: Avatar-Net reassembles style patterns according to the content image’s semantic spatial distribution, preserving recognizable faces, buildings, and skylines.The method handles styles ranging from global hue changes to local strokes and sketches.
- Qualitative Evaluations: Avatar-Net preserves multi-scale color distributions, brush strokes, and circular patterns better than the compared methods in close-up results.WCT distorts brush strokes and circular patterns, AdaIN loses color distribution, and Style-Swap fails in the cited example.
- Efficiency: Avatar-Net is more efficient than WCT because its hourglass architecture performs multi-scale processing in one feed-forward pass.Replacing the ZCA-based decorator with AdaIN reduces execution time to the same level as Style-Swap and AdaIN, while the economical decorator reduces style patches to 1/16 with similar performance but slightly fewer detailed patterns.
- Trade-off Between Content and Style: Stylization strength is controllable by interpolating either normalized stylized features or stylized features, with α = 0 reducing the former variant to WCT and α = 1 retaining only style patterns.Low-level shortcut-link adaptation is not adjusted in this experiment and can be interpolated to enable multi-scale transfer.
- Style Interpolation: Avatar-Net supports convex integration of multiple style images, although differing feature magnitudes affect the resulting interpolated stylization.The style adaptations in shortcut links are also extended to convex combinations of stylized features.
- Video Stylization: Per-frame video stylization remains stable and coherent across adjacent frames with Avatar-Net’s style decorator.The cited comparison reports face-region distortions and flickering artifacts for WCT.
6. Concluding Remarks
Avatar-Net combines a style decorator for semantic style feature propagation with a hourglass network for multi-scale holistic style adaptation, enabling fast and reliable zero-shot style transfer for arbitrary stylized images. Future work may replace the decorator’s projection and reconstruction steps with learnable modules to improve alignment robustness and execution efficiency.
- Contributions: Avatar-Net integrates a style decorator for semantic style feature propagation with a hourglass network for multi-scale holistic style adaptation.The method is designed as a fast and reliable multi-scale zero-shot style transfer approach.
- Experimental conclusion: Experimental results demonstrate Avatar-Net’s superiority in generating arbitrary stylized images.
- Future direction: Future work may replace the style decorator’s projection and reconstruction steps with learnable modules for increased alignment robustness and execution efficiency.
7. Appendix
The appendix analyzes normalization choices and patch parameters in the style decorator, showing their effects on matching robustness, style-pattern scale, diversity, and efficiency. It also provides additional stylization and video demonstrations under default settings.
- Projection and reconstruction transforms: ZCA applied to the covariance matrix is the paper’s style-decorator transform, while AdaIN and ZCA applied to the Gram matrix provide alternative normalization strategies.The normalized features follow a standard Gaussian distribution; covariance-based ZCA preserves feature-direction variation, whereas Gram-based ZCA orients features like the originals.
- Projection and reconstruction transforms: AdaIN leaves some content texture characteristics, limiting matched-style-patch diversity, whereas covariance- and Gram-based ZCA have very similar performance.AdaIN may still be used for real-time efficiency when its occasional similarity to the ZCA variants is acceptable; WCT requires covariance-based ZCA for satisfactory results.
- Patch Size: Larger patch sizes produce larger-scale, more global style patterns and greater distortion of content information; at P = 11, results resemble WCT but are more vivid.The appendix specifically reports that P = 11 yields a similar style-pattern scale to WCT while remaining more vivid.
- Patch Sampling: Patch-sampling density controls the diversity and completeness of recovered style patterns, while smaller strides cause only slight degradation when S is not larger than P −1.The method can use a stride 1 < S < P to reduce memory cost and computational complexity; larger strides discard many style patterns.
- Additional results: Additional stylization results use patch size P = 5 and blending weight α = 0.8 by default, alongside demonstrations of stylized videos generated by the proposed method.These defaults apply unless otherwise stated in the main article and supplementary material; the video demonstrations are provided through a YouTube link.