Source-linked AI summary
Universal Style Transfer via Feature Transforms
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang
TL;DR
Universal style transfer needs to generalize across arbitrary styles while maintaining visual quality and efficiency. The paper embeds whitening and coloring transforms in an image-reconstruction network to match content and style feature statistics during feed-forward inference. It reports favorable performance against state-of-the-art methods on arbitrary-style transfer, with applications to texture synthesis.
Problem
Universal style transfer remains difficult because existing methods trade off generalization, visual quality, and efficiency when applying style characteristics to content images.
Method
The method couples whitening and coloring transforms with a pre-trained encoder-decoder network to directly match content and style feature statistics without style-specific training.
Results
The algorithm achieves favorable performance against state-of-the-art methods in generalizing to arbitrary styles and is also effective for texture synthesis.
Takeaways & Limitations
Feed-forward feature-statistics matching provides a style-agnostic route to universal stylization and supports high-quality visual results and texture synthesis.
Takeaways & Limitations
The decomposition step is currently implemented on the CPU, and the authors identify more efficient GPU implementations as future work.
Abstract
from arXiv · showhide
Universal style transfer aims to transfer arbitrary visual styles to content images. Existing feed-forward based methods, while enjoying the inference efficiency, are mainly limited by inability of generalizing to unseen styles or compromised visual quality. In this paper, we present a simple yet effective method that tackles these limitations without training on any pre-defined styles. The key ingredient of our method is a pair of feature transforms, whitening and coloring, that are embedded to an image reconstruction network. The whitening and coloring transforms reflect a direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer. We demonstrate the effectiveness of our algorithm by generating high-quality stylized images with comparisons to a number of recent methods. We also analyze our method by visualizing the whitened features and synthesizing textures via simple feature coloring.
1 Introduction
Universal style transfer seeks arbitrary-style generalization without sacrificing visual quality and efficiency. The paper addresses this with whitening and coloring transforms embedded in an image-reconstruction network.
- Universal style transfer must reconcile generalization, visual quality, and efficiency when applying style characteristics to content images.
- Multi-level stylization matches style statistics across feature levels, while the approach also demonstrates high-quality visual results and universal texture synthesis.
- The method transforms intermediate content features so their statistical characteristics match those of style features at corresponding layers.
- Whitening and coloring transforms directly match content feature statistics to those of a style image in deep feature space.
- A fixed VGG encoder and pre-trained decoder networks reconstruct stylized images through simple feed-forward operations after feature transformation.
2 Related Work
Prior style-transfer methods trade off arbitrary-style flexibility, computational efficiency, or visual quality. This work uses feed-forward whitening and coloring to target arbitrary styles while retaining generalization and visual quality.
- Analogy-based methods often require content-style image pairs depicting the same scene type, limiting their scalability to arbitrary style images.
- Optimization-based arbitrary stylization matches deep-feature correlations but is inefficient because it requires iterative optimization.
- Feed-forward transformation networks improve efficiency but commonly require training one network per style because of limited generalization.
- Later multi-style methods use style-specific units, filters, or normalization parameters, while local feature swapping targets arbitrary style transfer.
- The proposed approach combines feed-forward efficiency with generalization and visual quality on arbitrary styles, using WCT rather than adaptive instance normalization.
3 Proposed Algorithm
The method formulates style transfer as image reconstruction with whitening and coloring transforms that match content-feature statistics to those of a style image. Multi-level processing combines coarse structural patterns from higher VGG layers with lower-level stylistic details.
- Reconstruction decoder: The reconstruction framework uses a fixed VGG-19 encoder and trained symmetric decoders to invert features back to RGB images.Five decoders are trained for features from Relu_X_1, X=1,2,3,4,5.
- Whitening and coloring transforms: WCT whitens content features and colors them with style statistics so their covariance matches the style feature covariance.Whitening removes feature correlations after centering; coloring applies the style covariance and re-centers with the style mean.
- Whitening and coloring transforms: Whitened features retain global content structures while removing style-related information, including detailed stroke patterns.The features are inverted with a reconstruction decoder to visualize what content and style information remains.
- Feature transform comparison: WCT captures salient style patterns more faithfully than channel-wise histogram matching, which transfers global color but breaks patterns and misrepresents local structures.The difference is attributed to WCT modeling correlations between feature channels through the covariance matrix.
- User control: The stylization strength is user-controllable by blending the transformed style feature with the original content feature using the style weight α.The blended feature is then passed to the decoder.
- Multi-level stylization: The multi-level pipeline processes higher VGG layers first for coarse stylization, then lower layers to refine details and low-level information.Higher layers capture complicated local structures or salient patterns, whereas lower layers carry colors and further improve details.
4 Experimental Results
Experiments evaluate stylization quality, efficiency, user control, and texture synthesis. The method supports multi-level and spatially controlled transfer, produces diverse textures, and compares favorably with prior approaches.
- The evaluation compares stylization methods across paintings, abstract styles, and styles with obvious texton elements, using α = 0.6 for our method.
- Our method captures salient style patterns while preserving and stylizing key content components such as bridges, eyes, and mouths.
- Lower covariance matrix difference Ls indicates stylized results closer to the given style statistics, and our method achieves lower Ls.
- A user study with 80 subjects and 1,200 votes found that our method received the most votes for better stylized results.
- Efficiency: The approach is efficient but slightly slower than [27] [15] because its eigenvalue decomposition is implemented on the CPU.
- User Controls: The method provides scale, weight, and spatial controls through feed-forward operations, including mask-based stylization of specified regions.
- Texture synthesis: For texture synthesis, repeated multi-level processing improves visual appeal, while sampled noise produces diverse results without the repetitive layouts observed for [27].
- Texture synthesis: Feature-space interpolation generates new mixed textural effects, whereas produces results described as overlaid combinations of two textures.
5 Concluding Remarks
The paper presents universal style transfer without per-style training by combining whitening and coloring transforms with a reconstruction network. It also extends the approach to multi-level stylization and texture synthesis, with favorable generalization to arbitrary styles.
- The algorithm avoids learning for each individual style by matching intermediate content and style feature statistics and correlations through whitening and coloring transforms.
- A multi-level stylization pipeline incorporates information from all style levels, and the approach is also effective for texture synthesis.