Source-linked AI summary
Dynamic Instance Normalization for Arbitrary Style Transfer
Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen
TL;DR
Existing arbitrary-style-transfer methods struggle with complex patterns while relying on costly shared encoders, limiting deployment on mobile terminals. The paper introduces Dynamic Instance Normalization, which combines adaptive dynamic convolution with a lightweight content encoder and achieves favorable style-transfer quality with a twentyfold computation-cost reduction.
Problem
Arbitrary-style-transfer methods remain limited by complex style patterns, fine-stroke generation, and costly encoders that hinder mobile-terminal deployment.
Method
Dynamic Instance Normalization encodes each style image into learned convolutional parameters through dynamic convolution, while a lightweight content encoder stylizes the content image.
Results
The approach achieves favorable visual and quantitative performance, especially for challenging style patterns, while reducing computation cost by a factor of twenty.
Takeaways & Limitations
DIN supports lightweight arbitrary style transfer and novel effects such as automatic spatial-stroke control through flexible convolutional operations.
Takeaways & Limitations
The paper identifies applying DIN to computer-vision tasks beyond style transfer as future work.
Abstract
from arXiv · showhide
Prior normalization methods rely on affine transformations to produce arbitrary image style transfers, of which the parameters are computed in a pre-defined way. Such manually-defined nature eventually results in the high-cost and shared encoders for both style and content encoding, making style transfer systems cumbersome to be deployed in resource-constrained environments like on the mobile-terminal side. In this paper, we propose a new and generalized normalization module, termed as Dynamic Instance Normalization (DIN), that allows for flexible and more efficient arbitrary style transfers. Comprising an instance normalization and a dynamic convolution, DIN encodes a style image into learnable convolution parameters, upon which the content image is stylized. Unlike conventional methods that use shared complex encoders to encode content and style, the proposed DIN introduces a sophisticated style encoder, yet comes with a compact and lightweight content encoder for fast inference. Experimental results demonstrate that the proposed approach yields very encouraging results on challenging style patterns and, to our best knowledge, for the first time enables an arbitrary style transfer using MobileNet-based lightweight architecture, leading to a reduction factor of more than twenty in computational cost as compared to existing approaches. Furthermore, the proposed DIN provides flexible support for state-of-the-art convolutional operations, and thus triggers novel functionalities, such as uniform-stroke placement for non-natural images and automatic spatial-stroke control.
Introduction
Arbitrary-style-per-model methods address style flexibility but remain limited by complex pattern handling and costly shared encoders. The proposed Dynamic Instance Normalization learns adaptive dynamic-convolution parameters, pairing an elaborate style encoder with a lightweight content encoder for more efficient, higher-quality arbitrary style transfer.
- Arbitrary-style-per-model methods use one model for new styles but struggle with complex patterns, fine strokes, and costly shared style-content encoders.
- DIN replaces manually computed affine-transformation parameters with a generalized dynamic convolution whose parameters are learned adaptively for arbitrary stylization.
- DIN produces favorable visual quality, particularly for finer strokes, sharper details, and challenging style patterns.The introduction reports superior quantitative and qualitative results against the state of the art.
- The method combines an elaborate style encoder with a lightweight content encoder to support more efficient and flexible arbitrary style transfer.
Related Work
Related work divides arbitrary style-per-model (ASPM) style transfer into non-parametric methods based on local patches and parametric methods based on summary statistics. Existing approaches generally rely on high-cost VGG encoders, while one exception incurs substantial extra memory by generating a separate 14-layer network for each style.
- ASPM Categories: ASPM algorithms aim to transfer arbitrary artistic styles to a photo using one trained model and one forward pass.The literature contains non-parametric and parametric ASPM categories.
- Non-Parametric ASPM: Non-parametric ASPM swaps content activation patches with their most similar style patches in VGG feature space before decoding.This approach transfers styles through local patch correspondences.
- Parametric ASPM: Parametric ASPM improves efficiency through global statistics matching, with AdaIN transferring channel-wise means and variances between style and content activations.Multi-scale stylization further extends AdaIN for improved visual quality.
- Parametric ASPM: WIN-based methods whiten VGG features to remove style information while preserving content structure, then apply coloring transforms to transfer desired patterns.This separates style removal from style insertion.
- Limitations: Existing state-of-the-art ASPM methods require high-cost VGG encoders, except a method generating a whole 14-layer style-specific network for every style at extra memory cost.The proposed approach instead targets lower encoder and memory costs while maintaining quality on challenging styles.
Proposed Method
The proposed Dynamic Instance Normalization (DIN) replaces manually defined affine parameter computation with style-conditioned dynamic convolutions, enabling generalized arbitrary stylization without complex shared encoders. Its flexible convolutional design also supports novel stylization functions and MobileNet-based lightweight deployment.
- Dynamic Instance Normalization: DIN combines instance normalization with dynamic convolution whose weights and biases are generated from each input style image.A sophisticated style encoder converts the style image into learnable convolutional parameters, which then stylize the content image.
- Generalized normalization: DIN generalizes existing normalization methods by treating IN, CIN, and AdaIN as specific parameterizations of its dynamic convolution.This broader formulation expands the search space for optimized normalization solutions and convergence.
- Encoder design: DIN removes the need for complex, shared content and style encoders because dynamic convolution avoids aligning meaningful feature means and variances.The approach instead permits a sophisticated style encoder to capture rich and complex style patterns adequately.
- Flexible convolutional operations: DIN supports different convolutional operations by changing the dynamic convolution type, enabling variants such as deformable and spatially-adaptive DIN.These variants are presented as examples of novel stylization functionalities enabled by the framework.
- Lightweight deployment: DIN enables the first MobileNet-based lightweight arbitrary stylization network for deployment on mobile and embedded devices.The network is built using depthwise separable convolutions from MobileNetV1.
Experiments
Experiments evaluate DIN on held-out content and style images against state-of-the-art arbitrary stylization methods, showing stronger transfer of challenging patterns, finer details, and efficient lightweight inference. Ablations further demonstrate faster convergence, comparable quality with depthwise separable convolutions, and extensions for spatially controlled strokes and non-natural images.
- Qualitative comparison: With the same VGG encoder, DIN transfers challenging style patterns more effectively and produces finer details than competing arbitrary stylization methods.The compared methods include Li et al., Huang et al., and Sheng et al.; competing results exhibit distorted patterns, missing details, or repeated textures.
- Efficiency: The hierarchical MobileNet-based network is about twenty times smaller than the VGG encoder while enabling arbitrary style transfer.The experiments compare both Ours (VGG) and Ours (MobileNet) under the same evaluation setting.
- Optimization: DIN achieves faster and better convergence than AdaIN under the same lightweight-network training settings.The comparison is reported through quantitative training curves.
- Ablation and variants: Depthwise separable convolutions retain visual quality comparable to standard convolutions despite lower computational complexity, while DIN variants enable spatially controlled strokes.Deformable DIN avoids random stroke placement according to visual attention, and spatially-adaptive DIN generates proper strokes for uniform pixel areas in non-natural images.
Discussions and Conclusions
The paper introduces Dynamic Instance Normalization (DIN) for flexible, efficient arbitrary style transfer, pairing an elaborate style encoder with a lightweight content encoder. Future work will investigate DIN beyond style transfer, particularly for domain adaptation.
- Conclusions: DIN enables flexible and more efficient arbitrary style transfer by combining rich style encoding with lightweight content encoding.This addresses the redundancy of shared encoders in previous methods.
- Future Work: Future work will apply DIN to other computer vision tasks, especially domain adaption, because it can replace normalization layers such as CIN and AdaIN.The paper characterizes style transfer as intrinsically a domain-adaption task.