PaperlayerView exact revision on arXiv

Source-linked AI summary

Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

Xun Huang, Serge Belongie

arXiv:1703.06868v2cs.CV

TL;DR

Neural style transfer is flexible but slow, while fast feed-forward methods are limited to predefined styles. This paper introduces AdaIN, which aligns content and style feature statistics to enable arbitrary style transfer in real time, with flexible user controls.

  • Problem

    Existing neural style transfer is either slow because of iterative optimization or restricted to predefined styles and unable to adapt to arbitrary new styles.

  • Method

    The method uses adaptive instance normalization to align content and style feature means and variances directly in feature space before decoding the result.

  • Results

    The approach transfers arbitrary new styles in real time at 56 FPS for 256 × 256 images and 15 FPS for 512 × 512 images.

  • Takeaways & Limitations

    A single feed-forward network supports arbitrary styles with speed comparable to the fastest style-specific feed-forward methods and enables flexible user controls.

  • Takeaways & Limitations

    AdaIN aligns only mean and variance, leaving higher-order feature statistics unmodeled and potentially limiting stylization quality.

Abstract

from arXiv · show

Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

1. Introduction

The paper addresses the flexibility–speed dilemma in neural style transfer: optimization-based methods support arbitrary styles but are prohibitively slow, while feed-forward methods are faster but usually style-restricted. It presents an approach for transferring arbitrary new styles in real time, inspired by instance normalization.

  • Motivation: Gatys et al. showed that image content and style are somewhat separable, enabling arbitrary images’ content and style to be combined.Their method can change style while preserving content, but relies on a prohibitively slow optimization process.
  • Problem: Most feed-forward style-transfer networks perform stylization in one forward pass but remain restricted to a single style.Recent methods either support only finite style sets or are much slower than single-style transfer methods.
  • Contribution: The paper presents the first neural style-transfer algorithm claimed to resolve the fundamental flexibility–speed dilemma.Its approach transfers arbitrary new styles in real time while combining optimization-based flexibility with speed similar to the fastest feed-forward approaches.
  • Contribution: The method is inspired by instance normalization, a layer reported to be surprisingly effective in feed-forward style transfer.The authors propose a new interpretation of instance normalization as performing style normalization.

2. Related Work

Related work spans classical style and texture transfer, optimization-based neural methods, and feed-forward approaches for faster or arbitrary style transfer. Prior work also explored alternative style losses and generative image-modeling frameworks, while arbitrary-style methods faced computational bottlenecks.

  • Style transfer: Style transfer originated in non-photorealistic rendering and is closely related to texture synthesis and transfer.Early methods included histogram matching on linear filter responses and non-parametric sampling.
  • Style transfer: Gatys et al.’s framework iteratively minimizes content and style losses, taking minutes to converge even with modern GPUs.Its optimization-based processing is therefore too slow for practical on-device mobile applications.
  • Style transfer: Feed-forward neural networks were proposed to replace slow optimization by training against the same style-transfer objective.These methods improve speed but the supplied passage introduces them as a common workaround rather than describing arbitrary-style capability here.
  • Style transfer: Chen and Schmidt enabled arbitrary style transfer with a style swap layer that replaces content features with closest-matching style features patch by patch.More than 95% of computation is spent on style swapping for 512 × 512 input images, creating a computational bottleneck.
  • Style loss functions: Style-transfer research has used diverse losses, including Gram-matrix, MRF, adversarial, histogram, CORAL, MMD, and channel-wise mean-and-variance matching.The original Gatys et al. framework matches second-order feature statistics through the Gram matrix.
  • Deep generative image modeling: Alternative image-generation frameworks include variational auto-encoders, auto-regressive models, and GANs, with GANs achieving the most impressive visual quality.GANs have also been applied to style transfer and cross-domain image generation.

3. Background

Normalization layers shape both the training behavior and style-transfer capabilities of feed-forward networks. Instance normalization improves stylization, while conditional instance normalization supports multiple fixed styles but scales poorly and cannot adapt to arbitrary new ones.

  • Batch normalization: Batch normalization (BN) accelerates feed-forward network training by normalizing feature statistics for each feature channel.BN computes channel-wise statistics across batch and spatial dimensions and uses learned affine parameters.
  • Instance normalization: Replacing BN with instance normalization (IN) significantly improves feed-forward style transfer.IN computes mean and standard deviation across spatial dimensions independently for each channel and sample, and remains unchanged at test time.
  • Conditional instance normalization: Conditional instance normalization (CIN) learns distinct affine parameters γs and βs for each style, enabling one convolutional network to generate different styles.During training, style indices are sampled from a fixed style set; the cited experiments used S = 32 styles.
  • Conditional instance normalization: CIN cannot adapt to arbitrary new styles without retraining the network.Its style-specific parameterization is tied to the fixed styles modeled during training.
  • Conditional instance normalization: 2FS additional parameters are required by CIN, and this overhead scales linearly with the number of styles.F is the network’s total number of feature maps, making large style collections such as tens of thousands challenging to model.

4. Interpreting Instance Normalization

Instance normalization appears to work for style transfer by normalizing feature statistics, rather than merely removing pixel-level contrast. Compared with batch normalization, it better aligns each sample with a common target style because it avoids intra-batch style differences.

  • Feature-statistics interpretation: Because IN operates in feature space, it can have deeper effects than pixel-space contrast normalization, and its affine parameters can completely change the output style.The passages identify both feature-space normalization and affine parameter control as reasons IN affects style beyond contrast.
  • Feature-statistics interpretation: IN performs a form of style normalization by normalizing convolutional feature statistics, whose channel-wise means and variances can support effective style transfer.Gatys et al. used second-order statistics, while other work found channel-wise mean and variance effective for style transfer.
  • Feature-statistics interpretation: IN remains effective after training images are equalized to the same luminance contrast, suggesting contrast invariance alone incompletely explains its success.In the reported experiment, the IN model also converged faster than the BN model before contrast equalization.
  • IN versus BN: BN normalizes feature statistics across a batch, effectively centering samples around one style while allowing each sample to retain different styles.This creates intra-batch style variation, which is undesirable when transferring all images to the same style and adds training challenges.

5. Adaptive Instance Normalization

AdaIN enables arbitrary style transfer by matching the channel-wise feature statistics of a content input to those of a style input. It performs this transfer with an IN-like layer that adds almost no computational cost while preserving content spatial structure.

  • Adaptive Instance Normalization: AdaIN aligns the channel-wise mean and variance of content features with those of style features for arbitrary style inputs.Unlike BN, IN, and CIN, AdaIN uses adaptive rather than learned affine transformations.
  • Adaptive Instance Normalization: AdaIN scales normalized content features by σ(y) and shifts them by µ(y), with statistics computed across spatial locations.The transformation adapts directly to the supplied style input y.
  • Adaptive Instance Normalization: AdaIN transfers style through feature statistics while preserving the content image’s spatial structure.Feature channels associated with style elements such as brushstrokes retain their spatial organization from the content features.
  • Adaptive Instance Normalization: AdaIN is as simple as an instance-normalization layer and adds almost no computational cost.This contrasts with the style swap operation, which is described as time-consuming and memory-consuming.

6. Experimental Setup

The experimental setup uses an encoder-decoder style-transfer network with AdaIN, trained on paired content and painting datasets using VGG-19-based losses. The design includes specific decoder and preprocessing choices to support image synthesis and stable training.

  • Network architecture: The network encodes content and arbitrary style images with the fixed initial layers of pre-trained VGG-19, then applies AdaIN to their feature maps.The encoder extends through relu4_1.
  • Network architecture: A randomly initialized decoder maps the AdaIN representation back to image space, while nearest up-sampling, reflection padding, and omitted normalization choices address reconstruction artifacts and style preservation.Nearest up-sampling replaces pooling to reduce checkerboard effects, and reflection padding avoids border artifacts.
  • Training procedure: Training uses roughly 80,000 MS-COCO content images and 80,000 painting style images, Adam optimization, batch size 8, and random 256 × 256 crops after resizing the smallest dimension to 512.The datasets are respectively drawn from MS-COCO and paintings mostly collected from WikiArt.
  • Objective: The decoder is trained with a weighted combination of content and style losses, using the AdaIN output as the content target for slightly faster convergence.The content loss is the Euclidean distance between target and output features, with style loss weight λ.
  • Objective: The style loss matches mean and standard-deviation statistics of style features across relu1_1, relu2_1, relu3_1, and relu4_1 with equal weights.This statistic-matching loss follows the behavior of the AdaIN layer rather than using the commonly used Gram matrix loss.

7. Results

The method achieves competitive stylization quality and real-time speed for arbitrary styles, while its AdaIN architecture outperforms alternative fusion and decoder-normalization choices. A single feed-forward network also supports runtime control over stylization strength, style interpolation, color preservation, and spatially varying styles.

  • Qualitative Examples: On unseen test styles, our stylized images are competitive with single-style and optimization-based methods, though slightly lower quality in some cases.The single-style baseline is fitted separately to each test style, whereas our network never observes those styles during training.
  • Quantitative evaluations: Our method and the single-style baseline have slightly higher but comparable average content and style losses, with style loss similar to optimization after 50–100 iterations.The comparison modifies the baselines’ loss functions to use the same instance-normalization statistics-based style loss.
  • Speed analysis: 56 FPS at 256 × 256 and 15 FPS at 512 × 512, excluding style encoding, enables arbitrary user-uploaded styles to be processed in real time.The method is nearly 3 orders of magnitude faster than optimization-based transfer and 1–2 orders faster than the patch-based method among arbitrary-style algorithms.
  • Ablation study: Replacing AdaIN with concatenation fails to disentangle style from content, while decoder BN or IN layers produce qualitatively worse results.The concatenation baseline preserves visible style-image object contours, reaches low style loss, but fails to reduce content loss.
  • Content-style trade-off: At runtime, changing α from 0 to 1 provides a smooth transition from content similarity to style similarity using the same network.α = 0 reconstructs the content image faithfully, while α = 1 produces the most stylized image.
  • Style interpolation and spatial/color control: The same feed-forward network interpolates arbitrary styles, preserves content colors through color alignment, and assigns different styles to different content regions.Spatially varying transfer applies AdaIN separately to regions and generalizes despite decoder training on homogeneous styles.

8. Discussion and Conclusion

The paper introduces AdaIN as a simple layer enabling arbitrary style transfer in real time by directly aligning feature statistics in one shot. The authors distinguish this approach from prior optimization- and feed-forward-based methods while identifying architectural, training, and statistical limitations for future work.

  • Contribution: AdaIN enables arbitrary style transfer in real time with a simple adaptive instance normalization layer.The authors present this as the first approach to achieve arbitrary style transfer in real time.
  • Conceptual difference: Unlike prior methods, the approach directly aligns feature-space statistics in one shot before inverting features back to pixel space.Gatys et al. optimize pixel values, while later feed-forward networks modify pixels indirectly to match feature statistics.
  • Future work: Future work will explore residual architectures, additional encoder skip connections, and incremental training schemes.These directions are proposed as ways to improve the method's network architecture and training process.
  • Limitation: AdaIN currently aligns only the most basic feature statistics: mean and variance.The authors identify this limitation while noting substantial room for improvement.
Loading 1703.06868v2…