Source-linked AI summary
Instance Normalization: The Missing Ingredient for Fast Stylization
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
TL;DR
Optimization-based stylization is computationally inefficient, while prior feed-forward generators did not match its quality. This paper revisits feed-forward stylization by replacing batch normalization with instance normalization, producing results comparable to slow optimization while enabling real-time generation on standard GPUs.
Problem
Prior feed-forward stylization methods were faster but did not achieve the quality of optimization-based stylization, which remained computationally inefficient.
Method
The paper replaces batch normalization with instance normalization in feed-forward stylization generators and retains instance normalization during testing.
Results
Instance normalization significantly improved both tested generator architectures, yielding stylization comparable to slow optimization while enabling real-time generation on standard GPUs.
Takeaways & Limitations
Replacing batch normalization with instance normalization can dramatically improve certain deep neural networks for image generation.
Takeaways & Limitations
The paper notes that it is unclear how the relevant function could be implemented using only ReLU and convolution operators.
Abstract
from arXiv · showhide
It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at https://github.com/DmitryUlyanov/texture_nets. Full paper can be found at arXiv:1701.02096.
1 Introduction
The paper revisits feed-forward neural style transfer to address the computational inefficiency of iterative optimization. It improves generator quality and enables real-time stylization by replacing batch normalization with instance normalization during both training and testing.
- 1 Introduction: Gatys et al.’s method transfers style by matching selected style and content statistics extracted from a pretrained classification network.Style statistics come from shallower layers and are spatially averaged, while content statistics are also extracted from the network.
- 1 Introduction: Gatys et al.’s optimization-based stylization takes several minutes for a 512 × 512 image, motivating feed-forward generator networks from Ulyanov et al. and Johnson et al.The feed-forward networks generate stylized images in a single pass.
- 1 Introduction: Replacing batch normalization with instance normalization produces results comparable to Gatys et al.’s slow method while enabling real-time generation on standard GPUs.The architectural change is presented as a small modification with a substantial qualitative improvement.
- 1 Introduction: Instance normalization is retained at test time and removes instance-specific contrast information from the content image, simplifying generation and improving images.This differs from batch normalization, whose test-time layers are frozen and simplified out.
2 Method
The method learns a convolutional generator to stylize arbitrary content images toward a fixed style, using feature-statistics losses and random Gaussian seeds. It replaces batch normalization with instance normalization throughout training and testing to discard content contrast information and simplify learning.
- Generator formulation: The generator g(x, z) applies a fixed style image x0 to arbitrary content images x, with random seed z producing sample stylizations.The generator is learned rather than optimized separately for each input image.
- Training objective: Training uses content images xt and a pre-trained CNN to compare style, content, and stylized-image feature statistics.The random variables zt are independent samples from a Gaussian distribution, zt ∼ N(0, 1).
- Prior limitations: Training on just 16 example images produced better qualitative results than training on thousands, while zero padding caused the most serious border artifacts.More complex padding techniques did not fully resolve the reported qualitative problems.
- Contrast normalization: The generator should discard content-image contrast because stylization is designed to match the stylized image’s contrast to the style image.The paper motivates contrast normalization as an efficient way to remove contrast information from the content image.
- Instance normalization: The method replaces batch normalization with instance normalization everywhere in g, preventing instance-specific mean and covariance shift and applying normalization at test time.Instance normalization is also called contrast normalization.
3 Experiments
The experiments replace batch normalization with instance normalization in two stylization-generator architectures and evaluate whether the modification generalizes across architectures. Both architectures improved significantly, and the Johnson et al. residual architecture was selected for subsequent results because it was somewhat more efficient and easier to use.
- Experimental setup: The study evaluates instance normalization as a replacement for batch normalization in the generator architectures of Ulyanov et al. (2016) and Johnson et al. (2016).The Johnson et al. architecture was reproduced from its paper because the original network was unavailable.
- Experimental results: Both architectures significantly improved after replacing batch normalization with instance normalization and retraining with the same hyperparameters.The improvement is shown in the second row of figure 5.
- Experimental results: The two generators produced similar quality, but the Johnson et al. residual architecture was somewhat more efficient and easier to use.The authors therefore adopted it for the results shown in figure 4.
4 Conclusion
Replacing batch normalization with instance normalization dramatically improves the performance of certain deep neural networks for image generation. The result also motivates experiments with similar ideas for image discrimination.
- 4 Conclusion: Replacing batch normalization with instance normalization dramatically improves certain deep neural networks for image generation.The authors describe this result as suggestive and report ongoing experiments applying similar ideas to image discrimination tasks.