Source-linked AI summary
Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
TL;DR
The paper addresses the quality and diversity gap between fast feed-forward generators and slow optimization-based texture synthesis and stylization. It introduces instance normalization and a Julesz-ensemble sampling formulation, and reports that their combination brings feed-forward results to a level comparable to optimization-based approaches while retaining feed-forward speed.
Problem
Feed-forward generators are much faster than optimization-based methods but produce lower-quality stylizations and insufficiently diverse texture samples.
Method
The paper introduces instance normalization for generator networks and a learning formulation that samples the Julesz ensemble rather than merely finding individual texture images.
Results
Combined, the two ideas improve feed-forward texture synthesis and image stylization to a level comparable to optimization-based approaches.
Takeaways & Limitations
The approach brings feed-forward texture synthesis and stylization closer to optimization-based quality while preserving the speed advantage of feed-forward generation.
Takeaways & Limitations
The entropy-loss weight must be tuned for each texture model: too little can prevent diversity, while too much can create artifacts.
Abstract
from arXiv · showhide
The recent work of Gatys et al., who characterized the style of an image by the statistics of convolutional neural network filters, ignited a renewed interest in the texture generation and image stylization problems. While their image generation technique uses a slow optimization process, recently several authors have proposed to learn generator neural networks that can produce similar outputs in one quick forward pass. While generator networks are promising, they are still inferior in visual quality and diversity compared to generation-by-optimization. In this work, we advance them in two significant ways. First, we introduce an instance normalization module to replace batch normalization with significant improvements to the quality of image stylization. Second, we improve diversity by introducing a new learning formulation that encourages generators to sample unbiasedly from the Julesz texture ensemble, which is the equivalence class of all images characterized by certain filter responses. Together, these two improvements take feed forward texture synthesis and image stylization much closer to the quality of generation-via-optimization, while retaining the speed advantage.
1. Introduction
Feed-forward networks make texture synthesis and stylization much faster, but initially sacrifice quality and diversity relative to optimization-based generation. This work combines instance normalization and Julesz-ensemble sampling to narrow those gaps.
- Motivation: Optimization-based texture synthesis and stylization can take tens of seconds to minutes for one image, motivating faster feed-forward generators.Once trained, feed-forward texture networks operate three orders of magnitude faster than optimization methods.
- Motivation: Earlier feed-forward generators produced less diverse texture samples and qualitatively and quantitatively worse stylizations than iterative optimization.These limitations define the two problems addressed in the paper.
- Contributions: Instance normalization substantially improves generator performance, especially stylization, reducing the quality gap with Gatys et al.'s iterative method.The improvement is reported both quantitatively and qualitatively.
- Contributions: A new formulation trains generators to uniformly sample the Julesz ensemble, the equivalence class of images matching specified filter statistics.The objective minimizes KL divergence to a quasi-uniform ensemble distribution and estimates generated-sample entropy differentiably.
- Results: Combined, the two contributions dramatically improve feed-forward texture synthesis and stylization to a level comparable to optimization-based approaches.The comparison uses extensive quantitative and qualitative experiments against gold-standard optimization results.
2. Background and related work
The background frames textures as distributions over images sharing filter-response statistics, with the Julesz ensemble collecting perceptually equivalent images. Existing optimization and feed-forward generators are useful but do not guarantee fair, diverse sampling, while deep CNN filters provide the paper's style representation.
- Julesz ensembles: A Julesz texture is a distribution over images whose local filter-response averages match characteristic values.The filter bank defines the statistics used to characterize the texture.
- Julesz ensembles: The Julesz ensemble contains texture images satisfying these constraints, and its images are treated as perceptually equivalent.The corresponding ideal distribution is uniform over the ensemble; an exponential form is used in practice.
- Generation by optimization: Generation-by-minimization finds a texture image, but random initialization and local non-convex optimization do not ensure fair sampling from the texture distribution.Different runs may land on different minimizers for incidental algorithmic reasons.
- Deep filter banks: Gatys et al.'s style loss uses empirical correlation matrices of deep CNN filter responses, with the filter bank and characteristic responses defining the texture.These learned filters are selected because they outperform earlier hand-designed filter candidates.
- Feed-forward generators: Feed-forward generators map random noise directly to optimization minima, greatly increasing speed, but they likewise lack a guarantee of fair texture-distribution sampling and tend to lack diversity.For stylization, the generator additionally takes a content image and minimizes texture plus content loss.
3. Julesz generator networks
The paper learns feed-forward generators that approximate sampling from the Julesz ensemble by combining style-loss matching with an entropy-based diversity objective. This targets the limited diversity of prior generators while retaining fast generation.
- Generation-by-optimization is faster than Monte Carlo sampling but generally finds individual ensemble points rather than samples from the ensemble.
- The proposed formulation trains generator networks to sample the Julesz ensemble with high visual fidelity and high diversity.The Julesz ensemble is the set of images sharing specified filter statistics.
- The target Gibbs distribution is approximated by minimizing the KL divergence between the generator distribution q and target distribution p.
- The KL objective combines expected style loss with the negative entropy of the generated distribution.
- The expected style loss is estimated over generated samples, while the negative-entropy term is estimated using nearest-neighbor distances within a batch.
- The resulting objective can be optimized with stochastic gradient descent over batches of noise samples passed through the generator.
4. Stylization with instance normalization
The paper replaces batch normalization with instance normalization in stylization generators to discard content-image contrast information. Instance normalization converges faster and produces results closer to iterative optimization than batch normalization.
- The stylization generator should discard contrast information from the content image because the output should match the texture’s contrast.
- Standard convolutional operators do not clearly implement the required contrast-normalization function, motivating a dedicated normalization layer.
- Instance normalization is introduced as a drop-in replacement for batch normalization that operates on individual instances.
- Instance normalization is applied throughout the architecture rather than only at the input, and Figure 2 compares this design with batch normalization and other variants.
5. Experiments
Experiments show that instance normalization improves stylization, while the diversity-inducing objective helps generators produce varied outputs without sacrificing texture quality. Combined, these methods improve feed-forward results relative to batch-normalized or low-capacity alternatives and approach iterative optimization.
- Effect of instance normalization: StyleNet IN converges faster than StyleNet BN and starts closer to the iterative-optimization solution, requiring less subsequent finetuning.The qualitative change after iterative optimization is small for IN but much larger for BN.
- Effect of instance normalization: IN is far superior to BN for stylization and much closer to Gatys et al.’s iterative results across content images and resolutions.The examples include 512 × 512 and 1024 × 1024 inputs; higher resolution generally yields visually better stylization.
- Effect of instance normalization: Instance normalization and batch normalization perform equally well for texture synthesis.The paper attributes the stylization-specific benefit of IN to normalizing highly variable content-image information.
- Effect of the diversity term: TextureNetV1 produces diverse but sometimes low-quality textures, whereas high-capacity TextureNetV2 produces higher-fidelity but nearly identical outputs under the baseline objective.TextureNetV1 is the low-capacity model used in prior work, while TextureNetV2 has higher capacity.
- Effect of the diversity term: TextureNetV2 with the diversity-inducing objective produces textures that are both high-quality and diverse.The diversity term is evaluated after validating the instance-normalization architecture.
- Effect of the diversity term: The diversity term also encourages stylization networks to produce different results from different noise inputs.Its weight must be tuned: values that are too small may fail to produce diversity, while values that are too large may create artifacts.
6. Summary
The paper advances feed-forward texture synthesis and stylization through instance normalization and a diversity-oriented learning formulation. The figures show improved diversity across texture generation and stylization settings.
- Instance normalization advances feed-forward texture synthesis and stylization networks.
- Texture synthesis: TextureNet V2 without a diversity term produces nearly identical textures, while TextureNet V1 achieves diversity but sometimes produces poor results.
- Image stylization: StyleNetV2 trained with λ > 0 generates substantially different stylizations for different input-noise values.