Source-linked AI summary
Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky
TL;DR
Existing descriptive networks can synthesize textures and stylize images but rely on slow, memory-intensive optimization. This paper trains compact feed-forward convolutional networks instead, achieving comparable texture quality with substantially greater efficiency.
Problem
Gatys et al.’s texture synthesis and style transfer methods require slow iterative optimization with high memory requirements.
Method
The paper trains compact feed-forward convolutional networks with expressive losses derived from pretrained descriptor CNN statistics for texture synthesis and image stylization.
Results
The generative approach produces textures with quality and diversity comparable to descriptive methods while being two orders of magnitude faster and one order of magnitude more memory efficient.
Takeaways & Limitations
Feed-forward networks can support complex image generation and stylization, including arbitrary-size processing and applications such as video-related tasks.
Takeaways & Limitations
For some styles, local convolutional operations cannot match the quality of the original optimization-based stylization because they cannot reproduce fixed global structure proportions.
Abstract
from arXiv · showhide
Gatys et al. recently demonstrated that deep networks can generate beautiful textures and stylized images from a single texture example. However, their methods requires a slow and memory-consuming optimization process. We propose here an alternative approach that moves the computational burden to a learning stage. Given a single example of a texture, our approach trains compact feed-forward convolutional networks to generate multiple samples of the same texture of arbitrary size and to transfer artistic style from a given image to any other image. The resulting networks are remarkably light-weight and can generate textures of quality comparable to Gatys~et~al., but hundreds of times faster. More generally, our approach highlights the power and flexibility of generative feed-forward models trained with complex and expressive loss functions.
1. Introduction
The paper replaces slow, memory-intensive optimization for texture synthesis and stylization with compact feed-forward generation networks. These networks aim to match descriptive methods’ quality while improving efficiency and supporting arbitrary-size outputs.
- 1. Introduction: Gatys et al.’s optimization-based image generation requires iterative backpropagation, takes several seconds for small images, and scales poorly because of memory demands.Feed-forward generation instead requires a single network evaluation without backpropagation.
- 1. Introduction: The proposed generative approach produces textures with quality and diversity comparable to descriptive methods.The comparison is stated for texture synthesis against Gatys et al.’s method.
- 1. Introduction: The method is two orders of magnitude faster and one order of magnitude more memory efficient than the descriptive approach.The paper attributes this efficiency to compact networks and single-pass generation.
- 1. Introduction: The authors introduce a multi-scale generative architecture tailored to texture synthesis and image stylization.The networks are fully convolutional, enabling texture generation and image processing at arbitrary sizes.
- 1. Introduction: The paper presents qualitative comparisons on challenging textures and images to evaluate the proposed approach.These comparisons are described as extensive and qualitative.
2. Background and related work
The background frames texture synthesis and style transfer as sampling or pre-image problems defined by image statistics. It reviews generator, adversarial, moment-matching, and descriptive texture models, emphasizing diversity and sampling difficulty as central concerns.
- 2. Background and related work: Texture synthesis samples images matching statistics extracted from one example, whereas style transfer matches a source style with a second image’s content.The corresponding conditional distributions are written as p(x|x0) and p(x|x0, x1).
- 2. Background and related work: Descriptive methods formulate generation as finding a pre-image x whose statistics Φ(x) approximately match those of a reference image x0.Sampling different pre-images provides diversity because the pre-image is usually not unique.
- 2. Background and related work: Deep descriptive approaches derive statistics from responses in pretrained neural-network layers, including correlations of convolutional maps.The paper reuses these correlation-based statistics.
- 2. Background and related work: Classical descriptive texture models use spatially invariant local statistics, while FRAME induces a maximum-entropy texture distribution whose marginals match empirical statistics.FRAME’s sampling difficulty motivated direct pre-image methods such as Portilla and Simoncelli’s approach.
- 2. Background and related work: Generator networks map random or deterministic parameters z directly to images x, but deterministic mappings require known parameter-image relations and may not express additional diversity.Random-input generators instead require a sample-quality measure that generalizes from available examples without penalizing diversity.
- 2. Background and related work: GANs use an adversarial network to distinguish generated samples from natural images, while moment-matching networks compare statistics averaged over generated and empirical samples.The paper positions its networks as similar to moment-matching networks but with different statistics and applications.
3. Texture networks
Texture networks train compact feed-forward generators for texture synthesis and style transfer, using losses derived from statistics in a fixed descriptor CNN. Multi-scale architectures generate arbitrary-sized outputs efficiently, while stylization combines texture matching with content preservation.
- A separate generator maps noise to texture samples, or maps noise and a content image to a stylized output.Once trained, each generator can synthesize arbitrarily many images of arbitrary size in a single feed-forward process.
- The reference architecture uses multiple noise tensors at different resolutions, convolutional blocks, upsampling, concatenation, and a final mapping to an RGB image.The implementation uses circular convolution for texture boundaries, with roughly 65K parameters.
- The training loss compares statistics from generated and reference images using a fixed pretrained descriptor CNN and its convolutional feature correlations.Gram matrices provide orderless correlations of local features for texture matching.
- Style-transfer training combines texture loss with content loss, whose spatially corresponding feature activations preserve content information.The content layers are chosen deeper than the texture layers so that high-level content is preserved.
- Multi-scale generator architectures produce lower texture loss and better perceptual quality while using fewer parameters and training faster than simpler chains.Different-resolution branches process noise tensors and join them through upsampling and channel concatenation.
- Stylization training uses a small pool of natural images, but local convolutional operations can struggle to match fixed global texture proportions across content images.Despite this limitation, stylized outputs are usually perceptually good, though some styles remain below optimization-based quality.
4. Experiments
The experiments compare texture synthesis and style transfer against optimization-based methods, showing comparable qualitative results with substantially faster feed-forward generation. The method also permits noise-controlled style-content trade-offs and uses markedly less memory.
- Style transfer: Scaling the input noise changes the balance between style and content without retraining the network.This provides test-time control despite the trade-off parameter not being directly available.
- Texture synthesis: The generator CNN produces texture results qualitatively comparable to Gatys et al. and superior to the other compared methods.The comparison covers two challenging textures and includes DCGAN and Portilla–Simoncelli baselines.
- Style transfer: Style-transfer results are qualitatively comparable to Gatys et al. across numerous style and content images.The comparisons use a fixed trade-off parameter and include examples from the earlier work.
- Speed and memory: 500× speed-up is achieved: feed-forward generation takes about 20ms, whereas iterative optimization takes about 10 seconds for comparable loss.The speed difference is attributed to a smaller generator, a single network evaluation, and avoidance of backpropagation.
- Speed and memory: 170 MB versus 1100 MB is used to generate a 256 × 256 sample, favoring the feed-forward method over Gatys et al.The memory comparison is reported alongside the speed evaluation.
5. Discussion
The paper presents feed-forward texture synthesis and image stylization that matches the texture-synthesis capability of iterative optimization. It identifies complex loss functions and expert descriptor networks as central to this result, while noting remaining stylization-quality gaps.
- Discussion: The approach generates complex textures and images purely feed-forward while matching Gatys et al.’s texture-synthesis capability.Gatys et al.’s method uses multiple forward-backward iterations, whereas this approach does not.
- Discussion: Complex loss functions using feed-forward architectures as experts assess the generator and support complex data generation.The descriptor networks provide the performance assessment used during generator training.
- Discussion: The authors plan better stylization losses because some current cases are less impressive than Gatys et al.’s stylization results.The limitation is stated for stylization quality, while texture synthesis is described as generally very good.
Supplementary material
The supplementary results examine texture-network behavior, including overfitting, multi-scale generation, arbitrary-resolution synthesis, and comparisons with Gatys et al. They also show that style-transfer quality can remain inferior for some styles.
- Additional qualitative results compare texture networks with Gatys et al. across texture synthesis and style transfer.
- Overfitted generators can produce samples with shared structure and a preference for diagonal, horizontal, and vertical lines.The shared structure appears even though individual stars differ in detail; the effect is associated with 3x3 filters.
- Lower texture loss does not guarantee preferable samples because generators may produce low-variance outputs, with no obvious validation-set control for this degradation.Increasing the noise amount can help in certain cases.
- In the multi-scale architecture, the effective input depth depends on texture-element size: depth K = 4 suffices for peppers, while starry night uses the deepest input blob.The generator is limited by VGG capacity and cannot capture larger texture elements.
- Texture networks trained on 256x256 samples can generate textures at different resolutions, reflecting the architecture’s fully convolutional design.Figure 10 shows samples at a different resolution, while Figure 14 applies a network trained on 256×256 images to a 1024×768 image.
- For some styles, the proposed style-transfer results are inferior to Gatys et al., leaving improvement from more complex losses or deeper networks unresolved.