Source-linked AI summary
Texture Synthesis Using Convolutional Neural Networks
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
TL;DR
The paper develops a texture model based on feature spaces from object-recognition CNNs, representing textures through correlations across layers. It reports improved texture quality over prior parametric synthesis and increasingly explicit object information across layers.
Problem
The paper addresses how to represent natural textures using feature spaces from convolutional neural networks trained for object recognition.
Source: §1 · p. 2Method
Results
The model substantially improves texture quality over prior parametric synthesis, while natural-image statistics and decodable object information increase across layers.
Source: §5 · pp. 6–7, §6 · p. 8Takeaways & Limitations
The layered representation connects natural-texture statistics with object information encoded in CNN feature representations.
Source: §5 · pp. 6–7The synthesis procedure is computationally more expensive than non-parametric texture synthesis, despite comparable texture quality.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition. Samples from the model are of high perceptual quality demonstrating the generative power of neural networks trained in a purely discriminative fashion. Within the model, textures are represented by the correlations between feature maps in several layers of the network. We show that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit. The model provides a new tool to generate stimuli for neuroscience and might offer insights into the deep representations learned by convolutional neural networks.
1 Introduction
Texture synthesis seeks to generate new samples that remain perceptually indistinguishable from an example. The paper proposes a parametric model using spatially invariant representations from a CNN trained for object recognition.
1 Introduction
- Texture samples are successful when human observers cannot distinguish them from the original texture.Human inspection is the usual quality criterion described here.Source: §1 · p. 1
- Non-parametric pixel or patch resampling can produce high-quality natural textures efficiently, but does not define an explicit texture model.It provides a procedure for randomizing a source texture while preserving perceptual properties.Source: §1 · p. 1
- The Portilla–Simoncelli parametric model uses handcrafted statistics of Steerable Pyramid filter responses, yet still fails to capture the full scope of natural textures.The paper presents its CNN-based model as an alternative to early-vision-inspired models.Source: §1 · p. 2
- The proposed parametric texture model builds on hierarchical, spatially invariant representations from a CNN trained for object recognition.It combines feature-response summary statistics with the CNN feature space.Source: §1 · p. 2
2 Convolutional neural network
The model uses VGG-19’s convolutional feature space, excluding its fully connected layers. Its alternating convolutions and pooling operations support image inputs of arbitrary size.
2 Convolutional neural network
- The feature space uses VGG-19’s 16 convolutional and 5 pooling layers, excluding all fully connected layers.The network was trained on object recognition and previously described in [25].Source: §2 · p. 3
- Linearly rectified 3 × 3 × k convolutions use stride and padding of one, preserving feature-map dimensions, while non-overlapping 2×2 max pooling halves them.Here k is the number of input feature maps.Source: §2 · p. 3
- Convolutions and pooling alternate; after each of the first three pooling layers, feature-map count doubles as spatial resolution halves.Using only convolutional layers allows arbitrarily large input images; average pooling gave slightly cleaner synthesis results than max pooling.Source: §2 · p. 3
3 Texture model
The texture model summarizes CNN feature responses with layerwise correlations that discard spatial position. A set of these Gram matrices provides a stationary description used to specify a texture.
3 Texture model
- The model extracts features at multiple scales, computes a stationary summary statistic, then uses gradient descent from white noise to match it.This procedure follows the conceptual framework of Portilla and Simoncelli.Source: §3 · p. 3
- Unlike Portilla and Simoncelli’s linear filter bank and handcrafted statistics, this model uses a deep network’s feature space and one statistic: within-layer feature-response correlations.The network is trained for object recognition.Source: §3 · p. 3
- Because textures are stationary, the model discards spatial information by summarizing correlations between feature responses.A layer’s filters produce feature maps, with each map vectorized across its spatial positions.Source: §3 · p. 3
- A set of Gram matrices from selected network layers provides the stationary description that fully specifies a texture in this model.The Gram matrix entries encode inner products between feature maps.Source: §3 · p. 4
4 Texture generation
Texture generation minimizes the distance between original and generated Gram-matrix representations, starting from white noise. Layer contributions are weighted, and pixel gradients are computed through backpropagation for numerical optimization.
4 Texture generation
- Starting from white noise, gradient descent minimizes the mean-squared distance between original and generated Gram-matrix entries.This optimization seeks an image matching the source image’s Gram-matrix representation.Source: §4 · p. 4
- The total objective combines layer-specific losses using weights that control each layer’s contribution.Each layer’s loss measures the difference between the original and generated Gram-matrix representations.Source: §4 · p. 4
- Analytic layer-loss derivatives are propagated to image pixels by standard backpropagation; L-BFGS then optimizes the high-dimensional image efficiently on GPUs.The procedure relies mainly on standard CNN forward and backward passes.Source: §4 · p. 4
5 Results
Matching texture representations across deeper VGG layers produces increasingly natural textures, while object identity becomes increasingly explicit in the representations. The model also admits substantial parameter reduction, though texture quality depends on architecture and learned features.
5 Results
- Matching through pool4 yields complex natural textures almost indistinguishable from originals; lowest-layer-only matching produces little structure.The rows progressively include constraints from more layers, with labels showing the highest included layer.Source: §5 · pp. 4, 6, Figure 2 · p. 5
- For non-texture images, synthesis preserves local spatial information but discards global arrangement; preserved regions grow with layer depth as receptive fields increase.Source: §5 · p. 6
- Some VGG feature maps encode image boundaries, possibly because of zero-padding, which may explain why a source's yellow-box location recurs in synthesized textures.Source: §5 · p. 6
- ∼177k parameters across one layer per scale retain nearly full quality; 64-component PCA reduces this to ∼10k, while layer averages alone (1,024) still produce interesting textures.Using all layers through pool4 requires ∼852k parameters; finding a minimal set that preserves full-model quality remains future work.Source: §5 · pp. 6–7
- The caffe reference network produces less accurate texture statistics and an artifactual grid, attributed to its larger receptive fields and strides.Source: Figure 3 · p. 6, §5 · p. 7
- Texture synthesis fails with random VGG weights, underscoring that learned feature spaces, as well as network architecture, matter.Source: Figure 3 · p. 6, §5 · p. 7
- 87.7% vs. 88.6% top 5 accuracy: pool5 Gram-matrix features approach full VGG-19 performance, with object identity increasingly decodable across layers.The classifiers were trained and evaluated on 224 × 224 centre crops without data augmentation.Source: §5 · pp. 7–8, Figure 4 · p. 7
6 Discussion
The model substantially improves parametric texture synthesis quality over the current state of the art. Its quality is comparable to non-parametric methods but comes at greater computational cost, while its feature representation and potential use in neuroscience motivate further applications.
6 Discussion
- The model substantially improves synthesized-texture quality over the current state of the art in parametric texture synthesis.The paper makes this comparison using Fig. 2, fourth row versus last row.Source: §6 · p. 8
- Texture synthesis with the model is computationally more expensive than non-parametric synthesis despite producing comparable-quality natural textures.The authors expect progress in making deep convolutional networks more efficient to transfer to their synthesis procedure.Source: §6 · p. 8
- The Gram-matrix representation makes CNN features stationary, but the authors did not evaluate it on texture-recognition benchmarks.They cite prior work on stationary feature representations and expect their representation may also work well for texture recognition.Source: §6 · p. 8
- The authors propose the model as a potential tool for studying visual information processing in biological systems.They connect this prospect to the use of biologically inspired texture models in neuroscience and to similarities between CNNs and visual neural systems.Source: §6 · p. 8