Source-linked AI summary
Non-Stationary Texture Synthesis by Adversarial Expansion
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, Hui Huang
TL;DR
Non-stationary textures with large-scale structures and spatial variation remain difficult for example-based synthesis. The paper trains a GAN to expand exemplar blocks, producing larger textures that preserve global arrangements and handle challenging examples.
Problem
Example-based synthesis remains an open problem for significantly non-stationary textures, including textures with large-scale structures, spatial variance, and inhomogeneity.
Method
A generative adversarial network trains a fully convolutional generator to expand k×k exemplar blocks into perceptually similar 2k × 2k blocks.
Results
The method captures and extends global structures in challenging non-stationary exemplars and handles an unprecedented array of examples that existing methods cannot handle, according to the authors.
Takeaways & Limitations
Once trained for an exemplar, the generator can expand the entire exemplar or sub-blocks in a single feed-forward pass and can also support texture transfer.
Takeaways & Limitations
The method may fail when training provides too few examples of a particular large-scale structure or pattern.
Abstract
from arXiv · showhide
The real world exhibits an abundance of non-stationary textures. Examples include textures with large-scale structures, as well as spatially variant and inhomogeneous textures. While existing example-based texture synthesis methods can cope well with stationary textures, non-stationary textures still pose a considerable challenge, which remains unresolved. In this paper, we propose a new approach for example-based non-stationary texture synthesis. Our approach uses a generative adversarial network (GAN), trained to double the spatial extent of texture blocks extracted from a specific texture exemplar. Once trained, the fully convolutional generator is able to expand the size of the entire exemplar, as well as of any of its sub-blocks. We demonstrate that this conceptually simple approach is highly effective for capturing large-scale structures, as well as other non-stationary attributes of the input exemplar. As a result, it can cope with challenging textures, which, to our knowledge, no other existing method can handle.
1 INTRODUCTION
Example-based synthesis remains difficult for non-stationary textures because preserving global structures and spatially varying local patterns is challenging. The paper proposes exemplar-specific adversarial expansion to address this gap.
- Non-stationary textures contain large-scale irregular structures, spatially varying attributes, or inhomogeneities that challenge example-based synthesis.
- Successful synthesis must preserve global structures while reproducing how local patterns vary across the spatial domain.
- The generator learns to double arbitrary texture blocks so expanded outputs resemble enclosing exemplar blocks, with similarity assessed by a discriminator.
- A trained fully convolutional generator expands the entire exemplar or sub-blocks, and can synthesize still larger textures by reusing its outputs.
- The method also supports texture transfer, preserving the guiding pattern’s large-scale structure when applied to another image or texture.
- The authors report that adversarial training handles an unprecedented array of highly non-stationary textures that existing methods cannot handle.
2 RELATED WORK
Prior texture synthesis methods reproduce many local or stationary patterns but struggle with large-scale, global, and inhomogeneous structure. Deep and GAN-based approaches broaden capabilities without resolving this challenge generally.
- Existing methods are unable to cope reliably with highly inhomogeneous textures or textures exhibiting large-scale or global structures.
- State-of-the-art classical approaches reproduce small- or middle-scale structure but fail to capture large-scale or global exemplar structure.
- Classical non-parametric methods typically assume a stationary MRF model, making highly inhomogeneous textures difficult without guidance maps.
- Methods explicitly modeling regular or near-regular deviations can address certain global structures, whereas this paper’s method makes no structural assumptions.
- Gram-based deep synthesis uses feature-channel correlations, but its loss captures regularities and larger structures poorly and remains inadequate for non-stationary structures.
- GAN-based texture methods generate patches or use periodic noise, but prior approaches had inconsistent quality or focused on periodic textures.
3 OUR APPROACH
The approach trains a fully convolutional generator to expand exemplar crops into larger enclosing blocks using self-supervised adversarial training. This enables exemplar-wide synthesis while combining perceptual and pixel-level losses.
- A k × k source block is mapped to a perceptually similar 2k × 2k enclosing target block taken directly from the exemplar.
- Training is self-supervised because target extended blocks are extracted from the input texture itself.
- The fully convolutional generator can process the entire exemplar or a sufficiently large portion to produce textures up to twice the input size.
- A dedicated GAN is trained for each exemplar, requiring considerable computational resources, although synthesis after training uses a single fast feed-forward pass.
- The method combines adversarial, L1, and style losses while the discriminator classifies real exemplar crops versus synthesized blocks.
- Training block size must balance capturing non-stationarity against retaining enough distinct target blocks for training; the implementation uses k = 128.
Network architecture
The generator uses a fully convolutional encoder–residual–decoder design, while a PatchGAN discriminator emphasizes spatial arrangement. Feature visualizations indicate that new large-scale structures emerge through the residual chain.
- Network architecture: Fully convolutional design supports arbitrary-sized test inputs, reduces parameters, and increases receptive-field-based capacity for large-scale non-stationarity.
- Feature-map analysis: New structure creation is typically complete before the residual chain ends, as shown by similar resblock_5 and resblock_6 feature maps.
- Network architecture: The generator combines convolutional downsampling, six residual blocks, channel expansion, and strided deconvolution to enlarge spatial dimensions.
- Feature-map analysis: The residual chain is where new large-scale structures emerge, with their number roughly doubling by resblock_6.
- Feature-map analysis: For leaf veins and wood rings, the generator learns horizontal or radial shifts after replication, respectively.
- Discriminator: PatchGAN replaces a loss network because it is more sensitive to spatial arrangement and spatially variant texture appearance.
Training procedure
Training alternates discriminator and generator updates on exemplar-derived target blocks, using adversarial, L1, and style losses. The trained network expands unseen blocks and supports texture transfer.
- Training alternates single discriminator and generator iterations while randomly selecting 256 × 256 target blocks from the exemplar.
- The network plausibly expands unseen texture blocks that differ in size and content from training examples, rather than merely memorizing patterns.
- The method also supports texture transfer, as demonstrated in Figure 15.
- The objective combines adversarial, L1, and style losses, with λ1 = 100 and λ2 = 1.The adversarial term is described as the main workhorse; L1 and style terms stabilize training and slightly reduce artifacts.
- Style loss uses Gram matrices from ReLU feature maps in five VGG-19 layers, weighted 0.244, 0.061, 0.015, 0.004, and 0.004.The VGG-19 model is pre-trained on ImageNet, and the layers range from relu1_1 through relu5_1.
4 RESULTS
The method captures and extends large-scale structure in challenging non-stationary textures, while also supporting diverse outputs, repeated expansion, and texture transfer. Comparisons and self-evaluations show strong results, with a 142×142 PatchGAN selected and rapid post-training synthesis.
- Computation: 4–5 milliseconds are required to double a 600×400 texture after training through one feed-forward generator pass.Training takes about 5 hours for 100,000 iterations in the reported implementation, though results often stop improving after roughly 36,000 iterations.
- Results: The method successfully captures and extends global structures in challenging non-stationary textures and also applies to regular, near-regular, and stochastic textures.The reported examples include irregular large-scale structures and inhomogeneities, alongside more stationary texture classes.
- Comparison: Compared with state-of-the-art methods, the approach handles large-scale non-stationarity better while producing comparable results on stationary or homogeneous textures.Figure 7 compares the method with classical optimization-based and recent deep learning-based approaches.
- Diversification: Different cropped subwindows or reshuffled tiles produce diversified expansion results from the same trained generator.Cropping generates distinct 512×512 outputs from 256×256 inputs, while tile shuffling diversifies results for sufficiently stationary textures.
- Ablation: Adversarial loss is central to expansion quality, while combining adversarial, L1, and style losses gives the best reported results.Without adversarial training, large-scale structures are not introduced and smaller structures are reproduced poorly; L1 reduces artifacts but can oversmooth, whereas style loss enhances details while adding artifacts and color distortions.
- PatchGAN: The method uses a 142×142 PatchGAN, balancing examination of larger structures with attention to local details.Larger receptive fields examine larger structures but very large patch sizes appear to reduce attention to local details.
- Self evaluation: Five expansion cycles enlarge a 64×64 crop to 2048×2048 while preserving the wood rings’ radial structure.The generator is reused across cycles without retraining, and the reported fourfold wood-ring expansion maintains the radial structure.
- Texture Transfer: Feeding a guiding image into a trained generator transfers its large-scale structure to the synthesized texture without modifying the method.The transferred result has twice the resolution of the guiding image, but no new large-scale structures are introduced in this setting.
5 SUMMARY
The method expands exemplar textures while preserving their global structures, but remains limited by training coverage, border artifacts, and training time. It also has documented failure cases for distinctive large-scale patterns.
- The method expands exemplar textures while faithfully preserving their global structures and local patterns.The generator learns to expand small exemplar subwindows into larger windows containing them.
- The approach can synthesize textures that adapt to large-scale structures in guiding images, including sketches and Perlin noise.
- Training time remains a limitation, with models trained for 100,000 iterations although results often stop improving after about 36,000.
- Artifacts tend to emerge near borders and corners, possibly because those regions have fewer training examples and convolutional padding effects.
- The method fails when training provides too few examples of distinctive large-scale structures, including stone-tile layouts and a sunflower-center singularity.
Non-Stationary Texture Synthesis by Adversarial Expansion • 49:13
The cited literature spans example-based texture synthesis, generative adversarial networks, image translation, texture analysis, and optimization. It includes foundational and application-oriented works across these areas.
- Several cited works use generative adversarial networks for texture synthesis or image translation.The bibliography includes GANs, Markovian GANs, spatial GANs, conditional adversarial networks, and cycle-consistent adversarial translation.
- The references also cover feed-forward texture or style synthesis, super-resolution, perceptual losses, and deep image representations.
- The references include foundational work on example-based texture synthesis and texture analysis.Examples include tree-structured vector quantization, graph-cut synthesis, texture optimization, and near-regular texture analysis.