Source-linked AI summary
Texture Synthesis with Spatial Generative Adversarial Networks
Nikolay Jetchev, Urs Bergmann, Roland Vollgraf
TL;DR
Texture synthesis needs models that generate high-quality new textures without conventional GANs’ fixed output dimensions. The paper introduces SGAN, which uses a spatial noise tensor and fully convolutional adversarial networks to synthesize scalable textures. Experiments report high-quality generation, real-time single-pass synthesis, arbitrary output sizes, and fusion of diverse source images, while regular non-mixing textures remain a limitation.
Problem
Conventional GANs hard-code output image size, while texture synthesis may require much larger textures and outputs of multiple or arbitrary sizes.
Method
SGAN extends GAN noise from a single vector to a spatial tensor and uses purely convolutional generator and discriminator architectures.
Results
SGAN produces high-quality textures with real-time single-pass generation, arbitrary desired output sizes, and combinations of separate source images into complex textures.
Takeaways & Limitations
The method provides a scalable data-driven approach for synthesizing large textures and combining diverse source images.
Takeaways & Limitations
SGAN cannot learn regular non-mixing textures because distant output pixels become independent under its strong mixing generation.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) are a recent approach to train generative models of data, which have been shown to work particularly well on image data. In the current paper we introduce a new model for texture synthesis based on GAN learning. By extending the input noise distribution space from a single vector to a whole spatial tensor, we create an architecture with properties well suited to the task of texture synthesis, which we call spatial GAN (SGAN). To our knowledge, this is the first successful completely data-driven texture synthesis method based on GANs. Our method has the following features which make it a state of the art algorithm for texture synthesis: high image quality of the generated textures, very high scalability w.r.t. the output texture size, fast real-time forward generation, the ability to fuse multiple diverse source images in complex textures. To illustrate these capabilities we present multiple experiments with different classes of texture images and use cases. We also discuss some limitations of our method with respect to the types of texture images it can synthesize, and compare it to other neural techniques for texture generation.
2 Introduction
Texture synthesis seeks to infer a generating process from example images, but existing approaches face trade-offs in modeling, computational cost, and output-size flexibility. The paper introduces SGAN, a fully convolutional adversarial model designed to synthesize textures at arbitrary sizes.
- Background: Texture synthesis infers a generating process from an example texture to produce new samples, judged mainly by visual quality and similarity.Application-specific criteria also include synthesis speed, diversity, arbitrary output size, and smooth texture morphing.
- Existing approaches: Non-parametric methods preserve perceptual properties by resampling pixels or patches but do not learn texture models and can be slow for large outputs.Their search routines make synthesis time-consuming as texture size increases.
- Existing approaches: Descriptor-based methods match image statistics, while Gatys et al. use pretrained convolutional filters and iterative optimization to generate expressive textures.Gatys-style optimization is costly in time and memory for each output texture.
- Problem: GANs offer high-quality image generation, but conventional architectures hard-code output dimensions and cannot directly produce arbitrary texture sizes.Pyramid-based extensions require increasingly complex models and predefined scale levels.
- Contribution: The paper introduces a fully convolutional adversarial model that learns texture images as the data distribution and supports desired output texture sizes.The authors present SGAN as a novel application of adversarial methods to arbitrarily large texture synthesis.
3 The SGAN Model
SGAN replaces the conventional GAN’s single noise vector with a spatial noise tensor and uses convolutional generator and discriminator networks. This design yields translation-invariant, strongly mixing generated data while enabling spatially scalable synthesis.
- Spatial generator: SGAN maps a spatial noise tensor Z ∈ R^l×m×d to an RGB image X ∈ R^h×w×3.Each spatial slice z_λµ is independently sampled from the prior distribution in the experiments.
- Spatial discriminator: The discriminator receives either a generated image or an equally sized random patch from a training image and outputs a two-dimensional real/fake probability field.Matching generated and real patch sizes was necessary for successful training.
- Scalability: Both networks are purely convolutional, allowing spatial dimensions to change without modifying weights and permitting larger images at deployment.A model trained on small images can generate much larger images matching local training statistics.
- Training: The model optimizes discriminator and generator objectives simultaneously over all spatial dimensions.The objective contains separate contributions for generated outputs and real image patches, with a practical non-saturating generator-loss variant.
- Stochastic-process properties: Because the generator is purely convolutional and spatial noise slices are identically distributed, SGAN generates translation-invariant data and is stationary with respect to translations.Non-overlapping output patches with non-overlapping projective fields are independently generated, establishing strong mixing and ergodicity.
4.1 Architectural details and speed
SGAN uses a convolutional architecture whose configurations vary in depth and spatial dimensions, enabling fast texture generation through a single generator forward pass. Training requires texture-dependent time and monitoring because overtraining can degrade image quality.
- Architecture: SGAN configurations use convolutional layers with stride 1 in the generator and stride 2 in the discriminator, with 5x5 kernels and zero padding.
- Architecture: Table 1 covers SGAN architectures with 4, 5, or 6 layers and reports their projective and receptive field sizes.
- Speed: Table 2 reports that SGAN is faster than TextureNet and Gatys, while time costs per calculated pixel scale sublinearly.
- Architecture: SGAN uses spatial dimensions h = rl and w = rm for the noise and image patches, with either 640x640 spatial dimensions or l = m = 4 working similarly well.
- Speed: 5ms is the reported SGAN forward-pass time at 256-pixel resolution, compared with 20ms for TextureNet despite SGAN using more filters.Generation with the trained generator is described as very fast because it requires a single forward pass.
- Training: Simple textures approach final subjective quality after roughly 10 minutes of training, whereas more complex textures require around 30 minutes.
- Training: Training time comparisons with TextureNet are indirect because the methods were not tested on the same machine and textures.The paper assumes SGAN trains more efficiently because its architecture is simpler.
- Training: GAN training may require monitoring and stopping because overtraining can cause degeneracy or image-quality degradation.
4.2 Examples of generated textures
SGAN generates textures from small and large single images, preserves salient spatial structure in satellite imagery, and can fuse multiple source images. Network depth controls mixing and structure, while composite results retain natural variety but may not blend smoothly.
- Single small image: SGAN and Gatys produce comparable results on small textures across different output sizes.The SGAN4 receptive field is 61 pixels, shown relative to the texture image sizes.
- Single large image: SGAN preserves Barcelona street-grid structure better than Gatys in generated satellite textures.The SGAN spatial autocorrelation resembles the original, whereas Gatys’ result is more isotropic and loses street-grid information.
- Single large image: Deeper SGAN models generate longer streets with less mixing and greater regularity at a given scale.Increasing depth from four to six layers increases receptive fields and allows larger structures to be learned.
- Composite textures from multiple images: SGAN can combine diverse flower images into natural-looking composite textures, but single-flower training data prevents smooth blending.The flowers experiment used 8,189 images and generated 1000x1000-pixel composites.
- Composite textures from multiple images: For five Amsterdam satellite images, SGAN combines orientations from all inputs and learns smooth spatial transitions between them.The angled segments join smoothly without data augmentation.
- Composite textures from multiple images: GANs naturally fuse several images by learning patch statistics, whereas descriptor-based methods are not straightforward to extend beyond one target image.This difference supports SGAN’s use for composite texture learning.
4.3 Extension using properties of the spatial GAN
SGAN’s local dependence on the spatial noise tensor enables seamless synthesis and memory-efficient generation. Large outputs can be computed in independent chunks, keeping GPU memory usage constant while storage remains the limiting factor.
- Spatial locality: Local independence in SGAN means each output pixel depends only on a subset of the spatial noise tensor.This property supports constructing textures with special properties.
- Seamless textures: Seamless textures can be synthesized at a desired size by matching the spatial dimensions of the noise tensor and generator output.The method defines a common scale ratio r between output and noise dimensions.
- Memory-efficient generation: Splitting the noise tensor lets SGAN generate large textures in independent chunks that use less GPU memory and can be parallelized.The chunks overlap before their edges are cropped and concatenated.
- Memory-efficient generation: The chunked procedure keeps GPU memory usage constant, leaving the number of storable pixels as the only limitation.The paper identifies real-time 3D engines as a potential application for generating visible parts of arbitrarily large textures.
5 Discussion
SGAN generates textures from locally consistent patches, offering scalable, fast synthesis and source-image combination while remaining limited by spatial dependence and regularity constraints.
- Method: SGAN learns locally consistent image patches, with mixing length determined by projective-field size and architecture choice depending on the texture.The paper notes a similar architecture-dependence for Gatys et al.'s parametric method through its selected network layers.
- Special properties: Periodic boundary conditions on Z enable straightforward seamless texture tiling.The illustrated 320x320 texture is tiled twice vertically and six times horizontally.
- Limitations: SGAN cannot learn statistical dependencies exceeding the projective-field size and struggles with regular non-mixing textures such as chess grids.For text images, it learns letter shapes but fails to align them into globally straight rows.
- Capabilities: SGAN supports real-time generation of high-quality textures in a single forward pass.
- Scalability: SGAN generates images of any desired size, with processing time scaling linearly with output pixels.
- Capabilities: SGAN can combine separate source images into complex textures.
- Future work: Future work targets longer spatial correlations, stronger regularity, conditional control, and applications including style transfer, in-painting, 3D data, and audio.Conditioning could support one network for multiple textures and blending them into novel textures.
Appendix I
Appendix I analyzes projective fields in SGANs and proves how spatial locality permits splitting large computations, including periodic inputs for seamless outputs.
- Projective fields: A single convolutional layer maps input interval [a, b) to output interval [2a − 2, 2b + 1).The interval-size relation is b′ − a′ = 2(b − a) + 3.
- Projective fields: After k convolutional layers, [a, b) maps to [a2^k − 2^(k+1) + 2, b2^k + 2^k − 1).
- Projective fields: A single input element has projective-field size PF(k) = 2^(k+2) − 3.This quantity is denoted PF/RF in Table 1.
- Splitting large computations: Two smaller input volumes can reproduce the corresponding halves of a large output exactly when their spatial inputs overlap by two elements.The appendix proves that this overlap is sufficient and necessary for the usual architecture.
- Periodic textures: For periodic textures, making Z periodic makes the output I periodic, requiring four identical elements along each dimension to provide two-element border overlaps.The construction sets Z[:,−4:] = Z[:,:4] and Z[−4:,:] = Z[:4,:].