Source-linked AI summary
Learning Texture Manifolds with the Periodic Spatial GAN
Urs Bergmann, Nikolay Jetchev, Roland Vollgraf
TL;DR
Texture synthesis methods have been limited in representing periodic textures, learning from multiple images, and producing plausible blended textures. PSGAN addresses these gaps by structuring the GAN noise tensor with spatial, global, and periodic dimensions. It learns diverse textures and supports smooth texture transitions, periodic-pattern synthesis, and arbitrarily large outputs, while representing only translational symmetries.
Problem
Existing texture-synthesis methods can reproduce periodic textures inaccurately, generally learn from a single example image, and have not demonstrated novel blended textures.
Method
PSGAN extends GANs by mapping a structured noise tensor to images with spatial, global, and periodic dimensions, using fully convolutional generator and discriminator architectures.
Results
PSGAN learns periodic and non-periodic textures from single images or large datasets, generates smoothly transitioning novel textures, and scales to potentially infinite output size.
Takeaways & Limitations
PSGAN supports flexible texture sampling across diverse image sources, including accurate periodic textures and texture manifolds with perceptual interpolation.
Takeaways & Limitations
PSGAN represents translational symmetries but not rotation or reflection symmetries.
Abstract
from arXiv · showhide
This paper introduces a novel approach to texture synthesis based on generative adversarial networks (GAN) (Goodfellow et al., 2014). We extend the structure of the input noise distribution by constructing tensors with different types of dimensions. We call this technique Periodic Spatial GAN (PSGAN). The PSGAN has several novel abilities which surpass the current state of the art in texture synthesis. First, we can learn multiple textures from datasets of one or more complex large images. Second, we show that the image generation with PSGANs has properties of a texture manifold: we can smoothly interpolate between samples in the structured noise space and generate novel samples, which lie perceptually between the textures of the original dataset. In addition, we can also accurately learn periodical textures. We make multiple experiments which show that PSGANs can flexibly handle diverse texture and image data sources. Our method is highly scalable and it can generate output images of arbitrary large size.
1. Introduction
Texture synthesis seeks to generate new images with properties of a source texture, but existing approaches face limitations in scalability, periodic structure, and blending multiple textures. PSGAN extends GAN-based synthesis to learn diverse textures from single images or datasets while supporting flexible sampling and arbitrary output sizes.
- Texture synthesis goals: Texture synthesis learns a generating process from example images so new images retain similar properties.Instance-based methods resample pixels or patches, while parametric methods optimize images against specified statistical criteria.
- Limitations of prior methods: Classical instance-based methods handle periodic images and produce good visual quality, but generating large textures has high runtime complexity.These methods resample and copy pixels or patches next to similar regions to form larger seamless textures.
- Limitations of prior methods: Parametric methods can synthesize periodic textures, but their runtime remains high and their texture blending produces patch-wise mixtures rather than homogeneous interpolations.Portilla and Simoncelli’s method includes phase variables for periodicity, while reported blending results did not form new homogeneous textures.
- Deep learning approaches: Deep texture-synthesis methods use learned descriptors or GANs, yet Gram-matrix approaches reproduce periodic textures inaccurately and generally learn from only one example image.SGAN improves scalability and performs well on some classes, but cannot handle certain textures or plausible texture morphing.
- PSGAN contribution: PSGAN learns periodic and non-periodic textures unsupervised from single images or large image datasets, then samples novel textures with smooth transitions and potentially infinite output size.The approach is introduced as extending the types of images neural texture synthesis can create.
2. Methods: Periodic GAN
PSGAN extends convolutional GANs with structured spatial noise and a spatial discriminator to model local variation, global texture identity, and periodicity. These components support scalable synthesis, texture selection and interpolation, and long-range periodic structure.
- Spatial GAN architecture: PSGAN maps a spatial noise tensor Z ∈ R^L×M×d to an image and uses a spatial discriminator field whose responses assess local image regions.The generator expands spatial dimensions L and M to image dimensions H and W, while each discriminator output responds to a local receptive field.
- Spatial GAN architecture: The discriminator objective averages GAN decisions over its L × M spatial output field and trains G and D through the usual minimax game.Generated images or real patches are evaluated spatially, with G minimizing and D maximizing the extended objective.
- Scalability and process properties: Exclusive convolutional layers let a trained generator roll out from small patches to arbitrarily large outputs, while periodic noise relaxes stationarity to represent long-range correlations.The resulting process is cyclostationary rather than stationary when periodic global structure is added.
- Structured noise: PSGAN concatenates local, global, and periodic noise parts, allowing independent spatial variation, whole-image structure selection, and periodic correlations.The three parts share spatial dimensions but can have different channel counts, with total dimension d = d_l + d_g + d_p.
- Global dimensions: Global noise vectors are repeated across spatial positions, so different regions of global space can generate different learned texture processes.For multiple texture images, the generator can associate regions of the global latent space with different texture styles.
- Global dimensions: Smooth variation in the global tensor produces smooth texture changes in the generated image, enabling transitions across the learned texture manifold.The latent field may vary over spatial coordinates provided neighboring global vectors do not change too rapidly.
- Spatially periodic dimensions: Periodic noise channels use plane waves whose wave vectors control direction and spatial frequency, while random phases model positional variation in extracted patches.For multiple textures, an MLP can make the wave vectors depend on global dimensions; for one texture, the wave numbers are direct parameters.
3. Experiments
Experiments show that PSGAN learns periodic and diverse textures from varied image sources, supports smooth texture transitions, and preserves periodic structure while changing appearance.
- Experimental Setup: PSGAN uses convolutional layers, structured global and periodic noise dimensions, and an MLP for learned wave numbers.The setup uses stride-1 convolutional layers and maps global noise into periodic dimensions through learned transformations.
- Periodic Textures: PSGAN captures periodic structure in text and honeycomb textures, including honeycomb autocorrelation periodicity that competing methods fail to reproduce.For text, PSGAN arranges content in regular lines while varying it horizontally; for honeycomb, it captures periodicity and random coloring inside cells.
- Periodic Textures: PSGAN learns more complex periodic patterns, including P6 wallpaper-group patterns with non-orthogonal symmetry axes.The method also generates large repeated honeycomb textures without aliasing, while PSGAN represents translational periodicities rather than rotation and reflection symmetries explicitly.
- Texture Manifolds: PSGAN extracts multiple textures from single large images and image sets, including bricks, grass, leaves, satellite scenes, and DTD textures.On the Merrigum House image, PSGAN produces varied textures, whereas SGAN and Gatys’ method produce a single whole-image mixing process.
- Disentangling Frequencies and Global Dimensions: Interpolating global dimensions changes visual texture appearance while fixed wave numbers preserve consistent periodic structure.The facades examples make the separation between texture identity and periodicity visually interpretable.
4. Discussion
PSGAN extends texture synthesis to diverse, periodic, and non-periodic textures while supporting smooth manifold interpolation and arbitrarily large outputs. Its main boundaries are mode dropping, difficult convergence, and inability to represent larger non-periodic structures.
- Training setting: The model remains fully unsupervised and uses periodic noise dimensions rather than class labels to control generation.Unlike CGANs, PSGAN does not require label information in the generator or discriminator.
- Key abilities: PSGAN learns highly variable textures from large images, periodic textures, and whole texture manifolds that support smooth blending into novel textures.The method is described as learning from image sets or single large images and generating interpolated textures.
- Scalability: PSGAN generates images of any desired size through a fast convolutional forward pass, with linear memory and speed scaling relative to output size.Large outputs can be rendered sequentially with constant GPU memory, and generated textures can be stitched seamlessly.
- Limitations: PSGAN cannot represent global trends, perspective projections, or aperiodic structures, and may drop modes when textures vary in scale or periodicity.The authors also report that convergence can be difficult, as with other GAN models.
- Disentangled control: The structured noise tensor separates global, periodic, and local effects, allowing material, color, and window periodicity to vary independently.With local dimensions fixed, changing global or periodic dimensions alters different visible properties in generated house textures.
- Limitations: Only translational symmetries are represented, so rotation and reflection symmetries are outside the current PSGAN formulation.The paper suggests adding reflection or rotation dimensions as future theoretical extensions.