Source-linked AI summary
TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
Anna Frühstück, Ibraheem Alhashim, Peter Wonka
TL;DR
Large-scale texture synthesis must handle many reference images while producing substantial outputs. TileGAN combines outputs from a limited-resolution GAN and provides artistic editing tools; the authors report fast, high-quality synthesis at hundreds-of-megapixels scale, with a testing-machine output limit of about 1.6 gigapixels.
Problem
Large-scale texture synthesis must process many reference images and generate large outputs while preserving variability and features at multiple scales.
Source: §1 · p. 1Method
Results
The authors report fast generation of high-quality textures comprising hundreds of megapixels, supported by early quantitative and qualitative results.
Source: §9 · p. 8Takeaways & Limitations
The framework uses a limited-resolution GAN as a building block for possibly infinite output textures and offers local and global artistic control.
Source: §1 · p. 2, §9 · p. 8The largest output generated on the testing machine was about 1.6 gigapixels; larger images would require additional memory-management procedures.
Source: §8 · p. 8
Abstract
from arXiv · showhide
We tackle the problem of texture synthesis in the setting where many input images are given and a large-scale output is required. We build on recent generative adversarial networks and propose two extensions in this paper. First, we propose an algorithm to combine outputs of GANs trained on a smaller resolution to produce a large-scale plausible texture map with virtually no boundary artifacts. Second, we propose a user interface to enable artistic control. Our quantitative and qualitative results showcase the generation of synthesized high-resolution maps consisting of up to hundreds of megapixels as a case in point.
1 INTRODUCTION
TileGAN addresses the challenge of using many reference images to create large-scale textures while preserving variability and multiscale features. It combines limited-resolution GAN outputs through latent-code transitions and adds tools for artistic control.
1 INTRODUCTION
- Large-scale synthesis must process substantial reference data to retain variability and features at multiple scales, and must also generate large outputs.The authors warn that insufficient reference data can yield homogeneous or repetitive results.Source: §1 · p. 1
- TileGAN blends neighboring tiles by manipulating lower-resolution GAN latent codes to create transition regions, rather than relying only on graph cuts, patch repair, or blending.Graph cuts can create artifacts when the blended tiles are insufficiently similar.Source: §1 · p. 2
- The authors propose latent brushes and other intuitive editing tools to provide global artistic control over large-scale texture maps.They contrast this with GAN-based texture synthesis approaches that offer no or minimal artistic control.Source: §1 · p. 2
- The framework uses a GAN of limited resolution as a building block for possibly infinite output textures and is reported to improve quality and speed over prior large-scale synthesis methods.The authors illustrate results with high-quality textures spanning hundreds of megapixels.Source: §1 · p. 2
2 RELATED WORK
The related-work review spans non-parametric and parametric synthesis, GAN-based texture methods, and other GAN applications. It positions TileGAN against prior approaches with limits in scalability, variability, and artistic control.
2 RELATED WORK
- The review groups related literature into multiple categories and refers readers to Akl et al. [2018] for a broader example-based texture synthesis review.The categories include non-parametric and parametric texture synthesis, GAN texture methods, and selected GAN applications.Source: §2 · p. 2
2.1 Non-parametric Texture Synthesis
- Non-parametric methods synthesize outputs by finding approximate input matches for each k × k output patch, often using fast correspondence methods such as PatchMatch [Barnes et al. 2009].The review notes strong visual results but highlights deep learning's promise for scaling input data and output size.Source: §2.1 · p. 2
2.2 Parametric Texture Synthesis
- Parametric approaches match feature statistics, including neural-network feature correlations and cross-scale features, while some train generators using feature-statistics losses.The cited work includes Gatys et al. [2015a; 2015b], Sendik and Cohen-Or [2017], Snelgrove et al. [2017], and Dosovitskiy and Brox [2016].Source: §2.2 · p. 2
2.3 GANs Trained for Texture Synthesis
- GAN texture methods include guidance-image matching and smoother patch transitions, but prior approaches remain limited in texture support, expected size, variability, or quality.TileGAN's synthesis overview depicts sampling latent tiles, storing downsampled versions, and retrieving tiles by similarity to guidance regions.Source: §2.3 · p. 3, Figure 3 · p. 3
2.4 Selected Applications of GANs
- GANs have been applied to image processing and graphics tasks, with terrain synthesis identified as the closest related problem.Other applications listed include image translation, caricatures, avatars, building details, and shape modeling.Source: §2.4 · p. 3
3 OVERVIEW
TileGAN uses a pretrained generator to create large-scale non-homogeneous textures by assembling compatible intermediate latent vectors into a latent field. It also provides editing tools that modify this field for artistic control.
3 OVERVIEW
- The framework combines generator and discriminator networks, with the generator expanding a random latent vector into an image and the discriminator assessing whether samples match the training distribution.The paper focuses on combining generator outputs for large-scale texture synthesis.Source: §3 · p. 3
- The synthesis method tiles compatible intermediate latent vectors into a latent field that a pretrained generator uses to produce a coherent large-scale texture.The field is optimized to follow a target guidance map and to replace tiles that are incompatible with their neighbors.Source: §3 · p. 3
- Users control synthesis by modifying the latent field through painting, shuffling, copying, and target-image matching.These operations offer different forms of artistic control.Source: §3 · p. 3
- The GenerateTextureMap procedure takes GBl, S, and M as inputs and returns the latent field F and output image I.The algorithm begins with initial tiling.Source: §3 · p. 4
4 SYNTHESIS
TileGAN synthesizes large textures by arranging intermediate GAN representations into a latent field, then applying the generator to that field. Its pipeline preprocesses and clusters samples, initializes and refines the field, and generates arbitrarily large outputs.
4 SYNTHESIS
- A split GAN generator maps latent vectors to intermediate tiles and then to final color images; the latent field is synthesized between these stages.The split level l determines where synthesis occurs.Source: §4 · p. 4
4.1 Preprocessing
- Preprocessing generates texture samples pairing intermediate latent tiles with downsampled images, then clusters them by visual appearance for fast lookup.More samples afford greater texture variability; k-means cluster centers serve as representative samples.Source: §4.1 · p. 4
4.2 Latent Field Synthesis
- Latent-field synthesis uses an MRF-style neighborhood objective so each output tile has a neighborhood similar to one in the input distribution.The framework separates synthesis into initialization and iterative refinement, allowing an initial result to appear while refinement runs in the background.Source: §4.2 · p. 4
4.2.1 Initialization.
- Initialization fills unassigned output tiles by selecting top matches to the guidance map, using a unary energy term.The guidance map supplies global content control, and the sample set and clustering are assumed to be prepared beforehand.Source: §4.2.1 · p. 4
4.2.2 Refinement.
- Refinement minimizes weighted unary and binary energies, comparing tiles with four-connected neighbors by appearance, latent representation, and cluster membership.The unary term measures visual similarity to the guidance map; refinement stops below an empirically set energy threshold intended to preserve feature variety.Source: §4.2.2 · pp. 4–5
- Cropping unstable outer latent regions reduces their influence; smaller latent tiles blend features more smoothly, while whole representative images support sliding-window matching.Experiments typically use latent tiles of 2 × 2 to 4 × 4.Source: §4.2.2 · p. 5
- The split level trades transition quality against the size of the affected region; levels 2–5 gave the authors their best visual results.Later-stage latent merging reduces the transition’s impact region.Source: §4.2.2 · p. 5, Figure 4 · p. 5
4.3 Texture Synthesis
- Applying the remaining convolutional generator to the latent field yields RGB textures of arbitrary size, with overlapping applications enabling large outputs.Most experiments use l = 3 and n = 9; overlapping output regions are discarded.Source: §4.3 · p. 5
5 ARTISTIC CONTROL
TileGAN provides interactive editing through direct latent-field manipulation and guidance-map editing. Latent brushes, cloning, and traditional image manipulation offer local and broader control over the resulting texture.
5 ARTISTIC CONTROL
- The interface provides two editing routes: directly manipulating the latent field and editing the guidance map.The preview updates locally as latent-field changes are made.Source: §5 · p. 6
- Users can drag cluster-sampled tiles onto the field, clone existing content, and shuffle, perturb, or interpolate latents for local control.Cluster representatives preview the expected tile appearance; cloning can add diversity through spatial shuffling.Source: §5 · p. 6
- Editing the guidance map with traditional image tools can vary output size, shading, and feature placement.Source: §5 · p. 6
6 IMPLEMENTATION DETAILS
The implementation builds on Progressive Growing of GANs, modifying the generator to expose intermediate latents for synthesis. The authors describe training on curated image tiles and report that synthesis and editing run interactively on a single GPU.
6 IMPLEMENTATION DETAILS
- The system modifies Progressive Growing of GANs to expose intermediate latents that can be modified and merged into large-scale non-homogeneous textures.The authors selected ProGAN for its image quality and stack of identical building blocks.Source: §6 · p. 6
- Training each dataset takes around four days on four NVIDIA V100 GPUs, while synthesis and editing run at interactive rates on one GPU.Preprocessing typically takes 30 minutes to sample 100K latent tiles and cluster representative images into 10 groups.Source: §6 · p. 6
- Training uses curated 512 × 512 tiles from terrain maps, satellite imagery, oil canvas, and night-sky imagery, without alignment or augmentation.The datasets contain between 18K and 65K tiles each.Source: §6 · p. 6
- In the informal comparison, the authors report that existing methods struggle with multi-scale features, while TileGAN’s result exhibits variations.The input image is guidance for TileGAN, not training input.Source: Figure 6 · p. 7
7 RESULTS
The authors argue that TileGAN produces high-quality large-scale textures, outperforming published methods in visual quality. In aerial-image comparisons, STTO produces repetitive output and NSTS does not produce high-quality results; TileGAN is faster when preprocessing is excluded, under the stated amortization assumption.
Visual quality
- The authors argue that their high-quality, low-artifact results surpass any published method; in aerial tests, STTO is repetitive and NSTS is not high-quality.The comparison used recommended settings for both competing methods.Source: §7 · p. 7
Visual comparison to the state of the art
- STTO and NSTS were tested using a single 628 × 425 input tile and their authors’ recommended settings.TileGAN used the fully trained network with the input image as a guidance map.Source: §7 · p. 7
Quantitative comparison
- TileGAN is faster than competing methods when preprocessing is excluded, assuming its cost is amortized across repeated applications of one trained GAN.The comparison also reports that STTO scales less well with input and output size, while NSTS spends substantial time analyzing a small input texture.Source: §7 · p. 7
Visual quality
- Apparently similar latent tiles can yield plausible but unintended salient boundaries when their transition is absent from the training data.The authors suggest this can occur when ice samples come from different continents.Source: Figure 7 · p. 8
8 LIMITATIONS
The framework has limits in output scale and tile quality, and refinement cannot eliminate all unpredictable transitions. Grid-like artifacts and repetition can also appear under particular tiling or guidance conditions.
Output scale
- The largest output on the testing machine is about 1.6 Gigapixels; larger images require additional memory-management procedures.Source: §8 · p. 8
Latent-tile quality
- Some latent tiles produce visual artifacts, which may stem from training, dataset sampling, or limitations of the chosen GAN implementation.Source: §8 · p. 8
Tile transitions
- Tile blending can create unpredictable transitions that refinement cannot eliminate, while MRF optimization to convergence may reduce diversity through repeated tiles.The authors stop optimization early to help avoid repeated, visually similar latent tiles.Source: §8 · p. 8
Tiling artifacts
- Grid-like artifacts are more prominent with larger latent tiles, while uniform-color guidance regions can produce repetitive tiling.The grid effect can be controlled through merge level, latent tile size, and MRF choices.Source: Figure 8 · p. 8, §8 · p. 8
9 CONCLUSIONS AND FUTURE WORK
TileGAN combines intermediate GAN latent tiles into large plausible textures and offers local and global artistic control. The authors report high-quality results reaching hundreds of megapixels, and identify additional data types and latent manipulation as future directions.
Conclusions
- TileGAN tiles compatible intermediate latent vectors and uses a generator to produce coherent, large-scale textures, with an interface for local and global artistic control.The authors describe the method as combining latent tiles at different GAN levels for interactive synthesis.Source: §9 · p. 8
Future work
- Future directions include other planetary and gigapixel object imagery, depth or multichannel data, stacked GANs for guide maps, and global appearance edits through latent manipulation.Source: §9 · p. 8
Conclusions
- The illustrated Google terrain and satellite-imagery results include large-scale outputs with multiple levels of detail and zoomed crops.The paper reports that outputs can reach hundreds of megapixels.Source: §9 · p. 8, Figure 9 · p. 9