PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Transformers for High-Resolution Image Synthesis

Patrick Esser, Robin Rombach, Björn Ommer

arXiv:2012.09841v3cs.CV

TL;DR

Transformers are expressive but prohibitively expensive for long image sequences because they model all pairwise interactions. This paper combines CNN-learned visual constituents with transformers for global composition, achieving high-resolution synthesis and strong performance across image-generation tasks.

  • Problem

    Transformers’ quadratic cost makes them computationally infeasible for high-resolution images, motivating efficient encoding of image structure without sacrificing flexibility.

  • Method

    The approach uses CNNs to learn a perceptually rich discrete codebook of image constituents, then applies transformers to model their global compositions.

  • Results

    The transformer consistently outperforms PixelSNAIL across unconditional and conditional image-modeling tasks, while synthesizing realistic high-resolution images across settings.

  • Takeaways & Limitations

    A reusable latent representation and transformer composition model provide an efficient, general-purpose mechanism for conditional image synthesis across tasks.

Abstract

from arXiv · show

Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers .

1. Introduction

Transformers flexibly learn complex relationships but lack CNNs’ locality and spatial-invariance biases, making image modeling costly. The paper combines CNNs for context-rich visual parts with transformers for their global composition.

  • Motivation: Transformers lack CNNs’ built-in locality prior, allowing complex input relationships but requiring them to learn all relationships.This generality contrasts with CNNs’ exploitation of strong local correlations.
  • Motivation: Transformers’ tendency to learn convolutional structures raises whether image inductive biases can be encoded efficiently without sacrificing transformer flexibility.The introduction frames this as avoiding relearning local image structure and regularity from scratch.
  • Motivation: CNNs provide locality and spatial invariance through shared weights, but this can hinder holistic understanding of inputs.The shared-weight design imposes the same spatial treatment across positions.
  • Key insight: The proposed strategy uses convolutions to learn a context-rich visual-part codebook, then uses transformers to model global compositions of those parts.The approach combines convolutional efficiency with transformer expressivity for compositional visual modeling.

2. Related Work

Transformers model long-range interactions through attention but incur quadratic complexity, making them difficult to apply to high-resolution images. Related work therefore explores convolutional locality and two-stage representations, while prior results show transformers outperform convolutional models at low resolution.

  • The Transformer Family: Transformers model interactions solely through attention, allowing inputs to interact regardless of their relative positions.The architecture was originally applied to language tokens but also operates on signals such as audio and images.
  • The Transformer Family: Autoregressive transformers use causal attention and predict logits for the next sequence element.Non-causal entries below the attention matrix diagonal are set to −∞ before the final point-wise linear transformation.
  • The Transformer Family: Quadratic attention complexity makes transformers quickly infeasible for high-resolution images, whose sequence length grows quadratically with resolution.Computing inner products between every pair of sequence elements enables long-range interactions but causes the computational bottleneck.
  • Convolutional Approaches: CNNs exploit image structure by restricting interactions to local neighborhoods, yielding costs linear in sequence length and quadratic in kernel size.This locality provides a computationally efficient alternative for image modeling.
  • Two-Stage Approaches: Although transformers consistently outperform convolutional counterparts on low-resolution images, prior convolutional and discrete-representation methods struggle to capture long-range interactions at high resolution.Two-stage approaches first learn a data encoding and then model its distribution, while VQVAE-based methods still use convolutional density estimation.

3. Approach

The approach combines a perceptually rich, CNN-based discrete image codebook with transformers that model global composition over shorter latent sequences. It supports autoregressive and conditioned synthesis, while patch-wise processing enables megapixel image generation within attention limits.

  • Latent representation: Images are represented as spatial collections of learned codebook entries rather than pixels, reducing description length so transformers can model global interrelations efficiently.The latent representation uses h · w codebook indices for an image encoded at spatial dimensions h × w.
  • Learning a Perceptually Rich Codebook: VQGAN combines vector quantization with discriminator and perceptual losses to learn a perceptually rich codebook at increased compression rates.The compression model uses an encoder, decoder, and codebook, with a single lowest-resolution attention layer to aggregate context.
  • Latent Transformers: The quantized encoding becomes a sequence of codebook indices, enabling image generation as autoregressive next-index prediction with likelihood p(si|s<i).The transformer predicts each next index conditioned on preceding indices and models the likelihood of the full latent representation.
  • Conditioned Synthesis: Conditioned synthesis incorporates class labels or images by learning the latent sequence likelihood given conditioning information c.Spatial conditioning is encoded with another VQGAN, whose index sequence r is prepended to the image sequence s.
  • Generating High-Resolution Images: Megapixel synthesis uses patch-wise training and cropping because transformer attention limits the feasible latent sequence length, while excessive downsampling degrades reconstruction quality.The downsampling factor m is constrained by dataset-dependent reconstruction degradation beyond a critical value.

4. Experiments

The experiments show that combining VQGAN representations with transformers retains transformer advantages over convolutional models while enabling diverse conditional and high-resolution image synthesis. Results further demonstrate that codebook quality strongly affects synthesis and that the approach compares favorably with established generative models.

  • 4.1 Transformer versus convolutional approaches: The experiments span unconditional ImageNet, Restricted ImageNet, LSUN, and conditional depth- and layout-guided synthesis.The same transformer-based methodology is applied across conditioning inputs and datasets, including class labels, depth maps, and semantic layouts.
  • 4.2 High-resolution synthesis: The sliding-window approach supports unconditional and conditional synthesis beyond 256 × 256 pixels and can in principle generate images of arbitrary ratio and size.Evaluations include LSUN-CT, FacesHQ, depth-guided Restricted ImageNet, COCO-Stuff, and Flickr landscapes.
  • 4.3 Codebook quality: Only the full receptive-field setting f = 16 synthesizes high-fidelity faces, whereas smaller receptive fields produce incoherent structures or inconsistent facial features and viewpoints.The result demonstrates the benefit of powerful VQGANs for increasing the transformer’s effective receptive field.
  • 4.4 Comparison to existing models: The model outperforms VQVAE-2 and DCTransformer in FID and IS, surpasses BigGAN and IDDPM at low rejection rates, and approaches state-of-the-art scores at higher rejection rates.Its ImageNet samples also exhibit realistic textures comparable to or better than BigGAN and VQVAE-2.

5. Conclusion

The paper proposes representing images as compositions of perceptually rich constituents to address transformers’ low-resolution limitations and avoid infeasible quadratic pixel-space modeling. CNNs model the constituents while transformers model their compositions, combining their complementary strengths.

  • 5. Conclusion: The approach represents images as compositions of perceptually rich image constituents.This representation addresses challenges that previously confined transformers to low-resolution images.
  • 5. Conclusion: Modeling images through constituents avoids the infeasible quadratic complexity of direct pixel-space modeling.The method targets the computational challenge of modeling high-resolution images directly as pixels.
  • 5. Conclusion: CNNs model the image constituents, while transformers model their compositions.The architecture combines CNNs’ constituent modeling with transformers’ composition modeling.

Supplementary Material

The supplementary material documents changes from a previous version, training hyperparameters and architectures, and additional evidence concerning perceptually rich codebooks. It also interprets codebook quality through a trade-off involving reconstruction fidelity and sampling.

  • Section A: Section A summarizes changes made since a previous version of the paper.
  • Section B: Section B presents the hyperparameters and architectures used to train the models.
  • Section C: Section C provides additional evidence for the importance of perceptually rich codebooks and interprets them as a trade-off involving reconstruction fidelity and sampling.

A. Changelog

This version documents implementation corrections, updated synthesis experiments, and expanded comparisons with subsequently available work. It also revises face-synthesis results and evaluates NLL-based overfitting detection.

  • Implementation corrections: A bug left the commitment-loss weighting at β = 1.0 for all models instead of the reported β = 0.25, so β was removed from Eq. (4).The implementation never used the β term, despite Tab. 8 reporting β = 0.25.
  • Updated experiments: ImageNet class-conditional results were updated using a model trained for 2.4 million steps, replacing a 1.0-million-step model and correcting first-token distribution modeling.Training used batch size 16 accumulated over 8 batches and took 45.8 days on one A100 GPU.
  • Expanded comparisons: The paper added VQVAE reconstruction results and newer class-conditional ImageNet sampling results from works released after the previous version.These additions appear in Tab. 5, Fig. 12, and Tab. 4; comparisons generally lacked available models or sampling results except for BigGAN and BigGAN-deep.
  • Updated experiments: Face-synthesis results were slightly improved with a revised implementation, and larger FFHQ and CelebA-HQ models were used to assess NLL-based overfitting detection.Nearest neighbors were shown for samples from checkpoints with the best validation-split and training-split NLL.

B. Implementation Details

The implementation uses VQGAN encoders and decoders with tunable compression, GPT2-style transformers with capacity controlled by depth, and practical training and sampling settings. Warm-up, precision, hardware, temperature, and top-k choices are specified for the experiments.

  • Training Setup: Most experiments are configured to train with batch-size at least 2 on a 12GB GPU, while training generally uses 2-4 GPUs totaling 48 GB VRAM.The c-IN (big), COCO-Stuff, and ADE20K models are exceptions; 16-bit precision is enabled when hardware permits.
  • VQGAN Architecture: VQGAN compression is tuned through the number of downsampling steps m.The encoder and decoder architecture is detailed in Table 7, while h = H 2^m, w = W 2^m, and f = 2^m.
  • VQGAN Architecture: The VQGAN loss weight λ is set to zero during warm-up, with at least one epoch recommended because longer warm-ups generally improve reconstructions.The models use a convolutional encoder and decoder, and the discriminator is patch-based.
  • Transformer Architecture: The transformer follows GPT2 and varies capacity mainly by changing the number of layers.Samples generally use temperature t = 1.0 and top-k k = 100, with higher top-k values for larger codebooks.

C. On Context-Rich Vocabularies

The section shows that context-rich image vocabularies require large downsampling factors for efficient long-range modeling while preserving perceptual fidelity. VQGAN provides sharper reconstructions than VQVAE, but the suitable compression rate depends on the dataset.

  • Compression factor trade-offs: Large downsampling factors f enable the transformer to model long-range interactions efficiently, but excessive compression reduces VQGAN reconstruction quality.Reconstruction quality is measured by LPIPS, with the degradation analyzed after a certain compression point.
  • Compression factor trade-offs: The VQGAN objective is to learn perceptually rich encodings that combine large f with perceptually faithful reconstructions.Figure 11 motivates this balance between efficient sequence modeling and reconstruction quality.
  • VQGAN reconstruction quality: At f = 8 with 8192 codebook entries, VQVAE and VQGAN both capture global structure, but VQGAN produces crisp, realistic textures instead of VQVAE’s blurry textures.The comparison highlights differences in details such as stone, fur, and squirrel-tail textures.
  • VQGAN reconstruction quality: Increasing VQGAN compression to f = 16 causes some reconstructed parts to become misaligned with the input.The paw of the squirrel is given as an example of imperfect alignment.
  • Dataset dependence: The appropriate f depends on the dataset: S-FLCKR samples improve as f increases, and perceptually faithful reconstructions persist at f32 unlike the corresponding face results.The passage suggests greater perceptual sensitivity to facial features than to textures may explain this difference.

D. Additional Results

The additional results provide qualitative comparisons with competing models, further evaluate the approach against ImageGPT, and present high-resolution samples across multiple datasets and conditioning settings.

  • Qualitative Comparisons: Qualitative comparisons cover VQVAE-2, MSP, BigGAN, and SPADE across ImageNet and semantic-synthesis benchmarks.Results are shown in figures corresponding to Tabs. 4, 6, and 2, including ADE20K and COCO-Stuff.
  • Comparison to Image-GPT: ImageGPT is used as a state-of-the-art generative-transformer comparison, but its pixel-space approach does not scale beyond a resolution of 192 ×.The comparison emphasizes ImageGPT’s substantial compute requirements and its pixel-space limitation.
  • Additional High-Resolution Results: Additional semantically guided high-resolution results are provided on S-FLCKR for f = 16 (m = 4) and f = 32 (m = 5).The figures include an enlarged version of a main-text result.
  • Additional High-Resolution Results: A sliding-window approach produces high-resolution depth-to-image samples on RIN and edge-to-image samples on IN.These results are included in additional figures for the corresponding conditioning settings.

E. Nearest Neighbors of Samples

Nearest-neighbor analysis shows that validation-NLL early stopping prevents overfitting in face synthesis, while FID can miss it. The class-conditional ImageNet model shows no validation-NLL overfitting and produces new, high-quality samples.

  • Face synthesis: Models were compared at checkpoints with the best validation NLL and best training NLL to test overfitting on FFHQ and CelebA-HQ.The best-validation checkpoints occurred at epochs 10 and 13, whereas the best-training checkpoints occurred at epoch 1000.
  • Face synthesis: Early stopping based on validation NLL can prevent overfitting, with dataset size limiting face-synthesis performance because the model nearly fits training data perfectly.The passage identifies dataset size as the bottleneck for face synthesis.
  • Face synthesis: FID fails to detect this overfitting: best-training-NLL checkpoints score 3.86 on CelebA-HQ and 2.68 on FFHQ, versus 10.2 and 9.6 for best-validation-NLL checkpoints.These comparisons show that lower FID can coincide with overfitting.
  • Class-conditional ImageNet: The class-conditional ImageNet model shows no overfitting according to validation NLL, and its nearest neighbors indicate new, high-quality samples.Nearest neighbors in Fig. 46 provide evidence for sample novelty and quality.

F. On the Ordering of Image Representations

Because image token sequences lack a canonical ordering, the authors evaluate six permutations designed to encode different spatial priors. Row-major ordering achieves the best negative log-likelihood, while the hierarchical subsample prior performs worst among the tested variants.

  • Motivation: Image representations have no obvious linear ordering, so the sliding-window approach motivates evaluating permutations of codebook-index sequences.The default sliding-window formulation depends on row-major ordering.
  • Ordering variants: The evaluated permutations are row major, spiral out, z-curve, subsample, alternate, and spiral in.These encode priors including centered objects, preserved locality, hierarchical subsampling, and alternating row direction.
  • Results: 4.767, row major achieves the best final negative log-likelihood among the tested ordering variants.The reported final values are 4.767, 4.889, 4.810, 5.015, 4.812, and 4.901 for variants (i)–(vi), respectively.
  • Results: 5.015, subsample has the worst final negative log-likelihood, indicating that its more hierarchical prior provides no helpful bias.Qualitative samples also identify subsample and spiral in as the two worst-performing models by NLL.
Loading 2012.09841v3…