PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Transformers for High-Resolution Image Synthesis

Patrick Esser, Robin Rombach, Björn Ommer

arXiv:2012.09841v3cs.CV

TL;DR

Transformers’ quadratic pairwise-interaction costs make high-resolution image modeling challenging; this paper represents images as perceptually rich constituents learned with CNNs and models their compositions with transformers. It reports better FID and IS than other autoregressive approaches and transformer-based image synthesis at megapixel scale.

  • Problem

    The open question is whether image-locality priors can be encoded efficiently without sacrificing transformer flexibility, given the quadratic costs of modeling all pairwise interactions.

    Source: §1 · p. 1

  • Method

    The paper uses CNNs to learn a codebook of context-rich visual parts, then transformers to model their global compositions.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The approach outperforms other autoregressive approaches on FID and IS and demonstrates transformer-based synthesis of images in the megapixel range.

    Source: §4.4 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The approach offers a unified, general-purpose mechanism for conditional image synthesis across tasks without task-specific architectures or modules.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers .

1. Introduction

The paper combines CNN-learned visual parts with transformer modeling of their global composition to synthesize high-resolution images, with control through object classes or spatial layouts.

  • Transformers lack a built-in locality prior and incur quadratic computational costs from pairwise interactions, making high-resolution images with millions of pixels difficult to scale to.CNNs exploit prior knowledge of strong local image correlations.Source: §1 · p. 1
  • The authors hypothesize that convolutional connectivity suits low-level image structure, while higher semantic levels need transformer flexibility; CNN weight sharing can hinder holistic understanding.CNNs bias toward spatial invariance through shared weights across positions.Source: §1 · pp. 1–2
  • CNNs learn a codebook of context-rich visual parts, and a transformer models their global compositions and long-range relations for high-resolution synthesis.An adversarial approach encourages the parts to capture perceptually important local structure, reducing the need for the transformer to model low-level statistics.Source: §1 · p. 2
  • The formulation supports conditioning on desired object classes or spatial layouts and is reported to outperform prior codebook-based state-of-the-art convolutional approaches.Source: §1 · p. 2

2. Related Work

Related work contrasts transformers’ global interactions and quadratic image cost with efficient but restrictive CNN locality, then situates the paper among two-stage and discrete-image modeling approaches.

The Transformer Family

  • Transformers model interactions across positions through attention, with layers combining attention and position-wise fully connected processing.For autoregressive maximum-likelihood learning, non-causal attention entries are masked and the output predicts logits for the next sequence element.Source: §2 · p. 2
  • Attention costs grow quadratically with sequence length; prior approaches restrict receptive fields or reduce costs to n√n [26], yet remain limited for high-resolution images.Restricted fields reduce expressivity and assume pixel independence; the n√n approaches still make resolutions beyond 64 pixels prohibitively expensive.Source: §2 · p. 2

Convolutional Approaches

  • CNNs exploit image locality for efficient computation, but their local-interaction bias can be too restrictive, as specialized synthesis layers suggest [53] [68].With a fixed small kernel such as 3×3, computation scales linearly with image sequence length and quadratically with kernel size.Source: §2 · p. 2
  • Although convolutional autoregressive models exist [10], transformers outperform convolutional counterparts on low-resolution images [26].The paper presents its approach as enabling efficient high-resolution transformer modeling while retaining their advantages over state-of-the-art convolutional approaches.Source: §2 · p. 2

Two-Stage Approaches

  • Two-stage generative approaches first encode data, then model the encoding distribution using VAE [13], normalizing-flow [18] [64], or GAN-based methods [43].[13] uses a VAE in both stages; [18] use unconditional and [64] conditional normalizing flows, while [43] learns a GAN on autoencoder representations and uses low-resolution wavelet coefficients decoded by a learned generator.Source: §2 · pp. 2–3
  • VQVAE learns discrete image representations for autoregressive convolutional modeling [72], later extended hierarchically, but convolutional density estimation hampers long-range interactions.Transformer image modeling found even 32 × 32 inputs computationally expensive and used VQVAE encodings up to 192 × 192 with a shallow, small-receptive-field model; this paper argues a context-rich first stage is critical for efficient high-resolution synthesis.Source: §2 · p. 3

3. Approach

The approach combines CNN-learned, perceptually rich discrete image codes with transformers that model their composition, reducing sequence length to enable high-resolution synthesis. It extends this framework to conditional generation and megapixel images.

3. Approach

  • CNNs learn a compact codebook of image constituents, whose compositions transformers model to synthesize realistic, globally consistent high-resolution images.The method targets megapixel-scale synthesis, beyond the 64 × 64 images previously demonstrated with transformers.Source: §3 · p. 3

3.1. Learning an Effective Codebook of Image Constituents for Use in Transformers

  • VQGAN uses a convolutional encoder-decoder and discrete codebook, with perceptual loss and a patch-based discriminator to preserve quality at increased compression.A single attention layer at the lowest resolution aggregates context and reduces latent sequence length. VQGAN is a variant of VQVAE; the discriminator is patch-based [28].Source: §3.1 · pp. 3–4

3.2. Learning the Composition of Images with Transformers

  • The transformer predicts codebook indices autoregressively and supports conditioning on labels or spatial inputs represented by a separate code sequence.For spatial conditioning, the method prepends the condition indices to the image sequence and trains on the image indices conditioned on that prefix.Source: §3.2 · p. 4
  • For megapixel synthesis, images are cropped into feasible patches for training and generated with a sliding attention window, while spatial conditioning can support aligned data.Increasing VQGAN downsampling eventually degrades reconstruction quality; sliding-window context is sufficient when dataset statistics are approximately spatially invariant or spatial conditioning is available, including image coordinates for unconditional synthesis on aligned data [42].Source: §3.2 · p. 5

4. Experiments

Experiments show that transformers retain an advantage over PixelSNAIL in the latent setting, while VQGAN representations enable one approach to handle diverse conditional and high-resolution synthesis. Context-rich codebooks improve coherence, and the reported benchmarks show competitive synthesis and reconstruction quality.

4.1. Attention Is All You Need in the Latent Space

  • Across unconditional and conditional tasks, transformers consistently outperform PixelSNAIL in NLL at equal training time, with a larger gap at equal training steps.PixelSNAIL trains roughly twice as fast; tasks include ImageNet, Restricted ImageNet, LSUN Churches and Towers, depth-conditioned Restricted ImageNet, and semantically conditioned Flickr landscapes.Source: §4.1 · p. 5, Table 1 · p. 5

4.2. A Unified Model for Image Synthesis Tasks

  • A single VQGAN-transformer approach supports semantic, depth/edge, pose-guided, superresolution, and class-conditional synthesis without task-specific architectures or modules.The experiments cover multiple datasets; stochastic superresolution uses an 8× upsampling factor, and the approach extends to megapixel semantic-layout synthesis on S-FLCKR.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3. Building Context-Rich Vocabularies

  • On FacesHQ, only the context-rich f = 16 representation synthesizes high-fidelity samples; smaller receptive fields fail to capture coherent structures, while f = 8 yields facial and viewpoint inconsistencies.The study varies VQGAN downsampling factor while keeping transformer sequence length fixed at 16 × 16.Source: §4.3 · pp. 6–7, Figure 7 · p. 8
  • With a fixed computational budget on CIFAR10, VQGAN latents improve FIDs by 18.63% and enable 14.08× faster image sampling than direct pixel modeling.The comparison uses a 512-value RGB dictionary for pixel modeling and a VQGAN latent code of size 16 × 16 = 256.Source: §4.3 · p. 7

4.4. Benchmarking Image Synthesis Results

  • On semantic and face synthesis benchmarks, the unified approach works across tasks, although some task-specialized GANs achieve better FID scores.The reported comparison also emphasizes that the approach can encode and reconstruct images, bridging adversarial and likelihood-based methods.Source: §4.4 · p. 7, Table 2 · p. 7, Table 3 · p. 8
  • On 256 × 256 ImageNet, the model outperforms VQVAE-2 and DCTransformer on FID and IS, surpasses BigGAN and IDDPM at low rejection rates, and approaches state-of-the-art scores at higher rates.The class-conditional model uses ≃10× fewer parameters than VQVAE-2; classifier rejection sampling can further increase quality.Source: §4.4 · p. 8
  • VQGAN outperforms non-adversarial discrete VAEs on ImageNet reconstruction FID with a sequence length of 256, versus 5120 for VQVAE-2 and 1024 for DALL-E.Larger codebooks or longer code sequences further improve reconstruction; a hierarchical VQGAN achieves the best reconstruction FID but requires a very long sequence.Source: §4.4 · p. 9, Table 5 · p. 9

5. Conclusion

The paper represents images as compositions of perceptually rich constituents, modeling them with CNNs and their compositions with transformers to enable high-resolution synthesis. Experiments demonstrate megapixel image synthesis and performance beyond state-of-the-art convolutional approaches, while the general conditional synthesis mechanism may support novel neural rendering approaches.

5. Conclusion

  • The approach synthesizes megapixel-range images with a transformer-based architecture and outperforms state-of-the-art convolutional approaches.The authors describe these as the first results on high-resolution image synthesis with a transformer-based architecture.Source: §5 · p. 9
  • Representing images as perceptually rich constituents and modeling their compositions with transformers overcomes the infeasible quadratic complexity of modeling images directly in pixel space.A CNN models the constituents, while a transformer models their composition, combining the architectures’ complementary strengths.Source: §5 · p. 9
  • The approach’s general mechanism for conditional synthesis offers opportunities for novel neural rendering approaches.Source: §5 · p. 9

Supplementary Material

The supplementary material covers paper revisions, model training details, codebook analysis, additional synthesis results, sample neighbors, and image-representation ordering.

Sections A–B

  • Sections A and B summarize changes from a previous paper version and provide the training hyperparameters and architectures.Source: Supplementary Material · p. 10

Section C

  • Section C adds evidence for perceptually rich codebooks and interprets them through the trade-off between reconstruction fidelity and sampling capability.Source: Supplementary Material · p. 10

Section D

Sections E–F

A. Changelog

The changelog corrects the commitment-loss description and revises synthesis evaluations, comparisons, and diagnostics. It also clarifies limits on reproducing results from other methods.

A. Changelog

  • The commitment-loss weight β was never used because of an implementation bug, so all models used β = 1.0 and β was removed from Eq. (4).The previous version had reported β = 0.25 for all models in Tab. 8.Source: A. Changelog · p. 10
  • ImageNet class-conditional results were updated using a revised model and training run, with FID now measured on 50k samples against all ImageNet training examples.The new model trained for 2.4 million steps over 45.8 days on one A100 GPU; the previous model trained for 1.0 million steps. The revised evaluation uses torch-fidelity, and qualitative samples and top-k/p and rejection-rate visualizations were updated or added.Source: A. Changelog · p. 10
  • The comparison adds VQVAE reconstruction results and class-conditional ImageNet results, but only BigGAN and BigGAN-deep [4] could be reevaluated with the same code.Class-conditional evaluation uses 50k samples against the full ImageNet training split; the authors note uncertainty about other implementations’ image resizing. Their own pipeline uses a bilinear, anti-aliased largest center crop resized to 256 × 256 with Pillow, then computes FID and Inception Scores with torch-fidelity.Source: A. Changelog · p. 10
  • Face-synthesis results were updated and slightly improved, and larger FFHQ and CelebA-HQ models were used to assess NLL-based overfitting detection.Nearest neighbors from checkpoints selected by best validation-split and training-split NLL are shown, and Fig. 10 visualizes how top-k tuning affects FID and IS.Source: A. Changelog · p. 10

B. Implementation Details

The implementation specifies VQGAN encoder–decoder design and training, GPT-2-based transformer capacity and sampling settings, and hardware constraints for the experiments.

Implementation Details

  • Most transformer models are configured for batch sizes of at least 2 on a 12GB-VRAM GPU, while training generally uses 2–4 GPUs totaling 48 GB of VRAM.This does not apply to c-IN (big), COCO-Stuff, and ADE20K models; 16-bit precision is enabled when hardware permits.Source: B. Implementation Details · p. 10

VQGAN Architecture

  • The VQGAN encoder and decoder follow the architecture in [25] without skip connections, while its discriminator is patch-based as in [28].Compression rate is controlled by tuning the number of downsampling steps m; setting λ = 0 for at least one epoch during warm-up is recommended, and longer warm-ups generally improve reconstructions.Source: B. Implementation Details · p. 11

Transformer Architecture

  • The transformer uses the GPT-2 architecture, with capacity varied mainly by changing the number of layers and samples generally produced at temperature t = 1.0 with top-k k = 100.Higher top-k values are used for larger codebooks.Source: B. Implementation Details · p. 11

C. On Context-Rich Vocabularies

VQGAN’s perceptually faithful reconstructions allow transformers to use large downsampling factors for efficient high-resolution synthesis, but the useful compression level depends on the reconstruction-fidelity limit of the dataset. Comparisons show that VQGAN preserves sharper textures than VQVAE at f=8 and can retain realistic reconstructions at larger factors.

C. On Context-Rich Vocabularies

  • Beyond f=16, severe reconstruction errors limit sample fidelity even when the transformer models image representations faithfully.Larger factors enable efficient modeling of long-range interactions, but the reconstruction capability sets an upper bound on generated-image quality.Source: C. On Context-Rich Vocabularies · p. 11
  • At f=8 with 8192 codebook entries, VQGAN textures are crisp versus blurry VQVAE; at f=16, slightly larger codebooks retain realism despite some misalignment.This demonstrates high-fidelity reconstruction at large factors, enabling efficient high-resolution synthesis with transformers.Source: C. On Context-Rich Vocabularies · pp. 11–12
  • On S-FLCKR, perceptually faithful reconstructions persist at f=32, unlike the face results, enabling more efficient high-resolution landscape generation.Sample quality also increases as f increases on S-FLCKR; the passage suggests greater perceptual sensitivity to facial features than textures may explain the dataset difference.Source: C. On Context-Rich Vocabularies · p. 12

D. Additional Results

The additional results compare image completions with ImageGPT and extend qualitative synthesis examples across high-resolution settings and semantic-synthesis benchmarks. They report more faithful, consistent completions than ImageGPT and illustrate outputs across several datasets and tasks.

Qualitative Comparisons

  • Qualitative comparisons accompany Tables 4 and 6, and Figures 40–41 compare semantic synthesis with SPADE on ADE20K and COCO-Stuff.For VQVAE-2 and MSP, results came from supplementary or provided samples; BigGAN samples were produced with its provided model.Source: D. Additional Results · pp. 12–13

Comparison to Image-GPT

  • Compared with ImageGPT, our approach produces consistent image completions of dramatically increased fidelity and supports resolutions beyond ImageGPT’s 192 × 192 limit.The approach uses image compression to obtain context-rich representations before transformer modeling; the comparison is shown in Figs. 27–28.Source: D. Additional Results · p. 13

Additional High-Resolution Results

  • Figures 29–44 provide additional high-resolution results spanning semantic, depth-, edge-, pose-, and class-guided synthesis, superresolution, and unconditional generation.Examples include semantically guided S-FLCKR at f = 16 and f = 32, sliding-window outputs for several tasks, and 256 × 256 results on ADE20K, COCO-Stuff, IN, and RIN.Source: D. Additional Results · p. 13

E. Nearest Neighbors of Samples

Nearest-neighbor comparisons show that face models selected by best training NLL reproduce training examples, while best-validation-NLL checkpoints generate new faces; the ImageNet model shows no comparable overfitting. Validation NLL can detect face-model overfitting, but FID cannot.

E. Nearest Neighbors of Samples

  • Best-training-NLL face checkpoints reproduce training examples, while best-validation-NLL checkpoints depict new faces absent from training data, based on LPIPS nearest neighbors.The checkpoints were selected on FFHQ and CelebA-HQ; best validation NLL occurred at epochs 10 and 13, while best training NLL occurred at epoch 1000.Source: E. Nearest Neighbors of Samples · p. 13
  • FID scores are 3.86 on CelebA-HQ and 2.68 on FFHQ for best-training-NLL checkpoints, versus 10.2 and 9.6 for best-validation-NLL checkpoints.Despite their lower FID scores, the best-training-NLL checkpoints reproduce training data, so FID cannot detect this overfitting.Source: E. Nearest Neighbors of Samples · p. 13
  • Early stopping by validation NLL can prevent face-synthesis overfitting, whose bottleneck is dataset size because the model can almost perfectly fit training data.The passage notes that it is unclear whether validation-NLL early stopping is optimal when sample quality is the main objective, and calls for metrics that distinguish novel high-quality samples from training-data reproduction.Source: E. Nearest Neighbors of Samples · p. 13
  • The class-conditional ImageNet model shows no overfitting according to validation NLL, and its nearest neighbors support that it generates new, high-quality samples.Source: E. Nearest Neighbors of Samples · p. 13

F. On the Ordering of Image Representations

Autoregressive codebook modeling is sensitive to token order: among the tested permutations, row-major ordering achieves the best validation negative log-likelihood.

F. On the Ordering of Image Representations

  • The study compares six sequence orderings, motivated by centered objects, preserved locality, subsampled prefixes [46], and additional context for predicting the image center.For a VQGAN trained on ImageNet, each ordering was tested with fixed initialization and computational budget; the sliding-window approach depends on row-major ordering.Source: F. On the Ordering of Image… · pp. 13–14
  • The two worst-NLL variants, subsample and spiral in, tend to produce more textural samples, while the other orderings yield more recognizable structures.Qualitative samples are shown in Figure 48.Source: F. On the Ordering of Image… · p. 14, Figure 48 · p. 48
  • Validation NLLs were 4.767 (row major), 4.889 (spiral out), 4.810 (z-curve), 5.015 (subsample), 4.812 (alternate), and 4.901 (spiral in); row major performed best.The results show that autoregressive codebook modeling is not permutation-invariant.Source: F. On the Ordering of Image… · p. 14, Figure 47 · p. 47

Related papers

Loading 2012.09841v3…