Source-linked AI summary
Image Transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran
TL;DR
Image generation requires models that combine tractable likelihoods with scalable computation and large receptive fields. The paper adapts the Transformer to autoregressive image modeling using local self-attention and an encoder-decoder for conditional generation. It reports a new ImageNet log-likelihood state of the art and stronger human evaluations for 4x super-resolution than prior work.
Problem
Image generation models must retain tractable likelihoods while addressing the computational difficulty of sequential prediction and the limited receptive fields of convolutional models.
Method
The Image Transformer models images autoregressively with locally restricted self-attention and uses an encoder-decoder configuration for image-conditioned generation.
Results
The Image Transformer attains a new ImageNet state of the art in log-likelihood and its 4x super-resolution outputs are judged convincingly natural significantly more often than prior autoregressive outputs.
Takeaways & Limitations
Local self-attention lets the model operate on image structures larger than sentences while improving unconditional ImageNet modeling and super-resolution.
Takeaways & Limitations
The paper identifies broader conditioning information, including free-form text, as future work.
Abstract
from arXiv · showhide
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our super-resolution model fool human observers three times more often than the previous state of the art.
1. Introduction
The paper adapts self-attention to autoregressive image generation, addressing the trade-off between parallel computation and long-range receptive fields. The resulting Image Transformer improves ImageNet image modeling and supports conditional generation tasks including super-resolution.
- Motivation: Image density models factor the joint pixel distribution into conditional distributions, but sequential recurrent prediction is computationally challenging.This formulation provides a tractable likelihood while creating scalability difficulties for recurrent models.
- Motivation: CNNs offer parallel computation but limited receptive fields, which can hinder modeling long-range image phenomena such as symmetry and occlusion.Expanding the receptive field increases parameters, computational cost, and training difficulty.
- Approach: The Image Transformer replaces recurrent and convolutional networks with locally restricted multi-head self-attention, interpreted as a sparsely parameterized gated convolution.Decoupling receptive-field size from parameter count permits larger receptive fields than PixelCNN.
- Results: The Image Transformer achieves a new ImageNet state of the art in log-likelihood despite comparatively low training-resource requirements.Experiments indicate significant improvements as the effective receptive field grows to 256 pixels, compared with 25 for PixelCNN using 5x5 filters.
- Conditional generation: The paper evaluates conditional image generation using image-class embeddings and a 4x super-resolution encoder-decoder configuration.Human evaluation found the super-resolution outputs convincingly natural significantly more often than those from recent autoregressive super-resolution work.
2. Background
The background situates Image Transformer among autoregressive likelihood-based image models and contrasts it with GANs. It highlights trade-offs involving output detail, training stability, diversity, and evaluability.
- Autoregressive models: Autoregressive image models factor pixel probabilities into conditional distributions and may model color channels as discrete multinomial outputs.This approach follows prior fully visible belief network, NADE, and PixelRNN formulations.
- Autoregressive models: PixelCNN++ achieves the CIFAR-10 state of the art using a discretized logistic mixture likelihood, whole-pixel conditioning, and architectural changes.The paper notes that these modifications could be applied to the Image Transformer in future work.
- Adversarial models: GANs can produce sharper images with realistic high-frequency detail than likelihood-based models in image generation and super-resolution tasks.Their generator is trained against a discriminator that distinguishes real from generated images.
- Adversarial models: GAN training is notoriously unstable, and mode collapse can make generated images fail to reflect training-set diversity.These drawbacks motivate methods aimed at making GAN training more robust.
- Evaluation: Because GANs lack a density in closed form, measuring diversity and objectively comparing design choices is more difficult.The paper contrasts this with models having a tractable likelihood.
3. Model Architecture
The Image Transformer represents image pixels and channels as positions processed by local self-attention and position-wise feed-forward layers. Encoder-decoder configurations support image-conditioned generation, while local memory blocks make attention scalable to larger images.
- 3.1. Image Representation: Pixel intensities are represented either as categorical channel values or ordinal values before adding coordinate encodings.Categorical representations use channel-specific embeddings for 0–255 values; ordinal representations combine the three channels with a strided convolution.
- 3.2. Self-Attention: The model stacks self-attention and position-wise feed-forward layers, with residual connections, dropout, and layer normalization.For image-conditioned generation, the encoder contextualizes the source image and the decoder autoregressively generates output intensities while consuming encoder representations.
- 3.2. Self-Attention: Each self-attention layer recomputes every pixel-channel representation by weighting representations from a memory of other positions.The mechanism compares a query with memory positions, applies a softmax attention distribution, transforms the memory, and computes a weighted average.
- 3.3. Local Attention: Local attention restricts each query position’s memory to a neighborhood, reducing the scalability burden while retaining parallel matrix-multiplication implementations.The attention operation has time complexity O(h · w · l_m · d), so limiting the memory size l_m is central to processing large images.
- 3.3. Local Attention: The experiments use two local-attention schemes that partition images into query blocks and associate each with a larger memory block.These schemes define different factorizations of the joint pixel distribution; 2D blocks extend above, left, and right of the query block.
4. Inference
Inference samples categorical channel intensities with a tempered softmax. The authors use τ = 1.0 for the highest perceptual quality in unconditioned and class-conditional generation, while reporting multiple temperatures for super-resolution.
- 4. Inference: Categorical sampling during decoding uses a tempered softmax with temperature τ > 0 applied to the channel-intensity logits.The temperature controls the concentration of the sampling distribution.
- 4. Inference: τ = 1.0 produced the highest perceptual quality for unconditioned and class-conditional image generation.For super-resolution, results are presented across different temperatures in Table 5.
5. Experiments
Experiments evaluate Image Transformer models for unconditional and class-conditioned image generation, plus CelebA super-resolution. The models achieve strong likelihood and perceptual results, with larger receptive fields improving image-modeling performance.
- Generative Image Modeling: 2.90 bits/dim is achieved by the best unconditional CIFAR-10 models, improving over PixelRNN and PixelCNN++ baselines.PixelSNAIL reaches 2.85 bits/dim on CIFAR-10, while the Image Transformer performs significantly better on ImageNet.
- Generative Image Modeling: 3.77 bits/dim establishes the Image Transformer’s ImageNet state of the art with checkpoint averaging.The model uses 12 self-attention and feed-forward layers, 512-dimensional representations, 8 attention heads, and 2048-dimensional feed-forward layers.
- Generative Image Modeling: Increasing receptive-field size significantly improves perplexity, while local self-attention keeps parameter count independent of receptive-field size.The experiments report significant improvements up to effective receptive-field sizes of 256 pixels.
- Conditioning on Image Class: Class-conditioned CIFAR-10 models have similar log-likelihoods but significantly higher perceptual quality than unconditioned models.The generated samples are realistic for categories including cars and trucks.
- Image Super-Resolution: The encoder-decoder super-resolution model enlarges low-resolution images and generates plausible, realistic high-resolution images across two datasets.The setup uses 8 × 8 inputs and 32 × 32 labels, trained end-to-end with a log-likelihood objective.
- Image Super-Resolution: 36.1% of human choices favored generated CelebA images, while the model achieved a Consistency score of 0.01 and an MS-SSIM score of 44.3.The human evaluation used paired generated and real images; the MS-SSIM analysis indicates synthesized details rather than merely copied training images.
6. Conclusion
The Image Transformer extends self-attention beyond text and improves probabilistic image modeling on ImageNet and super-resolution. The authors also identify broader conditioning, including free-form text, as future work.
- Self-attention models can operate beyond text and, through local attention, scale to structures larger than sentences.
- With fewer layers and larger receptive fields, the Image Transformer improves over the state of the art in ImageNet image modeling and super-resolution.
- Broader conditioning information, including free-form text, remains a direction for future work.