Source-linked AI summary
PixelSNAIL: An Improved Autoregressive Generative Model
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, Pieter Abbeel
TL;DR
Autoregressive generative models need architectures that can access long-range dependencies while retaining tractable likelihood computation. PixelSNAIL combines causal convolutions with self-attention for autoregressive modeling and demonstrates state-of-the-art density estimation on CIFAR-10 and ImageNet 32 × 32. Sampling remains comparable in speed to existing autoregressive models, leaving faster sampling without performance loss open.
Problem
Autoregressive models require neural architectures that can effectively access long-range dependencies in high-dimensional sequential data.
Source: §1 · p. 1Method
PixelSNAIL interleaves causal convolutions with masked self-attention in an autoregressive generative model.
Source: §1 · p. 2, §2 · pp. 2–3Results
PixelSNAIL demonstrates state-of-the-art density estimation performance on CIFAR-10 and ImageNet 32 × 32.
Source: §4 · p. 4Takeaways & Limitations
Combining causal convolutions with self-attention provides the paper’s supported architecture for autoregressive density estimation on the evaluated image datasets.
Source: §1 · p. 2, §4 · p. 4PixelSNAIL sampling remains comparable in speed to existing autoregressive models because autoregressive generation samples each pixel sequentially.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions, which offer better access to earlier parts of the sequence than conventional RNNs. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this note, we describe the resulting model and present state-of-the-art log-likelihood results on CIFAR-10 (2.85 bits per dim) and $32 \times 32$ ImageNet (3.80 bits per dim). Our implementation is available at https://github.com/neocxi/pixelsnail-public
1 Introduction
Autoregressive models offer tractable likelihoods for high-dimensional data, but their architectures must access long-range dependencies effectively. PixelSNAIL combines causal convolutions and self-attention to address their complementary limitations in autoregressive generation.
Autoregressive modeling
- Autoregressive models factor a high-dimensional joint distribution into conditional distributions over sequentially generated elements.An RNN models each conditional distribution given preceding elements, optionally with additional global information.Source: §1 · p. 1
Architectural limitations
- Traditional RNNs struggle with long-range relationships because information propagates through a temporally linear hidden-state dependency.This limitation applies to architectures such as GRUs and LSTMs.Source: §1 · p. 1
- Causal convolutions provide high-bandwidth access to earlier sequence elements but have finite receptive fields and attenuate distant information.Masking or shifting ensures current predictions depend only on previous elements.Source: §1 · p. 1
- Self-attention offers an unbounded receptive field and undeteriorated access to distant information, but accesses only small amounts at a time and needs positional information.Source: §1 · p. 2
PixelSNAIL motivation
- Interleaving causal convolutions with self-attention combines high-bandwidth local aggregation with long-range attentive lookup.The convolutional pathway aggregates context for the subsequent attention operation.Source: §1 · p. 2
- PixelSNAIL applies this combined architecture to autoregressive image modeling and reports state-of-the-art results on CIFAR-10 and ImageNet 32 × 32.Source: §1 · p. 2
2 Model Architecture
PixelSNAIL interleaves causal residual convolutional blocks with masked self-attention blocks. Its implementation uses fixed block widths, dataset-specific regularization, Polyak averaging, and mixture-logistic output distributions.
Building blocks
- PixelSNAIL is primarily composed of residual blocks and attention blocks that are interleaved throughout the architecture.Source: §2 · p. 2, Figure 2 · p. 3
Residual block
- Residual blocks use masked 2D convolutions so each pixel accesses only pixels to its left and above, with gated activations.The model uses four convolutions per block and 256 filters in each convolution.Source: §2 · pp. 2–3
Attention block
- Attention blocks project inputs into keys and values, then perform masked softmax attention over previously generated pixels.Keys have size 16 and values have size 128.Source: §2 · p. 3
Full architecture
- The full architecture uses 12 blocks for both datasets, with 10 mixture components for CIFAR-10 and 32 for ImageNet.Source: Figure 2 · p. 3
Training and output
- Dropout is applied only to CIFAR-10, while both datasets use Polyak averaging with dataset-specific exponential moving average weights.The weights are 0.9995 for CIFAR-10 and 0.9997 for ImageNet.Source: §2 · p. 3
3 Experiments
PixelSNAIL is evaluated using negative log-likelihood on CIFAR-10 and ImageNet 32 × 32 against autoregressive architectures based on recurrent networks, causal convolutions, or attention alone.
Results
- PixelSNAIL outperforms PixelRNN, PixelCNN, PixelCNN++, and Image Transformer on negative log-likelihood across CIFAR-10 and ImageNet 32 × 32.The comparison metric is negative log-likelihood measured in bits per dimension.Source: §3 · p. 3, Table 1 · p. 4
Comparisons
- The evaluated baselines represent LSTM-based, causal-convolution-only, and attention-only autoregressive models.Source: §3 · p. 3
Evaluation
- Table 1 reports average negative log-likelihoods for both datasets in bits per dimension.Source: Table 1 · p. 4
4 Conclusion
The paper concludes that PixelSNAIL combines causal convolutions and self-attention for state-of-the-art density estimation on CIFAR-10 and ImageNet 32 × 32. Sampling remains as slow as in existing autoregressive models.
Conclusion
Limitation
- Autoregressive sampling remains slow because each pixel must be sampled sequentially.PixelSNAIL’s sampling speed is comparable to that of existing autoregressive models.Source: §4 · p. 4, Figure 4 · p. 5
Related papers
This paper cites
Cited by
- Post-training Quantization for Hybrid Iterative Generative Models2026
- A Continuous Time Framework for Discrete Denoising Models2022
- Video Diffusion Models2022
- NVAE: A Deep Hierarchical Variational Autoencoder2020
- Efficient Content-Based Sparse Attention with Routing Transformers2020
- Generating Long Sequences with Sparse Transformers2019