Source-linked AI summary
PixelSNAIL: An Improved Autoregressive Generative Model
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, Pieter Abbeel
TL;DR
Autoregressive generative models need architectures that can access long-range dependencies while retaining tractable likelihood computation. PixelSNAIL combines causal convolutions with self-attention for autoregressive modeling and demonstrates state-of-the-art density estimation on CIFAR-10 and ImageNet 32 × 32. Sampling remains comparable in speed to existing autoregressive models, leaving faster sampling without performance loss open.
Problem
Autoregressive models require neural architectures that can effectively access long-range dependencies in high-dimensional sequential data.
Method
PixelSNAIL interleaves causal convolutions with masked self-attention in an autoregressive generative model.
Results
PixelSNAIL demonstrates state-of-the-art density estimation performance on CIFAR-10 and ImageNet 32 × 32.
Takeaways & Limitations
Combining causal convolutions with self-attention provides the paper’s supported architecture for autoregressive density estimation on the evaluated image datasets.
Takeaways & Limitations
PixelSNAIL sampling remains comparable in speed to existing autoregressive models because autoregressive generation samples each pixel sequentially.
Abstract
from arXiv · showhide
Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions, which offer better access to earlier parts of the sequence than conventional RNNs. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this note, we describe the resulting model and present state-of-the-art log-likelihood results on CIFAR-10 (2.85 bits per dim) and $32 \times 32$ ImageNet (3.80 bits per dim). Our implementation is available at https://github.com/neocxi/pixelsnail-public
1 Introduction
Autoregressive models offer tractable likelihoods for high-dimensional data, but their architectures must access long-range dependencies effectively. PixelSNAIL combines causal convolutions and self-attention to address their complementary limitations in autoregressive generation.
- Autoregressive modeling: Autoregressive models factor a high-dimensional joint distribution into conditional distributions over sequentially generated elements.An RNN models each conditional distribution given preceding elements, optionally with additional global information.
- Architectural limitations: Traditional RNNs struggle with long-range relationships because information propagates through a temporally linear hidden-state dependency.This limitation applies to architectures such as GRUs and LSTMs.
- Architectural limitations: Causal convolutions provide high-bandwidth access to earlier sequence elements but have finite receptive fields and attenuate distant information.Masking or shifting ensures current predictions depend only on previous elements.
- Architectural limitations: Self-attention offers an unbounded receptive field and undeteriorated access to distant information, but accesses only small amounts at a time and needs positional information.
- PixelSNAIL motivation: Interleaving causal convolutions with self-attention combines high-bandwidth local aggregation with long-range attentive lookup.The convolutional pathway aggregates context for the subsequent attention operation.
- PixelSNAIL motivation: PixelSNAIL applies this combined architecture to autoregressive image modeling and reports state-of-the-art results on CIFAR-10 and ImageNet 32 × 32.
2 Model Architecture
PixelSNAIL interleaves causal residual convolutional blocks with masked self-attention blocks. Its implementation uses fixed block widths, dataset-specific regularization, Polyak averaging, and mixture-logistic output distributions.
- Building blocks: PixelSNAIL is primarily composed of residual blocks and attention blocks that are interleaved throughout the architecture.
- Residual block: Residual blocks use masked 2D convolutions so each pixel accesses only pixels to its left and above, with gated activations.The model uses four convolutions per block and 256 filters in each convolution.
- Attention block: Attention blocks project inputs into keys and values, then perform masked softmax attention over previously generated pixels.Keys have size 16 and values have size 128.
- Full architecture: The full architecture uses 12 blocks for both datasets, with 10 mixture components for CIFAR-10 and 32 for ImageNet.
- Training and output: Dropout is applied only to CIFAR-10, while both datasets use Polyak averaging with dataset-specific exponential moving average weights.The weights are 0.9995 for CIFAR-10 and 0.9997 for ImageNet.
3 Experiments
PixelSNAIL is evaluated using negative log-likelihood on CIFAR-10 and ImageNet 32 × 32 against autoregressive architectures based on recurrent networks, causal convolutions, or attention alone.
- Results: PixelSNAIL outperforms PixelRNN, PixelCNN, PixelCNN++, and Image Transformer on negative log-likelihood across CIFAR-10 and ImageNet 32 × 32.The comparison metric is negative log-likelihood measured in bits per dimension.
- Comparisons: The evaluated baselines represent LSTM-based, causal-convolution-only, and attention-only autoregressive models.
- Evaluation: Table 1 reports average negative log-likelihoods for both datasets in bits per dimension.
4 Conclusion
The paper concludes that PixelSNAIL combines causal convolutions and self-attention for state-of-the-art density estimation on CIFAR-10 and ImageNet 32 × 32. Sampling remains as slow as in existing autoregressive models.
- Conclusion: PixelSNAIL combines causal convolutions with self-attention in an autoregressive generative model.
- Conclusion: PixelSNAIL achieves state-of-the-art density estimation performance on CIFAR-10 and ImageNet 32 × 32.The implementation is publicly available.
- Limitation: Autoregressive sampling remains slow because each pixel must be sampled sequentially.PixelSNAIL’s sampling speed is comparable to that of existing autoregressive models.