Source-linked AI summary
Pixel Recurrent Neural Networks
Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu
TL;DR
Modeling natural-image distributions requires models that are expressive, tractable, and scalable. The paper introduces PixelRNNs and related PixelCNNs that sequentially model discrete raw RGB pixels while preserving their dependencies. The models improve likelihood results on MNIST and CIFAR-10, establish ImageNet benchmarks, and generate sharp, coherent samples.
Problem
Natural-image distributions are difficult to estimate because image data are high-dimensional and highly structured, while useful models must remain expressive, tractable, and scalable.
Method
The paper develops fast two-dimensional LSTM-based PixelRNNs, a masked-convolutional PixelCNN, discrete softmax modeling of raw RGB values, and residual connections for deep recurrent networks.
Results
PixelRNNs significantly improve the state of the art on MNIST and CIFAR-10, provide new ImageNet generative-model benchmarks, and produce sharp, coherent samples.
Takeaways & Limitations
The samples and completions indicate that PixelRNNs model both spatially local and long-range correlations in natural images.
Abstract
from arXiv · showhide
Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent.
1. Introduction
The paper develops PixelRNNs and PixelCNNs to model natural-image distributions while balancing expressiveness, tractability, and scalability. These models preserve pixel dependencies, use discrete RGB distributions, and improve reported likelihood results on standard datasets while introducing ImageNet benchmarks.
- Generative image modeling is challenging because natural images are high-dimensional, highly structured, and require models that are expressive, tractable, and scalable.
- PixelRNNs advance two-dimensional recurrent modeling with Row LSTM and Diagonal BiLSTM layers for large-scale natural-image modeling.The networks use up to twelve LSTM layers and apply convolutions along rows or diagonals.
- PixelCNN provides a simplified fully convolutional sequence model with fifteen masked-convolutional layers that preserve spatial resolution and output conditional distributions at each location.
- Both architectures encode complete pixel inter-dependencies, including dependencies among RGB channels, without the independence assumptions used by latent-variable models.
- The models treat raw pixel channels as discrete variables with multinomial softmax distributions, which the authors report provides representational and training advantages.
- PixelRNNs obtain considerably better log-likelihood scores than previous results on MNIST and CIFAR-10, and provide generative-model likelihood benchmarks for ImageNet.
2. Model
The model factorizes an image distribution into conditional pixel predictions generated row by row and pixel by pixel. It shares prediction parameters across positions while modeling RGB-channel dependencies with discrete softmax distributions.
- The network estimates an image distribution by scanning rows and pixels sequentially, predicting each pixel from the previously scanned context with shared parameters.
- The joint image distribution is factorized into a product of conditional distributions over a row-by-row pixel sequence.For pixel xi, the conditional distribution is p(xi|x1, ..., xi−1).
- Each RGB pixel is modeled through a conditional product in which red, green, and blue values depend on prior pixels and previously generated channels.
- Training and evaluation compute pixel-value distributions in parallel, whereas image generation proceeds sequentially.
- Each pixel channel takes one of 256 values, and every conditional distribution is modeled as a multinomial discrete distribution with a softmax layer.
3. Pixel Recurrent Neural Networks
PixelRNNs model images autoregressively with recurrent or convolutional architectures that preserve pixel dependencies, using specialized spatial layers, residual connections, and masked color conditioning.
- LSTM Architectures: PixelRNN architectures use convolutional LSTM layers to compute spatial states efficiently, including Row LSTM and Diagonal BiLSTM variants.Row LSTM processes rows with one-dimensional convolutions, while Diagonal BiLSTM processes image diagonals in two directions.
- LSTM Architectures: Diagonal BiLSTM skews the input map for diagonal parallelization, applies a 2 × 1 state-to-state convolution, and shifts the output back to image coordinates.The skewed map has size n × (2n −1), and offset positions are removed after computation.
- LSTM Architectures: The Diagonal BiLSTM captures the entire available context for any image size, so larger recurrent kernels do not broaden its receptive field.Its minimal 2 × 1 kernel processes limited information at each step while retaining a global receptive field.
- Residual Connections: Residual connections pass information between deep LSTM layers and use a 1 × 1 convolution to restore the input feature width before addition.The paper trains PixelRNNs with up to twelve layers and uses residual connections to improve convergence speed and signal propagation.
- Masked Convolution: Masked conditioning preserves autoregressive validity across RGB channels: R uses prior spatial context, G also uses R, and B uses both R and G.Mask A is used in the first convolution, while mask B is used later and permits a color to connect to itself.
- PixelCNN: PixelCNN provides a faster fully convolutional alternative using stacked masked convolutions with a bounded receptive field and spatially preserved feature maps.Convolutional layers compute features for all pixel positions simultaneously, unlike sequential recurrent state updates.
- Multi-Scale PixelRNN: The Multi-Scale PixelRNN first generates a subsampled s × s image, then conditions a second PixelRNN on it to generate an n × n image.The conditional network incorporates an upsampled version of the smaller image as layer-wise bias.
4. Specifications of Models
The experiments evaluate four PixelRNN-family architectures across MNIST, CIFAR-10, and two ImageNet resolutions, with architecture depth and hidden size adjusted by dataset.
- Model Types: The evaluated models are Row LSTM PixelRNN, Diagonal BiLSTM PixelRNN, fully convolutional PixelCNN, and Multi-Scale PixelRNN.These four network types define the experimental model specifications.
- Single-Scale Architectures: Single-scale networks begin with a 7 × 7 type-A masked convolution, followed by recurrent or PixelCNN layers using type-B masking where applicable.The output stack uses ReLU and 1 × 1 convolution layers, with residual and layer-to-output connections across all three single-scale networks.
- Dataset Configurations: MNIST uses a 7-layer Diagonal BiLSTM with h = 16, while CIFAR-10 uses 12-layer LSTMs or a 15-layer PixelCNN with h = 128.The ImageNet models use a 12-layer Row LSTM with h = 384 for 32 × 32 images and a 4-layer Row LSTM with h = 512 for 64 × 64 images.
5. Experiments
Experiments evaluate PixelRNN and PixelCNN architectures, discrete softmax outputs, residual connections, depth, and performance across MNIST, CIFAR-10, and ImageNet. The models achieve strong likelihood results while producing samples that capture local and global image structure.
- 5.3. Discrete Softmax Distribution: Discrete softmax outputs achieve 3.06 bits/dim on CIFAR-10, versus 3.22 bits/dim for an MCGSM output in the same Row LSTM model.The discrete formulation also supports multimodal, skewed, peaked, and long-tailed distributions without probability mass outside [0, 255].
- 5.4. Residual Connections: Residual connections are as effective as skip connections in the 12-layer CIFAR-10 Row LSTM, while combining both preserves the advantage.The comparison is reported in bits/dim on the CIFAR-10 validation set.
- 5.4. Residual Connections: With both residual and skip connections, Row LSTM performance improves as depth increases through the 12 LSTM layers tested.The depth comparison uses negative log-likelihood on the CIFAR-10 validation set.
- 5.7. ImageNet: Samples capture local spatial dependencies, while ImageNet models better capture global structure and multi-scale conditioning improves coherence at 64×64 resolution.The 64×64 single-scale and multi-scale models have similar log-likelihood, but the multi-scale samples appear more globally coherent.
- 5.5. MNIST: The Diagonal BiLSTM achieves the best reported result on binary MNIST according to the paper.MNIST is used as a sanity check because it has substantial prior comparative work.
- 5.6. CIFAR-10: The Diagonal BiLSTM gives the best performance on CIFAR-10, followed by the Row LSTM and PixelCNN, consistent with their respective receptive-field sizes.All proposed-network results were obtained without data augmentation.
- 5.7. ImageNet: On ImageNet, validation performance improves with model size and depth, while computation time and GPU memory constrain model size.The paper reports ImageNet log-likelihood benchmarks without data augmentation.
- 5.7. ImageNet: ImageNet images are less compressible than CIFAR-10 images because they are less blurry and therefore have less predictable neighboring pixels.The authors note that the downsampling method can influence compression performance.
6. Conclusion
The paper develops deeper recurrent generative models for natural images using two-dimensional LSTM layers, discrete pixel modeling, masked convolutions, and residual connections. PixelRNNs improve prior results on MNIST and CIFAR-10, establish ImageNet benchmarks, and generate sharp, coherent images capturing local and long-range correlations.
- 6. Conclusion: PixelRNNs use Row LSTM and Diagonal BiLSTM layers, discrete softmax modeling of raw RGB values, masked convolutions, and residual connections in networks up to 12 LSTM layers.These components are presented as architectural improvements for scalable natural-image generation.
- 6. Conclusion: The models significantly improve state-of-the-art results on MNIST and CIFAR-10 while providing new generative image-modeling benchmarks on ImageNet.The conclusion reports these outcomes without supplying the corresponding numerical table values here.
- 6. Conclusion: Samples and completions indicate that PixelRNNs model local and long-range spatial correlations and produce sharp, coherent images.The conclusion connects these properties to generated samples and image completions.