Source-linked AI summary
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P. Kingma
TL;DR
PixelCNN offers flexible, tractable image likelihoods, but its 256-way sub-pixel output and local architecture motivate simpler and more efficient alternatives. PixelCNN++ introduces discretized logistic mixtures, whole-pixel conditioning, multiscale connectivity, skip connections, and dropout, achieving state-of-the-art CIFAR-10 log-likelihood with coherent global structure.
Problem
PixelCNN’s 256-way sub-pixel softmax is costly and can yield sparse early gradients without representing proximity among neighboring intensity values.
Method
PixelCNN++ modifies PixelCNN with a discretized logistic mixture likelihood, whole-pixel conditioning, multiscale downsampling and upsampling, long-range skip connections, and dropout.
Results
PixelCNN++ achieves state-of-the-art log-likelihood on CIFAR-10 and generates images with coherent global structure.
Takeaways & Limitations
The reported experiments demonstrate the usefulness of PixelCNN++’s modifications for CIFAR-10 image generation, and the released code can be adapted to other datasets.
Abstract
from arXiv · showhide
PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.
1 INTRODUCTION
PixelCNN is a powerful image generative model with tractable likelihood, flexible shared convolutional conditionals, and parallel evaluation for observed images. The paper introduces an implementation with structural modifications intended to simplify the model and improve performance.
- PixelCNN factorizes an image probability distribution over sub-pixels using shared-parameter convolutional neural networks for the conditional distributions.
- Its flexible conditionals and GPU-parallel evaluation make PixelCNN effective for likelihood-based generative modeling.
- The authors developed and released an implementation that modifies the base PixelCNN to simplify its structure and improve performance.Experiments report state-of-the-art log-likelihood results supporting the usefulness of the modifications.
2 MODIFICATIONS TO PIXELCNN
PixelCNN++ replaces the costly sub-pixel softmax with a discretized logistic mixture and reorganizes computation around whole pixels and multiple spatial resolutions. Downsampling, skip connections, and dropout further shape the architecture and optimization behavior.
- 2.1 DISCRETIZED LOGISTIC MIXTURE LIKELIHOOD: The standard 256-way sub-pixel softmax is memory-intensive, can produce sparse early gradients, and does not encode proximity between neighboring values.
- 2.1 DISCRETIZED LOGISTIC MIXTURE LIKELIHOOD: PixelCNN++ models a continuous latent intensity that is rounded to the observed 8-bit value, using a logistic mixture to obtain smooth discretized probabilities.
- 2.1 DISCRETIZED LOGISTIC MIXTURE LIKELIHOOD: The discretized mixture avoids probability mass outside [0, 255], models edge values naturally, and uses relatively few components to produce denser gradients.The authors report faster convergence, especially early in training, but cannot isolate whether this applies to the original PixelCNN architecture.
- 2.2 CONDITIONING ON WHOLE PIXELS: Whole-pixel conditioning shares a mixture indicator across the three color channels and samples the channels sequentially within that component.
- 2.4 ADDING SHORT-CUT CONNECTIONS: The architecture uses downsampling and upsampling so inner layers operate at larger spatial scales, while long-range skip connections link corresponding early and late layers.The network is organized into sequences of layers separated mostly by sampling operations.
- 2.5 DROPOUT: Dropout regularizes the high-capacity model to help avoid overfitting and its associated low perceptual quality.
3 EXPERIMENTS
On CIFAR-10, PixelCNN++ achieves state-of-the-art log-likelihood and generates images with coherent global structure.
- PixelCNN++ achieves state-of-the-art log-likelihood results on CIFAR-10 while generating images with coherent global structure.
3.1 UNCONDITIONAL GENERATION ON CIFAR-10
The modified PixelCNN is applied to unconditional CIFAR-10 generation using a multiresolution residual architecture with dropout. The model achieves state-of-the-art test log-likelihood and produces samples illustrating the model's generative behavior.
- The CIFAR-10 model uses three residual blocks of five layers, 2 × 2 downsampling in encoding, upsampling in decoding, 192 feature maps, and dropout rate 0.5.
- The model achieves state-of-the-art test log-likelihood on CIFAR-10.
- Figure 3 presents samples generated by the PixelCNN trained on CIFAR-10.
3.2 CLASS-CONDITIONAL GENERATION
The model is also evaluated conditionally on CIFAR-10 class labels, with generated samples compared against real images. Class conditioning makes overfitting harder to avoid and yields a best test log-likelihood of 2.94.
- Class conditioning projects a one-hot class label into a separate bias vector for each convolutional unit.
- 2.94 is the best test log-likelihood reported for the class-conditional model.
- Figure 4 compares class-conditional PixelCNN samples on the left with real CIFAR-10 images on the right.
3.3 EXAMINING NETWORK DEPTH AND FIELD OF VIEW SIZE
Experiments test whether large receptive fields are necessary for CIFAR-10 generation. Small-receptive-field PixelCNNs can remain competitive when capacity is restored through additional within-field connections, although their samples lack global structure.
- The original PixelCNN's blind spot is relevant because receptive-field size and blind-spot removal had been hypothesized to affect performance.
- The study examines receptive fields of 11x5 and 15x8 in PixelCNN++ models without downsampling blocks and with fewer layers.
- Limiting receptive-field size reduces network capacity because the model contains many fewer layers.
- NIN inserts gated ResNet blocks with 1x1 convolutions, while Autoregressive Channel adds gated 1x1-convolution skip connections between channel sets.
- Both capacity-increasing modifications improve log-likelihood, but small-receptive-field samples lack global structure.
3.4 ABLATION EXPERIMENTS
Ablation experiments evaluate the logistic mixture likelihood, discretization, shortcut connections, and dropout. The results show benefits from discretization and shortcuts, while removing shortcuts prevents training and removing dropout leads to severe overfitting.
- Replacing the discretized logistic mixture with a 256-way softmax tests the contribution of the likelihood parameterization.
- Removing discretization and adding standard uniform noise yields 3.11 bits per dimension versus 2.92 bits per dimension with discretization.
- The continuous model's 3.11 bits per dimension versus 2.92 bits per dimension shows the benefit of discretization in the likelihood model.
- Removing the additional shortcut connections causes the model to fail to train, likely because subsampling discards information that is difficult to recover.
- Without dropout, training log-likelihood falls below 2.0 bits per sub-pixel while final test log-likelihood rises above 6.0 bits per sub-pixel.
- The intentionally overfitted model produces poor perceptual quality despite its low training log-likelihood.
4 CONCLUSION
PixelCNN++ modifies PixelCNN with a discretized logistic mixture likelihood and other changes, achieving state-of-the-art results on CIFAR-10.
- PixelCNN++ uses a discretized logistic mixture likelihood on pixels alongside other modifications to PixelCNN.
- The modifications were demonstrated to be useful through state-of-the-art results on CIFAR-10.
- The released code is available at https://github.com/openai/pixel-cnn and can be adapted for other datasets.