Source-linked AI summary
Conditional Image Generation with PixelCNN Decoders
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu
TL;DR
Image modelling often needs to generate images conditioned on labels, descriptions, states, or incomplete observations rather than unconditionally. The paper introduces Gated and Conditional PixelCNNs, combining efficient gated convolutions with latent-vector conditioning, and demonstrates strong likelihoods and diverse conditional generation. The models also support portrait variation and image decoding in autoencoders.
Problem
Practical image-modelling applications require generation conditioned on prior information, while PixelCNNs generally perform worse than PixelRNNs despite being faster to train.
Method
The paper introduces Gated PixelCNN with gated activations and dual convolutional stacks, then conditions it on latent vectors to model image distributions and decode autoencoder representations.
Results
The models generate diverse, realistic class-conditional images and varied portraits of the same person, while Gated PixelCNN matches or outperforms PixelRNN with lower computational cost.
Takeaways & Limitations
Conditional PixelCNN provides a flexible image generator and decoder for class labels, image embeddings, and autoencoder representations.
Takeaways & Limitations
The paper identifies caption-conditioned generation and variational autoencoders as future directions rather than demonstrated settings.
Abstract
from arXiv · showhide
This work explores conditional image generation with a new image density model based on the PixelCNN architecture. The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks. When conditioned on class labels from the ImageNet database, the model is able to generate diverse, realistic scenes representing distinct animals, objects, landscapes and structures. When conditioned on an embedding produced by a convolutional network given a single image of an unseen face, it generates a variety of new portraits of the same person with different facial expressions, poses and lighting conditions. We also show that conditional PixelCNN can serve as a powerful decoder in an image autoencoder. Additionally, the gated convolutional layers in the proposed model improve the log-likelihood of PixelCNN to match the state-of-the-art performance of PixelRNN on ImageNet, with greatly reduced computational cost.
1 Introduction
The paper develops conditional image modelling with PixelCNN, targeting applications that require images generated from prior information. It introduces Gated PixelCNN for faster training and Conditional PixelCNN for class- and embedding-based generation.
- Conditional image models support applications including visual-environment planning, denoising, deblurring, inpainting, super-resolution, colorization, and content generation.
- PixelCNN models explicit image probability densities with autoregressive pixel-by-pixel connections, unlike alternatives such as generative adversarial networks.
- Gated PixelCNN matches PixelRNN log-likelihood on CIFAR and ImageNet while requiring less than half the training time.
- Conditional PixelCNN generates diverse images from class labels or latent embeddings, including different poses of the same person from a single image.
2 Gated PixelCNN
The architecture combines autoregressive masked convolutions with gated activations, dual convolutional stacks, and conditioning on latent descriptions. These changes address PixelCNN’s receptive-field blind spot and support image decoding in autoencoders.
- PixelCNN: PixelCNN factorizes an image distribution into conditional pixel distributions ordered in raster-scan order, with each pixel depending on pixels above and to its left.
- PixelCNN: Masked convolutions enforce the dependency structure, while convolutions enable parallel prediction during training and sequential feedback during sampling.
- Gated convolutional layers: The Gated PixelCNN replaces rectified linear units with gated activations combining tanh and sigmoid transformations through element-wise multiplication.
- Blind spot in the receptive field: The original PixelCNN can ignore as much as a quarter of its potential receptive field because of a blind spot created by masked convolutions.
- Blind spot in the receptive field: Two convolutional stacks remove the blind spot by combining a horizontal stack for the current row with a vertical stack for rows above.
- Conditional PixelCNN: Conditional PixelCNN adds latent-vector-dependent terms before nonlinearities, allowing class labels or descriptions to control image content without specifying pixel locations.
- Image decoding: Conditional PixelCNN can replace a deconvolutional decoder in an end-to-end trained autoencoder, modelling diverse multimodal reconstructions.
3 Experiments
Experiments evaluate Gated PixelCNN on CIFAR-10 and ImageNet, then test conditional generation using class labels, portrait embeddings, and autoencoder bottlenecks. The results show improved likelihood, diverse conditional samples, and qualitatively different reconstructions from a PixelCNN decoder.
- Unconditional Modeling with Gated PixelCNN: 0.11 bits/dim improvement lets Gated PixelCNN outperform PixelCNN on CIFAR-10 and approach PixelRNN performance.The reported comparison uses bits/dim, where lower values are better.
- Unconditional Modeling with Gated PixelCNN: 60 hours using 32 GPUs achieves similar ImageNet performance to PixelRNN in less than half the training time.The ImageNet comparison reports performance in bits/dim and attributes the result to the larger Gated PixelCNN model.
- Conditioning on ImageNet Classes: Class conditioning produces distinct and diverse ImageNet scenes across eight classes, including varied viewpoints and lighting conditions.The model generalized from roughly 1000 images per animal or object class to new renderings.
- Conditioning on Portrait Embeddings: Portrait embeddings from a supervised convolutional network support generating varied new faces of an unseen person with preserved facial features.The generated portraits vary in poses and lighting conditions, while the source portrait data had widely varying quality.
- PixelCNN Auto Encoder: PixelCNN autoencoder reconstructions can sample different but similar-looking indoor scenes instead of exactly reproducing the input.The comparison used 10- or 100-dimensional bottlenecks against a convolutional autoencoder optimized for MSE.
4 Conclusion
The work establishes Gated PixelCNN as an efficient PixelCNN improvement and demonstrates Conditional PixelCNN for class-conditional generation, portrait variation, and image decoding. It also identifies single-example object generation and improved caption-conditioned generation as future directions.
- 4 Conclusion: Gated PixelCNN matches or outperforms PixelRNN while requiring less computation, using dual convolutional stacks and gating to address blind spots and improve convergence.The architecture achieves similar performance to PixelRNN on CIFAR-10 and state-of-the-art results on ImageNet 32x32 and 64x64 datasets.
- 4 Conclusion: Conditional PixelCNN generates diverse, realistic images across classes, new portraits of the same person, and varied autoencoder reconstructions.The demonstrated settings include class-conditional generation, portrait synthesis from a single image embedding, and PixelCNN decoding in an autoencoder.
- 4 Conclusion: A future direction is generating images of a specified animal or object from only a single example image.The paper also suggests combining Conditional PixelCNN with variational inference and using image captions instead of class labels.
- 4 Conclusion: Conditional PixelCNN is proposed as a possible way to improve blurry caption-conditioned samples from alignDRAW-like models.This direction is presented as a future possibility rather than an evaluated result.
Appendix
The appendix figures show class-conditional samples at two resolutions, portrait generation from latent representations, embedding interpolation, and PixelCNN autoencoder reconstructions.
- Appendix: Class-conditional samples are shown at multi-scale 64 × 64 resolution.
- Appendix: Class-conditional samples are also shown at 32 × 32 resolution.
- Appendix: Portrait figures pair a source image with new portraits generated from its high-level latent representation.
- Appendix: Embedding interpolations use the leftmost and rightmost image embeddings as endpoints decoded by PixelCNN.
- Appendix: Autoencoder comparisons show original images, MSE reconstructions, and conditional PixelCNN samples for 10- and 100-dimensional bottlenecks.