Source-linked AI summary
Parallel Multiscale Autoregressive Density Estimation
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, Nando de Freitas
TL;DR
PixelCNN offers strong density estimation but still generates pixels sequentially, making inference costly. This paper introduces a multiscale PixelCNN that models selected pixel groups as conditionally independent and reports efficient, high-resolution generation across image and video tasks. Its central result is O(log N) sampling instead of O(N), with competitive density estimation and fast-sampling performance across benchmarks.
Problem
PixelCNN inference remains costly because generation is serial by pixel even after caching activations, motivating more parallel sampling without discarding useful dependencies.
Method
The paper uses a coarse-to-fine multiscale PixelCNN that models selected pixel groups as conditionally independent given lower-resolution images and contextual information.
Results
The model achieves competitive density estimation across CUB, MPII, MS-COCO, ImageNet, and Robot Pushing videos, surpassing other density models that admit fast sampling.
Takeaways & Limitations
Sampling scales as O(log N) instead of O(N), enabling orders-of-magnitude speedups and diverse generation up to 512 × 512.
Takeaways & Limitations
In action-conditional video generation, the O(T log N) model degraded on testing configurations with novel arms and cameras, apparently overfitting training backgrounds and camera positions.
Abstract
from arXiv · showhide
PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N) for N pixels. This can be sped up by caching activations, but still involves generating each pixel sequentially. In this work, we propose a parallelized PixelCNN that allows more efficient inference by modeling certain pixel groups as conditionally independent. Our new PixelCNN model achieves competitive density estimation and orders of magnitude speedup - O(log N) sampling instead of O(N) - enabling the practical generation of 512x512 images. We evaluate the model on class-conditional image generation, text-to-image synthesis, and action-conditional video generation, showing that our model achieves the best results among non-pixel-autoregressive density models that allow efficient sampling.
1. Introduction
PixelCNN trains efficiently but samples sequentially, motivating a multiscale formulation that models selected pixel groups as conditionally independent. The proposed approach cuts spatial dependencies to obtain O(log N) sampling while retaining competitive density estimation.
- Inference bottleneck: PixelCNN inference remains serial by pixel even after caching hidden activations, limiting sampling efficiency.The naive approach requires a full network evaluation per pixel; caching reduces computation but not serial generation.
- Motivation: Parallel sampling requires modeling selected pixels as independent, while preserving strong dependencies between immediately neighboring pixels.The method targets weak dependencies because neighboring pixels tend to be highly correlated.
- Approach: Multiscale generation models pixel groups as conditionally independent given lower-resolution images and contextual information such as preceding video frames.The implementation must also address global information transfer, group selection, and border artifacts during merging.
- Results: The model samples N pixels in O(log N) time instead of O(N), producing orders of magnitude speedup in practice.For video with high-resolution previous frames, sampling can reach O(1) time and outperform comparably fast baselines.
- Formulation: The paper’s group-factor formulation merges per-pixel factors, trading some expressiveness for parallel sampling within each group.Dividing N pixels into G groups yields a product of corresponding group factors.
2. Related work
The work builds on autoregressive, convolutional, adversarial, flow-based, and multiscale density models while targeting faster inference and higher resolution. Its stated contributions combine conditional-independence-based speedups with broad conditional-generation evaluations.
- Autoregressive models: Autoregressive LSTMs and causally structured convolutional networks established strong density estimation and improved training speed and scalability.PixelCNN and WaveNet preserve causal structure while allowing factor learning in parallel for a given image.
- Non-autoregressive generators: Non-autoregressive generator networks, including GANs, support conditional image generation but optimize objectives other than likelihood.The cited applications include text- and spatial-structure-to-image generation.
- Multiscale modeling: Multiscale structures have been used in GANs and earlier multiscale PixelRNN or PixelCNN models for coarse-to-fine image generation.Earlier multiscale autoregressive models condition each resolution on the next lower-resolution image.
- Density-model alternatives: Real NVP and related tractable density models provide alternative efficient-generation mechanisms, including invertible mappings and multiscale structures.Real NVP maps images to a simple noise distribution through invertible coupling layers.
- This work: The paper contributes faster inference through conditional-independence structure, higher-resolution scaling, and evaluations across class-, text-, structure-conditional, and video generation.The work positions these contributions across a diverse set of challenging benchmarks.
3. Model
The model orders pixels from coarse to fine, grouping spatially local pixels so groups are generated sequentially while pixels within each group are processed in parallel. Recursive upscaling yields O(log N) image generation, with conditioning added throughout the factors.
- Coarse-to-fine model: Successively higher-resolution images are generated conditioned on the previous resolution in a coarse-to-fine ordering.Pixels are grouped to exploit spatial locality at each resolution.
- Factorization: The joint distribution factorizes over pixel groups and scales, allowing training losses and gradients to be computed in parallel.The training objective maximizes log P(x; θ).
- Pixel grouping: Four groups at each scale correspond to the upper-left, upper-right, lower-left, and lower-right corners of tiled 2 × 2 blocks.No two adjacent high-resolution pixels share a group, while additional dependencies help capture local textures and avoid border artifacts.
- Inference: At test time, inference proceeds sequentially over pixel groups but in parallel within each group, with channel dependencies modeled by masking.The same grouped structure enables parallel loss and gradient computation during training.
- Upscaling variants: Type-A upscaling uses three network evaluations per pixel group, while type-B divides feature maps into M × M patches and uses M^2 small evaluations per color channel.The reported implementation uses M = 4 with shared shallow PixelCNN weights across patches.
- Artifact control: Non-overlapping predicted patches avoid border artifacts because each pixel receives adjacent context and generated patches are separated by output rows or columns.The merge and split operations only marshal data and have no learnable parameters.
- Complexity: Recursive generation across doubled resolutions requires O(1) evaluations per scale, giving O(log N) complexity for N-pixel images.The first group is generated recursively down to a base resolution using a standard PixelCNN.
- Conditional generation: Conditional likelihood adds context c, such as text, segmentation, or previous video frames, as an input to every distribution factor and upscaling network.Text uses a character-CNN-GRU, spatial structure a convolutional network, and previous frames a ConvLSTM.
4. Experiments
Experiments evaluate the model across conditional image and video generation, density estimation, and sampling speed. Results show competitive likelihood, high-resolution synthesis, substantial speedups, and a video generalization limitation under novel camera configurations.
- Experimental setup: The model is evaluated on ImageNet, CUB, MPII, MS-COCO, and Robot Pushing datasets using class, text, structure, and action conditioning.Samples use validation queries not seen during training.
- Conditional image generation: The model achieves likelihood scores not far from original PixelCNN while increasing sample resolution by 8×, with qualitatively greater realism at higher resolution.This result is reported for text- and structure-conditioned image generation.
- Action-conditional video generation: For video, the O(T log N) model outperforms the O(T) model on validation but underperforms the original O(TN) model and drops on testing configurations.The testing drop is attributed to possible overfitting to training-arm backgrounds and camera positions.
- Density estimation: On 32 × 32 and 64 × 64 ImageNet, it achieves significantly better likelihood scores than reported non-pixel-autoregressive models that also support efficient sampling.The comparison includes ConvDRAW and Real NVP.
- Sampling speed: Sampling is two orders of magnitude faster than standard PixelCNN at 32 × 32, with greater speedups at higher resolution and O(log N) network evaluations.The resulting computation graph can fit in memory for reasonable batch sizes.
5. Conclusions
The paper presents a parallelized, multiscale PixelCNN that delivers competitive density estimation and high-resolution generation across several tasks. It also reports broad qualitative capabilities, including text-to-image synthesis, video generation, and super-resolution.
- The multiscale PixelCNN achieves competitive density estimation on CUB, MPII, MS-COCO, ImageNet, and Robot Pushing videos.
- It surpasses other density models that admit fast sampling while supporting diverse generation tasks.
- The model produces qualitative results in text-to-image synthesis, video generation, and super-resolution up to 512 × 512.
6. Appendix
The appendix provides additional generated samples spanning conditional image generation, video generation, and upscaling. These examples cover multiple datasets, resolutions, and conditioning settings.
- Additional MPII samples are randomly chosen from the validation set.
- Additional MS-COCO samples are randomly chosen from the validation set.
- The appendix includes label-conditional 128 × 128 ImageNet samples and additional upscaling samples.