PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

Jacob Menick, Nal Kalchbrenner

arXiv:1812.01608v1cs.CVcs.GRcs.LGstat.ML

TL;DR

High-fidelity unconditional generation of large images remains difficult because models must preserve global coherence and detailed accuracy while encoding extensive context. The paper introduces Subscale Pixel Networks and Multidimensional Upscaling to generate images through interleaved slices and staged growth in size and bit depth. It achieves strong likelihood results and high-fidelity samples across CelebAHQ and ImageNet, including previously unexplored large-image settings.

  • Problem

    High-fidelity unconditional generation of large images remains open because autoregressive models must encode vast context while preserving global coherence and exact detail.

  • Method

    Subscale Pixel Networks generate equal-sized image slices sequentially, while Multidimensional Upscaling grows images through stages in spatial size and channel depth.

  • Results

    The model achieves state-of-the-art likelihood results in multiple settings and generates high-fidelity unconditional samples for CelebAHQ-256 and ImageNet up to 256 × 256.

  • Takeaways & Limitations

    The approach demonstrates high-resolution unconditional samples with strong semantic coherence and detailed accuracy across complex image domains.

Abstract

from arXiv · show

The unconditional generation of high fidelity images is a longstanding benchmark for testing the performance of image decoders. Autoregressive image models have been able to generate small images unconditionally, but the extension of these methods to large images where fidelity can be more readily assessed has remained an open problem. Among the major challenges are the capacity to encode the vast previous context and the sheer difficulty of learning a distribution that preserves both global semantic coherence and exactness of detail. To address the former challenge, we propose the Subscale Pixel Network (SPN), a conditional decoder architecture that generates an image as a sequence of sub-images of equal size. The SPN compactly captures image-wide spatial dependencies and requires a fraction of the memory and the computation required by other fully autoregressive models. To address the latter challenge, we propose to use Multidimensional Upscaling to grow an image in both size and depth via intermediate stages utilising distinct SPNs. We evaluate SPNs on the unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32 to 256. We achieve state-of-the-art likelihood results in multiple settings, set up new benchmark results in previously unexplored settings and are able to generate very high fidelity large scale samples on the basis of both datasets.

1 INTRODUCTION

The paper targets high-fidelity unconditional generation of large images, where autoregressive models face difficult fidelity–likelihood trade-offs and prohibitive context costs. It introduces Subscale Pixel Networks and Multidimensional Upscaling to address these challenges, and reports strong likelihood and sample-fidelity results on CelebAHQ and ImageNet.

  • Motivation: Large-image autoregressive generation must model 196,608 image positions, creating severe memory and computation demands.The paper also notes that maximum likelihood allocates capacity across the full empirical distribution, including regions less relevant to visual fidelity.
  • Approach: The method focuses generation on salient low-resolution slices and significant bits, then maps between them through size and depth upscaling.For ImageNet, the example pipeline grows a 32 × 32 3-bit RGB image to 128 × 128 3-bit and then to 8-bit depth.
  • Approach: Subscale Pixel Networks generate equal-sized interleaved image slices sequentially while conditioning on previously generated slices.The architecture uses a conditioning network for previous slices and a decoder that predicts each target slice, with shared decoding weights across structurally similar slices.
  • Evaluation: The paper evaluates SPNs and multidimensional upscaling on unconditional CelebAHQ-256 and ImageNet at sizes up to 256.It reports state-of-the-art MLE results on CelebAHQ-256 and ImageNet-64, while establishing MLE baselines for ImageNet-128 and ImageNet-256.
  • Results: The models produce high-fidelity CelebAHQ-256 samples and successful unconditional ImageNet-128 samples with strong visual coherence.The CelebAHQ-256 samples have visual fidelity similar to GAN-based methods, while ImageNet-128 provides a fidelity baseline for future methods.

2 MODEL

The model introduces subscale ordering and multidimensional upscaling to organize autoregressive image generation across spatial slices and channel-depth stages. These designs compactly capture long-range dependencies, support shared slice decoding, and prioritize visually salient image information.

  • 2.1 CONVENTIONAL GENERATION ORDERING: The conventional autoregressive ordering generates each pixel’s three 8-bit channels from top left to bottom right using all previously generated intensity values.This raster-scan ordering is parametrized by a deep neural network for each conditional distribution.
  • 2.2 SUBSCALE ORDERING IN IMAGES: Subscale ordering divides a large image into equally sized interleaved slices generated sequentially, compactly encoding long-range spatial dependencies.With scaling factor S, the image yields S^2 slices of size H/S × W/S, each identified by row and column offsets.
  • 2.2 SUBSCALE ORDERING IN IMAGES: A shared decoder can process all slices because they have the same spatial structure, while smaller slices permit self-attention without local contexts.The ordering also induces an implicit size-upscaling structure over the original image.
  • 2.3 SIZE UPSCALING IN SUBSCALE ORDERING: Explicit size upscaling generates the first subscale slice with a single-slice decoder, then generates the remaining slices with the main network.The single-slice model may train on first slices or on slices from all positions because the slices share spatial structure.
  • 2.4 DEPTH UPSCALING: Multidimensional upscaling extends staged generation across image height, width, and channel depth using distinct networks for successive bit groups.Later stages generate less significant bits conditioned on previously generated more significant bits.

3 ARCHITECTURE

The Subscale Pixel Network generates images as sequences of equal-sized slices, encoding spatial dependencies while keeping computation and memory effectively constant as image size grows. Multidimensional upscaling extends this process across image size and channel depth.

  • Motivation: Existing autoregressive models require superlinear computation and memory, with self-attention becoming limiting beyond 32 × 32 images.Reducing these requirements can sacrifice global context.
  • Subscale Pixel Network: SPN divides an H × W × 3 × D image into S^2 interleaved slices of size H/S × W/S × 3 × D, typically choosing 32 × 32 slices.For the larger images, the paper uses S = 8 at 256 × 256 and S = 4 at 128 × 128.
  • Subscale Pixel Network: The SPN embedding network conditions the current slice on preceding slices, preserving their relative meta-positions with padding and target-position embeddings.This design gives the context network offset equivariance and a fixed input depth across target slices.
  • Subscale Pixel Network: A self-attention and residual-block context network emits a slice-sized feature map, which the hybrid masked-convolution and self-attention decoder uses in raster-scan order.The decoder receives position-preserving representations from preceding slices.
  • Multidimensional Upscaling: The SPN supports size upscaling by initializing its first slice with an externally generated subimage and depth upscaling by conditioning later channel-depth slices on a low-bit-depth SPN.The log-likelihood is optimized with a uniformly sampled target slice and its conditional log-probability.

4 EXPERIMENTS

Experiments evaluate SPN and multidimensional upscaling on ImageNet and CelebAHQ across resolutions and bit depths. The models achieve strong likelihood results and high-fidelity samples, including large unconditional ImageNet images.

  • Results: The experiments extend high-fidelity unconditional generation to high-resolution ImageNet, including a first benchmark at 256 × 256 and strong global coherence.The paper reports state-of-the-art 128 × 128 log-likelihoods by a large margin.
  • Experimental setup: The model uses a 5-layer convolutional context network, 6–8 self-attention layers, a 15-layer PixelCNN decoder, and an 8–10-layer 1D Transformer.Layer counts vary by dataset for the context network and Transformer.
  • Downsampled ImageNet: 3.52 bits/dim is the state-of-the-art log-likelihood achieved on 64 × 64 Downsampled ImageNet.SPN scores similarly at 3.53 bits/dim in the reported comparison.
  • ImageNet at 128 × 128: 3.08 bits/dim improves on Parallel Multiscale PixelCNN’s 3.55 bits/dim for 128 × 128 ImageNet.Depth-only and complete multidimensional upscaling have the same NLL, while multidimensional upscaling appears to increase sample success.
  • CelebAHQ: At 256 × 256, SPN produces high-fidelity CelebAHQ celebrity-face samples and significantly improves previously reported MLE scores.The reported sample quality compares favorably with Glow and GAN-based methods.

5 CONCLUSION

The SPN and Multidimensional Upscaling model advance unconditional generation of complex natural images by combining strong likelihood results with high-fidelity large-scale samples.

  • The model attains state-of-the-art MLE scores on CelebAHQ-256 and ImageNet-128.
  • It generates high-fidelity full 8-bit samples at 128 × 128 and 256 × 256 without altering the sampling process.
  • The large-scale samples exhibit semantic coherence and exactness of detail.

APPENDIX A

Sampling temperature is used as an analysis control, while the accompanying figures show CelebA-HQ samples at several bit depths and temperatures.

  • A temperature of 0.95 means dividing trained-model logits by 0.95 during sampling.
  • Figure 6 shows 8-bit 256 × 256 RGB CelebA-HQ samples from SPN with Depth-Upscaling, using temperature 0.99 for the low-bit-depth image and 1.0 otherwise.

APPENDIX B

The appendix directs readers to a table containing the detailed hyperparameters used in the experiments.

  • Table 4 contains all detailed hyperparameters.
  • The appendix uses Table 4 as the reference for experiment-specific configuration details.
  • Hyperparameter details are presented outside the surrounding appendix prose.

APPENDIX C

The appendix describes the computational scale, parameter counts, and sample figures associated with SPN experiments and multidimensional upscaling.

  • Experiments use batches up to 2048 and TPU pods, with 64 tensorcores for ImageNet 32 and 128 for ImageNet 64, 128, and 256.
  • SPN architectures contain approximately 50M to 250M parameters depending on the dataset.
  • Multidimensional upscaling reaches approximately 650M total parameters for ImageNet 128.
  • Figure 9 shows 256 × 256 CelebA-HQ 3-bit samples from SPN sampled at temperature 0.95.
  • Figures 10–12 show 128 × 128 ImageNet 3-bit samples from upscaled slices and SPN.
  • Table 4 lists SPN hyperparameters, including learning-rate schedules, attention settings, and residual-channel counts.
Loading 1812.01608v1…