Source-linked AI summary
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus
TL;DR
Natural images are difficult to model at high resolution, motivating a method that combines GANs with a Laplacian pyramid. LAPGAN generates images through independently trained, coarse-to-fine refinements and produces higher-quality samples, including scenes from LSUN; its evaluation also excludes training-only images from one noted sample set.
Problem
High-resolution natural-image generation is difficult because images are complex and high dimensional, while standard GANs cannot directly handle large images.
Method
LAPGAN integrates conditional GAN models into a Laplacian pyramid, training separate convolutional generators at successive scales for coarse-to-fine sampling.
Results
LAPGAN produces higher-quality samples than alternate deep generative approaches and generates credible 64 × 64 LSUN scenes capturing long-range structure.
Takeaways & Limitations
Breaking generation into plausible successive refinements provides a conceptually simple route to high-quality natural-image samples and may benefit other multiscale signal modalities.
Takeaways & Limitations
One reported sample evaluation used validation images to start generation, and those images were not used during training.
Abstract
from arXiv · showhide
In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion. At each level of the pyramid, a separate generative convnet model is trained using the Generative Adversarial Nets (GAN) approach (Goodfellow et al.). Samples drawn from our model are of significantly higher quality than alternate approaches. In a quantitative assessment by human evaluators, our CIFAR10 samples were mistaken for real images around 40% of the time, compared to 10% for samples drawn from a GAN baseline model. We also show samples from models trained on the higher resolution images of the LSUN scene dataset.
1 Introduction
Modeling natural images remains difficult because images are complex and high dimensional, while existing generative methods often target patches or simpler datasets. The paper addresses this gap with a multiscale convolutional GAN approach for generating plausible scenes at 32 × 32 and 64 × 64.
- Natural-image generation is a fundamental computer-vision problem complicated by image complexity and high dimensionality.
- Many existing approaches generate image patches rather than modeling entire high-resolution scenes.
- LAPGAN uses multiple conditional GAN-based convolutional models within a Laplacian pyramid to generate scenes across scales.
- The proposed approach generates plausible-looking scenes at 32 × 32 and 64 × 64 resolutions through coarse-to-fine multiscale sampling.
- Earlier deep generative models showed convincing samples mainly on simple datasets such as MNIST and NORB, possibly because training complexities limit larger-image applications.
2 Approach
LAPGAN combines conditional GANs with a Laplacian pyramid to generate images from coarse to fine, training separate models at each scale. Each finer model generates high-frequency structure conditioned on an upsampled coarser image.
- LAPGAN integrates conditional GANs into a Laplacian pyramid, using separate generative convolutional models for coefficient distributions at different scales.The pyramid represents images with band-pass coefficients and a low-frequency residual.
- The Laplacian coefficients capture scale-specific structure, and reconstruction follows Ik = u(Ik+1) + hk from the coarsest level back to the original image.The finest image is reconstructed by adding each finer-scale difference image to the upsampled coarser image.
- Sampling begins with a generated coarse residual and repeatedly upsamples it, adds a generated difference image, and proceeds to finer levels until full resolution.For a four-model, K = 3 pyramid, this produces a 64 × 64 sample.
- At each pyramid level, the generator receives noise and an upsampled coarse image, while the discriminator judges real or generated high-frequency coefficients with that image as context.At the final, sufficiently small scale, a standard GAN directly models the low-frequency residual.
- Independent level-wise training avoids training a discriminator on complete cascades and makes memorization of training examples more difficult.The method instead focuses on making each successive refinement plausible.
- Class-conditional variants add a one-hot class vector as an additional conditioning variable for each generator and discriminator.
3 Model Architecture & Training
The experiments evaluate LAPGAN on CIFAR10, STL, and LSUN using dataset-specific pyramid resolutions and architectures. Model sizes and sampling setups vary with image resolution and available training data.
- LAPGAN is evaluated on CIFAR10, STL, and LSUN, spanning 32 × 32 object images, 96 × 96 images, and 64 × 64 natural scenes.LSUN contains approximately 10M images across 10 scene types, while CIFAR10 and STL each provide 100k training samples.
- Architectures are selected using a combination of log-likelihood and visual appearance, while all model noise vectors are sampled uniformly from [-1,1].
- The initial 8 × 8 scale uses densely connected generator and discriminator networks with two hidden layers and ReLU nonlinearities.The initial-scale noise vector zK has 100 dimensions, and the discriminator uses dropout.
- For CIFAR10, four 28 × 28 crops expand the training set, with subsequent pyramid scales of 8 → 14 and 14 → 28.The CIFAR10 and STL models use convolutional generators and discriminators at subsequent scales.
- STL uses four pyramid levels progressing from 8 → 16 → 32 → 64 → 96, with convolutional generators and discriminators at the subsequent scales.
- LSUN models use four subsequent scales, 4 → 8 → 16 → 32 → 64, with substantially larger shared architectures enabled by the dataset’s larger training set.Separate LAPGAN models are trained for each of the 10 scene classes, and evaluation begins from validation images downsampled to 4 × 4.
4 Experiments
The experiments evaluate LAPGAN through likelihood estimates, generated samples, and human judgments across CIFAR10, STL, and LSUN. Results indicate stronger likelihood estimates and visually credible samples, including scene structure at higher resolutions.
- Evaluation setup: Three evaluation methods measure held-out log-likelihood, generated sample quality, and human discrimination between generated and real images.The human experiment compares LAPGAN, class conditional LAPGAN, standard GAN, and real CIFAR10 images.
- Log-likelihood evaluation: LAPGAN achieves significantly higher Parzen-window log-likelihood estimates than the standard GAN on both CIFAR10 and STL.The estimates use the model’s multi-scale structure and validation-selected Parzen window size.
- CIFAR10 samples: CIFAR10 LAPGAN samples appear more object-like and have more clearly defined edges than standard GAN samples, while class conditioning improves object structure.Nearest-neighbor comparisons indicate the model is not simply copying training examples.
- STL samples: STL samples remain sharp despite losing clear object shape, and the figure also presents their coarse-to-fine generation chain.The model is trained on STL images and displays random samples alongside the generation process.
- LSUN samples: LSUN models generate 64 × 64 tower, bedroom, and church-front images with long-range scene structure and credible recombinations of scene elements.Ten different samples per category illustrate variation from the model.
- Human evaluation: The human evaluation tests whether 15 volunteers can distinguish samples from three CIFAR10 GAN variants from real images.Subjects see randomly selected images and report whether each appears real.
5 Discussion
The discussion frames LAPGAN as a simple multi-scale alternative to global image fidelity, using successive plausible refinements to generate complex images. CIFAR10 comparisons, STL generation chains, LSUN samples, and human evaluation illustrate this approach.
- Discussion: LAPGAN gives up a global notion of fidelity and instead generates images through plausible successive refinements.The authors identify multiscale structure as a potentially useful principle for other signal modalities.
- CIFAR10 comparison: CIFAR10 figures compare class conditional LAPGAN, LAPGAN, and standard GAN samples, with training-set nearest neighbors shown beside samples.The nearest-neighbor column supports visual comparison between generated images and training examples.
- STL generation: STL figures show random 96 × 96 LAPGAN samples alongside a coarse-to-fine generation chain.The chain visualizes the successive stages used to produce samples.
- LSUN generation: LSUN figures organize 64 × 64 samples by tower, bedroom, and church-front categories, beginning each row with a 4 × 4 validation image.Subsequent columns show different draws from each model, exposing variation across generated scenes.
- Human evaluation: Human evaluation compares real CIFAR10 images with standard GAN, LAPGAN, and class conditional LAPGAN samples using error bars for inter-subject variability.The class conditional model fools human evaluators around 40% of the time, versus ≤10% for the standard GAN and >90% for real images.
Appendix A
The appendix constructs a normalized image density by decomposing each image into a coarsened component and a high-pass component, then estimating conditional densities with Parzen windows across pyramid scales.
- Appendix A: Each image I is decomposed into a coarsened image l=d(I) and a high-pass component h=I−u(d(I)).The appendix uses blockwise averaging for d and a corresponding upsampling operator u in its likelihood derivation.
- Appendix A: The model defines an image density from a coarse-scale density q1(l) and a conditional high-pass density q0(l,h).The functions are assumed nonnegative before their specific estimators are introduced.
- Appendix A: The density construction is normalized by integrating the conditional high-pass density and the coarse-scale density.The appendix explicitly checks that the resulting image density has unit integral.
- Appendix A: Parzen window approximations estimate the densities at each pyramid scale from training samples.The construction begins with q1 at the coarsest scale and continues analogously for finer scales.
- Appendix A: The likelihood uses the true low-pass component at each scale and compares each true high-pass component with high-pass samples generated by the model.For a K-level pyramid, the final log likelihood combines the density contributions across levels.