PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space

Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, Jason Yosinski

arXiv:1612.00005v2cs.CV

TL;DR

High-resolution, realistic image generation remains difficult, and prior activation-maximization methods can converge to only a few modes. PPGN combines a latent-code prior with a replaceable condition network, improving sample quality and diversity over DGN-AM.

  • Problem

    Generating realistic, high-resolution images remains challenging, while DGN-AM tends to produce samples with limited diversity.

    Source: §1 · p. 2

  • Method

    PPGN frames iterative generation as sampling under priors and conditions, pairing a generator with a replaceable condition network.

    Source: §1 · p. 2

  • Results

    PPGN improves sample quality and diversity over DGN-AM, and Noiseless Joint PPGN-h empirically produces the best image quality.

    Source: §1 · p. 2, §3.5 · p. 6

  • Takeaways & Limitations

    A replaceable condition network supports class- and caption-conditioned image generation, and the authors state that the approach should generalize to other data types.

    Source: §1 · p. 2

    Activation-maximization methods that vary only their starting conditions can converge to a single mode or a small number of modes.

    Source: S7. Probabilistic interpretation of… · p. 14

Abstract

from arXiv · show

Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x227) than previous generative models, and does so for all 1000 ImageNet categories. In addition, we provide a unified probabilistic interpretation of related activation maximization methods and call the general class of models "Plug and Play Generative Networks". PPGNs are composed of 1) a generator network G that is capable of drawing a wide range of image types and 2) a replaceable "condition" network C that tells the generator what to draw. We demonstrate the generation of images conditioned on a class (when C is an ImageNet or MIT Places classification network) and also conditioned on a caption (when C is an image captioning network). Our method also improves the state of the art of Multifaceted Feature Visualization, which generates the set of synthetic inputs that activate a neuron in order to better understand how deep neural networks operate. Finally, we show that our model performs reasonably well at the task of image inpainting. While image models are used in this paper, the approach is modality-agnostic and can be applied to many types of data.

1. Introduction

High-resolution, realistic image generation remains difficult, and DGN-AM tends to produce samples with limited diversity. PPGN adds a latent-code prior and a replaceable condition network, improving sample quality and diversity while supporting different conditioning tasks.

1. Introduction

  • DGN-AM often converges to one mode across random initializations, producing images with similar compositions despite small variations.The paper illustrates this with cardoon images that vary in flower count or viewing angle but retain a close-up composition.Source: §1 · p. 2
  • Adding a prior on the latent code improves DGN-AM sample quality and diversity by keeping optimization on the manifold of realistic-looking images.For cardoons, PPGN images better represent the diversity of random class images than DGN-AM images, which resemble top-activating examples.Source: §1 · p. 2, Figure 2 · p. 2
  • PPGN casts activation-maximization approaches as energy-based models whose energy combines priors with conditions, and samples iteratively using approximate Metropolis-adjusted Langevin sampling.Priors can encourage realism, while conditions can specify a class or multiple classes.Source: §1 · p. 2
  • PPGN’s replaceable condition network can support class- or caption-conditioned image generation by swapping in a classification or image-captioning network.The paper describes attaching a recurrent image-captioning network to the generator output and performing iterative sampling.Source: §1 · p. 2
  • Although this paper focuses on image generation, the authors state that the approach should generalize to other data types.The paper also notes its code and trained networks are published.Source: §1 · p. 3

2. Probabilistic interpretation of iterative image generation methods

The paper interprets iterative image generation as approximate sampling from probabilistic models. It formulates activation maximization as conditional sampling with prior, condition, and noise terms, then uses that framework to characterize earlier methods.

2. Probabilistic interpretation of iterative image generation methods

  • MALA-approx provides a Markov-chain sampler whose stationary distribution approximates a target distribution, forming the paper’s framework for interpreting iterative image generation.The framework is used to derive a probabilistic formulation of activation maximization and interpret other iterative methods.Source: §2 · p. 3

2.1. Probabilistic framework for Activation Maximization

  • For activation maximization, fixing a chosen class yields conditional image sampling in which image realism, class confidence, and noise contribute separate update terms.The p(x) term moves toward a generic image, p(y = yc|x) toward higher confidence in the chosen class, and noise encourages diversity.Source: §2.1 · p. 3

2.2. Interpretation of previous models

  • The framework interprets methods without an image prior as using no prior term, while L2-decay methods use a Gaussian image prior; both omit noise and sacrifice sample diversity.The authors note that the sampler’s stationary distribution converges appropriately when step sizes are chosen appropriately, aside from errors from omitting a reject step.Source: §2.2 · pp. 3–4

3. Plug and Play Generative Networks

The section develops learned priors for iterative PPGN sampling, progressing from pixel-space and latent-space DAEs to joint denoising through the generator and encoder. Joint PPGN-h improves sample quality and mixing, while its noiseless variant produces substantial diversity with a modest mixing trade-off.

3. Plug and Play Generative Networks

  • PPGN replaces hand-engineered sampling priors with explicitly learned priors modeled by denoising autoencoders, whose reconstructions approximate the log-density gradient used in sampler updates.The DAE does not evaluate p(x) directly; with sufficient capacity and training, its gradient approximation becomes perfect as the training-noise variance approaches zero.Source: §3 · p. 4

3.1. PPGN-x: DAE model of p(x)

  • Directly learning p(x) with a DAE produced poor data modeling and slow chain mixing, motivating reparameterization into lower-dimensional h-space.The section proposes sampling through p(h)p(x|h) rather than directly in pixel space.Source: §3.1 · p. 5, §3.2 · p. 5

3.2. DGN-AM: sampling without a learned prior

  • DGN-AM optimizes generator latent h for class activation with a hand-designed Gaussian/L2 prior, but often converges to the same mode and mixes poorly.The sampling update conditions on the target class through the generator and classifier.Source: §3.2 · p. 5

3.3. PPGN-h: Generator and DAE model of p(h)

  • A DAE trained directly on fc6 latent h mixes faster than pixel-space PPGN-x, but remains qualitatively similar to DGN-AM and lacks sample quality and diversity.The authors hypothesize that the DAE poorly models the latent distribution.Source: §3.3 · pp. 5–6

3.4. Joint PPGN-h: joint Generator and DAE

  • Joint PPGN-h mixes substantially faster and produces better-quality samples than all previous PPGN treatments, including PPGN-h.Training treats the model as three interleaved DAEs sharing parameters for h, h1, and x; sampling adds noise at all three variables.Source: §3.4 · p. 6

3.5. Ablation study with Noiseless Joint PPGN-h

  • Noiseless Joint PPGN-h gave the best image quality with no training noise and pool5 feature-matching, adversarial, and image-reconstruction losses.Removing adversarial loss yielded blurrier images; pool5 feature matching led to better quality than fc6 or their combination.Source: §3.5 · p. 6
  • Noiseless Joint PPGN-h produces substantial diversity, mixes faster than DGN-AM but slightly slower than Joint PPGN-h, and larger sampling noise often worsens quality.Noise sweeps found no qualitatively better reconstructions; even the no-noise h-autoencoder appeared contractive, and diversity evaluation used 10 chains of 200 steps alongside Table S3 comparisons. Why it succeeds on high-resolution 1000-class ImageNet remains for future work.Source: §3.5 · pp. 6–7

4. Additional results

The additional results show how replaceable condition networks extend PPGNs to scene classes, captions, hidden-neuron visualization, and inpainting. Across these tasks, the model can generate images beyond its original class-conditioning setup, although caption-conditioned image quality has limits.

4.1. Generating images with different condition networks

  • Replacing prior and condition components lets PPGNs take on new image-generation tasks, including conditions specified at test time.The paper demonstrates class, caption, hidden-neuron, and image-context conditions.Source: §4.1 · p. 7

Generating images conditioned on classes

Generating images conditioned on captions

  • Caption conditioning with an image-captioning network produces reasonable images in many cases, but quality is lower than with class conditioning and some prompts fail.Failures include “people” and “giraffe”; the authors also report unrecognizable images that nevertheless receive high-scoring captions.Source: Generating images conditioned on… · pp. 7–8, Figure 5 · p. 7

Generating images conditioned on hidden neurons

  • Conditioning on a hidden neuron enables Multifaceted Feature Visualization, revealing diverse inputs that activate the neuron rather than only class-output examples.For neuron 196, examples include human and dog faces as well as objects that only barely resemble faces.Source: Generating images conditioned on… · p. 8, Figure 6 · p. 8

4.2. Inpainting

  • With class and image-context conditioning, PPGN-context matches pixels around masked regions better than PPGN and often fills them more semantically than Photoshop’s Context-Aware Fill.The authors report that inpainting suggests the model understands image semantics rather than merely memorizing images.Source: Figure 7 · p. 8, §4.2 · p. 8

5. Conclusion

PPGNs’ central property is that a replaceable condition network can specify what images to generate at test time. The paper points to applications including image synthesis for videos and art, and connects this flexibility to generators combined with pretrained condition networks.

5. Conclusion

  • A replaceable condition network lets users specify image-generation conditions at test time.The paper suggests this could support video synthesis or art, including using one or multiple condition networks.Source: §5 · p. 8
  • DGN-AM had already enabled scientists and amateurs to use pretrained condition networks for art and scientific visualizations.The authors attribute this in part to the generality of the fc6 features that the generator was trained to invert.Source: §5 · p. 8
  • The authors argue that flexible generators are valuable because they can be combined with pretrained condition networks.They connect this possibility to fc6 features covering the set of natural images.Source: §5 · p. 8

S6. Markov chain Monte Carlo methods and derivation of MALA-approx

When IID sampling is difficult, the section reviews MCMC methods and proposes MALA-approx, which uses SGD stochasticity in place of an accept-reject step. Decoupling step size and injected noise provides a way to tune the trade-off between sample diversity and quality.

S6. Markov chain Monte Carlo methods and derivation of MALA-approx

  • MCMC is presented as an alternative when an IID sampler is not readily available for the target distribution.The section summarizes sampler properties under Gaussian proposal distributions and uses MALA-approx for the paper's samples.Source: S6. Markov chain Monte Carlo methods… · p. 12, Table S1 · p. 13
  • MALA preferentially steps toward higher-probability regions, improving over simple Metropolis-Hastings proposals, but still requires evaluating p(x) to calculate acceptance.The simple Metropolis-Hastings sampler uses a Gaussian proposal and requires p(x) to compute its acceptance ratio.Source: S6. Markov chain Monte Carlo methods… · p. 12
  • MALA-approx skips rejection and relies on SGD stochasticity as implicit noise, producing asymptotically unbiased samples when the step size is annealed or minibatches gradually increase.The authors report reasonable samples with moderate step sizes, although an accept ratio of 1 is reached only in the zero-step-size limit.Source: S6. Markov chain Monte Carlo methods… · p. 12
  • Decoupling the gradient step and normal-noise scales allows independent tuning of variance and bias, a trade-off the authors report helps balance sample diversity and quality.SGD noise is not perfectly normal, and the authors suggest decoupling may compensate for this and for sampling without a reject step.Source: S6. Markov chain Monte Carlo methods… · pp. 12–13

S7. Probabilistic interpretation of previous models (continued)

The section interprets activation-maximization methods as samplers with different prior, condition, and noise terms. It also motivates a log-softmax class gradient and notes that noiseless methods rely on initialization for diversity.

S7. Probabilistic interpretation of previous models (continued)

  • The framework represents activation maximization without an image prior as a class-gradient update with no added noise, a setting that can yield adversarial or fooling images.The described parameter setting is (ϵ1, ϵ2, ϵ3) = (0, 1, 0).Source: S7. Probabilistic interpretation of… · p. 13
  • L2 decay during sampling is interpreted as a Gaussian image prior, which pulls images toward the origin while the class term favors higher probability for the chosen class.The class term is computed as the derivative of the log of a softmax unit.Source: S7. Probabilistic interpretation of… · p. 13
  • The log-softmax derivative is the theoretically justified class-gradient variant under the probabilistic interpretation, and the authors recommend it for future activation-maximization studies.The paper reports using this third derivative variant in its experiments; Table S2 compares it with derivatives of logits and softmax.Source: S7. Probabilistic interpretation of… · p. 14, Table S2 · p. 14
  • Noiseless methods obtain diversity only from different initial conditions, so samples tend to converge to one mode or a small number of modes.The section notes that this limitation applies to the methods considered there.Source: S7. Probabilistic interpretation of… · p. 14

S8. Comparing feature matching losses

The section compares feature-matching losses at different AlexNet layers for training an image generator. Pool5 matching was selected for the main experiments, while the ablation identifies the GAN loss as important for avoiding blurry output.

S8. Comparing feature matching losses

  • Feature-matching losses compare generated and real images in a neural feature space rather than pixel space and have been used to improve sample sharpness and realism.The cited prior work includes feature spaces at different depths of a deep neural network.Source: S8. Comparing feature matching losses · p. 14
  • Pool5 feature matching produced the best image quality among the tested layers, so its generator was selected for the main experiments.The comparison trained generators with pool5, fc6, or both feature-matching losses.Source: S8. Comparing feature matching losses · p. 14, Figure S8 · p. 15
  • Removing the GAN loss yielded blurry results in the loss-combination ablation.The figure compares generators trained with different combinations of pixel, pool5, and fc6 reconstruction losses, as well as the GAN loss.Source: Figure S8 · p. 15

S9. Training details

The training details specify shared GAN training settings and separate procedures for PPGN-x, PPGN-h, and two joint PPGN-h variants. The joint variants use reconstruction and adversarial losses, with the noisy version adding latent-code denoising and reporting instability at larger noise levels.

S9.1. Common training framework

  • The common framework uses Adam with β1 = 0.9, β2 = 0.999, γ = 0.5, an initial learning rate of 0.0002, and batch size 64; GAN training pauses a network when the discriminator-to-generator loss ratio is extreme.The thresholds are pause D if r < 0.1 and pause G if r > 10; the authors did not find BatchNorm helpful and had not tried all discussed stabilization techniques.Source: S9.1. Common training framework · p. 16

S9.2. Training PPGN-x

  • PPGN-x uses a four-layer convolutional denoising autoencoder trained on images corrupted with Gaussian noise σ = 25.6 as its p(x) prior.The autoencoder maps images to AlexNet conv1 and decodes them with three upconvolutional layers; it is trained for 25,000 mini-batch iterations.Source: S9.2. Training PPGN-x · p. 16

S9.3. Training PPGN-h

  • PPGN-h models p(h) with a seven-hidden-layer denoising autoencoder and uses Gaussian noise σ = 1, about 10% of mean fc6 activation.The network was trained for 100,000 mini-batch iterations; larger networks were tested but this configuration worked best.Source: S9.3. Training PPGN-h · p. 16

S9.4. Training Noiseless Joint PPGN-h

  • Noiseless Joint PPGN-h trains G as a decoder of frozen AlexNet fc6 features using adversarial, image-reconstruction, and pool5 feature-matching losses.The generator maps high-level fc6 codes to images; the cited training procedure follows Dosovitskiy & Brox [9].Source: S9.4. Training Noiseless Joint PPGN-h · p. 16

S9.5. Training Joint PPGN-h

  • Joint PPGN-h trains G by treating h, h1, and x as three interleaved denoising autoencoders, adding an h reconstruction loss to the three losses retained from noiseless Joint PPGN-h.Noise levels for x, h1, and h were σ = {1, 4, 1}; larger noise levels caused training instability, and removing noise from x did not qualitatively improve results.Source: S9.5. Training Joint PPGN-h · p. 17

S10. Inpainting

PPGN-context fills a masked image by combining class-conditioned sampling with clamping of the observed pixels. It often produces semantically appropriate content, though local details may not match the surrounding image.

S10. Inpainting

  • The method masks a 100 × 100 patch in a real image, applies class-conditioned sampling, and clamps the uncorrupted pixels during each update.The image variable is updated during the forward pass, while the observed region is kept fixed.Source: S10. Inpainting · p. 17
  • Filled regions are often semantically correct but can mismatch local context because fc6 features encode global object structure more than local detail.The predicted pixels may not transition smoothly into the surrounding image.Source: S10. Inpainting · p. 17
  • An added pixel-space context cost improves matching to the surrounding image, but PPGN often matches local features worse than Photoshop CS6 while filling semantic objects better in some cases.The comparison cites bird and bell-pepper examples.Source: S10. Inpainting · p. 17

S11. PPGN-x: DAE model of p(x)

The PPGN-x variant uses a denoising autoencoder to model the image distribution directly, but its samples blur and its chain mixes slowly. Combining an encoder and generator into an image-space autoencoder performs better, though not as well as Joint PPGN-h.

S11. PPGN-x: DAE model of p(x)

  • A 4-layer convolutional DAE trained on unlabeled ImageNet images gave its best results with 20% pixel-range noise, corresponding to ϵ3 = 25.6.The DAE models p(x) directly.Source: S11. PPGN-x: DAE model of p(x) · p. 17
  • PPGN-x samples become blurry over time and change only slightly across hundreds of sampling steps, indicating poor distribution modeling and slow mixing.The reported sampling parameters are (ϵ1, ϵ2, ϵ3) = (1, 10^5, 25.6).Source: S11. PPGN-x: DAE model of p(x) · p. 17
  • The Noiseless Joint PPGN-h generator reconstructs images from compressed features and is trained with image-reconstruction, feature-matching, and adversarial losses.This variant is used for the paper's main experiments.Source: Figure S9 · p. 18
  • Joint PPGN-x performs better than PPGN-x, but worse than Joint PPGN-h.Joint PPGN-x forms an image-space denoising autoencoder by composing separately trained E and G as E(G(.)); Joint PPGN-h is the stronger variant.Source: S11. PPGN-x: DAE model of p(x) · p. 18

S12. Why PPGNs produce high-quality images

The paper attributes image plausibility to combining adversarial and reconstruction losses and to its supervised encoder–generator mapping. PPGN samples outperform AC-GAN on Inception accuracy, but real images exceed PPGN on all four reported quality and diversity metrics.

S12. Why PPGNs produce high-quality images

  • The authors attribute sharper, more plausible reconstructions to GAN loss pushing them toward real-image modes, while reconstruction losses help combat GAN missing-mode behavior.The GAN objective alone may move samples far from their originals and collapse multiple modes into fewer modes.Source: S12. Why PPGNs produce high-quality… · pp. 18–19
  • The encoder–generator one-to-one mapping helps prevent latent-to-input missing-mode collapse, and supervised ImageNet features are hypothesized to provide a semantically meaningful distribution for generation.The paper proposes testing other feature distributions, such as random or PCA features, as future work.Source: S12. Why PPGNs produce high-quality… · p. 19
  • PPGN samples have higher Inception accuracy than AC-GAN; diversity is similar (847/1000 vs. 829/1000 by Percent of classes), while real images outperform PPGN on all four metrics.The table reports Inception scores, Inception accuracies, MS-SSIM, and Percent of classes.Source: Table S3 · p. 20
  • PPGN produces higher-quality, substantially more diverse class samples than DGN-AM, with volcano and pool-table examples showing varied scenes and conditions.The volcano samples vary in activity, plume color, peaks, terrain, and sky; pool-table samples vary in felt color, lighting, camera angle, and interior.Source: Figure S10 · p. 21, Figure S11 · p. 22, Figure S12 · p. 23
  • Noiseless Joint PPGN-h substantially improves mixing speed and sample quality over DGN-AM; the three-DAE model mixes slightly faster but produces slightly worse samples.A tiny amount of sampling noise also outperforms larger noise levels for Noiseless Joint PPGN-h.Source: Figure S13 · p. 24, Figure S15 · p. 26
  • With an ImageNet-trained prior, PPGN generates high-quality images for unseen MIT Places categories, and caption conditioning can describe objects outside ImageNet categories.The Places classifier is the condition network in the unseen-category examples; caption conditioning uses an image-captioning network.Source: Figure S17 · p. 28, Figure S18 · p. 29
  • For multifaceted feature visualization, PPGN produces diverse, high-quality preferred stimuli for hidden neurons, revealing unexpected detector facets such as houses activating a face neuron.The paper describes this as state of the art relative to previous methods.Source: Figure S19 · p. 30, Figure S20 · p. 31
  • Inpainting is often semantically good but can miss local context, and generated image sets have substantially lower diversity than real-image sets.The latter observation aligns with the paper's quantitative measure in Table S3.Source: Figure S24 · p. 35, Figure S25 · p. 36

Related papers

Loading 1612.00005v2…