Source-linked AI summary
Stacked Generative Adversarial Networks
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, Serge Belongie
TL;DR
Deep discriminative models learn useful representations but do not capture complex data distributions, and existing generative models struggle with large variation. SGAN inverts hierarchical representations with a top-down stack of conditional GANs, representation discriminators, conditional loss, and entropy loss. The paper reports higher-quality images than non-stacked GANs, while noting limitations of Inception score diversity sensitivity and the current label-conditioned formulation.
Problem
Discriminative networks learn transferable representations without capturing data distributions, while deep generative models struggle to model large variations.
Method
SGAN uses a top-down stack of conditional GANs that generates lower-level representations from higher-level ones, with representation discriminators and conditional and entropy losses.
Results
SGAN achieves significantly higher Inception scores and better sample quality than baseline DCGANs, with joint training slightly improving Inception score over SGAN-no-joint.
Takeaways & Limitations
The stacked approach decomposes image-distribution estimation into easier conditional representation-generation tasks and provides intermediate supervision from discriminative representations.
Takeaways & Limitations
Inception score correlates with visual quality but can be insensitive to diversity, and the current formulation uses a discriminative model and label-conditioned generator.
Abstract
from arXiv · showhide
In this paper, we propose a novel generative model named Stacked Generative Adversarial Networks (SGAN), which is trained to invert the hierarchical representations of a bottom-up discriminative network. Our model consists of a top-down stack of GANs, each learned to generate lower-level representations conditioned on higher-level representations. A representation discriminator is introduced at each feature hierarchy to encourage the representation manifold of the generator to align with that of the bottom-up discriminative network, leveraging the powerful discriminative representations to guide the generative model. In addition, we introduce a conditional loss that encourages the use of conditional information from the layer above, and a novel entropy loss that maximizes a variational lower bound on the conditional entropy of generator outputs. We first train each stack independently, and then train the whole model end-to-end. Unlike the original GAN that uses a single noise vector to represent all the variations, our SGAN decomposes variations into multiple levels and gradually resolves uncertainties in the top-down generative process. Based on visual inspection, Inception scores and visual Turing test, we demonstrate that SGAN is able to generate images of much higher quality than GANs without stacking.
1. Introduction
Discriminative networks learn powerful transferable representations but do not capture data distributions, while existing deep generative models struggle with large variation. SGAN addresses this gap by generating lower-level representations conditionally through a top-down stack guided by discriminative representations.
- Bottom-up discriminative models learn useful representations but are incapable of capturing the data distribution.
- Current deep generative models still fail when data distributions contain large variations.
- SGAN uses a top-down stack of GANs to generate plausible lower-level representations conditioned on higher-level representations.
- Representation discriminators align generated intermediate representations with the representation space of a bottom-up discriminative network.
- Conditional and entropy losses encourage generators to use higher-level information and produce diverse representations.
2. Related Work
Prior work spans deep generative models, factorized GANs, feature-based generator regularization, intermediate-representation matching, and visualization. SGAN differs by regularizing internal representations adversarially within a top-down generative model rather than only its output.
- Deep Generative Image Models: Deep generative image modeling includes autoregressive models, variational auto-encoders, and generative adversarial networks.
- Deep Generative Image Models: GANs remain challenged by generating diverse images with sufficient detail, motivating factorized approaches that divide generation into tractable subtasks.
- Deep Generative Image Models: LAPGAN factorizes generation across resolutions, with each GAN producing a higher-resolution residual conditioned on a lower-resolution image.
- Using Discriminative Models: Feature-based generator regularization encourages reconstructed outputs to match originals in discriminative feature space or style-feature statistics.
- Matching Intermediate Representations Between Two DNNs: Unlike prior approaches that regularize generator outputs, SGAN regularizes internal representations using adversarial loss rather than L2 reconstruction loss.
- Matching Intermediate Representations Between Two DNNs: SGAN transfers knowledge from a bottom-up discriminative network to a top-down generative model, unlike knowledge transfer between bottom-up networks.
3. Methods
SGAN inverts a pretrained bottom-up encoder with a top-down stack of conditional GANs, using representation matching and complementary losses to model hierarchical variation. The model is trained independently by stack and then jointly end-to-end.
- Pre-trained Encoder: A pretrained encoder maps input images x through hierarchical representations h_i to classification output y, with stack boundaries chosen by the authors.The encoder is defined by mappings h_i+1 = E_i(h_i), where h_0 = x and h_N = y.
- Stacked Generators: SGAN uses top-down generators G_i to invert encoder mappings, producing lower-level features from higher-level features and noise.Each generator is first trained independently with encoder features as conditions, then jointly with upper-generator outputs.
- Sampling and Variation Decomposition: Different noise variables represent different levels of variation, and SGAN samples images by stacking generators top-down and feeding noise to each generator.The factorization decomposes image entropy into multiple conditional entropy terms plus label entropy.
- Training Objectives: SGAN combines adversarial, conditional, and entropy losses for each generator, with adversarial losses also providing intermediate supervision during joint training.The conditional loss feeds generated lower-level representations back through the encoder and compares recovered representations with the conditioning representations.
- Representation Adversarial Training: Each generator is paired with a representation discriminator D_i that distinguishes generated representations from real encoder representations.The adversarial objective aligns generated intermediate representations with the corresponding representation distribution.
- Conditional and Entropy Losses: The conditional loss encourages use of higher-level information, while the entropy loss prevents generators from ignoring noise and promotes diverse outputs conditioned on higher-level features.The entropy objective maximizes a variational lower bound on H(hat h_i|h_i+1) using an auxiliary posterior model Q_i(z_i|hat h_i).
4. Experiments
SGAN is evaluated on MNIST, SVHN, and CIFAR-10 using visual inspection, Inception scores, visual Turing tests, and ablations. Across these evaluations, stacking, joint training, and entropy loss improve generation quality or diversity.
- Experimental setup: SGAN is evaluated on MNIST, SVHN, and CIFAR-10 using a two-stack generator and a CNN encoder.The top GAN generates fc3 features conditioned on labels, while the bottom GAN generates images conditioned on those features.
- Samples: MNIST samples are diverse, visually indistinguishable from real images, and different from corresponding nearest-neighbor training images.This visual comparison supports generation beyond direct memorization of the nearest training examples.
- Samples: Fixed fc3 features yield similar coarse outlines with varied low-level properties, whereas removing entropy loss makes the bottom GAN ignore noise and generate deterministically.On SVHN and CIFAR-10, the varied properties include lighting conditions and background clutter.
- Visual Turing test: SGAN’s visual Turing-test error rate is 24.4%, compared with 15.6% for DCGAN (Ladv + Lcond + Lent), based on 9,000 votes per model.The authors report this as further confirmation that stacking improves image quality over GAN without stacking.
- More ablation studies: Ablations show that SGAN outperforms baseline DCGANs, joint training slightly improves over SGAN-no-joint, and entropy loss improves diversity.Without entropy loss, DCGAN (Ladv + Lcond) collapses to one image per category; adding entropy loss produces diverse images, while Inception score can miss this difference.
5. Discussion and Future Work
SGAN leverages representations from a pre-trained discriminative network through a top-down generative framework. It decomposes image-distribution estimation into easier conditional tasks and uses intermediate supervision and entropy regularization to improve conditional generation.
- SGAN leverages representational information from a pre-trained discriminative network in a top-down generative framework.
- The model decomposes image-distribution estimation into multiple easier tasks that generate plausible representations conditioned on higher-level representations.
- Representation discriminators provide intermediate supervision at different training hierarchies.
- SGAN introduces an entropy loss to address conditional GANs ignoring noise.
- The entropy loss is proposed for applications including diverse future-frame synthesis and image generation conditioned on label maps.