Source-linked AI summary
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas
TL;DR
GANs struggle with stable, diverse high-resolution image generation. The paper proposes StackGAN-v1’s staged text-to-image synthesis and StackGAN-v2’s multi-stage, tree-structured approximation of related distributions, reporting improved image quality and training stability over prior methods. The approach reaches 256×256 photo-realistic text-to-image synthesis and improves inception scores on CUB and Oxford-102.
Problem
GAN training is unstable, mode collapse is common, and high-resolution image distributions are difficult to approximate.
Method
StackGAN uses a two-stage sketch-refinement model and a tree-like multi-stage architecture that jointly approximates related image distributions.
Results
The methods improve image quality and training stability, with StackGAN-v1 generating 256×256 photo-realistic text-to-image results and improving IS from 2.88 to 3.70 on CUB.
Takeaways & Limitations
StackGAN decomposes high-resolution generation into manageable sub-problems and supports photo-realistic text-to-image synthesis.
Takeaways & Limitations
Stage-II assumes randomness is already preserved in the Stage-I result and therefore does not use a separate noise vector.
Abstract
from arXiv · showhide
Although Generative Adversarial Networks (GANs) have shown remarkable success in various tasks, they still face challenges in generating high quality images. In this paper, we propose Stacked Generative Adversarial Networks (StackGAN) aiming at generating high-resolution photo-realistic images. First, we propose a two-stage generative adversarial network architecture, StackGAN-v1, for text-to-image synthesis. The Stage-I GAN sketches the primitive shape and colors of the object based on given text description, yielding low-resolution images. The Stage-II GAN takes Stage-I results and text descriptions as inputs, and generates high-resolution images with photo-realistic details. Second, an advanced multi-stage generative adversarial network architecture, StackGAN-v2, is proposed for both conditional and unconditional generative tasks. Our StackGAN-v2 consists of multiple generators and discriminators in a tree-like structure; images at multiple scales corresponding to the same scene are generated from different branches of the tree. StackGAN-v2 shows more stable training behavior than StackGAN-v1 by jointly approximating multiple distributions. Extensive experiments demonstrate that the proposed stacked generative adversarial networks significantly outperform other state-of-the-art methods in generating photo-realistic images.
1 INTRODUCTION
GANs can model complex data but are unstable and prone to mode collapse, especially at high resolution. StackGAN addresses this through staged generation and multi-distribution approximation.
- GAN training is unstable and sensitive to hyper-parameter choices.
- High-resolution generation makes distribution overlap unlikely, while mode collapse produces repetitive color or texture patterns.
- Most prior methods approximate image distributions at one scale, limiting their ability to generate high-resolution images.
- StackGAN-v1 uses a two-stage sketch-refinement process, generating low-resolution images before refining them into 256×256 images from text.
- StackGAN-v2 uses a tree-like architecture to jointly approximate related distributions across multiple image scales and conditional settings.
- The proposed systems improve generated-image quality and training stability, with StackGAN-v1 reaching 256×256 photo-realistic text-to-image synthesis.
2 RELATED WORK
Prior work explored generative modeling, conditional image synthesis, and multi-stage GANs, but high-resolution detail generation remained difficult. StackGAN directly generates high-resolution images conditioned on low-resolution inputs.
- Generative image modeling is a fundamental computer-vision problem, with GANs producing sharp images but unstable high-resolution training.
- Conditional image-generation methods use attributes, class labels, images, or domains as conditioning information.
- Multiple GANs have been combined to improve sample quality by separating structure, style, foreground, background, or resolution stages.
- Unlike LAPGANs, StackGAN directly generates high-resolution images conditioned on low-resolution inputs rather than generating residual images.
3 PRELIMINARIES
GANs train a generator and discriminator in competition, while conditional GANs additionally provide conditioning variables to both models.
- GAN training is a two-player minmax game between a generator reproducing the data distribution and a discriminator distinguishing real from synthetic images.
- The generator receives a noise vector sampled from a prior distribution, while the discriminator evaluates generated and real images.
- Conditional GANs give both generator and discriminator an additional variable c, enabling image generation conditioned on c.
4 STACKGAN-V1: TWO-STAGE GENERATIVE ADVERSARIAL NETWORK
StackGAN-v1 decomposes text-to-image synthesis into a low-resolution sketching stage and a high-resolution refinement stage. Conditioning Augmentation and text reuse help preserve variation and recover omitted details.
- Stage-I generation: Stage-I GAN generates a low-resolution image containing the object’s rough shape and basic colors from text, with background structure from noise.
- Stage-II refinement: Stage-II GAN conditions on the Stage-I image and text embedding to correct defects and produce a high-resolution photo-realistic image.
- Conditioning Augmentation: Conditioning Augmentation samples perturbed Gaussian conditioning variables, encouraging a smoother latent manifold and diverse outputs.
- Stage-I generation: Stage-I simplifies high-resolution synthesis by focusing first on rough shape and correct object colors.
- Stage-II refinement: Stage-I may omit text details and contain shape distortions, which Stage-II addresses through additional text conditioning.
- Stage-II refinement: Stage-II reuses randomness preserved in the Stage-I result rather than receiving a separate noise vector.
5 STACKGAN-V2: MULTI-DISTRIBUTION GENERATIVE ADVERSARIAL NETWORK
StackGAN-v2 models multiple image distributions with generators and discriminators arranged in a tree-like structure. Its branches produce images at increasing scales while jointly handling conditional and unconditional generation and enforcing cross-scale color consistency.
- 5.1 Multi-scale image distributions approximation: StackGAN-v2 uses multiple generators and discriminators in a tree-like structure to model multi-scale image distributions.Images are generated from low-resolution to high-resolution branches, with each branch capturing the distribution at its scale.
- 5.1 Multi-scale image distributions approximation: The hidden features are computed progressively, with each later branch receiving the preceding features and the noise vector, and generators producing samples from small to large scales.The noise vector is concatenated with preceding hidden features to capture information omitted by earlier branches.
- 5.1 Multi-scale image distributions approximation: Each generator is paired with a discriminator, and the discriminators are trained in parallel while generators jointly approximate image distributions at multiple scales.Each discriminator focuses on a single image scale and distinguishes real images from generated samples at that scale.
- 5.2 Joint conditional and unconditional distribution approximation: Conditional StackGAN-v2 feeds conditioning variables into the generator and uses discriminator losses that separately assess image realism and image-condition matching.The conditioning vector replaces the noise input in later transformations, and the objective combines unconditional and conditional losses.
- 5.3 Color-consistency regularization: Color-consistency regularization minimizes differences in pixel means and covariances across scales to preserve similar basic structure and colors.The regularizer is important for unconditional generation with α = 50.0 but is not needed for text-to-image synthesis, where α = 0.0.
- 5.4 Implementation details: The implementation progressively transforms a 4×4×64Ng feature tensor into 64×64×4Ng, 128×128×2Ng, and 256×256×1Ng tensors for multi-scale image generation.Intermediate features generate images at corresponding scales through 3×3 convolutions.
6 EXPERIMENTS
Experiments evaluate StackGAN using quantitative metrics, human judgments, visualizations, and ablations across conditional and unconditional image-generation tasks. StackGAN-v1 performs strongly for text-to-image synthesis, while StackGAN-v2 improves stability and diversity-related behavior but can converge harder on complex datasets.
- Text-to-image synthesis: StackGAN-v1 achieves the best FID*, IS, and average human rank on CUB, Oxford-102, and COCO against previous GAN models.Its IS improves from 2.88 to 3.70 on CUB and from 2.66 to 3.20 on Oxford-102.
- Text-to-image synthesis: 28.47% improvement in IS is achieved on CUB, from 2.88 to 3.70, while Oxford-102 improves 20.30%, from 2.66 to 3.20.At matched 64×64 resolution, StackGAN-v1 has higher IS* than GAN-INT-CLS but slightly lower IS* than GAWWN, which uses additional supervision.
- Diversity and failure analysis: StackGAN-v2 produces no collapsed nonsensical modes in the reported CUB comparison, whereas StackGAN-v1 has two small collapsed modes.The authors note that MS-SSIM does not reliably detect these small-scale collapses, while t-SNE visualization exposes them.
- StackGAN-v1 versus StackGAN-v2: StackGAN-v2 improves stability and FID or inception scores on most datasets, but StackGAN-v1 can converge faster, use less GPU memory, and appear slightly better on COCO.StackGAN-v2’s jointly optimized multiple distributions make convergence harder on complex datasets such as COCO, while StackGAN-v1 has more moderate or severe failures.
- Unconditional image synthesis: 9.55 ± 0.11 is StackGAN-v2’s inception score on ImageNet Dog, versus 8.19 ± 0.11 for DCGAN.Qualitative comparisons also report more photo-realistic 256×256 samples for StackGAN-v2 on LSUN bedroom and against EBGAN-PT.
- Ablation and qualitative analysis: Removing Conditioning Augmentation lowers the CUB inception score from 3.70 to 3.31 and causes 256×256 Stage-I samples to collapse to nonsensical images.With CA, Stage-I GAN generates more diverse poses and viewpoints from the same text embedding, while Stage-II adds higher-resolution details to rough Stage-I outputs.
7 CONCLUSIONS
StackGAN decomposes realistic high-resolution image generation into manageable sub-problems through two stacked architectures. StackGAN-v1 enables 256×256 photo-realistic text-to-image synthesis, while StackGAN-v2 jointly approximates multiple distributions and improves conditional and unconditional image generation.
- The proposed stacked architectures decompose difficult realistic high-resolution image generation into more manageable sub-problems.
- StackGAN-v1 uses a sketch-refinement process to generate 256×256 photo-realistic images from text descriptions.Conditioning Augmentation is incorporated into the text-to-image architecture.
- StackGAN-v2 jointly approximates multi-scale image distributions and jointly conditional and unconditional image distributions.Color-consistency regularization facilitates this multi-distribution approximation.
- Extensive quantitative and qualitative results show that the proposed methods significantly improve the state of the art in conditional and unconditional image generation tasks.