Source-linked AI summary
StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas
TL;DR
Generating detailed, photo-realistic images from text remains difficult because existing approaches may capture overall meaning without vivid object details. StackGAN uses a two-stage sketch-refinement process with Conditioning Augmentation, and experiments report higher-resolution, more photo-realistic, and more diverse images, including 256×256 outputs.
Problem
Text-to-image systems can roughly reflect descriptions but struggle to generate high-resolution images containing necessary details and vivid object parts.
Method
StackGAN decomposes synthesis into Stage-I sketching of shapes and colors, Stage-II refinement using images and text, and Conditioning Augmentation for smoother conditioning.
Results
StackGAN generates higher-resolution images with more photo-realistic details and diversity than existing text-to-image generative models, including 256×256 outputs.
Takeaways & Limitations
The sketch-refinement design enables Stage-II to correct Stage-I defects and add details while Conditioning Augmentation improves diversity and supports stable conditional-GAN training.
Takeaways & Limitations
Failure cases primarily arise when Stage-I GAN cannot generate plausible rough object shapes or colors.
Abstract
from arXiv · showhide
Synthesizing high-quality images from text descriptions is a challenging problem in computer vision and has many practical applications. Samples generated by existing text-to-image approaches can roughly reflect the meaning of the given descriptions, but they fail to contain necessary details and vivid object parts. In this paper, we propose Stacked Generative Adversarial Networks (StackGAN) to generate 256x256 photo-realistic images conditioned on text descriptions. We decompose the hard problem into more manageable sub-problems through a sketch-refinement process. The Stage-I GAN sketches the primitive shape and colors of the object based on the given text description, yielding Stage-I low-resolution images. The Stage-II GAN takes Stage-I results and text descriptions as inputs, and generates high-resolution images with photo-realistic details. It is able to rectify defects in Stage-I results and add compelling details with the refinement process. To improve the diversity of the synthesized images and stabilize the training of the conditional-GAN, we introduce a novel Conditioning Augmentation technique that encourages smoothness in the latent conditioning manifold. Extensive experiments and comparisons with state-of-the-arts on benchmark datasets demonstrate that the proposed method achieves significant improvements on generating photo-realistic images conditioned on text descriptions.
1. Introduction
StackGAN addresses the difficulty of generating detailed, high-resolution images from text by decomposing synthesis into staged sketching and refinement. Its Conditioning Augmentation technique further stabilizes conditional-GAN training and improves sample diversity.
- Motivation: Text-conditioned GANs struggle to generate stable, photo-realistic high-resolution images, often producing nonsensical outputs or lacking vivid object details.Simply adding upsampling layers generally destabilizes training; prior text-to-image work succeeded at 64×64 but lacked details such as bird beaks and eyes.
- Method: StackGAN decomposes text-to-image synthesis into two tractable stages: Stage-I sketches rough shapes and colors, while Stage-II refines them into detailed high-resolution images.Stage-II also reprocesses the text description to recover information omitted by Stage-I and correct defects.
- Method: Conditioning Augmentation introduces small random perturbations in the latent conditioning manifold to stabilize training and increase synthesized-image diversity.The technique addresses sparsity caused by the limited number of training text-image pairs.
- Contributions: StackGAN is presented as the first method to generate 256×256 images with photo-realistic details from text descriptions.The authors report extensive qualitative and quantitative experiments demonstrating the effectiveness of the model and its components.
2. Related Work
Related work spans generative image models, conditional image generation, and multi-stage GANs. StackGAN differs by using text-conditioned refinement to correct defects and produce 256×256 photo-realistic images.
- Generative image modeling: VAEs, autoregressive models, and GANs have advanced generative image modeling, but GAN training instability makes high-resolution synthesis difficult.Several techniques target more stable GAN training and compelling image generation.
- Conditional generation: Conditional image-generation methods use attributes, class labels, or images, while super-resolution methods add limited details and cannot correct large defects.The cited comparison positions StackGAN's text-conditioned refinement as addressing defects beyond ordinary super-resolution.
- Multi-stage GANs: Prior multi-stage GANs factorized scene structure and style or reconstructed multi-level representations, whereas StackGAN completes object details and corrects Stage-I defects from text.The distinction concerns the role of the second stage rather than merely stacking GAN components.
- Resolution: Earlier stacked-GAN work reached 32×32 images, while StackGAN generated 256×256 images with photo-realistic details using a simpler architecture.The cited passage contrasts the resolutions and notes that StackGAN uses sixty-four times more pixels.
3. Stacked Generative Adversarial Networks
StackGAN decomposes text-to-image synthesis into two conditional GAN stages: Stage-I sketches a low-resolution image, while Stage-II refines it into a high-resolution image by correcting defects and adding details. Conditioning Augmentation supplies stochastic text-conditioned variables to improve training and preserve variation.
- StackGAN decomposes high-resolution text-to-image synthesis into two more tractable stages.
- Stage-I GAN: Stage-I GAN conditions rough object shape and basic colors on text while using noise to draw the background, producing a low-resolution image.
- Conditioning Augmentation: Conditioning Augmentation samples Gaussian latent variables from text-dependent distributions, creating additional conditioning variables and encouraging robustness to perturbations.
- Stage-II GAN: Stage-II GAN takes the Stage-I image and text conditioning to correct distortions, recover omitted information, and generate a realistic high-resolution image.
- Stage-II GAN: Stage-II omits a separate noise input because randomness is assumed to be preserved in the Stage-I result, while separate conditioning layers capture additional text information.
- Stage-II GAN: The Stage-II generator uses an encoder-decoder with residual blocks to combine Stage-I image features and text features before producing the high-resolution output.
4. Experiments
Experiments compare StackGAN with prior text-to-image methods across CUB, Oxford-102, and COCO, using inception scores, human rankings, qualitative examples, retrieval, and component analyses. StackGAN achieves stronger reported image quality while its stacked refinement and Conditioning Augmentation improve detail, stability, and diversity.
- Evaluation setup: StackGAN is evaluated against GAN-INT-CLS and GAWWN on CUB, Oxford-102, and COCO using quantitative and qualitative comparisons.The evaluation includes inception scores, average human ranks, representative examples, and several baseline models.
- Quantitative and qualitative results: StackGAN achieves the best inception score and average human rank on all three datasets.Compared with GAN-INT-CLS, inception score improves 28.47% on CUB, from 2.88 to 3.70, and 20.30% on Oxford-102, from 2.66 to 3.20.
- Quantitative and qualitative results: On CUB, GAN-INT-CLS captures general bird shape and color but often lacks vivid parts and convincing detail, while GAWWN performs slightly below StackGAN and requires location constraints.StackGAN generates 256×256 photo-realistic images from text descriptions alone.
- Stage-wise refinement: Stage-I produces rough, blurry shapes and colors, whereas Stage-II generates four-times-higher-resolution images, adds details, and can correct Stage-I defects by processing the text again.Examples include adding beak, tail, and leg details and correcting a blue crown to the reddish brown crown described in the text.
- Generalization analysis: Nearest-neighbor retrieval indicates that generated images share characteristics with training samples but are essentially different, supporting capture of language-image relations rather than simple memorization.Visual features are extracted using the Stage-II discriminator, with L2 distances used for retrieval.
- Component analysis: Removing Conditioning Augmentation lowers the inception score from 3.70 to 3.31 and produces unstable, collapsed samples, while CA supports varied poses and viewpoints from the same text embedding.Directly using Stage-I at 256×256 also fails to produce plausible samples without CA, and reducing output resolution to 128×128 lowers the inception score from 3.70 to 3.35.
5. Conclusions
StackGAN uses a sketch-refinement process with Conditioning Augmentation to synthesize higher-resolution, more photo-realistic and diverse images from text.
- StackGAN decomposes text-to-image synthesis into a sketch-refinement process.Stage-I sketches object shape and colors, while Stage-II corrects defects and adds details for higher-resolution output.
More Results of Birds and Flowers
The bird and flower results illustrate text-conditioned samples across varied visual descriptions, while failure cases mainly arise when Stage-I produces implausible rough shapes or colors.
- Birds: The additional results cover varied bird descriptions involving plumage, wings, bills, body colors, and markings.
- Birds: The bird examples include black, brown, gray, yellow, red, orange, and white visual attributes.
- Flowers: The flower results describe differences in petal color, layering, shape, texture, and central structures.
- Failure Cases: The main failure cases occur when Stage-I GAN fails to generate plausible rough object shapes or colors.
Beyond Birds and Flowers: Results on MS COCO
On MS COCO, StackGAN produces diverse samples for text descriptions and demonstrates generalization to multiple objects and complex backgrounds, though image quality is lower than for birds and flowers.
- Results on MS COCO: StackGAN demonstrates generalization to MS COCO images containing multiple objects and complex backgrounds.
- Results on MS COCO: Diverse samples can be generated for each text description.
- Results on MS COCO: The method synthesizes reasonable images across various MS COCO cases.
- Results on MS COCO: Image quality on complex scenes is lower than the results for birds and flowers.