Source-linked AI summary
Generating Images Part by Part with Composite Generative Adversarial Networks
Hanock Kwak, Byoung-Tak Zhang
TL;DR
Image generation must model complex hierarchical structure, but conventional generation reconstructs whole images from latent variables. The paper proposes CGAN, which uses recurrently coordinated generators and alpha blending to generate image parts sequentially; experiments suggest that it can learn image factors without labels, although evaluation is mainly qualitative and the approach remains scoped to hierarchical data and proposed extensions.
Problem
Image generation requires reconstructing complex, hierarchically structured images from latent variables, while unsupervised disentanglement of image factors remains insufficiently solved.
Method
CGAN connects multiple generators through an RNN and sequentially combines their RGBA outputs with alpha blending; CGAN+VAE adds encoders for latent variables.
Results
CGAN successfully disentangles image factors without labels and generates image parts such as backgrounds, faces, hair, and flowers through different generators.
Takeaways & Limitations
The experiments suggest that hierarchical image structure can be learned empirically by generating images part by part with multiple generators.
Takeaways & Limitations
Evaluation relies mainly on qualitative analysis because generative-model objectives vary, and the model's demonstrated scope is hierarchical image data.
Abstract
from arXiv · showhide
Image generation remains a fundamental problem in artificial intelligence in general and deep learning in specific. The generative adversarial network (GAN) was successful in generating high quality samples of natural images. We propose a model called composite generative adversarial network, that reveals the complex structure of images with multiple generators in which each generator generates some part of the image. Those parts are combined by alpha blending process to create a new single image. It can generate, for example, background and face sequentially with two generators, after training on face dataset. Training was done in an unsupervised way without any labels about what each generator should generate. We found possibilities of learning the structure by using this generative model empirically.
Related Works
The paper situates CGAN among models for learning image structure, generation, and latent representations. Unlike recurrent models that construct images through feedback, CGAN combines outputs from multiple generators sequentially.
- Earlier probabilistic graphical models, including RBMs, DBMs, and DBNs, modeled latent structure but faced computational costs or simple-dataset restrictions.
- VAEs reconstruct data from continuous latent variables, while DRAW and related recurrent variants process image generation recurrently with attention or feedback.
- Recurrent adversarial networks sequentially add generated images, but RGB-channel addition intermixes pixels; CGAN adds an alpha channel to avoid this issue.
- GAN variants improve generated-image quality, while VAE/GAN uses discriminator features for reconstruction and Structure-GAN plus Style-GAN separate structure from style.
Model
CGAN connects multiple alpha-producing generators through an RNN and combines their RGBA outputs sequentially, while CGAN+VAE adds encoders for latent variables and feature-based reconstruction. Alpha loss constrains generator participation so intermediate outputs remain separable rather than allowing one generator to dominate.
- Alpha Blending: Alpha blending covers the previous image with the next using RGB values and alpha transparency, retaining the next image where alpha is near one and the previous where it is near zero.The new image is assumed opaque, with its alpha value fixed at one.
- Composite Generative Adversarial Networks: CGAN uses multiple generators connected by an RNN; each receives sequential hidden states and produces an RGBA image that is alpha-blended into the final output.The RNN preserves consistency among generators, and the final image is formed from intermediate outputs.
- Model: CGAN training retains the GAN objective while updating the discriminator and each generator from sampled data and latent vectors; different learning rates can stabilize training.
- Combining Variational Autoencoder: CGAN+VAE adds n encoders that generate latent variables under the same prior as CGAN, while its VAE objective combines prior regularization with reconstruction terms.The discriminator's hidden representation can provide feature-based reconstruction information for the VAE loss.
- Model: Figure 3 presents CGAN samples trained on CelebA and Oxford 102 Flowers, with checkerboards indicating transparent-image backgrounds.
- Alpha Loss: Because a single generator can govern the final output despite multiple generators, alpha loss constrains total alpha and pushes individual alpha values toward zero or one.The resulting variants are called CGAN+A and CGAN+VAE+A.
Experiments
Experiments evaluate CGAN qualitatively and with SSIM on face, flower, and cartoon datasets. The results show comparable image quality to GAN, part-wise generation across multiple domains, and unsupervised disentanglement of image factors.
- Experimental Setup: Experiments use 64 × 64 images and an LSTM-based RNN with DCGAN-like generators and discriminators, including CelebA, Oxford 102 Flowers, and Pororo data.CelebA contains 202,599 face images, Oxford 102 Flowers covers 102 flower classes, and Pororo supplies diverse cartoon frames with recurring backgrounds.
- Evaluation: CGAN output quality is similar to GAN according to Table 1.SSIM is used as a quantitative measure, with the largest sample similarity retained for each test image.
- Generating Images Part by Part: CelebA samples separate backgrounds, faces, and hair across three generators, while flower samples separate backgrounds and flowers across two.CGAN also generates images from the more complex Pororo cartoon-video dataset.
- Generating Images Part by Part: The CelebA third generator can fail to produce meaningful images, but alpha loss diminishes this problem and makes CGAN+VAE intermediate images less blurry and more separable.These observations concern CGAN with three generators and the CGAN+VAE variant.
- Disentanglement of Factors in Images: The first latent variable determines overall image outlines, while later variables control remaining variations conditioned on it.VAE visualization shows common factors influencing final images, and CGAN disentangles image factors without labels.
Discussion
The paper argues that unlabeled learning can capture interacting image factors through hierarchical structure, while noting dataset scale limits finer understanding and broader extensions remain prospective.
- The authors caution that 100K–1M images may be too small for learning disentangled factors as effectively as humans exposed to years of real-life video.
- Figure 6 illustrates that encoding one image while fixing another produces reconstructions reflecting the fixed image's features.
- Unlabeled learning can capture interacting image factors by constructing hierarchical image structures.
- The model is proposed for extension to video, text, audio, and multimodal settings using specialized encoders and generators.