Source-linked AI summary
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
Jianwei Yang, Anitha Kannan, Dhruv Batra, Devi Parikh
TL;DR
Image generators often produce unrealistic foregrounds and fail to model the layered, contextual structure of scenes. LR-GAN recursively generates backgrounds and foregrounds while factoring appearance, shape, and pose; experiments find more natural, human-recognizable images than DCGAN.
Problem
Existing generators produce foregrounds that can be deformed, blended into backgrounds, and less recognizable, while overlooking contextual background-foreground structure.
Method
LR-GAN recursively generates a background and successive foregrounds, separately modeling each foreground's appearance, shape, and pose for contextual composition.
Results
LR-GAN generates globally natural images with realistic, recognizable object and background structures, and AMT judges preferred its images 68.4% of the time versus 31.6% for DCGAN.
Takeaways & Limitations
The model's compositional generation produces images free of blending artifacts, with clear shapes and contextually relevant foreground objects.
Takeaways & Limitations
The model assumes that the number of objects is known for each dataset.
Abstract
from arXiv · showhide
We present LR-GAN: an adversarial image generation model which takes scene structure and context into account. Unlike previous generative adversarial networks (GANs), the proposed GAN learns to generate image background and foregrounds separately and recursively, and stitch the foregrounds on the background in a contextually relevant manner to produce a complete natural image. For each foreground, the model learns to generate its appearance, shape and pose. The whole model is unsupervised, and is trained in an end-to-end manner with gradient descent methods. The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than DCGAN.
1 INTRODUCTION
LR-GAN addresses unrealistic foregrounds and missing scene structure by recursively generating backgrounds and contextually composed foreground layers. It is trained unsupervised and evaluated for naturalness and recognizability across multiple datasets.
- Existing image generators often produce foreground objects that are deformed, blended into backgrounds, and difficult to recognize.
- Images reflect layered 3D-world structure, including backgrounds and foreground objects placed in contextually relevant ways.
- LR-GAN recursively generates a background, then foreground appearance, mask, and affine pose conditioned on the current composite image.
- The model learns object-shape foreground-background masks without object-mask supervision and factorizes appearance from transformation.
- LR-GAN is evaluated on MNIST-ONE, MNIST-TWO, CIFAR-10, and CUB-200 using samples, metrics, and human studies.
2 RELATED WORK
LR-GAN combines sequential generation with GAN flexibility through recursive, compositional modeling of appearance, shape, and pose. It differs from related recursive approaches through explicit pose modeling, factored shape generation, and contextual conditioning.
- Sequential generation has improved image-generation models, but recurrent VAE approaches had not shown scalability to natural images.
- LR-GAN combines recursive sequential generation with GANs to model image composition through appearance, shape, and pose.
- Unlike related approaches, LR-GAN explicitly models foreground poses with a dedicated generator for natural-image generation.
- LR-GAN separates shape and appearance generators, allowing more flexible generated scenes.
- Its recursive framework conditions later objects on prior hidden vectors and generated objects for explicit contextual modeling.
3 PRELIMINARIES
The preliminaries motivate inductive biases for GAN generation and represent images as layered compositions. LR-GAN uses foreground appearance, shape, and pose to model these layers.
- 3.1 GENERATIVE ADVERSARIAL NETWORKS: GANs train a generator and discriminator with competing goals, using a minmax value function.
- 3.1 GENERATIVE ADVERSARIAL NETWORKS: The discriminator supplies gradient information based on whether generated samples lie on the data manifold rather than matching specific training examples.
- 3.1 GENERATIVE ADVERSARIAL NETWORKS: Effective GAN generators benefit from inductive biases, motivating recurrent image generation that produces objects over successive time steps.
- 3.2 LAYERED STRUCTURE OF IMAGE: A layered image can factorize a foreground and background using appearance and a mask that depicts layer shapes.
- 3.2 LAYERED STRUCTURE OF IMAGE: An alternative layered representation uses foreground and background appearance together with an affine transformation and spatial transformation operator.
- 3.2 LAYERED STRUCTURE OF IMAGE: LR-GAN models foreground appearance, shape, and pose simultaneously within layered composition.
4 LAYERED RECURSIVE GAN (LR-GAN)
LR-GAN uses a layered recursive generator to construct images from a background and successive foreground objects, explicitly modeling each object's appearance, shape, and pose. Its evaluation combines established and human-annotation-based metrics for comparing generated and real distributions.
- Layered recursive generator: LR-GAN's layered recursive generator composes an image by pasting successive object layers onto a generated background.The background is generated once, while foreground generation recurs over time with shared parameters.
- Generator architecture: The architecture separates a single-use background generator from a recurrent foreground generator, with a discriminator trained in the GAN minmax framework.The background becomes the canvas for subsequent foregrounds, and the two generators do not share parameters.
- Layered recursive generator: Each foreground layer is parameterized by canonical appearance, shape or mask, and pose or affine transformation.The spatial transformation operator warps the appearance and mask before composition.
- Temporal connections: The model's temporal connections use an LSTM over noise vectors and previous object response maps so foreground generation receives information about earlier generated content.These connections let the model see previously generated objects while adding new layers.
- Evaluation metrics: The evaluation uses standard GAN metrics plus adversarial accuracy and adversarial divergence based on human-labeled categories.Adversarial accuracy compares classifier accuracies, while adversarial divergence measures KL divergence between generated and real category posteriors.
5 EXPERIMENT
Across MNIST-ONE, MNIST-TWO, CIFAR-10, and CUB-200, LR-GAN generates layered images with separate backgrounds, foregrounds, masks, and context-dependent composition. Qualitative, human, and quantitative evaluations report recognizable structures, improved realism over DCGAN, and useful contextual dependencies.
- Evaluation setup: LR-GAN is evaluated on MNIST-ONE, MNIST-TWO, CIFAR-10, and CUB-200 using qualitative samples, evaluation metrics, and Amazon Mechanical Turk studies.The model uses dataset-specific timesteps and assumes the number of objects is known.
- Qualitative results: CIFAR-10 and CUB-200 samples show foregrounds with clear shapes, sharper bird forms, and fewer blending artifacts between backgrounds and foregrounds.The qualitative results attribute these properties to the model’s compositional generation process.
- MNIST-ONE and MNIST-TWO: MNIST-ONE results nearly perfectly disentangle digits from backgrounds, while trained masks become nearly binary and accurately carve out the foreground digits.MNIST-TWO results separately generate two foreground digits while incorporating context from earlier timesteps.
- MNIST-ONE and MNIST-TWO: In MNIST-TWO, the second digit tends to appear on the opposite side when the first digit is placed left or right.This demonstrates context-sensitive placement across recursive foreground-generation timesteps.
- Human evaluation: 68.4% of AMT judges selected LR-GAN images as more realistic, compared with 31.6% for DCGAN.The comparison used 1000 matched image pairs, with nine judges evaluating each pair by majority vote.
- Human evaluation: LR-GAN reduced CIFAR-10 non-recognizability by 10% absolute: 67.3% for LR-GAN versus 77.7% for DCGAN.The study used five judges per image and quality levels based on majority agreement.
- Quantitative evaluation: LR-GAN outperformed DCGAN across the reported Inception Score, Adversarial Accuracy, and Adversarial Divergence comparisons.The paper cautions that Inception Score varies with the classifier model used.
- Quantitative evaluation: Discriminator features achieved 62.09%±0.01% 1-NN accuracy for LR-GAN versus 56.05%±0.02% for DCGAN.Features came from the discriminator’s last convolutional layer, using cosine similarity.
6 APPENDIX
Algorithm 1 describes stochastic layered recursive image generation: a background is generated first, then recurrent steps add transformed foreground layers to the evolving composite.
- The process samples an initial latent vector and uses the background generator to produce x0.
- At each recurrent step, a new latent vector is sampled and processed through the model’s recurrent state.
- The algorithm applies nonlinear embedding layers and predicts representations used by the foreground generator.
- Foreground shape is generated from the recurrent representation, while appearance, mask, and pose are combined through spatial transformation.
- The transformed foreground is composited with the previous image using the spatially transformed mask.
6.2 MODEL CONFIGURATIONS
The appendix specifies dataset-dependent generator and discriminator architectures, while fixing random and hidden vector dimensions and comparing parameter counts with DCGAN.
- Random vectors and hidden vectors are set to dimension 100 for all datasets.
- The configurations cover MNIST-ONE, MNIST-TWO, CUB-200, and CIFAR-10 with separate background, composition, foreground, and discriminator components.
- The appendix compares LR-GAN and DCGAN parameter counts, reporting LR-GAN before the slash and DCGAN after it.
- MNIST-ONE: MNIST-ONE uses a background generator, composition generator, and discriminator with the listed convolutional architectures.
- MNIST-TWO: MNIST-TWO uses deeper listed background and discriminator configurations than the MNIST-ONE setup.
- CUB-200 and CIFAR-10: CUB-200 and CIFAR-10 receive distinct architecture specifications for background, composition, foreground, and discriminator generators.
6.3 RESULTS ON MNIST-ONE
MNIST-ONE is evaluated with human recognition studies and GAN metrics, with LR-GAN outperforming DCGAN on adversarial measures while standard Inception Score is unsuitable for this dataset.
- 1,000 images from LR-GAN and DCGAN, plus 1,000 real images, were judged by five unique workers for digit recognition.
- Human annotations recorded recognized digit categories and quality levels for the MNIST-ONE generations.
- LR-GAN achieves higher adversarial accuracy and lower adversarial divergence than DCGAN on MNIST-ONE.
- The standard Inception Scores are low for all three image sets because the ImageNet-trained Inception network differs from MNIST’s data distribution.
- The authors therefore argue that standard Inception Score is not suitable for some image datasets such as MNIST.
6.4 MORE RESULTS ON CUB-200
CUB-200 results show that lowering the minimum object scale allows larger foregrounds while preserving automatically generated crisp bird-like masks.
- Setting the minimal allowed object scale to 1.1 permits the model to generate larger foreground objects.
- Under the 1.1 scale setting, the model automatically generates crisp bird-like masks, as with the 1.2 constraint.
- The generated CUB-200 examples are presented as backgrounds, foregrounds, masks, masked foregrounds, transformed foregrounds, composites, and nearest-neighbor real images.
6.5 MORE RESULTS ON CIFAR-10
On CIFAR-10, LR-GAN produces varied images without memorizing training examples and generates foregrounds that remain compatible with fixed backgrounds.
- Qualitative results: LR-GAN does not memorize CIFAR-10 training data under a minimal allowed object scale of 1.1.Generated images were compared with the closest training images using cosine similarity in pixel space.
- Latent foreground space: With the same generated background, LR-GAN consistently generates contextually compatible foregrounds.Grass-like backgrounds tend to produce horses and deer, while blue-sky backgrounds tend to produce airplane-like objects.
- Human studies: Human studies on CIFAR-10 collected both list-based image names and free-form one-word descriptions from five people per image.Word clouds were produced for real images, DCGAN samples, and LR-GAN samples.
6.6 RESULTS ON LFW FACE DATASET
On LFW face images, LR-GAN separately generates backgrounds and faces, learns where to place faces, and models varied affine transformations across datasets.
- LFW generation: LR-GAN generates original LFW images containing substantial backgrounds rather than relying on cropped and aligned faces.This configuration evaluates modeling of object appearance, shape, and pose.
- LFW generation: The model learns to place generated faces so the complete LFW image looks natural.The generation process includes separate background and face components before composition.
- Comparison: Removing transformation modeling causes generation results to degrade much relative to the comparison method.The cited comparison is Kwak and Zhang (2016), which does not model transformation.
- Transformation statistics: Affine transformation statistics show varied scaling, translation, and rotation across MNIST-ONE, CUB-200, CIFAR-10, and LFW.The six parameters are scaling, translation, and rotation in the x and y coordinates.
6.8 CONDITIONAL IMAGE GENERATION
Conditional LR-GAN decomposes input images into backgrounds and foregrounds, learning object masks on CIFAR-10 and face regions on LFW without supervision.
- Conditional model: Conditional LR-GAN uses encoded input images, residual foreground information, reconstruction loss, and adversarial loss to reconstruct CIFAR-10 images.The background generator receives the image, while the foreground generator receives the residual after background generation.
- CIFAR-10: On CIFAR-10, the conditional model successfully decomposes input images into background and foreground.The background generator tends to inpaint a complete background, while the foreground generator produces an object mask.
- LFW: On LFW, conditional LR-GAN consistently learns to generate face regions despite large background portions in the input images.The authors describe this as successfully learning to detect faces in images.
- Implications and scope: The authors argue that the model may support unsupervised generative image segmentation and object detection.They identify verification on high-resolution and more complicated datasets as future work.