Source-linked AI summary
Image Generation from Layout
Bo Zhao, Lili Meng, Weidong Yin, Leonid Sigal
TL;DR
Generating realistic images with multiple objects in specified layouts is difficult because layouts underspecify appearance, interactions, and overlapping objects. Layout2Im generates diverse images from bounding boxes and categories, and outperforms baselines across three evaluation metrics while preserving object locations and recognizability.
Problem
Complex layout-based image generation is a difficult one-to-many problem because bounding boxes and categories underspecify object appearance, interactions, and overlapping pixels.
Method
Layout2Im samples appearance and relational latent codes for each object, combines them with category embeddings and layouts using convolutional LSTM, and decodes the fused representation into an image.
Results
Layout2Im significantly outperforms baselines on all three evaluation metrics, with qualitative examples showing recognizable objects in the input locations, including multiple instances of object types.
Takeaways & Limitations
The method generates realistic, diverse complex images and allows users to control generation by adding or moving objects in the layout.
Takeaways & Limitations
The hidden feature map must encode all objects at desired locations, coordinate their representations, and plausibly fill unspecified regions such as backgrounds.
Abstract
from arXiv · showhide
Despite significant recent progress on generative models, controlled generation of images depicting multiple and complex object layouts is still a difficult problem. Among the core challenges are the diversity of appearance a given object may possess and, as a result, exponential set of images consistent with a specified layout. To address these challenges, we propose a novel approach for layout-based image generation; we call it Layout2Im. Given the coarse spatial layout (bounding boxes + object categories), our model can generate a set of realistic images which have the correct objects in the desired locations. The representation of each object is disentangled into a specified/certain part (category) and an unspecified/uncertain part (appearance). The category is encoded using a word embedding and the appearance is distilled into a low-dimensional vector sampled from a normal distribution. Individual object representations are composed together using convolutional LSTM, to obtain an encoding of the complete layout, and then decoded to an image. Several loss terms are introduced to encourage accurate and diverse generation. The proposed Layout2Im model significantly outperforms the previous state of the art, boosting the best reported inception score by 24.66% and 28.57% on the very challenging COCO-Stuff and Visual Genome datasets, respectively. Extensive experiments also demonstrate our method's ability to generate complex and diverse images with multiple objects.
1. Introduction
Layout2Im addresses the difficult one-to-many problem of generating complex images from coarse layouts, where categories and locations leave object appearance and interactions unspecified. It disentangles category from appearance and composes object representations to generate realistic images with desired placements.
- Motivation: Existing text-to-image methods often generate plausible results only on simple datasets, limiting their effectiveness for complex real-world images.
- Contribution: Layout2Im conditions generation on bounding boxes and object categories to place desired objects at correct locations.The paper presents layouts as a more controllable and flexible conditioning modality than textual descriptions.
- Motivation: Complex layout-based image generation must represent many possible appearances and interactions consistent with the same bounding boxes and categories.The task is difficult because layouts provide limited information and nearby objects may overlap.
- Method: Each object is disentangled into a category encoded by a word embedding and an appearance represented by a low-dimensional vector sampled from a normal distribution.Object feature maps are then composed using convolutional LSTM into a complete-layout representation for image decoding.
2. Related Work
Prior conditional image-generation work uses modalities such as text, scene graphs, and layouts, but layout generation without detailed segmentation masks remains comparatively direct and practical. Layout2Im uses coarse layouts and disentangled object representations for diverse image generation.
- Conditional Image Generation: Conditional image generation has used source images, sketches, scene graphs, dialogues, and text descriptions, alongside VAEs, autoregressive models, and GANs.
- Image Generation from Layout: Prior layout-based methods often use layouts as intermediate or complementary representations, and some require detailed instance segmentation masks for training.The cited prior approaches combine coarse layouts with text or scene graphs, while mask acquisition is described as time-consuming and labor-intensive.
- Disentangled Representations: Layout2Im uses coarse layouts consisting of bounding boxes and category labels as the fundamental input modality for diverse image generation.Its object representations separate category-related and appearance-related information.
3. Image Generation from Layout
Layout2Im trains an end-to-end layout-conditioned generator with posterior and prior object codes, reconstructing input images while sampling new appearances from a normal prior. Latent-code regression and adversarial discrimination support consistency and realism.
- Training Pipeline: For each object, Layout2Im samples a posterior code conditioned on the cropped object and a prior code from a normal distribution.Category labels are represented by word embeddings, while bounding boxes guide object feature-map construction.
- Training Pipeline: Object feature maps are processed by an object encoder and object fuser to produce a hidden feature map containing information from all specified objects.The decoder reconstructs the ground-truth image from posterior codes and generates a new image from prior codes.
- Training Objectives: Latent-code regression recovers sampled prior codes from generated objects, encouraging a consistent code-to-output mapping and more diverse results.
- Training Objectives: Image- and object-level discriminators classify real and generated results as real or fake during adversarial training.The object discriminator also encourages generated objects to appear at their desired bounding-box locations.
- Object Latent Codes: The object estimator predicts posterior means and variances from cropped, resized objects, then samples latent codes from those estimated distributions.
3.2. Object Feature Map Composition
Layout2Im composes each object’s category and appearance information inside its bounding box, then fuses object maps with a spatially preserving convolutional LSTM. The fused representation supports coordinated scene decoding.
- 3.2. Object Feature Map Composition: The object composer creates a feature map whose bounding-box region contains the object’s identity and appearance representation.
- 3.2. Object Feature Map Composition: Category labels become word embeddings, which are concatenated with latent vectors encoding the appearance of a specific object instance.
- 3.3. Object Feature Maps Fusion: A good fused hidden map must preserve object locations, coordinate representations across objects, and fill unspecified regions such as background.
- 3.3. Object Feature Maps Fusion: A convolutional LSTM fuses downsampled object feature maps while preserving spatial information through feature-map hidden and cell states.
- 3.3. Object Feature Maps Fusion: The final convolutional-LSTM output is the fused hidden layout H, incorporating location and category information from all objects.
3.4. Image Decoder
The image decoder produces an output image from the fused layout representation through reconstruction and sampling paths. These paths differ in latent-code estimation but share the same layout, allowing varied appearances.
- The image decoder generates a result image from the fused hidden feature map.
- The reconstruction path uses posterior latent codes conditioned on input objects, whereas the sampling path draws latent codes from normal priors.
- Both paths preserve the same layout even when their generated objects differ in appearance.
3.5. Object Latent Code Regression
The model regresses sampled latent codes from generated objects to maintain a consistent connection between latent codes and outputs. This constraint is intended to prevent many-to-one mappings and promote diverse results.
- The model recovers randomly sampled latent codes from objects generated along the sampling path.
- This regression constraint helps prevent many-to-one mappings from latent codes to outputs and produces more diverse results.
- Generated objects are cropped using the input bounding boxes and passed to a shared latent-code estimator.
- The estimator’s computed mean vectors serve as regressed latent codes and are compared with the sampled codes for every object.
3.6. Image and Object Discriminators
Two discriminators assess generated images and objects to improve realism, recognizability, and spatial placement. The image discriminator distinguishes real from generated images, while the object discriminator evaluates cropped objects and their categories.
- The model uses image and object discriminators to make generated images realistic and objects recognizable.
- The adversarial objective rewards discriminating real samples from generated ones, while the generator minimizes the same adversarial loss.
- The image discriminator classifies input, reconstructed, and sampled images as real or fake.
- The object discriminator assesses real, reconstructed, and sampled objects for quality and category.
- Because generated objects are cropped according to the input boxes, the object discriminator also encourages their desired locations.
3.7. Loss Function
The generator and discriminators are trained end-to-end with a weighted combination of six losses. These losses constrain latent distributions, image reconstruction, latent-code recovery, adversarial realism, object quality, and object-category recognition.
- The generator and two discriminators are trained end-to-end in an adversarial manner.
- The generator minimizes a weighted sum of six losses, with λ_i balancing their contributions.
- The KL loss measures divergence between the posterior object-latent distribution and a normal distribution.
- The image reconstruction loss penalizes the L1 difference between the ground-truth image and reconstructed image.
- The object latent-code reconstruction loss penalizes the L1 difference between sampled and re-estimated latent codes.
- Image and object adversarial losses distinguish real images or objects from reconstructed and sampled outputs, while auxiliary classification encourages category recognition.
3.8. Implementation Details
The model uses specified architectural components and fixed training settings for 64 × 64 image generation.
- SN-GAN provides stable training, while batch normalization, ReLU, conditional batch normalization, and residual blocks support the architecture.Conditional batch normalization normalizes object feature maps according to category, and six residual blocks refine hidden image feature maps.
- The model sets both m and n to 64, with image and crop sizes of 64 × 64.
- Training uses Adam with learning rate 0.0001 and batch size 8 for 300,000 iterations.Training takes about 3 days on a single Titan Xp GPU.
4. Experiments
Experiments evaluate Layout2Im against pix2pix and sg2im on COCO-Stuff and Visual Genome using realism, recognition, and diversity measures. Qualitative, quantitative, and ablation results show stronger layout adherence, diversity, and overall performance than the baselines.
- Experimental setup: Experiments compare Layout2Im with pix2pix and sg2im on COCO-Stuff and Visual Genome.The models use multiple-object bounding-box annotations with labels, and sg2im is evaluated using ground-truth layouts.
- Evaluation metrics: Four metrics assess generated-image quality: Inception Score, Fréchet Inception Distance, Object Classification Accuracy, and Diversity Score.These metrics target realism, similarity to real images, object recognizability, and diversity from the same input.
- Qualitative results: Layout2Im generates complex images with multiple objects, including multiple instances of the same object type.Examples include two boats, two cows, and two people.
- Qualitative results: Generated objects respect input bounding-box locations and remain recognizable and consistent with their input labels.
- Qualitative results: Progressively adding or moving bounding boxes places new objects at desired locations while keeping existing objects consistent.
- Qualitative results: Sampling three images from the same layout produces different appearances while preserving the shared layout.The results illustrate the model’s ability to represent multiple realizations of one layout.
- Quantitative results: The proposed method significantly outperforms baselines across Inception Score, Fréchet Inception Distance, and Object Classification Accuracy.On COCO, its Diversity Score is 0.15 versus 0.02 for sg2im; pix2pix is deterministic with Diversity Score 0.
- Ablation study: Removing any loss term decreases overall performance, while the full model balances realism, object recognition, and diversity.Removing classification loss harms recognition and increases diversity, whereas removing adversarial losses substantially decreases Inception Score.
5. Conclusion
The paper introduces an end-to-end method for generating diverse, realistic images from layouts, with recognizable objects at desired locations. Experiments on COCO-Stuff and Visual Genome demonstrate its ability to generate realistic complex images, while high-resolution generation remains future work.
- Layout2Im generates diverse images from layouts consisting of bounding boxes and categories.
- The method produces realistic images with recognizable objects at their desired locations.
- The generation process can be controlled by adding or moving objects in the layout.
- Qualitative and quantitative results on COCO-Stuff and Visual Genome demonstrate realistic complex-image generation.
- High-resolution image generation from layouts remains future work, alongside finer-grained instance-attribute control.