PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Image Synthesis from Semantic Layouts via Conditional IMLE

Ke Li, Tianhao Zhang, Jitendra Malik

arXiv:1811.12373v2cs.CVcs.GRcs.LG

TL;DR

For semantic layouts, existing conditional image synthesis methods often generate only one or a fixed number of plausible images. This paper uses IMLE to generate arbitrarily many images from the same layout, and reports greater diversity with fewer artifacts than the leading approach despite using the same architecture.

  • Problem

    For semantic layouts, existing methods generate only one or a fixed number of plausible images, limiting outputs for inputs with multiple plausible appearances.

    Source: §1 · p. 1

  • Method

    The method applies IMLE to semantic-layout image synthesis, sampling different noise vectors with a single neural network.

    Source: §1 · pp. 1–2

  • Results

    The method generates more diverse images with fewer artifacts than the leading approach, despite using the same architecture.

    Source: §5 · p. 8, §4.4 · p. 8

  • Takeaways & Limitations

    The model learns a sensible latent space without supervision, with interpolations producing continuous changes and shared noise vectors producing consistent styles across layouts.

    Source: §5 · p. 8

Abstract

from arXiv · show

Most existing methods for conditional image synthesis are only able to generate a single plausible image for any given input, or at best a fixed number of plausible images. In this paper, we focus on the problem of generating images from semantic segmentation maps and present a simple new method that can generate an arbitrary number of images with diverse appearance for the same semantic layout. Unlike most existing approaches which adopt the GAN framework, our method is based on the recently introduced Implicit Maximum Likelihood Estimation (IMLE) framework. Compared to the leading approach, our method is able to generate more diverse images while producing fewer artifacts despite using the same architecture. The learned latent space also has sensible structure despite the lack of supervision that encourages such behaviour. Videos and code are available at https://people.eecs.berkeley.edu/~ke.li/projects/imle/scene_layouts/.

1. Introduction

For semantic layouts, the paper addresses the challenge of generating multiple plausible images when most methods produce only one or a fixed number. It proposes conditional image synthesis based on IMLE, which supports arbitrary sampling and uses a single network for more stable training.

1. Introduction

  • Most methods generate only one image per input, although the same input can be consistent with multiple plausible images and datasets usually provide only one ground-truth image.The model must learn to generate other plausible images without multiple ground-truth examples for each input.Source: §1 · p. 1
  • For semantic layouts, the method uses IMLE instead of GANs, which are prone to mode collapse in conditional synthesis.Mode collapse can make a model generate only one plausible image per input regardless of latent noise.Source: §1 · p. 1
  • Unlike, it can generate an arbitrary number of images per input by sampling different noise vectors.Unlike, it uses one neural network rather than three networks with opposing roles, making training more stable.Source: §1 · p. 2

2. Related Work

Prior work on image synthesis includes unimodal predictors, fixed-output models, and approaches intended to capture arbitrary modes. The section describes limits of these approaches, including mode collapse, fixed output counts, training difficulty, and challenges with global image consistency.

2.1. Unimodal Prediction

  • Conditional GANs and regression methods generally produce a single image per input; attribute-conditioned approaches may also require supervision or manual input changes.Regression is unimodal because neural nets produce point estimates, while semantic-layout GANs can suffer mode collapse.Source: §2.1 · p. 2

2.2. Fixed Number of Modes

  • Branch- or model-based methods generate a fixed number of outputs and cannot generate continuous changes.Examples include and Chen and Koltun.Source: §2.2 · p. 2

2.3. Arbitrary Number of Modes

  • Learned regularizers address mode collapse with trade-offs: Bi-GAN/ALI [6] [7] can still collapse, VAE-GAN [18] has a train–test latent mismatch, and Bicycle-GAN is difficult to train.Bicycle-GAN trains a generator, discriminator, and encoder simultaneously.Source: §2.3 · p. 2
  • Colorization methods model per-pixel color marginals but do not learn the joint distribution needed for global consistency.Correlations between colors of different pixels make consistency difficult to ensure.Source: §2.3 · p. 2

3. Method

The method replaces GAN-based conditional synthesis, which can ignore plausible outputs through mode collapse, with IMLE’s real-to-generated matching. Conditional IMLE matches each ground-truth image only to generated outputs from its corresponding input layout.

3. Method

  • The method addresses GAN-based conditional synthesis, where mode collapse can prevent a model from learning the distribution of plausible outputs.The paper frames multimodal image synthesis as generating diverse images for the same semantic layout.Source: §3 · pp. 2–3

3.1. Why Mode Collapse Happens

  • GAN mode collapse can make a conditional generator ignore latent noise and produce only one output, even when multiple outputs are plausible.The paper illustrates GAN mode collapse as generated images being pushed toward real images while some real images are never selected.Source: §3 · p. 3, §3.1 · p. 3, Figure 2 · p. 3

3.2. IMLE

  • IMLE matches each real image to a generated image, then makes the matched image more similar, ensuring no real images are left unmatched.Unlike GAN training, which matches each generated image to a real image, IMLE reverses the matching direction.Source: §3.2 · p. 3

3.3. Conditional IMLE

  • Conditional IMLE feeds the input label alongside latent noise and matches each ground-truth output only to samples generated from its corresponding label.Outputs generated from a different input label cannot be matched to the current ground-truth output.Source: §3.3 · pp. 3–4

3.4. Probabilistic Interpretation

  • IMLE approximately maximizes a lower bound on log-likelihood without a discriminator or adversarial training, making training more stable and reducing hyperparameter tuning.The paper contrasts this with GANs, which do not approximately maximize likelihood and can collapse modes.Source: §3.4 · p. 4

3.5. Formulation

  • For semantic layouts, the goal is to estimate the full conditional distribution p(y|x), since one layout can admit multiple plausible images.Here, x is the semantic segmentation map and y is the generated image; the method uses a VGG-19 perceptual loss [31].Source: §3.5 · p. 4

3.6. Architecture

  • The architecture adapts Cascaded Refinement Networks into an implicit probabilistic model by adding random-noise inputs and a noise encoder.The encoder maps lower-dimensional noise to the size of the noise input, which the paper says addresses the high dimensionality of concatenated noise.Source: §3.6 · pp. 4–5

3.7. Dataset and Loss Rebalancing

  • Because datasets can favor common object appearances, the authors propose two strategies to rebalance the dataset and loss.The strategies are described in the supplementary material.Source: §3.7 · p. 5

4. Experiment

Experiments compare the method with CRN and other baselines, using GTA-5 for multimodal evaluation and assessing diversity, image quality, latent-space structure, and style consistency. The results report greater diversity, fewer artifacts, and smooth latent interpolations.

4.1. Dataset

  • GTA-5 has greater hue variation than Cityscapes, which is concentrated around one hue mode; it also contains more than 20,000 images.The authors use GTA-5 for multimodal evaluation and also show results trained on BDD100K.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2. Experimental Setting

  • The evaluation uses 12,403 GTA-5 training images and 6,383 validation images at 256 × 512 resolution, with CRN as the multimodal baseline.Human image-quality evaluation compares CRN and the proposed method on 20 layouts with 62 subjects; Table 2 reports the percentage judged to have more obvious synthetic patterns, where lower is better.Source: §4.2 · p. 5, §4.3 · p. 7, Table 2 · p. 7

4.3. Quantitative Comparison

  • The proposed method achieves the highest diversity in the LPIPS comparison and outperforms baselines; ablations perform worse without the noise encoder or rebalancing scheme.Diversity is measured by average perceptual distance between output pairs for each layout.Source: §4.3 · pp. 5, 7, Table 1 · p. 7

4.4. Qualitative Evaluation

  • Qualitatively, the method produces diverse images without BicycleGAN’s distortions, while Pix2pix-HD outputs are nearly identical and CRN has little diversity.Compared with CRN, the method also produces fewer grid-like artifacts despite using the same architecture and distance metric; the ablation study indicates each component matters.Source: §4.4 · pp. 7–8
  • Interpolating between daytime and nighttime noise vectors yields a smooth transition, suggesting the latent space is sensibly ordered along time of day.The authors also report that using the same random vector across semantic maps preserves style, including in scene-editing examples.Source: Figure 6 · p. 7, Figure 7 · p. 7, §4.4 · p. 8

5. Conclusion

The paper presents an IMLE method for multimodal image synthesis from semantic layouts. It generates arbitrarily many images per layout and, compared with the leading approach, produces more diverse images with fewer artifacts while learning a sensible latent space without supervision.

5. Conclusion

  • The IMLE method generates arbitrarily many images for the same semantic layout and is easy to train.The paper reports more diversity and fewer artifacts than the leading approach despite using the same architecture.Source: §5 · p. 8
  • Interpolating noise vectors produces continuous changes, and reusing a vector across layouts yields images with consistent style.Figure 9 illustrates scene edits including adding a car, changing class labels, and deleting or replacing objects.Source: §5 · p. 8, Figure 9 · p. 8
  • The paper also presents generated images on BDD100K.Figure 10 shows examples generated using the method on that dataset.Source: Figure 10 · p. 8

Appendix A. Implementation Details

The implementation encodes semantic layouts as one-hot maps and uses appearance-based dataset and loss rebalancing to address rare object appearances. It measures image distance with a VGG-19 perceptual loss.

Appendix A. Implementation Details

  • Training inputs encode each segmentation-map pixel as a one-hot representation of its semantic category.This encoded map is fed to the neural network.Source: Appendix A · p. 11
  • The distance metric in the IMLE objective is a VGG-19 perceptual loss using features from five convolutional layers.The layers are conv1_2 through conv5_2, with weights set so each layer contributes equally.Source: Appendix A · p. 11
  • For each category, the method estimates a KDE over its average colors across training images and scores an image by the density of that category's color.The color is a three-dimensional vector, and the score is defined when the category appears in the image.Source: Appendix A · p. 11
  • Training batches reserve portions for the five categories with the largest dataset-wide areas, sampling images in proportion to their rarity scores.The authors choose large-area categories because they tend to occur more often and be visually prominent; reserving batch space for rare categories could overfit images containing them.Source: Appendix A · pp. 11–12
  • A normalized rarity mask upweights loss contributions from pixels associated with rare appearances.The mask is downsampled to match each feature-map size before element-wise weighting of the loss.Source: Appendix A · p. 12

Appendix B. Baselines with Proposed Rebalancing Scheme

Applying the proposed rebalancing to baseline methods does not close the diversity gap with the paper's method. The comparisons show only a slight CRN improvement and no Pix2pix-HD improvement.

Appendix B. Baselines with Proposed Rebalancing Scheme

  • CRN's LPIPS rises only from 0.12 to 0.13 with dataset and loss rebalancing, while the proposed method reaches 0.19.Figure 11 also shows that rebalanced CRN and Pix2pix-HD remain less diverse, while BicycleGAN is less diverse and has more artifacts than the proposed method.Source: Appendix B · p. 13, Figure 11 · p. 13
  • Pix2pix-HD shows no diversity improvement after dataset rebalancing and continues to ignore the latent input noise vector.The authors conclude that the baselines do not take advantage of the rebalancing scheme, unlike their method.Source: Appendix B · p. 13

Appendix C. Additional Results

The appendix provides videos of additional results, including smooth transitions between different renderings of the same scene. The interpolation video is represented by sampled frames.

Appendix C. Additional Results

  • The appendix makes videos of its additional results available online.The listed videos include smooth transitions between renderings of the same scene.Source: Appendix C · p. 14, §C.1 · p. 14

C.1. Video of Interpolations

  • A generated video shows smooth transitions between different renderings of the same scene.Figure 12 presents frames from a video generated by interpolating between latent noise vectors.Source: §C.1 · p. 14, Figure 12 · p. 14

C.2. Video Generation from Evolving Scene Layouts

With evolving scene layouts, the method can preserve appearance by reusing a latent noise vector across frames. The paper contrasts this with pix2pix, which does not control its mode and produces inconsistent appearance.

C.2. Video Generation from Evolving Scene Layouts

  • Reusing the same latent noise vector across changing car layouts selects a consistent mode across frames.Figure 13 shows frames with cars varying in size; the caption also describes interpolating between vectors associated with daytime and nighttime scenes and reports consistent style.Source: §C.2 · p. 15, Figure 13 · p. 15
  • Unlike the proposed model, pix2pix does not maintain consistent appearance across frames because it arbitrarily picks a mode without control over which mode it generates.Figure 14 compares adjacent-frame differences; darker pixels indicate smaller differences and lighter pixels indicate larger ones.Source: §C.2 · p. 16, Figure 14 · p. 16

Appendix D. More Generated Samples

Figures 15 and 16 show 19 generated samples each, all conditioned on the same semantic layout shown in the top-left corner.

Appendix D. More Generated Samples

  • Figure 15 presents 19 samples generated from the semantic layout shown in its top-left corner.Source: Figure 15 · p. 17
  • Figure 16 presents 19 samples generated from the semantic layout shown in its top-left corner.Source: Figure 16 · p. 18

Related papers

Loading 1811.12373v2…