PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem

arXiv:1612.00215v1cs.CV

TL;DR

Outdoor image synthesis needs models that can generate realistic scenes while controlling objects, layouts, and global attributes. The paper proposes AL-CGAN, which conditions a GAN on semantic layouts and transient attributes. It reports realistic outdoor images with clear object boundaries and controllable day-night and weather appearance.

  • Problem

    Realistic image synthesis remains limited across object and scene types, motivating controllable generation using richer conditioning information.

  • Method

    AL-CGAN uses spatially replicated transient attributes and pixel-based semantic labels as conditioning vectors for a conditional GAN.

  • Results

    The model generates realistic outdoor scenes with objects drawn within specified semantic boundaries and transforms global appearance across day-night and weather conditions.

  • Takeaways & Limitations

    Semantic layouts control where objects appear, while transient attributes control how the scene’s global appearance is rendered.

  • Takeaways & Limitations

    Transient Attributes images lack semantic layout annotations, so the study uses one manually annotated layout for all images from each webcam and omits dense small-element annotations.

Abstract

from arXiv · show

Automatic image synthesis research has been rapidly growing with deep networks getting more and more expressive. In the last couple of years, we have observed images of digits, indoor scenes, birds, chairs, etc. being automatically generated. The expressive power of image generators have also been enhanced by introducing several forms of conditioning variables such as object names, sentences, bounding box and key-point locations. In this work, we propose a novel deep conditional generative adversarial network architecture that takes its strength from the semantic layout and scene attributes integrated as conditioning variables. We show that our architecture is able to generate realistic outdoor scene images under different conditions, e.g. day-night, sunny-foggy, with clear object boundaries.

1. Introduction

The paper addresses outdoor image synthesis by conditioning a GAN on semantic layouts and transient scene attributes. AL-CGAN aims to generate realistic scenes with controllable object boundaries and global appearance.

  • Realistic photo synthesis remains unsolved across many object types despite progress on digits, faces, chairs, rooms, videos, and text-conditioned images.
  • Conditioning on object type, visual properties, and location can increase a generator’s descriptive power while providing diversity in generated images.
  • AL-CGAN combines transient attributes such as sunny or foggy with semantic layouts that specify where scene objects should be drawn.
  • The model learns outdoor scene transformations automatically from deep convolutional networks rather than relying on hand-crafted procedures.
  • The model generates realistic images with objects inside their segments and can transform scenes, such as imagining a day scene at night.

2. Related Work

Prior work explored GANs, VAEs, and outdoor scene editing with different conditioning sources. This paper instead proposes an attribute- and layout-conditioned GAN for automatic outdoor scene editing and generation.

  • Image Generation: Earlier image-generation systems synthesized diverse objects, scenes, and videos using GANs, VAEs, recurrent architectures, and textual descriptions.
  • Image Generation: Conditional GANs incorporated class names, captions, bounding boxes, and keypoints to control generated content and location.
  • Outdoor Scene Editing: Outdoor scene editing previously used regressors and manual adjustment of transient attributes across webcam scenes.
  • Our Work: The proposed approach differs by learning attribute-controlled scene generation and editing from scratch with an attribute- and layout-conditioned GAN.

3. Model Architecture

AL-CGAN extends conditional GANs by jointly conditioning generation and discrimination on semantic layouts and transient attributes. Its generator synthesizes images from noise and conditions, while a Siamese discriminator fuses image and condition features.

  • 3.1. Preliminary: Generative Adversarial Nets: GANs train a generator and discriminator jointly in a two-player min-max game, with the generator synthesizing realistic images and the discriminator distinguishing real from generated images.
  • 3.1. Preliminary: Generative Adversarial Nets: A conditional GAN adds side information c so both generation and discrimination are controlled by contextual variables.
  • 3.2. AL-CGANs: AL-CGAN learns scene layout and content from ground-truth semantic layouts and transient attributes using deconvolutional and convolutional networks.
  • 3.2. AL-CGANs: The generator concatenates tiled noise-and-attribute inputs with 128 × 128 semantic layout maps before convolutional and deconvolutional processing.
  • 3.2. AL-CGANs: The discriminator is a Siamese network whose image and attribute-layout branches produce features that are fused before the final decision layer.
  • 3.2. AL-CGANs: Models were trained with mini-batch SGD using batches of 64, Adam learning rate 2 × 10^-4, momentum 0.5, and 400 epochs.

4. Experiments

The experiments evaluate AL-CGAN on outdoor scenes using semantic layouts, transient attributes, and noise to control realism, diversity, scene conditions, and incremental detail. Results show unseen-layout synthesis, attribute-driven transformations, scene refinement, non-memorization, and sharper images with combined conditioning.

  • Datasets and Data Preprocessing: The model is trained and evaluated on outdoor-scene data from ADE20K and Transient Attributes, using 18 predefined semantic labels.ADE20K contributes 9,201 selected outdoor images; Transient Attributes lacks semantic annotations, which are manually collected for one aligned image per webcam and reused across that webcam’s images.
  • Generating Realistic Outdoor Scenes: Unseen semantic layouts from SIFTflow and LMSun produce outdoor scenes with sharp object boundaries and realistic color distributions under a fixed clear-sunny-day condition.The results distinguish towers, skyscrapers, ground, sky, and tree leaves without observed boundary blurring.
  • Generating Realistic Outdoor Scenes: Varying noise with a fixed layout generates diverse visual appearances, while varying attributes changes scene conditions such as darkness, rain grayness, and cloud density.The experiments vary noise horizontally and attributes vertically, with neither layouts nor corresponding images seen during training.
  • Generating Realistic Outdoor Scenes: Increasing night, sunset, cloud, or rain attributes transforms the same scene while preserving relevant structure and modifying illumination, sky regions, or global tone.Night illuminates buildings, sunset darkens them, clouds modify the relevant sky portion, and rain gradually makes the scene grayish.
  • Incrementally Adding/Deleting Scene Elements: Adding scene elements to coarse layouts progressively produces more detailed images, while erasing elements yields convincing but less detailed simplified scenes.Examples add mountains, trees, lakes, rocky islands, grasslands, and trees; the authors associate semantic layout guidance with greater detail and realism.
  • Comparing with Other GAN Architectures: Nearest-neighbor comparisons indicate that AL-CGAN usually generates novel scenes rather than memorizing training images, and combined conditioning yields sharper, more diverse, realistic results.Compared with scene-label and attribute-only variants, layout conditioning improves boundaries and semantics, while full attribute-layout conditioning adds detail and color variation.

5. Conclusion

AL-CGAN generates realistic outdoor scenes by jointly conditioning on semantic layouts and transient attributes. It controls object placement, global appearance, attribute transitions, scene detail, and image quality through its architecture components.

  • 5. Conclusion: AL-CGAN uses semantic layouts to specify where background and objects are drawn, while transient attributes guide global appearance.Its condition vectors combine spatially replicated transient scene attributes with pixel-based semantic labels.
  • 5. Conclusion: Varying semantic layouts controls objects within specified boundaries, while varying attributes produces conditions such as sunny and cloudy scenes.The model also supports smooth transitions between transient attributes.
  • 5. Conclusion: AL-CGAN generates more detailed images by gradually adding new scene elements.
  • 5. Conclusion: The ablation study finds that every framework component is necessary for higher-quality images.

Additional Results

Additional results demonstrate attribute adjustment across rain, sunny, night, sunset, clouds, and fog conditions. They also show synthesis with novel layouts, smooth attribute-manifold changes, and comparisons against nearest training images.

  • Attribute adjustment: Figures 10–12 adjust rain, sunny, night, sunset, clouds, and fog attributes in AL-CGAN-generated outdoor scenes.The model is trained on ADE20K and the Transient Attributes dataset using layouts seen during training.
  • Incremental scene construction: Figure 13 incrementally adds new scene elements using semantic layouts that are novel and user-provided.These layouts were not seen during training.
  • Attribute-manifold comparison: Figure 14 contrasts smooth attribute-manifold interpolation with nearest training images that lack reasonable transformations.Some nearest images even come from different scenes.
Loading 1612.00215v1…