Source-linked AI summary
Image Synthesis From Reconfigurable Layout and Style
Wei Sun, Tianfu Wu
TL;DR
Layout-to-image synthesis must produce sharp, realistic images while preserving one-to-many style variation and adapting to layout changes. LostGANs address this with weakly supervised masks and ISLA-Norm, and achieve state-of-the-art performance on COCO-Stuff and Visual Genome.
Problem
Reconfigurable layout-to-image synthesis must preserve multiple plausible styles and adapt to layout or style changes, but prior methods mainly operate at low resolution with insufficient diversity.
Method
LostGANs use an end-to-end layout- and style-based GAN with weakly supervised fine-grained masks and object instance-specific layout-aware feature normalization.
Results
State-of-the-art performance is obtained on COCO-Stuff and Visual Genome across inception score, Fréchet Inception Distance, diversity score, and classification accuracy.
Takeaways & Limitations
LostGANs demonstrate reconfigurable scene generation with layout changes and instance-level style control.
Abstract
from arXiv · showhide
Despite remarkable recent progress on both unconditional and conditional image synthesis, it remains a long-standing problem to learn generative models that are capable of synthesizing realistic and sharp images from reconfigurable spatial layout (i.e., bounding boxes + class labels in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors), especially at high resolution. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of a layout and style latent code. In this paper, we present a layout- and style-based architecture for generative adversarial networks (termed LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. Inspired by the vanilla StyleGAN, the proposed LostGAN consists of two new components: (i) learning fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and (ii) learning object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. In experiments, the proposed method is tested on the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained. The code and pretrained models are available at \url{https://github.com/iVMCL/LostGANs}.
1. Introduction
Layout-to-image synthesis must generate multiple plausible styles while preserving object placement and adapting to layout changes. LostGANs address this with weakly supervised masks and instance-specific, layout-aware style normalization, achieving state-of-the-art results on COCO-Stuff and Visual Genome.
- Motivation and Objective: Conditional layout-to-image synthesis remains challenging because layouts require one-to-many generation, consistent multi-object synthesis, occlusion handling, and high-resolution outputs.Existing reconfigurable methods mainly target 64 × 64 resolution and insufficient style diversity.
- Motivation and Objective: LostGANs generate images from reconfigurable layouts and style codes while adapting to added or moved bounding boxes.The model preserves multiple plausible images for a layout and responds to layout and style perturbations.
- Method Overview: LostGANs use ResNet backbones and jointly apply image and object adversarial hinge losses in end-to-end training.The discriminator combines a shared ResNet backbone with image-head and object-head classifiers.
- Method Overview: The generator predicts fine-grained object masks weakly supervisedly, creating a layout-to-mask-to-image pathway that captures object geometry.The learned masks help place objects with fine-grained geometric properties without requiring ground-truth masks.
- Method Overview: ISLA-Norm extends adaptive instance normalization with object-instance-specific, layout-aware affine transformations for spatially distributed multi-object style control.It computes batch statistics while recalibrating affine parameters for each sample and object instance.
- Experiments: State-of-the-art performance is reported on COCO-Stuff and Visual Genome across inception score, Fréchet Inception Distance, diversity score, and classification accuracy.The experiments support the effectiveness of LostGAN and its ISLA-Norm design.
2. Related Works
Prior work studied conditional and layout-based image synthesis, but reconfigurable layout-to-image generation remains technically difficult. LostGANs combine established conditional and unconditional GAN practices with ISLA-Norm and report state-of-the-art results on two datasets.
- Conditional Image Synthesis: Conditional GANs incorporate labels, images, text, or other information through generator inputs, discriminator features, or projection-based conditioning.LostGANs feed layout conditions to the generator through ISLA-Norm and object information to a projection-based discriminator.
- Image Synthesis from Layout: Prior layout-based methods use layouts and object information for text-to-image synthesis, often generating semantic layouts before synthesizing images.Some approaches require pixel-level masks, whereas LostGANs target direct synthesis from reconfigurable layouts and style.
- Contributions: LostGANs integrate practices from conditional and unconditional GANs for image synthesis from reconfigurable layout and style.The architecture is presented for a relatively new task with multiple objects and controllable styles.
- Contributions: ISLA-Norm explicitly incorporates layout information into object-instance-specific affine transformations, combining ideas from conditional BatchNorm and StyleGAN's AdaIN.This design targets instance-level, spatially distributed style control.
- Results: State-of-the-art inception score, Fréchet Inception Distance, diversity score, and classification accuracy are reported on COCO-Stuff and Visual Genome.The comparison spans two widely used datasets and four evaluation measures.
3. The Proposed Method
LostGAN defines layout-to-image generation with image- and object-level style reconfiguration, then combines mask prediction, ISLA-Norm, and image/object adversarial discrimination. Its generator uses layout, global style, and instance-style inputs, while its discriminator scores both complete images and cropped objects.
- 3.1. Problem Formulation: LostGAN models images from labeled bounding-box layouts, a global style code, and per-object style codes.The target is a generation function approximating the conditional distribution p(I|L, zimg, Zobj).
- 3.1. Problem Formulation: The method evaluates reconfigurability through image-style changes, object-style changes, and layout edits such as adding objects or moving boxes.When adding an object, the method also samples a new object-style code.
- 3.2.1 The Generator: The generator uses a ResNet backbone whose residual blocks progressively upsample features; 64×64 generation uses four blocks and 128×128 generation uses five.The global style code is projected into the initial feature tensor before upsampling.
- 3.2.2 The ISLA-Norm: ISLA-Norm normalizes feature responses and applies object-instance-specific, layout-aware affine parameters that are recalibrated for each sample.The method computes gamma and beta from label embeddings and object-style codes, predicts soft masks, resizes them to boxes, and averages overlapping contributions.
- 3.2.3 The Discriminator: The discriminator combines a shared ResNet backbone with image and object heads, producing an image score and an average score for cropped objects.Its object head uses ROI Align, and a separate label embedding supports the object adversarial hinge loss.
- 3.2.4 The Loss Functions: Training uses hinge adversarial losses with a weighted image/object objective, where λ controls the trade-off and is set to 1 in the experiments.The discriminator processes real and synthesized image-layout pairs, while the generator uses fake-data expectations.
4. Experiments
Experiments evaluate LostGAN on COCO-Stuff and Visual Genome at 64×64 and 128×128 resolutions using image quality, diversity, classification accuracy, and reconfigurability tests. LostGAN consistently outperforms Layout2Im and supports layout, image-style, instance-style, and weakly supervised mask reconfiguration.
- Experimental setup: Experiments use COCO-Stuff and Visual Genome, evaluating generated images at 64×64 and 128×128 resolutions.Comparisons include Layout2Im, sg2im, and pix2pix.
- Quantitative results: LostGAN outperforms Layout2Im on Inception Score and Diversity Score, with improved 64×64 visual quality and diversity.The evaluation also reports FID and classification accuracy.
- Quantitative results: At 128×128 resolution, LostGAN obtains consistently better results than the compared methods.The model is further reported to synthesize higher-quality images at this resolution.
- Layout reconfiguration: Adding objects or moving bounding boxes produces reasonable objects at desired positions while preserving existing-object appearance and style consistency.The reported tests demonstrate layout reconfiguration under fixed styles for existing objects.
- Style reconfiguration: Multiple samples from one layout show varied visual appearance while preserving object locations, and instance-style interpolation changes one object while leaving others unaltered.Sky style changes from blue to dusk, while grass changes from green to withered.
- Weakly-supervised mask prediction: LostGAN learns semantically reasonable fine-grained masks weakly, without using ground-truth object masks during training, including overlapping objects.For overlapping boxes, labels are assigned according to predicted mask weights.
5. Conclusion
LostGANs is an end-to-end layout- and style-based GAN architecture for generating images from reconfigurable layouts and styles. It combines weakly supervised fine-grained mask learning with ISLA-Norm for multi-object style generation and achieves state-of-the-art performance on COCO-Stuff and Visual Genome.
- LostGANs generates images from reconfigurable layouts and styles through an end-to-end layout- and style-based GAN architecture.
- Weakly supervised fine-grained mask maps bridge the gap between layouts and images.
- ISLA-Norm enables multi-object style generation through object instance-specific layout-aware feature normalization in the generator.
- LostGANs achieves state-of-the-art performance on the COCO-Stuff and Visual Genome datasets.