Source-linked AI summary
LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, Xi Li
TL;DR
Complex multi-object layout-to-image generation requires precise control over both global layouts and individual objects, which remains difficult for text-guided diffusion models. LayoutDiffusion unifies structural image patches with layouts and uses LFM and OaCA for object-aware, position-sensitive fusion. It outperforms prior methods on COCO-stuff and VG, while realistic distortion-free generation remains difficult for complex layouts.
Problem
Text-guided diffusion models can miss objects and misplace, misshape, or misclassify objects in complex multi-object scenes.
Method
LayoutDiffusion converts multi-resolution structural image patches with position and size information into special layouts, then fuses them with input layouts using LFM and OaCA.
Results
46.35%, 9.61%, and 26.70% relative improvements on COCO-stuff, and 44.29%, 11.30%, and 41.82% on VG, are reported for FID, DS, and CAS.
Takeaways & Limitations
Experiments on COCO-stuff and Visual Genome show significant improvements across image quality, diversity, and controllability metrics over GAN-based and diffusion-based methods.
Takeaways & Limitations
Realistic images without distortion and overlap remain difficult for complex multi-object layouts, and the model requires dataset-specific training with detection labels.
Abstract
from arXiv · showhide
Recently, diffusion models have achieved great success in image synthesis. However, when it comes to the layout-to-image generation where an image often has a complex scene of multiple objects, how to make strong control over both the global layout map and each detailed object remains a challenging task. In this paper, we propose a diffusion model named LayoutDiffusion that can obtain higher generation quality and greater controllability than the previous works. To overcome the difficult multimodal fusion of image and layout, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form. Moreover, Layout Fusion Module (LFM) and Object-aware Cross Attention (OaCA) are proposed to model the relationship among multiple objects and designed to be object-aware and position-sensitive, allowing for precisely controlling the spatial related information. Extensive experiments show that our LayoutDiffusion outperforms the previous SOTA methods on FID, CAS by relatively 46.35%, 26.70% on COCO-stuff and 44.29%, 41.82% on VG. Code is available at https://github.com/ZGCTroy/LayoutDiffusion.
1. Introduction
LayoutDiffusion addresses the difficulty of controlling complex multi-object image generation by unifying image patches and layouts, with object-aware fusion for spatially precise conditioning. It reports improvements over prior layout-to-image methods on COCO-stuff and VG.
- Text-guided diffusion can miss objects or misgenerate their positions, shapes, and categories in complex multi-object scenes.
- Layout provides bounding-box positions and object categories, supplying spatial and high-level semantic information for stronger controllability.
- LayoutDiffusion treats each multi-resolution image patch as a special object and transforms the patched image into a layout for unified fusion.
- LFM and OaCA fuse image patches with layouts using object-aware, position-sensitive relationships in a unified coordinate space.
- Experiments across quality, diversity, and controllability metrics show significant gains over state-of-the-art GAN-based and diffusion-based methods.
- 46.35%, 9.61%, and 26.70% relative improvements on COCO-stuff, and 44.29%, 11.30%, and 41.82% on VG, are reported for FID, DS, and CAS.
2. Related work
Related work covers layout-to-image generation and diffusion models, emphasizing the field’s progression from GAN-based approaches toward diffusion-based layout conditioning.
- Layout-to-image generation evolved from layouts used as complementary features or intermediate representations to direct generation from category- and bounding-box-annotated layouts.
- GAN-based methods such as LostGAN and PLGAN improve object control or separate stuff and instance layouts, but the area largely remained GAN-based.
- LDM supports coarse-layout conditioning and represents an early diffusion-based direction for layout-guided image generation, while GAN approaches may suffer unstable convergence and mode collapse.
- Diffusion models have achieved state-of-the-art sample quality across class-conditional, text-to-image, and image-to-image generation tasks.
3. Method
LayoutDiffusion embeds object categories and bounding boxes, models interactions among layout objects, and fuses layout with image features through global and local conditioning. Its structural image patches share unified spatial representations with layout objects, while OaCA incorporates object category, position, and size information.
- Framework: LayoutDiffusion comprises layout embedding, Layout Fusion Module, Image-Layout Fusion Module, and a layout-conditional diffusion model.The framework also includes training and accelerated sampling methods.
- Layout Embedding: Each layout object is represented by a bounding box and category, padded to fixed length, then projected into layout embeddings containing positional and category information.The layout embedding L is the sum of bounding-box embedding BL and category embedding CL.
- Layout Fusion Module: The Layout Fusion Module uses transformer self-attention to model interactions among multiple objects before the layout enters subsequent conditioning stages.It outputs a fused layout embedding L′ that captures information about individual objects and related objects.
- Image-Layout Fusion Module: Structural image patches add region information encoding position and size, transforming image patches into special layout objects in the same spatial space as the input layout.Image-patch bounding boxes use normalized row and column coordinates, enabling unified positional representations for image patches and layout objects.
- Image-Layout Fusion Module: Global conditioning directly adds the fused layout representation to multi-resolution image features, while OaCA performs local conditioning between image patches and layout features.OaCA constructs attention inputs from image and layout features, incorporating category, fused layout, and positional information.
- Image-Layout Fusion Module: OaCA makes cross-attention object-aware and position-sensitive by representing image patches and layout objects as bounding boxes in a unified coordinate space.Its layout keys and values combine category embeddings with fused layout embeddings, while standard attention uses query, key, and value representations.
4. Experiments
Experiments on COCO-Stuff and Visual Genome evaluate LayoutDiffusion across generation quality, diversity, and controllability, including qualitative comparisons, quantitative benchmarks, and ablations. The method achieves stronger results than prior methods, while precise control introduces a diversity trade-off.
- Datasets and metrics: Experiments use COCO-Stuff and Visual Genome to evaluate LayoutDiffusion with metrics covering quality, diversity, and controllability.COCO-Stuff uses 3,097 validation images, while Visual Genome uses 5,096 test images in the reported evaluation setup.
- Qualitative results: LayoutDiffusion produces more recognizable objects in layout-specified positions than Grid2Im, LostGAN-v2, and PLGAN, especially for complex multi-object layouts.The qualitative comparisons report improved generation of trains, zebras, buses, and a laptop placed at the requested location.
- Qualitative results: Images generated from the same layout show high quality and diversity across lighting, textures, colors, and details.The diversity visualization compares multiple outputs conditioned on identical layouts.
- Quantitative results: 46.35% and 29.29% are the maximum improvements over the SOTA model in FID and IS, respectively.The reported metrics also include maximum improvements of 122.22% in YOLOScore, 41.82% in CAS, and 11.30% in DS.
- Quantitative results: LayoutDiffusion achieves better performance than LDM at different cost levels, with the full model reaching an FID of 31.6 at higher cost.Ours-small uses comparable GPU resources to LDM-8, with fewer parameters and better throughput, while outperforming LDM-4 in all respects.
- Ablation studies: Applying both LFM and OaCA yields 13.37 / 6.58 / 39.77 / 27.00 on FID / IS / CAS / YOLOScore, while precise control slightly reduces DS relative to baseline.The ablation study attributes the diversity decrease to the constraints imposed by precise controllability.
5. Limitations & Societal Impacts
The paper acknowledges difficulty generating realistic, distortion-free images for complex multi-object layouts and raises copyright concerns from training on real-world datasets.
- Realistic images without distortion or overlap remain difficult, especially for complex multi-object layouts.
- LayoutDiffusion is trained from scratch on specific datasets requiring detection labels.
- Combining text-guided diffusion with parameters pretrained on massive text-image datasets remains future work.
- Societal Impacts: Training on COCO and Visual Genome raises potential copyright infringement concerns.
6. Conclusion
LayoutDiffusion is a one-stage, end-to-end diffusion model for layout-to-image generation that improves object control and image quality. It uses structural image patches, Layout Fusion Module, and Object-aware Cross Attention, with experiments showing advantages over GAN-based and diffusion-based methods.
- LayoutDiffusion is a one-stage, end-to-end diffusion model designed for layout-to-image generation.
- Structural image patches encode region information and treat each patch as a special object for unified image-layout fusion.
- Layout Fusion Module and Object-aware Cross Attention model relationships among objects and fuse patched image features with layouts at multiple resolutions.
- The method reports higher quality and better controllability than prevailing GAN-based and diffusion-based methods on COCO-stuff and Visual Genome.
- Layout edits can add objects or modify their positions, sizes, and categories.
A.4. More comparision with previous methods
The appendix provides additional visual comparisons across COCO-stuff and Visual Genome at 128×128 and 256×256 resolutions, alongside Gaussian diffusion formulation details. It also describes direct noisy-sample construction and learned reverse-process approximation.
- More comparision with previous methods: Additional comparisons cover COCO-stuff and Visual Genome at 128×128 and 256×256 resolutions.
- More comparision with previous methods: Additional COCO-stuff visualizations use 25-step DPM-Solver sampling, with scale=1.0 at 256×256 and scale=0.6 for comparisons at 128×128.
- More comparision with previous methods: Additional Visual Genome comparisons use 25-step DPM-Solver sampling, with scale=0.5 at 128×128 and scale=1.0 at 256×256.
- Denoising Diffusion Probabilistic Model: Gaussian diffusion adds noise through a forward Markov process and learns pθ(x_{t-1}|x_t) to approximate the reverse process.
- Denoising Diffusion Probabilistic Model: The noisy sample q(x_t|x_0) can be drawn directly as a Gaussian rather than by repeatedly applying every forward step.
- Denoising Diffusion Probabilistic Model: DDPM trains εθ(x_t,t) to predict the added noise using a simplified squared-error objective, then derives μθ from that prediction.
B.2. Classifier-free Method for Layout-conditional Training and Sampling
The classifier-free method trains one diffusion model for both conditional and unconditional sampling by randomly replacing layouts with an empty layout. During sampling, a scale-adjusted combination of scores controls condition strength without classifier gradients.
- A single model supports conditional and unconditional sampling by using a null condition for the unconditional case.
- During training, the condition is randomly replaced with the unconditional identifier with probability p_uncond.
- Classifier-free sampling replaces the conditional score with a linear combination of conditional and unconditional score estimates.
- The scale s changes the effect of the condition, while avoiding classifier-gradient computation during sampling.
- For layout-to-image generation, the condition is the layout l and the unconditional input is the empty layout l_pad.
C. Implementation details
Tables 4 and 5 list the hyperparameters used for LayoutDiffusion and its ablation study, with training performed on eight RTX 3090 GPUs.
- Table 4 reports LayoutDiffusion hyperparameters for the quantitative experiments.
- Table 5 reports the hyperparameters used in the ablation study.
C.2. Analysis of Training Resources and Sampling Speed
Table 6 compares LayoutDiffusion with the SOTA diffusion-based method LDM on COCO-stuff at 256×256 using matched image counts for fairness.
- Table 6 compares LayoutDiffusion and LDM on COCO-stuff at 256×256.
- Both methods generate the same 2048 images for a fair comparison.
- LDM uses DDIM sampling, whereas LayoutDiffusion uses DPM-Solver sampling.
D.1. Datasets
The experiments use filtered COCO-Stuff and Visual Genome datasets, with dataset size depending on the object-filtering protocol.
- COCO-Stuff contains 80 thing and 91 stuff categories, with a standard split of 40K/5k/5k train/validation/test-dev images.
- The main COCO-Stuff filtering retains images with 3 to 8 non-crowd objects covering more than 2% of the image, yielding 25,210 training and 3,097 validation images.
- Alternative COCO-Stuff filtering protocols produce 24,972/3,074 or 74,777/3,097 train/validation images.
- Visual Genome version 1.4 comprises 108,077 densely annotated images, divided into 80%/10%/10% train/validation/test sets.
- The Visual Genome experiments use bounding boxes and retain object and relationship categories occurring at least 2000 and 500 times in training, respectively.
D.2. Evaluation Metrics
The evaluation uses six metrics spanning image-level quality, diversity, object recognizability, and layout control, with metric direction specified for interpretation.
- Six metrics evaluate generated-image quality and layout control from image-level to layout-level perspectives.
- FID measures the Fréchet distance between Gaussian feature distributions of real and generated images, where lower scores indicate smaller differences.
- IS evaluates generated-image clarity and diversity using outputs from an ImageNet-1000-pretrained Inception-V3 network, with higher scores preferred.
- DS measures perceptual diversity between two images generated from the same layout using LPIPS, with higher scores indicating greater diversity.
- YOLO Score evaluates generated-object alignment and fidelity through bbox mAP on 80 thing categories using a pretrained YOLOv4 detector.
- CAS measures classification accuracy in ground-truth layout areas, where higher scores indicate better layout control.