Source-linked AI summary
Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation
Abdelrahman Eldesokey, Peter Wonka
TL;DR
Existing layout control is limited to static 2D layouts and does not preserve generated images under layout changes, restricting interactive 3D object-wise editing. Build-A-Scene sequentially generates scenes with 3D boxes, uses Dynamic Self-Attention to add objects while preserving existing contents, and applies consistent 3D translation to preserve object identity under layout changes. 2x higher object generation success rate than LooseControl and ∼15% higher than Layout-Guidance; Build-A-Scene also outperforms both in preserving objects under layout changes on all metrics.
Problem
Existing layout control is limited to static 2D layouts and does not preserve generated images under layout changes, restricting interactive 3D object-wise editing.
Source: §1 · p. 2Method
Build-A-Scene sequentially generates scenes with 3D boxes, uses Dynamic Self-Attention to add objects while preserving existing contents, and applies consistent 3D translation to preserve object identity under layout changes.
Source: §1 · pp. 2–3Results
2x higher object generation success rate than LooseControl and ∼15% higher than Layout-Guidance; Build-A-Scene also outperforms both in preserving objects under layout changes on all metrics.
Source: §1 · p. 3Takeaways & Limitations
The approach provides interactive control over individual objects in 3D through a multi-stage generation process.
Source: §6 · p. 10Unsuitable box aspect ratios can produce duplicate or distorted objects, and objects extending beyond scene boundaries can be distorted.
Source: §5 · pp. 9–10
Abstract
from arXiv · showhide
We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control. Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects' placement and relationships from text descriptions. Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes. This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control. We replace the traditional 2D boxes used in layout control with 3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages. We achieve this through our proposed Dynamic Self-Attention (DSA) module and the consistent 3D object translation strategy. Experiments show that our approach can generate complicated scenes based on 3D layouts, boosting the object generation success rate over the standard depth-conditioned T2I methods by 2x. Moreover, it outperforms other methods in comparison in preserving objects under layout changes. Project Page: \url{https://abdo-eldesokey.github.io/build-a-scene/}
1 INTRODUCTION
Text-to-image diffusion models struggle with object count, placement, and relationships, while existing layout controls are static and mostly 2D. Build-A-Scene introduces interactive 3D layouts and reports higher object-generation success and better preservation under layout changes than the compared methods.
1 INTRODUCTION
- Diffusion models struggle to follow prompts accurately, particularly for object count, placement, and relationships.These difficulties are reported across multiple studies.Source: §1 · p. 2
- Existing layout controls use 2D inputs and static layouts, limiting 3D positioning and orientation and failing to preserve images when objects move or scale.Moving a sofa box in Layout-Guidance changes both the sofa and the rest of the scene, such as the door.Source: §1 · p. 2
- LooseControl conditions generation on 3D boxes but is not designed for diverse-object layouts or interactive object movement, and layout changes can alter objects or cause artifacts.It relies on text prompts for image contents, and objects may be omitted or misplaced in multi-object scenes.Source: §1 · p. 2
- Build-A-Scene is an interactive, training-free text-to-image approach that replaces 2D boxes with 3D boxes and generates scenes sequentially.Dynamic Self-Attention adds objects while preserving existing contents, and consistent 3D translation preserves object identity under layout changes.Source: §1 · pp. 2–3
- Build-A-Scene outperforms LooseControl by 2x on object generation success rate and Layout-Guidance by ∼15%, and outperforms both in preserving objects under layout changes on all metrics.It also outperforms LooseControl on adherence to user-provided 3D boxes.Source: §1 · p. 3
2 RELATED WORK
Prior work addresses layout control through 2D annotations and consistent generation through personalization or attention-based preservation. Build-A-Scene differs by targeting interactive 3D layouts and preserving scene contents during object insertion.
2.1 LAYOUT CONTROL IN T2I DIFFUSION MODELS
- Layout control methods use annotations such as points, scribbles, boxes, and masks, incorporated through fine-tuning or training-free methods.Fine-tuned approaches add computational and data-annotation costs.Source: §2.1 · p. 3
- Training-free 2D methods steer cross-attention or use segmentation guidance, but do not provide 3D control or preserve the rest of an image during layout edits.Build-A-Scene is introduced to address these limitations with interactive 3D layout control.Source: §2.1 · p. 3
2.2 CONSISTENT OBJECT GENERATION IN T2I
- Personalized-generation approaches fine-tune models or train adapters using subject images, whereas Build-A-Scene targets layout- and prompt-driven generation with edits that preserve the generated image.Other training-free methods manipulate self-attention to preserve overall style, but not every individual object's details.Source: §2.2 · pp. 3–4
- Build-A-Scene uses Dynamic Self-Attention with latent blending to insert objects while preserving existing scene contents.The pipeline illustrates scene setup, adding a prompted 3D box, and translating the object while preserving its identity.Source: §2.2 · p. 4, Figure 3 · p. 4
3 METHOD
Build-A-Scene turns image generation into an interactive sequence of 3D layout edits, with the goal of preserving existing objects as new ones are added or layouts change. It combines depth-conditioned 3D boxes, Dynamic Self-Attention, and a consistent 3D translation strategy.
3 METHOD
- The method targets interactive 3D layout control and object consistency under layout changes.The user specifies 3D bounding boxes and corresponding prompts for the objects.Source: §3 · p. 4
3.1 FROM 2D TO 3D LAYOUT CONTROL
- Generation begins with a plane-defined background; each later stage adds one prompted 3D box whose type, size, location, or orientation the user can change.The scene is rendered into a depth map and masks for the object being generated at that stage.Source: §3.1 · pp. 4–5
3.2 INTERACTIVE 3D LAYOUT CONTROL
- Dynamic Self-Attention combines prior-stage keys with masked current-stage keys, preserving existing image contents while allowing a new object style.It is plug-and-play with a pretrained diffusion model and does not require fine-tuning or guidance that backpropagates through the model.Source: §3.2 · pp. 5–6
3.3 CONSISTENT 3D TRANSLATION
- For 3D translation, the method segments the existing object, warps it using box-coordinate correspondences, and blends its inverted latents so it can be regenerated at the new location while preserving background.The latent blending is performed for a limited number of denoising steps.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments evaluate 3D layout control and object consistency under layout changes, using quantitative and qualitative comparisons, plus ablations of pipeline components.
4.1 EXPERIMENTAL SETUP
- The evaluation compares against LooseControl and Layout-Guidance; quantitative runs use 20 denoising steps, while qualitative runs use 40.The protocol samples 100 layouts and runs each with five seeds; its metrics assess text alignment, object presence, and fit within layout boxes.Source: §4.1 · pp. 6–7, §4.2 · p. 7
4.2 3D LAYOUT CONTROL RESULTS
- Build-A-Scene scores 2x LC and 15% above Layout-Guidance on OA, beats LC and matches Layout-Guidance on CLIPT2I, and exceeds both on mIOU.The mIOU comparison is reported without numerical values; the authors attribute its improvement to DSA forcing generated objects within their boxes.Source: Table 1 · p. 7, §4.2 · pp. 7–8
4.3 OBJECT CONSISTENCY UNDER LAYOUT CHANGE RESULTS
- For object consistency, Build-A-Scene and LC have comparable CLIPI2I, but Build-A-Scene performs significantly better on SSIM and PSNR; Layout-Guidance performs worst on all metrics.The results are averaged over five seeds.Source: Table 1 · p. 7, §4.3 · p. 9
4.2 3D LAYOUT CONTROL RESULTS
- The 3D layout protocol samples 16 COCO objects and 10 scene prompts, then places objects using randomized left, right, or above relations.The sampled boxes match object aspect ratios, and placements avoid occlusion and out-of-bounds positions.Source: §4.2 · p. 7
- Figure 5 shows greater layout faithfulness than Layout-Guidance, while LooseControl struggles to generate all objects; the examples also show objects integrated across stages.The authors point to reflections and shadows as evidence of seamless integration between stages.Source: §4.2 · p. 8, Figure 5 · p. 8
4.3 OBJECT CONSISTENCY UNDER LAYOUT CHANGE RESULTS
- Figure 6 shows Build-A-Scene preserving objects during moves and zooms, while LC distorts or subtly changes them and Layout-Guidance changes the image or pose.Build-A-Scene also seamlessly inserts the object at its new location during scaling changes.Source: §4.3 · p. 9, Figure 6 · p. 9
4.4 ABLATION STUDY
- Disabling DSA prevents new-object insertion, skipping latent blending changes background details, and AdaIN harmonizes generated-object colors with the background.Source: §4.4 · p. 9
- In the translation-strategy blend sweep, T = 0.8T seamlessly blends the sofa while retaining fine details; T = T introduces boundary artifacts.At T = 0.4T, the sofa is integrated but cushion details remain imperfect.Source: §4.4 · p. 9
5 LIMITATIONS AND FUTURE WORK
The approach is sensitive to box aspect ratio and segmentation success, and boundary-crossing objects can be distorted. The authors also note multi-stage overhead and propose automated layout generation and identity-preserving in-plane rotation as future work.
Limitations
- Box aspect-ratio mismatches can produce duplicate or distorted objects, while objects extending beyond box boundaries can also be distorted.The authors recommend reasonable box definitions to obtain the desired results.Source: §5 · pp. 9–10
- If object segmentation fails, the Consistent 3D Translation strategy cannot preserve objects.Source: §5 · p. 10
- Generation takes 2 seconds per stage, and a 5-object layout takes the same time as Layout-Guidance.The authors acknowledge that the multi-stage pipeline adds computational overhead.Source: §5 · p. 10
Future Work
- Future work includes using an LLM for automated layout generation and supporting in-plane rotations while preserving object identity.Source: §5 · p. 10
6 CONCLUSION
The paper presents an interactive 3D layout-control approach built on a pretrained T2I diffusion model. It reformulates generation as a multi-stage process and reports quantitative and qualitative improvements over LooseControl and recent 2D layout-control methods.
6 CONCLUSION
- Build-A-Scene introduces interactive 3D layout control with multi-stage generation and object preservation under layout changes, outperforming LooseControl and recent 2D layout-control methods.Source: §6 · p. 10