PaperlayerView exact revision on arXiv

Source-linked AI summary

LACONIC: A 3D Layout Adapter for Controllable Image Creation

Léopold Maillard, Tom Durand, Adrien Ramanana Rahary, Maks Ovsjanikov

arXiv:2507.03257v2cs.CV

TL;DR

Existing text- and image-based controls do not inherently preserve consistent 3D scene structure. LACONIC conditions a pretrained text-to-image model on semantic 3D layouts and camera views; without text prompting, it reports state-of-the-art results across all metrics, with narrower advantages under text prompting.

  • Problem

    Text- and image-based controls do not inherently preserve consistent, unambiguous 3D scene structure, including off-screen context.

    Source: §1 · p. 2

  • Method

    LACONIC combines semantic 3D bounding-box layouts, camera control, an adapter, and supervised training to guide pretrained text-to-image diffusion models.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 5

  • Results

    Without text prompting, LACONIC reports state-of-the-art results across all metrics and significantly outperforms SceneCraft; with text prompting, it beats SceneCraft on most metrics.

    Source: §4.1 · p. 8

  • Takeaways & Limitations

    The method supports camera control, consistent 3D-aware synthesis, and object-level scene editing and restyling.

    Source: §1 · p. 2, §4.2 · p. 8, §5 · p. 8

    Outputs remain bounded by the training data distribution, so a model trained on bedroom layouts is unlikely to produce plausible kitchens with fine-grained control.

    Source: §5 · p. 8

Abstract

from arXiv · show

Existing generative approaches for guided image synthesis of multi-object scenes typically rely on 2D controls in the image or text space. As a result, these methods struggle to maintain and respect consistent three-dimensional geometric structure, underlying the scene. In this paper, we propose a novel conditioning approach, training method and adapter network that can be plugged into pretrained text-to-image diffusion models. Our approach provides a way to endow such models with 3D-awareness, while leveraging their rich prior knowledge. Our method supports camera control, conditioning on explicit 3D geometries and, for the first time, accounts for the entire context of a scene, i.e., both on and off-screen items, to synthesize plausible and semantically rich images. Despite its multi-modal nature, our model is lightweight, requires a reasonable number of data for supervised learning and shows remarkable generalization power. We also introduce methods for intuitive and consistent image editing and restyling, e.g., by positioning, rotating or resizing individual objects in a scene. Our method integrates well within various image creation workflows and enables a richer set of applications compared to previous approaches.

1. Introduction

Existing text- and image-based controls do not inherently preserve consistent, unambiguous 3D structure, while 3D-aware methods face data and conditioning challenges. LACONIC introduces a 3D layout representation, adapter, and training framework for controllable image synthesis and evaluates its effectiveness and object-level conditioning.

1. Introduction

  • Text- and image-based controls do not inherently preserve consistent, unambiguous 3D structure, including viewpoint, orientation, occlusion, and off-screen context.The introduction identifies nested object arrangements and scene-wide stylistic and functional coherence as further requirements.Source: §1 · p. 2
  • Accurate 3D semantic annotations are limited, and encoding 3D inputs as controls for pretrained diffusion models presents additional technical challenges.Source: §1 · p. 2
  • LACONIC injects explicit 3D geometry through a semantic bounding-box representation, an adapter compatible with existing conditioning, and training for camera and object-level control.The representation supports consistent structure across views without depth estimators, multi-view images, or panoramas.Source: §1 · p. 2
  • The paper evaluates the method across varied settings and introduces an evaluation methodology for adherence to object-level conditioning.Source: §1 · p. 2

2. Related Work

Prior work addresses 3D-aware generation through multi-view or multi-frame data, depth maps, 2D layouts, and diffusion adapters, each with stated constraints. LACONIC combines explicit 3D scene structure with semantic control in a unified framework.

2. Related Work

  • Existing 3D-aware methods use multi-view or multi-frame datasets, depth maps, or costly score distillation, which can limit acquisition, flexibility, or applicability.Source: §2 · pp. 2–3
  • 2D conditioning can struggle with complex or nested arrangements and unseen scene context, while projected maps may omit occluded items or restrict object types.Figure 2 contrasts text, 2D boxes, projected semantic and depth maps, and LACONIC’s view-consistent 3D layout.Source: §2 · p. 3, Figure 2 · p. 3
  • Diffusion adapters support pretrained-model control, but LACONIC jointly learns a 3D semantic layout encoder and attention-based adaptation module.Source: §2 · p. 3
  • LACONIC aims to provide comprehensive control of object semantics and underlying 3D geometry in one framework.Source: §2 · p. 3

3. Method

LACONIC conditions a pretrained text-to-image diffusion model on semantic 3D layouts and camera views, using learned scene representations and decoupled cross-attention. Its training and application mechanisms support camera control, multiple consistent views, text-driven restyling, and object-level edits.

3. Method

  • The framework targets pretrained text-to-image models using an explicit, view-independent 3D representation, with feature extractors and adapters learned jointly.Source: §3 · p. 3

3.1. Preliminaries

  • A pretrained diffusion denoiser is trained with additional conditioning while its backbone remains frozen, incorporating layout guidance into the denoising objective.The method builds on the text-to-image diffusion training objective and trains additional components for conditioning.Source: §3 · p. 3, §3.1 · p. 4

3.2. Semantic Layout Representation

  • Each scene is an unordered set of objects represented by semantic 3D bounding boxes; an optional floor plan supplements object geometry, and camera intrinsics are assumed consistent.Each box includes center position, dimensions, rotation, and a natural-language semantic description.Source: §3.2 · p. 4

3.3. Adapter Architecture

  • The encoder embeds object geometry and captions, optionally encodes the floor plan, and uses a transformer to produce scene tokens for decoupled cross-attention with image features.A scalar γ controls scene-layout conditioning strength relative to the global caption.Source: §3.3 · p. 5

3.4. Supervised Training

  • Training reframes object positions and rotations into the target camera coordinate system, then conditions denoising on that scene; layout inputs are randomly dropped and objects use free-form captions.Training pairs scenes with rendered images and camera poses; caption supervision may come from dataset annotations or vision-language models.Source: §3.4 · p. 5

3.5. Application Scenarios

  • The same 3D scene can be rendered from multiple target viewpoints by transforming its conditioning input for each camera, supporting structurally consistent views.Source: §3.5 · p. 6
  • Users can restyle scenes with global text prompts and iteratively add, remove, move, resize, or recaption objects while retaining 3D structure.Object attributes can be independently adjusted; Figure 4 also reports that global-prompt concepts appear on relevant objects and that removing a window affects global illumination.Source: §3.5 · p. 6, Figure 4 · p. 6

4. Experiments

The experiments evaluate LACONIC against layout-guided synthesis baselines using image-quality and object-level alignment measures. Results report advantages over baselines, alongside qualitative demonstrations of editing and out-of-distribution generalization.

4.1. 3D Layout-Guided Image Synthesis

  • The evaluation uses HyperSim layouts and renderings, compares against SceneCraft and a layout-conditioned diffusion model trained from scratch, and reports image-quality, diversity, realism, and text-alignment metrics.HyperSim has 326 unique 3D layouts paired with 24,383 images; the authors add 72,000 bedroom layouts paired with one image and viewpoint each for qualitative results.Source: §4.1 · pp. 6–7
  • The new Scene Object CLIP score measures semantic alignment between individual layout objects and their locations in generated images.It projects 3D boxes using camera parameters, filters for sufficient visibility, and scores crops of the selected objects with CLIP; obstructed and out-of-bounds objects are excluded.Source: §4.1 · pp. 7–8, Figure 7 · p. 7
  • Without text prompting, LACONIC achieves state-of-the-art results across all metrics and outperforms SceneCraft; with text prompting, it beats SceneCraft on most metrics and DM-FS on all metrics.The adapter-based approach also produces more natural images with fewer artifacts and greater fidelity to the 3D layout and viewpoint.Source: Figure 5 · p. 7, §4.1 · p. 8, Table 1 · p. 8

4.2. Additional Qualitative Results

  • Additional qualitative results demonstrate iterative object editing, out-of-distribution generalization, 3D-space manipulations, and scene modeling that accounts for off-screen objects.Even at low adapter strength, the generated images are reported to be more geometrically and semantically sound than base text-to-image outputs while retaining textual adherence.Source: §4.2 · p. 8

5. Conclusion, Limitations & Future Work

The conclusion presents LACONIC as a 3D layout-conditional framework for image generation and editing. Its output domain remains bounded by the training data distribution, and the authors identify continuous scene representations and cross-view consistency as future directions.

5. Conclusion, Limitations & Future Work

  • LACONIC maps 3D representations into the image domain and integrates them with a pretrained text-to-image model for fine-grained structural and semantic control.The authors describe the framework as supporting image generation and editing.Source: §5 · p. 8
  • The model's output domain remains bounded by its training distribution, so bedroom-layout training is unlikely to yield plausible kitchens with fine-grained control.The authors note generalization to unseen concepts, while retaining this domain boundary.Source: §5 · p. 8
  • Future directions include deriving continuous 3D scene representations that follow user-defined structures and improving geometric and visual consistency across camera views.The authors cite NeRFs as an example of continuous representations.Source: §5 · p. 8

Supplementary Material

The supplement details LACONIC’s pretrained backbone integration, 3D layout encoding, and HyperSim data setup. It also provides additional synthesis examples across prompt settings.

Supplementary Material

  • LACONIC generates images from 3D layouts and prompts across no-prompt, in-domain, and out-of-distribution settings.The supplement presents these as additional examples of adherence to both layout and text conditions.Source: Figure 8 · p. 13

A.1.1. Pretrained Models

  • The implementation uses Stable Diffusion v1.5 with a pretrained UNet denoiser and CLIP text encoder for global captions and object descriptions.The pretrained models and implementation come from HuggingFace Diffusers.Source: §A.1.1 · p. 13

A.1.2. 3D Layout Encoder

  • Each object token concatenates 192-dimensional position, rotation, dimension, and semantic embeddings into a 768-dimensional representation.Position and dimension are each encoded to 192 dimensions; rotation is represented continuously, encoded, and projected to 192 dimensions, while semantic descriptions are projected by an MLP.Source: §A.1.2 · p. 13
  • The encoder represents an optional floor plan by sampling 100 points along room boundaries and projecting a PointNet output into one floor token.The point cloud is three-dimensional, and the PointNet output has 1024 dimensions before projection.Source: §A.1.2 · p. 13
  • A four-layer Transformer processes the unordered object and floor-token sequence, with zero-padding for variable scene sizes and a 50-object maximum.The encoder uses six attention heads and does not add token positional encodings.Source: §A.1.2 · p. 14

A.1.3. Cross-Attention Layers

  • The layout encoder and attention-based adaptation module are jointly learned, while the layout sequence supplies keys and values to decoupled cross-attention in Stable Diffusion UNet residual blocks.The adapter’s key and value projections are learned dense layers.Source: §A.1.3 · p. 14

A.2.1. HyperSim Dataset

  • The filtered HyperSim subset contains 323 scenes and 24,383 captioned images with NYU40-typed 3D boxes; its scenes average 121 objects, with a median of 54.The original dataset has 461 scenes and 77,400 rendered images; the subset discards unbounded scenes and those with excessively large scales.Source: §A.2.1 · p. 14
  • HyperSim conditioning prioritizes visible objects when scenes exceed the 50-object limit, then uses box dimensions if visible objects still exceed it; floor-plan input is omitted.Object descriptions use the NYU40 category labels because object-level semantic information is limited to those labels.Source: §A.2.1 · p. 14

A.2.2. Custom Bedroom Dataset

The authors create a 72,000-layout bedroom dataset to support object-level learning beyond the limited HyperSim layouts. Each layout has a rendering, floor information, and object descriptions.

A.2.2. Custom Bedroom Dataset

  • The custom dataset contains 72,000 human-designed 3D indoor bedroom layouts, each paired with one rendering.The authors motivate it by noting that limited HyperSim layouts may let the model learn global scene and camera mappings while overlooking individual objects.Source: §A.2.2 · p. 14
  • Floor information is represented as a 3D point cloud for each layout.Source: §A.2.2 · p. 14
  • Object-level semantic descriptions are extracted from the renderings with LLaVA using an instruction to describe each object without analyzing its background.Source: §A.2.2 · p. 14

A.3. Training Protocol

The training protocol optimizes the adapter while freezing the pretrained denoiser, and the appendix specifies inference settings and comparison baselines. A from-scratch baseline is motivated by limited annotated-image data and implemented with a smaller, text-free model.

A.3. Training Protocol

  • The adapter is trained with a DDPM epsilon-prediction objective while the text-to-image denoiser remains frozen.Layout conditioning is randomly dropped at rate 0.15; training runs for 250 epochs, and mixed-precision training was empirically unstable and inconsistently convergent.Source: §A.3 · p. 14

A.4. Inference Settings

  • Quantitative evaluation uses DPM-Solver with 30 denoising steps and guidance scale 5.0, while qualitative images use DDIM with 50 steps.Source: §A.4 · p. 15

A.5.1. SceneCraft

  • SceneCraft renders 3D boxes from the target camera into semantic and depth maps, then trains ControlNet modules on those conditions.The evaluated checkpoint uses Stable Diffusion 2.1 and was trained on the same HyperSim subset as LACONIC.Source: §A.5.1 · p. 15

A.5.2. Diffusion Model from Scratch

  • The from-scratch baseline tests the adapter rationale that limited 3D-annotated image data makes photorealism difficult, especially for uncommon camera views, while pretrained priors may generalize across domains.Source: §A.5.2 · p. 15
  • The from-scratch baseline jointly trains a randomly initialized UNet and layout encoder, omits text conditioning, and uses a downsized UNet.Its layout sequence is cross-attention-conditioned; the UNet has four downsampling and upsampling residual blocks, with two cross-attention bottleneck blocks.Source: §A.5.2 · p. 15

A.6. Evaluation Metric

The appendix describes the Scene Object CLIP score for assessing object-level conditioning and a joint-attention adaptation for Stable Diffusion 3. The DiT examples show layout and viewpoint adherence with this backbone.

A.6. Evaluation Metric

  • The Scene Object CLIP score assesses whether generated objects match their conditioned positions, sizes, and semantic attributes.Source: §A.6 · p. 15
  • The score identifies visible objects using layout, camera pose, and ground-truth 2D boxes, then compares cropped generated objects with their semantic descriptions using CLIP.Projected 3D boxes handle objects partly outside image bounds; the evaluation filters crops by visibility, size, and descriptive NYU40 labels.Source: §A.6 · pp. 15–16

B. Generalization to DiT Backbones

B.1. Background on Joint-Attention

  • Stable Diffusion 3 jointly attends over image and caption tokens, concatenating their projected queries, keys, and values to support bidirectional information flow.Source: §B.1 · p. 16

B.2. Layout Conditioning

  • Layout conditioning adds a joint-attention pass using queries from text conditioning and new key-value pairs from image states and encoded layout features.The resulting image and layout states are processed and merged with primary-pass outputs using the paper’s linear combination.Source: §B.2 · p. 16

B.3. Qualitative Results

LACONIC’s adapted Stable Diffusion 3 model follows the specified 3D layout and viewpoint, rendering scenes in accordance with objects’ spatial and semantic features. The results also show multimodal conditioning and object-level editing capabilities.

B.3. Qualitative Results

  • The Stable Diffusion 3 DiT adapter takes the input 3D layout and viewpoint into account and renders objects according to their spatial and semantic features.Figure 10 presents the synthesized images.Source: §B.3 · pp. 16–17
  • The Build-A-Scene comparison uses a common 3D layout and viewpoint, with cozy-bedroom and Van-Gogh-style prompts as generation conditions.Build-A-Scene also uses an individual prompt for each bounding box.Source: Figure 11 · p. 17
  • A global text caption can condition generation alongside the layout, with its influence adjustable according to Equation (6).Source: §B.3 · p. 17

C.1. Ablation Study

The ablation identifies camera-coordinate reframing as a key design choice, while the perceptual study favors LACONIC over SceneCraft for realism and layout adherence. Qualitative comparisons also report stronger layout fidelity and advantages in complex scenes.

C.1. Ablation Study

  • The ablation identifies transforming the input layout into camera coordinates as a key contributor to performance.The ablation evaluates camera-coordinate reframing and the adapter’s transformer encoder; Table 2 reports metrics on the custom bedroom dataset.Source: §C.1 · p. 17, Table 2 · p. 17

C.2. Baseline Comparison

  • Against Build-A-Scene on the same bedroom layout, LACONIC more accurately reflects the 3D structure, showing all and only the specified objects.Build-A-Scene is a training-free baseline that iteratively adds objects to a background using a depth-conditioned generator and manipulated attention maps.Source: §C.2 · p. 17

C.3. Perceptual Study

  • The perceptual study used a randomized two-alternative forced-choice test, asking participants to judge realism and adherence to the layout and viewpoint.The SceneCraft baseline was trained on the custom bedroom dataset using depth and semantic maps rendered from layouts at the target camera view.Source: §C.3 · p. 17, Figure 12 · p. 18
  • In 638 preference votes from 15 participants across 319 image pairs, LACONIC was favored for realism in 71.2% of comparisons and achieved an 89.0% preference rate for adherence to the input 3D layout.Source: §C.3 · p. 17

C.4. Qualitative Results

  • Qualitative comparisons report out-of-distribution generalization, prompt adherence, and greater realism, detail, and layout fidelity, especially in complex scenes with many objects.The text attributes the complex-scene advantage to combining the 3D layout encoder’s expressiveness with the pretrained text-to-image backbone’s detailed visual features.Source: §C.4 · p. 18

D. Limitations

LACONIC can produce missing objects, visual artifacts, and distorted perspectives; the authors suggest inconsistent camera-intrinsics assumptions may contribute to perspective failures. The societal-impact discussion notes potential benefits alongside misuse, displacement, and bias concerns.

D. Limitations

  • Generated images can omit objects, contain visual artifacts, or distort perspective; the authors associate unnatural perspective with an assumption of consistent camera intrinsics across samples.Examples include missing windows, inaccurate complex floor shapes, and inconsistent wall patterns.Source: D. Limitations · p. 18, Figure 13 · p. 18

E. Societal Impact

  • The authors cite accessible 3D creation and lower training costs as potential benefits, while warning of deceptive misuse, job displacement, and stereotype reinforcement.They note that the approach does not natively incorporate recent generative-model watermarking advances.Source: E. Societal Impact · p. 18
Loading 2507.03257v2…