PaperlayerView exact revision on arXiv

Source-linked AI summary

Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation

Xiaoming Zhu, Xu Huang, Qinghongbing Xie, Zhi Deng, Junsheng Yu, Yirui Guan, Zhongyuan Liu, Lin Zhu, Qijun Zhao, Ligang Liu, Long Zeng

arXiv:2510.15564v1cs.CV

TL;DR

Generating coherent, visually appealing layouts from predefined assets remains challenging, with manual rules constraining diversity. Imaginarium fine-tunes image generation for asset-aligned guidance and parses images to retrieve and optimize 3D assets; the authors report improved layout quality over previous methods.

  • Problem

    Generating logically coherent and visually appealing layouts from predefined assets is challenging, while manual priors require artistic expertise and may limit diverse scene combinations.

    Source: §1 · p. 2

  • Method

    The system fine-tunes Flux on curated scene data to create image guidance, then parses it and uses semantic matching, pose estimation, and scene-graph-based optimization to build 3D layouts.

    Source: §1 · p. 2

  • Results

    The authors report significant layout-quality improvements over previous methods, and user studies with 100 art students and 20 professional artists show significant performance advantages over current SoTA methods.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The system generates coherent, aesthetic scenes from text or Canny images within 240 seconds, compared with the 2.5-hour time typical in professional workflows.

    Source: §6 · p. 12

    Fine-tuning for consistency across multiple objects in complex scenes remains challenging, and single-image pose estimation is difficult under severe occlusion.

    Source: §6 · p. 12

Abstract

from arXiv · show

Generating artistic and coherent 3D scene layouts is crucial in digital content creation. Traditional optimization-based methods are often constrained by cumbersome manual rules, while deep generative models face challenges in producing content with richness and diversity. Furthermore, approaches that utilize large language models frequently lack robustness and fail to accurately capture complex spatial relationships. To address these challenges, this paper presents a novel vision-guided 3D layout generation system. We first construct a high-quality asset library containing 2,037 scene assets and 147 3D scene layouts. Subsequently, we employ an image generation model to expand prompt representations into images, fine-tuning it to align with our asset library. We then develop a robust image parsing module to recover the 3D layout of scenes based on visual semantics and geometric information. Finally, we optimize the scene layout using scene graphs and overall visual semantics to ensure logical coherence and alignment with the images. Extensive user testing demonstrates that our algorithm significantly outperforms existing methods in terms of layout richness and quality. The code and dataset will be available at https://github.com/HiHiAllen/Imaginarium.

1 Introduction

Imaginarium targets the difficulty of generating coherent, visually appealing layouts from predefined assets, where manual rules constrain diversity and existing generative approaches have other limitations. It uses image guidance, a curated asset collection, and visual-semantic and geometric analysis to generate and refine 3D layouts.

1 Introduction

  • Manual scene priors take time and artistic expertise to define and may limit diverse combinations, while learned generators face limited training data and LLM-based methods struggle with spatial and geometric precision.The cited LLM-based methods also struggle to represent complex spatial relationships, model object poses, and follow aesthetic principles.Source: §1 · p. 2
  • The authors curated 2,037 indoor and outdoor assets and had professional artists create 147 high-quality scene layouts for an intended open-source dataset.The collection is intended to address asset-library limitations including poor mesh quality, limited stylization, and reliance on composite assets.Source: §1 · p. 2
  • The system transfers image models’ controllable 2D generative capabilities to 3D layout generation.The image-generation model is fine-tuned to produce guidance more consistent with the asset collection.Source: §1 · p. 2
  • The pipeline fine-tunes Flux on scene-layout data to create a guide image, parses its semantics and geometry, retrieves matching assets, estimates their transformations, and optimizes the layout.Scene-graph logic and image-semantic parsing are used in final consistency optimization.Source: §1 · p. 2
  • The approach improves layout quality over previous methods by varying object poses and placements and arranging assets within other assets.The paper attributes more natural, detailed, and visually appealing layouts to these capabilities.Source: §1 · p. 2

2 Related Work

Related work covers rule-based and learned layout generation, language-driven scene synthesis, and novel-object pose estimation. The paper positions its approach as combining image guidance with continuous pose estimation and geometric and scene-logic constraints.

2.1 Data-Driven Scene Layout Generation

  • Manual scene priors encode design guidelines and object distributions but require time-intensive design and constrain model expressiveness.Learned generators include autoregressive models, GANs, VAEs, and diffusion approaches, but remain challenged by diversity, stability, realism, and scarce 3D data [Nie et al. 2023; Paschalidou et al. 2021b; Wang et al. 2018, 2021; Yang et al. 2021b; Purkait et al. 2020; Yang et al. 2021c,a; Xiao et al. 2021; Dahnert et al. 2024; Tang et al. 2024].Source: §2.1 · p. 3

2.2 Language-Driven Scene Layout Generation

  • LLM-based scene synthesis uses predicted object properties, formatted layouts, multi-agent collaboration, or generated scripts, while multimodal methods add vision and optimization [Yang et al. 2024b; Feng et al. 2024; Çelen et al. 2024; Hu et al. 2024; Huang et al. 2025; Deng et al. 2025; Wang et al. 2024a; Sun et al. 2024].The paper identifies instability, complexity and aesthetic limits, fixed orientations, hard relational constraints, and mismatches between generated imagery and available assets; it contrasts these with style-consistent guidance and continuous pose estimation.Source: §2.2 · p. 3

2.3 Pose Estimation of Novel Objects

  • Novel-object pose methods range from geometric hashing and CAD alignment to 2D template matching and large-dataset optimization, but CAD-library dependence limits open-set applicability [Drost et al. 2010; Kuo et al. 2020; Gümeli et al. 2022; Langer et al. 2022; Gao et al. 2024; Nguyen et al. 2022; Thalhammer et al. 2023; Labbé et al. 2022; Örnek et al. 2024; Nguyen et al. 2024a].For category-based rotation estimation, the method uses GigaPose’s fine-tuned DINOv2 with geometric constraints and scene logic; the overview traces image analysis through asset retrieval and transformation estimation to layout optimization.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 Method

The system turns a user prompt and predefined 3D assets into a scene layout through image guidance, scene analysis, asset retrieval, transformation estimation, and refinement. Its method combines a purpose-built scene dataset, visual and geometric cues, scene-graph constraints, and physical optimization.

3 Method

  • The system maps a textual prompt and predefined asset set to a 3D layout, using a guided image, scene analysis, asset retrieval, transformation estimation, and final refinement.The output represents each object as an asset with rotation, translation, and scale.Source: §3 · p. 4

3.1 Prompt Expander under Predefined Assets

  • Fine-tuned Flux expands prompts into images aligned with the asset collection’s style, supporting later retrieval and transformation estimation.Training uses rendered scenes and a unique tag; the generated images retain layout flexibility while learning global patterns and moderate object-level features.Source: §3.1 · p. 4
  • The dataset contains 2,037 models across 500 classes and 237 categories, used in 147 expert-designed layouts across 20 types.Compared with 3D-Future, it has 500 versus 34 asset classes and 31.86 versus 5.09 objects per scene.Source: §3.1 · p. 4

3.2 Scene Image Analysis

  • Scene analysis parses foreground objects and geometry, detects walls, floors, and ceilings, and constructs support and wall-proximity relationships.The method estimates depth and fits oriented bounding boxes, then builds support trees and refines floor-supported object boxes using scene-graph relations.Source: §3.2 · p. 5

3.3.1 3D Asset Retrieval.

  • For each masked image region, the system retrieves an asset using inverse category mapping, visual-feature similarity, and size compatibility.The retrieved assets form the basis for the coarse reconstructed layout.Source: §3.3.1 · p. 6, §3.3 · p. 6

3.3.2 Transformations Estimation Module.

  • Transformation estimation combines visual-semantic matching with geometric cues to select rotations, approximate translations from box centers, and scale assets by object type.It ranks 162 rendered viewpoints, uses homography to suppress symmetry-related correspondence errors, and refines rotation candidates with depth-derived geometry, particularly for cuboid-like assets.Source: §3.3.2 · p. 6

3.4.1 Local Transformation Refinement based on Scene Graph.

  • Scene-graph relationships refine rotation and scale by aligning objects with supports, walls, or ceilings and adjusting contained objects to container capacity.The support tree enables recursive rotation adjustment, while internal subspaces guide resizing of contained objects.Source: §3.4 · p. 7, §3.4.1 · p. 7

3.4.2 Global Post-optimization for Translation Transformations.

  • A constrained translation optimization preserves visual alignment while enforcing non-intersection, support, ceiling, and wall constraints; physics simulation further promotes realistic object behavior.The optimization preprocesses support and wall constraints before simulated annealing with voxel-based intersection calculations.Source: §3.4.2 · p. 7, §3.4.3 · p. 7

4 Experiments

The experiments evaluate layout quality, reconstruction fidelity, rotation estimation, and the effects of system components. User studies favor the generated layouts, while ablations examine asset alignment, pose estimation, and scene refinement.

4 Experiments

  • The evaluation combines user studies and experiments on quality, single-image rotation estimation, and ablations; Flux fine-tuning used 147 scenes rendered at 1024×1024 pixels.Fine-tuning ran for 15 epochs on two A100 GPUs using LoRA; the full system takes approximately 240 seconds on one A100.Source: §4 · p. 7, §4.1 · p. 7

4.2.1 Evaluation by Senior Art Students.

  • Against DiffuScene, HOLODECK, LayoutGPT, and InstructScene, our method achieves preference rates of 79.22%, 77.71%, 76.60%, and 65.36% for reasonableness and realism, and 80.38%, 76.73%, 79.33%, and 72.49% for aesthetic quality.Source: §4.2.1 · p. 8

4.2.2 Evaluation by Professional Artists on Richness.

  • Our method consistently outperforms all baseline approaches, scoring 3.34 from human artists and 3.06 from GPT-4o, indicating performance on par with or slightly better than professional standards.Source: §4.2.2 · p. 8

4.2.3 Fidelity and Similarity of 3D Scene Layout Reconstruction.

  • 92.31% primary-object recovery, 95.83% category preservation, and 93.26% scene-graph accuracy accompany 74.83% rotation AUC@60° and 84.32% translation AUC@0.5m.Secondary-object recovery is 70.41%; the paper attributes its lower rate to resolution limits and detection constraints on smaller objects.Source: Table 3 · p. 9, §4.2.3 · p. 9

4.3 Comparison of Rotation Transformation Estimation

  • 70.06% category-level and 81.44% instance-level AUC@60° surpass all benchmarks; category-level mAP is 50.5%, 65.5%, and 80.5% at 5°, 15°, and 45° thresholds.The paper relates stronger instance-level results to greater query-template similarity and reports Orient Anything at 56.24% AUC among non-template approaches.Source: Table 4 · p. 9, §4.3 · p. 9

4.4.1 Ablation study of finetuned Flux.

  • Flux fine-tuning improves asset-library alignment, retrieval accuracy, and pose estimation while maintaining comparable visual and layout diversity to vanilla Flux.The reported pose-estimation AUC@60° rises from 70.06% to 81.44% when query objects match CAD models; fine-tuning exhibits minimal overfitting and preserves layout diversity.Source: §4.4.1 · pp. 9–10, Figure 10 · p. 10

4.4.2 Ablation study of rotation transformation estimation.

  • Homography improves rotation estimation, reaching 76.07% mAP@45° and 59.42% mAP@15°; adding geometry raises mAP@5° from 5.23% to 36.22%.The complete model combines AENet, homography, and geometry and achieves the best performance across all reported metrics.Source: §4.4.2 · pp. 10–11

4.4.3 Ablation study of scene layout refinement.

  • Global optimization is the most critical refinement step for fixing support relationships and reducing object interference while maintaining logical plausibility.The study assesses local refinement, global optimization, and physical constraints using support correctness, intersection-pair counts, and GPT-4o quality scores.Source: §4.4.3 · p. 11, Table 8 · p. 11

5 Application

The paper presents image-based re-editing as a way to control generated 3D layouts. Users can repaint selected regions, then re-retrieve and re-estimate poses for objects in those regions while keeping objects outside them fixed.

5 Application

  • Re-editing supports global scene completion, object replacement, and local object addition through masked-region repainting and subsequent object retrieval and pose estimation.After repainting, objects outside the selected mask are fixed while the algorithm re-retrieves and estimates poses for objects within it.Source: §5 · p. 11, Figure 12 · p. 11

6 Conclusion and Discussion

The system generates coherent, aesthetic 3D layouts from text or Canny images in 240 seconds, and user studies found significant advantages over current state-of-the-art methods. The authors identify cross-object image consistency and pose estimation under severe occlusion as ongoing challenges.

6 Conclusion and Discussion

  • 240 seconds per layout, versus the 2.5 hours typical in professional workflows; user studies with 100 art students and 20 professional artists found significant advantages over current state-of-the-art methods.The reported system accepts text or Canny images as input.Source: §6 · p. 12
  • Fine-tuning the image generation model to maintain consistency across multiple objects in complex scenes remains a primary challenge.The authors also note accurate pose estimation from single images is challenging, particularly under severe occlusions.Source: §6 · p. 12
  • The authors propose multi-view perspective information as a path toward more robust scene analysis addressing pose ambiguity.They also identify automated 3D data generation from 2D placement knowledge as a potential application.Source: §6 · p. 12

A.1 Technical Implementation Details

Scene graph construction extracts geometric logical relationships from foreground regions. The process combines qualitative image analysis using multimodal models with precise geometric methods.

A.1 Technical Implementation Details

  • The scene graph construction extracts geometric logical relationships from foreground regions in the scene image.The foreground regions are denoted S_fg = {m_i}.Source: §A.1.1 · p. 15
  • The method combines qualitative image analysis using multimodal models with precise geometric methods to handle complex shapes and occlusions.Source: §A.1.1 · p. 15
  • The construction process is organized into three steps.The passage introduces the steps but does not describe them individually.Source: §A.1.1 · p. 15

A.1.1 Scene Graph Construction.

The scene graph combines support relationships and wall contacts with geometric checks, while excluding regions without clear logical relationships. Object bounding boxes are also corrected using scene graph relationships to improve spatial accuracy.

A.1.1 Scene Graph Construction.

  • 91.95% accuracy was achieved in floor-support analysis, which uses GPT-4o to identify floor-supported objects and recursive geometric search to build a support tree.The tree retains vertical relative distances for later asset placement.Source: §A.1.1 · p. 15
  • GPT-4o prompting identifies ceiling-supported objects, which the authors report typically exhibit singular logical relationships in their experiments.Source: §A.1.1 · p. 15
  • For wall relationships, GPT-4o identifies objects contacting walls, then distances from their oriented bounding boxes to structural planes determine the wall-specific object sets.Source: §A.1.1 · p. 15
  • Regions without clear logical relationships are excluded to improve controllability of the scene layout.Source: §A.1.1 · p. 15
  • Scene graph relationships correct depth-derived bounding boxes that become inaccurate under occlusion, using floor geometry to restore perpendicularity and contact for floor-supported objects.The authors report that this correction significantly improves spatial accuracy in the final layout.Source: §A.1.1 · p. 15

A.1.2 3D Asset Retrieval.

Asset retrieval first maps an image-mask category to relevant library categories, then selects an asset using visual similarity and size compatibility. The matching strategy compares masks with rendered asset images and adjusts similarity for absolute size differences.

A.1.2 3D Asset Retrieval.

  • For each mask region, the system uses the inverse category merge map to identify relevant asset categories before matching an asset.Source: §A.1.2 · pp. 15–16
  • Asset matching uses semantic visual-feature similarity between a mask and rendered asset images, adjusted by absolute size difference to address cases such as substantial occlusion.The cited example is a bedside table obstructed by a bed.Source: §A.1.2 · p. 16
  • The matching score combines rendered-view feature similarity with a dimension-difference penalty, using 20 sampled viewpoints and α = 0.1.The passage defines the rendered-view features as those extracted by the original DINOv2 model and the dimension estimates as GPT-4o outputs.Source: §A.1.2 · p. 16

A.1.3 Scale Transformation.

Scale transformations are optimized using object OBBs, with artist-informed choices that balance scene layout needs and each asset’s design characteristics.

A.1.3 Scale Transformation.

  • OBB-based scale optimization follows professional artists’ layout practices while balancing scene fit against preserving assets’ design traits.The scale variable changes the retrieved asset’s OBB, and the optimization is informed by three asset-scaling scenarios.Source: §A.1.3 · p. 16
  • Height-adjustable scaling preserves the original length-to-width ratio and suits decorative objects and lighting fixtures.This mode has two degrees of freedom.Source: §A.1.3 · p. 16
  • Slender objects use scaling along their two longest OBB axes, with the third dimension set proportionally from their average.Examples include picture frames, wooden boards, and curtains.Source: §A.1.3 · p. 16
  • Furniture such as tables, cabinets, and beds can be freely scaled in length, width, and height.This mode has three degrees of freedom.Source: §A.1.3 · p. 16

A.1.4 Global Translation Optimization.

Global translation optimization first enforces support and wall constraints, then uses simulated annealing on the remaining variables with voxel proxies for faster intersection calculations.

A.1.4 Global Translation Optimization.

  • Support and wall constraints are processed first, fixing supported objects’ vertical positions and wall-adjacent objects’ positions along the wall normal.Support heights are updated from the support-tree root; wall-adjacent objects are moved into close contact with the wall.Source: §A.1.4 · p. 16
  • Simulated annealing addresses the remaining highly nonlinear translation optimization after constrained variables are fixed.Supported objects’ z values and wall-normal movement are excluded from subsequent optimization.Source: §A.1.4 · p. 16
  • Voxel representations serve as proxies for polygon meshes to reduce the computational complexity of intersection calculations.Source: §A.1.4 · p. 16

A.1.5 Scene Layout Similarity Function.

This supplementary section presents a grid-based scene-layout similarity measure and documents system prompts, dataset materials, visualizations, qualitative examples, and failure cases. It also includes generated 3D layout examples for living-room and bedroom prompts.

A.1.5 Scene Layout Similarity Function.

  • Layout similarity is the maximum fraction of matching furniture-class cells across padded, aligned floor grids after 90°, 180°, or 270° rotations and mirroring.The method projects objects onto the floor plane, standardizes orientation using a wall direction, and uses 0.1m × 0.1m grid cells; Fig. 13 illustrates layout and visual neighbors.Source: §A.1.5 · pp. 16–17, Figure 13 · p. 18

A.1.7 Additional Visualization of Intermediate Results.

  • The supplementary visualizations show the workflow and scene graph, plus intermediate processing from scene parsing and asset retrieval through object rotation estimation.Fig. 14 describes object detection and segmentation, plane fitting, top-two asset retrieval, and rotation estimation combining visual-semantic matching with OBB and scene-graph constraints.Source: §A.1.7 · p. 17, Figure 14 · p. 19

A.2.2 Prompt for Object Extraction.

  • The object-extraction prompt requests thorough, area-based detection using predefined categories and a three-level JSON hierarchy of areas, parent objects, and child objects.It also directs the model to match objects outside the categories to the closest available category.Source: §A.2.2 · p. 17

A.2.3 Prompt for Scene Layout Analysis; A.2.4 Prompt for GPT4 evalutation

  • The scene-analysis prompt encodes whether each object is against a wall, on the floor, or hanging from the ceiling; expert evaluation rates composition and semantic logic on 1–5 scales.Layout matching is rated on a 1–10 scale.Source: §A.2.3 · p. 19, §A.2.4 · p. 20

A.3 Dataset Details; A.4 Analysis of Failure Cases

  • The dataset offers handcrafted scenes across diverse room functions and asset variants, while the failure analysis identifies asset-library mismatch and severe occlusion as limitations.Novel object topologies absent from the finite library can cause incorrect retrieval and invalidate downstream constraints; occlusion can make pose estimates unreliable and difficult to correct.Source: §A.4 · pp. 20–21

A.5 More Qualitative Results

  • Additional examples demonstrate diverse generated layouts, including scenes with bookshelves and abstract artwork and a casino interior with a poker table and slot machines.The paper presents these as further qualitative examples of the method’s ability to generate diverse 3D scene layouts.Source: §A.5 · p. 21
  • Additional qualitative results include generated layouts for living-room and bedroom prompts.The figure caption identifies these as additional 3D scene layouts generated by the system.Source: Figure 19 · p. 24, §A.5 · p. 24

Related papers

Loading 2510.15564v1…