Source-linked AI summary
Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis
Yixuan Yang, Zhen Luo, Wanshui Gan, Jinkun Hao, Junru Lu, Jinghao Yan, Zhaoyang Lyu, Xudong Xu
TL;DR
Top-down image-to-3D room synthesis requires precise spatial alignment, while existing approaches remain limited by handcrafted rules or unstable holistic generation. Code-as-Room addresses this with staged Blender-code synthesis, cross-stage memory, and visual feedback, consistently improving VLM-based generation and human-perceived quality.
Problem
Existing room-generation methods are limited by handcrafted rules or unstable image-conditioned holistic synthesis, despite the need for realistic, functional 3D indoor rooms.
Method
Code-as-Room parses top-down images into spatially related scene elements and synthesizes editable Blender code through staged layout, geometry, material, and lighting refinement with shared memory.
Results
Code-as-Room consistently improves VLM-based image-to-3D room generation and human-perceived quality, with Gemini3.1-Pro achieving the best similarity, usability, and acceptability scores.
Takeaways & Limitations
Structured decomposition, cross-stage memory, and visual feedback support more reliable and spatially faithful code-based 3D room synthesis.
Takeaways & Limitations
The framework targets top-down views and is not yet optimized for arbitrary-view inputs, while procedural code remains insufficient for faithfully generating many real-world objects.
Abstract
from arXiv · showhide
Designing realistic and functional 3D indoor rooms is essential for a wide range of applications, including interior design, virtual reality, gaming, and embodied AI. While recent MLLM-based approaches have shown great potential for 3D room synthesis from textual descriptions or reference images, text-based methods struggle to capture precise spatial information, and existing image-conditioned agents suffer from instability and infinite looping when tasked with holistic room generation from top-down views. To address these limitations, we propose Code-as-Room, an MLLM-based agentic framework equipped with a structured execution harness, which represents 3D rooms with Blender codes. Given a top-down room image, the framework parses the reference image to extract scene elements and their spatial relationships, and synthesizes executable Blender code for geometry, materials, and lighting in a principled, multi-stage pipeline. A cross-stage memory module is maintained throughout to mitigate context forgetting inherent to existing agent-based frameworks. We further introduce a dedicated benchmark for code-based 3D room synthesis, encompassing various evaluation protocols. Based on our benchmark, comprehensive comparisons against existing agent-based methods are conducted to validate the effectiveness of our proposed execution harness.
1 Introduction
Code-as-Room addresses the labor and spatial-reasoning challenges of 3D indoor-room creation by using top-down images as global spatial priors and synthesizing executable Blender code through a structured MLLM agentic framework. The paper also introduces a benchmark targeting visual understanding, spatial reasoning, and vision-to-code generation.
- Motivation: Manual 3D room construction is labor-intensive because it requires expertise in object modeling, spatial arrangement, material design, and lighting adjustment.Traditional procedural, rule-based, and constraint-driven graphics methods have sought to reduce this manual effort.
- Related Approaches: MLLM-based scene-generation approaches represent indoor scenes in JSON or other structured formats to predict spatial information.The introduction motivates leveraging MLLMs for 3D room generation but does not specify the full capabilities of these approaches in the supplied passage.
- Image-Guided Synthesis: Top-down layouts, sketches, and floor-plan-like images provide rich spatial priors and holistic scene appearance for 3D room creation.The passage describes designers iteratively consulting such references while refining rooms and motivates image-conditioned MLLM agents.
- Code-as-Room: Code-as-Room parses a top-down reference image and generates executable Blender code representing major furniture, attached accessories, and interior finishes.The identified finishes include doors, windows, and walls.
- Evaluation: The Image-to-3D Room synthesis benchmark evaluates visual understanding, spatial relationship reasoning, and vision-to-code generation capability.The benchmark also supports comprehensive comparisons between existing MLLMs or agent-based methods and the proposed framework.
- Contributions: The structured execution harness orchestrates an MLLM agent for stable and coherent code-based 3D room synthesis.The contribution is framed as a top-down image-guided paradigm in which the input image guides complete indoor-room generation.
2 Related Work
Prior indoor-scene synthesis spans rule-, constraint-, and optimization-driven methods, while newer LLM- and image-conditioned approaches enable controllable layouts or 3D generation. Code-as-Room instead targets complete indoor rooms from top-down images, representing the full scene as executable Blender code.
- Procedural and Data-driven Indoor Scene Synthesis: Early indoor-scene synthesis formulated generation as rule-based, constraint-based, or optimization-driven problems.Constraint-based systems support semantic and geometric placement, while furniture-layout methods incorporate design guidelines, ergonomic objectives, and spatial priors.
- LLM- and Agent-based 3D Scene Generation: LLM-based methods use commonsense reasoning and open-vocabulary planning to generate room layouts, object selections, spatial relations, or scene graphs from language instructions.Examples include Holodeck, LLplace, LAYOUTVLM, and I-Design.
- Image-conditioned 3D Generation and Code-based 3D Scene Representation: Image-conditioned 3D generation uses diffusion models, neural fields, meshes, and other learned representations, but mainly targets single objects or relatively simple scenes.These outputs are often designed for reconstruction or visual synthesis rather than structured generation of a complete indoor room.
- Image-conditioned 3D Generation and Code-based 3D Scene Representation: Code-as-Room generates a complete indoor room from a top-down image, representing the full scene as executable Blender code.This distinguishes the work from approaches focused primarily on language-driven layouts or image-conditioned object and simple-scene generation.
3 Method
Code-as-Room formulates top-down image-to-executable-Blender-code generation as a coarse-to-fine agentic pipeline. It separates global layout alignment from local geometry and appearance synthesis using shared cross-stage memory and rendered visual feedback.
- Overview: Given a room-level top-down image I, the framework generates executable Blender code C specifying room structure, object placement, geometry, materials, and lighting.The task is formulated as an agentic image-to-code generation process using the proposed VLM-agent harness.
- Coarse-to-fine decomposition: The pipeline first constructs a structured scene state and coarse layout, then enriches that layout with image-grounded object descriptions and final room-program details.The coarse stage fixes room structure and object placement, while the fine stage adds editable geometry, appearance, and rendering code.
- Cross-stage memory: A persistent cross-stage memory M stores typed artifacts, while downstream stages read predefined memory views to preserve consistency and reduce prompt noise and hallucinated dependencies.This memory is shared across all modules throughout the pipeline.
- Scene understanding: Stages 1 and 2 convert the image into a schema-constrained description and an object-centric scene graph with architectural, major-object, hierarchical, and minor-object information.The process preserves fixed spatial references for walls, doors, windows, openings, and built-in structures, and filters invalid graph relations before writing the result to memory.
4 Experiments
Experiments evaluate Code-as-Room through a benchmark, comparisons with direct VLM generation and VIGA, human assessment, image re-rendering, and ablations. Results show that structured agentic execution improves spatially coherent, complete, and usable 3D room generation, while memory is important for layout preservation.
- Benchmark: The benchmark evaluates three VLM backbones on 41 diverse scenes using metrics for visual understanding, spatial reasoning, code generation, and holistic scene quality.Scenes span residential and specialized environments, with varying room complexity and image styles.
- Benchmark Results: Code-as-Room consistently improves VLM-based image-to-3D generation, producing more complete structures, clearer functional regions, and better-aligned furniture layouts than direct prompting.Gemini models become stable and competitive with the structured workflow, while direct GPT-5.5 remains spatially inconsistent and unstable in Blender execution.
- Human Evaluation: Code-as-Room achieves higher human-perceived quality than direct generation and VIGA, with the Gemini3.1-Pro variant obtaining the best similarity, usability, and acceptability.VIGA has comparable lighting but weaker similarity and usability because of template-like scenes, missing details, and inaccurate object placement.
- Image Re-rendering: Image-level re-rendering preserves the generated scenes’ original layout and geometric and semantic consistency across multiple views.The Blender scenes provide structural priors including room structure, object layout, spatial relations, and camera-consistent geometry for visual refinement.
- Ablation Studies: Removing cross-stage memory degrades all metrics, causing missing objects and weaker layout preservation, with the largest drop occurring in Layout IoU.The ablation uses the same VLM backbone and input images across variants for fair comparison.
5 Conclusion
Code-as-Room presents an MLLM-based agentic framework that synthesizes editable, renderable, and structured 3D indoor rooms from top-down reference images as executable Blender code. Its current scope is limited by top-down inputs and the difficulty of faithfully generating some real-world objects procedurally.
- Contributions: Code-as-Room synthesizes realistic and functional 3D indoor rooms from top-down reference images using executable Blender code.The representation supports editable, renderable, and structured 3D room synthesis.
- Contributions: The framework uses input images as explicit spatial priors rather than relying on text-driven room generation.This design targets precise spatially grounded room synthesis from top-down views.
- Limitations and future work: The framework currently targets global 3D scene synthesis from top-down view images, limiting applicability to arbitrary-view real-world settings.It is not yet optimized for arbitrary-view inputs.
- Limitations and future work: Procedural code remains insufficient for faithfully generating many real-world objects because alignment between code-generation models and 3D assets is limited.Retrieval-based asset insertion may therefore be needed for higher geometric fidelity.