PaperlayerView exact revision on arXiv

Source-linked AI summary

I-Design: Personalized LLM Interior Designer

Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, Xi Wang

arXiv:2404.02838v1cs.AI

TL;DR

Turning user preferences into an interior requires choosing appropriate objects and arranging them plausibly, a challenge for non-professionals. I-Design uses LLM agents, scene graphs, placement, and asset retrieval to build 3D designs; experiments report better design quality and alignment with abstract user concepts than existing methods.

  • Problem

    Realizing interior preferences requires identifying suitable and common room objects and arranging them in plausible spatial relationships from user input.

    Source: §1 · pp. 1, 3

  • Method

    I-Design uses LLM agents to turn plain-text preferences into scene graphs, places objects, retrieves 3D assets, and composes the result in 3D.

    Source: §1 · p. 3, §3.2 · p. 6

  • Results

    I-Design outperforms existing methods in delivering high-quality 3D design solutions aligned with abstract concepts in user input.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The scene graph makes proposed designs interpretable through a transparent breakdown of the process, facilitating an accessible design process.

    Source: §5 · p. 15

    The pipeline can fail to find placements for many objects in small scenes, and retrieved assets may have variable quality, invalid orientations, or size mismatches that cause resizing artifacts.

    Source: §5 · pp. 15–16

Abstract

from arXiv · show

Interior design allows us to be who we are and live how we want - each design is as unique as our distinct personality. However, it is not trivial for non-professionals to express and materialize this since it requires aligning functional and visual expectations with the constraints of physical space; this renders interior design a luxury. To make it more accessible, we present I-Design, a personalized interior designer that allows users to generate and visualize their design goals through natural language communication. I-Design starts with a team of large language model agents that engage in dialogues and logical reasoning with one another, transforming textual user input into feasible scene graph designs with relative object relationships. Subsequently, an effective placement algorithm determines optimal locations for each object within the scene. The final design is then constructed in 3D by retrieving and integrating assets from an existing object database. Additionally, we propose a new evaluation protocol that utilizes a vision-language model and complements the design pipeline. Extensive quantitative and qualitative experiments show that I-Design outperforms existing methods in delivering high-quality 3D design solutions and aligning with abstract concepts that match user input, showcasing its advantages across detailed 3D arrangement and conceptual fidelity.

1 Introduction

I-Design addresses the challenge of turning unstructured user preferences into feasible, personalized 3D interiors. It combines LLM agents, scene-graph layout, and asset retrieval, and reports stronger design quality and abstract-concept alignment than existing methods.

  • Prior generative and data-driven methods are constrained by limited datasets and often require structured text with predefined grammar, complicating designs for unseen interiors.Source: §1 · p. 2
  • Realizing user preferences requires selecting suitable objects, adding common room items, and establishing plausible spatial relationships across an extensive range of interiors.The system also needs broad knowledge of design preferences and 3D objects’ functionality, dimensions, and style.Source: §1 · p. 3
  • I-Design uses multiple LLM agents to transform plain-language preferences into a scene graph of room items, properties, and relative relationships, then places objects and retrieves assets for 3D composition.Source: §1 · p. 3
  • I-Design outperforms existing methods in delivering high-quality 3D designs aligned with abstract concepts in user input.Source: §1 · p. 4
  • The approach contributes a procedural transformation from relative scene-graph relationships to absolute 3D representations and an interpretable pipeline for iterative design without restarting the entire process.Source: §1 · p. 4
  • The paper also proposes a vision-language-model evaluation protocol for 3D scenes.Source: §1 · p. 4

2 Related Work

Prior 3D indoor scene synthesis work spans LLM-based, generative, and data-driven approaches. The paper positions its LLM-based approach against limits in geometric reasoning, view consistency, asset selection, and dataset-constrained layout synthesis.

3DISS via LLMs

  • LLM-based scene synthesis has progressed from encoding text into vectors to LayoutGPT’s coordinate-based generation and dataset-free open-vocabulary layouts, but geometric reasoning can still yield overlaps or out-of-bounds objects.Source: §2 · p. 4

Specialized multi-agents

  • Multi-agent LLM systems have been applied across several domains, and this work uses multi-agent reasoning to address 3D indoor scene synthesis challenges.Source: §2 · p. 4

3DISS via Generative Models

  • Image-generation approaches create multi-view images or panoramas and convert them into 3D, while 2D methods can struggle with geometric constraints, depth uncertainty, and semantic consistency across views.Source: §2 · pp. 4–5

3DISS through Prior Learning

  • Asset selection methods either retrieve models from datasets or generate assets; I-Design chooses retrieval from existing repositories for direct real-world applicability.Source: §2 · p. 5
  • Furniture-layout methods include rule-based placement, procedural grammars, interactive editing, and learned autoregressive or diffusion-based synthesis from expert-designed layouts.Source: §2 · p. 5
  • Because dataset-trained layout methods inherit dataset constraints, the paper opts for LLM-based methods, whose commonsense knowledge can support a wider range of reasonable designs.Source: §2 · p. 5

3 Method

I-Design turns free-form design preferences into a scene graph, places objects within room constraints, and assembles a 3D design from retrieved assets. Its pipeline combines specialized LLM agents, backtracking layout, and a replayable output bundle.

3.1 Problem Formulation

  • I-Design represents objects and their relative spatial relationships in a scene graph, with the goal of producing a functional, coherent, and 3D-consistent scene aligned with user requests.The graph also represents room-layout elements and object properties such as materials, dimensions, and position.Source: §3.1 · p. 6

3.2 LLM Multi-agents

  • Five specialized LLM agents transform free-form input into a relative scene graph, correcting implausible connections and refining ambiguous object relationships.The agents work through successive stages, passing user input and prior-stage JSON representations forward.Source: §3.2 · pp. 6–7, Figure 2 · p. 7

3.3 Scene Graph Layout

3.4 3D Asset Retrieval

  • I-Design retrieves each 3D asset by matching text descriptions of object names, styles, and materials with database representations, then adjusts assets to fit their bounding boxes.The descriptions are embedded with CLIP and compared with OpenShape encodings.Source: §3.4 · p. 10

3.5 3D Composer

  • An off-the-shelf renderer visualizes the assembled design, while the output bundle includes the scene graph, floor plan, rendered views, prompt, and agent communication log.The bundle supports inspection, replay of selected stages, design variations, and quantitative studies of user-preference alignment.Source: §3.5 · p. 10

4 Experiments

The experiments compare I-Design with LayoutGPT, test prompt types, and combine GPT-4V ratings with a user study and qualitative examples. Results favor I-Design overall, while prompt type affects alignment and the gallery illustrates how designs reflect user preferences.

4.2 Quantitative Evaluation

  • The evaluation defines NObj, OOB, and BBL for object count and layout feasibility, and uses GPT-4V ratings for functionality, layout, scheme, and atmosphere.GPT-4V grades scenes from 0 to 10 using two concatenated viewpoints and the user prompt; each scene is rated three times.Source: Metrics · pp. 10–11

4.1 Implementation Details

  • The benchmark extends LayoutGPT to retrieve and place Objaverse objects, compares bedroom and living-room scenes at varied room sizes, and uses matched rendering settings.Both methods generate 10 scenes of each room type; the implementation uses AutoGen agents equipped with GPT-4, OpenShape retrieval, and Blender rendering.Source: §4.1 · p. 10, Experimental Settings · p. 11, Baseline · p. 11

4.2 Quantitative Evaluation

  • I-Design outperforms LayoutGPT across GPT-4V metrics, proposes larger furniture sets, and has zero OOB and minimal BBL values.The comparison covers bedrooms and living rooms with 10 scenes per room type for each method.Source: Table 1 · p. 13, Results · p. 13
  • Functional prompts yield the best results in all four grading aspects, while atmosphere and scheme information somewhat diminish synthesis quality.The authors attribute the latter to unmodeled room-surface materials and textures and the lack of object re-texturing.Source: Table 2 · p. 13, Results · pp. 13–14

4.3 User Study

  • The user study’s Bradley-Terry preferences favor I-Design over LayoutGPT for both bedrooms and living rooms.Participants evaluated 100 scene pairs, with 1,254 bedroom-pair answers and 660 living-room-pair answers.Source: §4.3 · p. 14, Table 3 · p. 15

4.4 Qualitative Study

  • In the gallery, generated rooms reflect explicit furniture and position requests as well as implicit needs, such as placing a crib beside a king-sized bed for a toddler’s room.The figure pairs prompts with scene-graph layouts, post-retrieval top-down renders, and corner views.Source: Figure 4 · p. 12, §4.4 · p. 15

5 Conclusion

I-Design uses LLM agents to turn text preferences into structured furniture choices and spatial relationships, with a transparent scene-graph workflow. Placement can fail in crowded small scenes, and retrieved assets can introduce quality and sizing problems.

5 Conclusion

  • I-Design interprets textual preferences through LLM agents, infers suitable furniture and common-sense relationships, and represents the design as a scene graph.The authors describe the stepwise breakdown as making the process interpretable and accessible.Source: §5 · p. 15
  • Placement may fail when many objects must fit in a small scene, and spatial conflicts or insufficient space can persist after postprocessing.Retrieved assets may have variable quality or invalid default orientations, and size mismatches can create resizing artifacts.Source: §5 · pp. 15–16
  • The authors propose learning-based placement instead of trial-and-error backtracking and generative asset methods to improve retrieval diversity and flexibility.Source: §5 · p. 16

6 Additional Qualitative Results

The supplementary renders illustrate I-Design’s outputs across generic and varied prompts, as well as the conversion from a generated scene graph to a 3D representation.

6 Additional Qualitative Results

  • The additional gallery presents qualitative outcomes across generic prompts, varied prompt types, and scene-graph-to-3D conversion.The figures show living-room renders for “Design me a living room,” renders generated through elaborate prompting, and a synthesized scene graph with corresponding renders.Source: §6 · p. 21, Figure 6 · p. 21, Figure 7 · p. 22
  • Figure 6 shows living-room renders generated with the common prompt “Design me a living room.”Source: Figure 6 · p. 21
  • Figure 7 presents rendered samples generated from prompts varying across atmosphere, scheme, layout, and functionality.Source: Figure 7 · p. 22
  • Figure 8 illustrates a user prompt transformed into a scene graph and then into its corresponding 3D representation.Source: Figure 8 · p. 23

7 Implementation of the Multi-agent Pipeline

The multi-agent pipeline distributes scene-planning and structuring tasks across LLM instances. The section motivates this design as a way to improve subproblem handling, reduce hallucinations, and manage output limits.

7 Implementation of the Multi-agent Pipeline

  • A divide-and-conquer approach has each LLM solve a simpler subproblem using earlier agents’ outputs, while feedback helps address unwanted behavior and hallucinations.Source: §7 · p. 24
  • Distributing work across agents mitigates output-token constraints and allows more objects to be generated while reducing misalignment with user and system prompts.The passage notes GPT-4’s 4,000-token output limit and reports that attention to prompts decreases as outputs approach it.Source: §7 · p. 24
  • The Interior Designer and Interior Architect process all objects together in one pass; incremental alternatives produced repetitive object suggestions.GPT-4 agents use JSON mode to restrict outputs to JSON and reduce tokens spent per iteration.Source: §7 · p. 24
  • The Engineer structures each object independently, so its schema-formatting step proceeds iteratively rather than in a large output chunk.Source: §7 · p. 24

8 Implementation of the GPT-4V Evaluator

The evaluator grades room renders in four categories, and its findings align with a user study. The system prompt asks it to assess renders against user preferences and return structured results.

8 Implementation of the GPT-4V Evaluator

  • The GPT-4V evaluator grades renders in atmosphere, scheme, layout, and functionality, and its findings align with the user study.Source: §8 · p. 24
  • The evaluator averages grades from three runs, using two opposing-corner renders and prompting it to comment before assigning scores.Source: §8 · p. 25

8.1 GPT-4V Evaluator System Prompt

  • The system prompt requests 0–10 grades across realism and geometric consistency, functionality, layout, color and materials, and overall aesthetic, conditioned on user preference.It requests results in a specified JSON format.Source: §8.1 · p. 25

9 Discussion

The discussion identifies limits in relational planning, matching inferred object properties to retrieved assets, texture control, and bounding-box placement. These constraints can produce unreasonable relationships, visual mismatch, or spatial artifacts.

9 Discussion

  • As scenes grow more complex, LLMs may struggle to consider multiple placement factors, increasing the likelihood of unreasonable relational edges and object overflow.Source: §9 · p. 25
  • Differences between inferred and retrieved asset dimensions or orientations can cause visually implausible distortion during resizing.The passage gives open versus closed laptops and ambiguity about a desk’s front side as examples.Source: §9 · pp. 25–26
  • Retrieved-object textures may not match user preferences, limiting precise texture and geometry control; re-texturing is suggested as future work.Source: §9 · p. 26
  • Bounding-box placement simplifies spatial representation and prevents collisions, but can make child objects spatially inconsistent, including floating artifacts.The paper suggests exploring mesh surfaces for more precise placement.Source: §9 · p. 26

10 System Prompts for Agents

The system prompts give agents distinct roles and guide their responses and structured outputs. The Interior Designer prompt asks for essential objects suited to user preferences, room function, and room size.

10 System Prompts for Agents

  • Agents have distinct roles within the overall generation process.Source: §10 · p. 26
  • System prompts impart those roles to agents through tailored instructions.Source: §10 · p. 26
  • The prompts shape agents’ responses to user prompts and instill individual characteristics.Source: §10 · p. 26
  • The prompts define each agent’s functionality and guide generation of structured outputs.Source: §10 · p. 26
  • The agent prompts are reproduced verbatim, while their JSON schemas are available in supplementary files and on the project webpage.Source: §10 · p. 26

10.1 Interior Agent System Prompt

  • The Interior Designer is instructed to suggest {n} essential objects based on user preference, room functionality, and room size.Source: §10.1 · p. 26

2. Architecture style (ex., modern, classic, etc.)

This section describes how room-object preferences are translated into structured placements and how evaluation prompts are generated across room types and design criteria, including ten bedroom styles defined by distinct palettes, materials, furnishings, and accents.

10.2 Interior Architect System Prompt

  • The Interior Architect assigns each object a relative placement, adjacency status, and facing wall, specifying placements separately for multiple instances.Placement uses a restricted set of prepositions for objects and room elements; the agent gives each instance’s placement in one list item.Source: §10.2 · p. 27

10.3 Engineer System Prompt

  • The Engineer converts Admin-suggested objects into schema-conforming JSON, outputting only objects to place and recording each instance separately when quantities exceed one.It maps the input “Placement” key to JSON “placement,” follows the specified prepositions, and uses only the supplied schema.Source: §10.3 · pp. 27–28

10.4 Layout Corrector System Prompt

  • The Layout Corrector changes scene_graph and facing_object when spatial conflicts make a user-provided object unsuitable, then validates the JSON against the schema.It is instructed to use the specified placement prepositions.Source: §10.4 · p. 28

10.5 Layout Refiner System Prompt

  • The Layout Refiner adds a second relative placement for child objects, relating them to one another and to their parent while considering the initial placement.For example, when children are initially “on” a parent, the refiner also specifies their positions relative to one another and a second relation to the parent.Source: §10.5 · p. 28

11 Room Synthesis Prompt Generation for Evaluation

  • GPT-4 generated ten prompts per evaluation criterion, varying room type and dimensions for functionality and holding them fixed while varying layout, color/material, or atmosphere.The prompts support a GPT-4V evaluation criterion; the paper frames them around functionality, layout, color scheme with material choice, and overall atmosphere.Source: §11 · p. 29

Minimal Preference Generation Prompt

  • Minimal preference prompts ask for ten bedroom or living-room dimension sets, while object specifications restrict bounding-box keys to Length, Width, and Height and prohibit door- or window-related objects.The bounding-box dimensions are in meters; the object instructions also include a quantity field.Source: §3 · pp. 26–27, Minimal Preference Generation Prompt · p. 29

Functionality-related Preference Generation Prompt

11.2 Complete Lists of Prompts

  • The evaluation prompt collections cover minimal bedroom and living-room preferences, plus functionality, layout, color/material, and atmosphere preferences.Table 4 covers prompts 1–10 for bedrooms and 11–20 for living rooms; Table 5 groups prompts 1–10 by functionality, 11–20 by layout, 21–30 by color/material, and 31–40 by atmosphere.Source: §11.1 · p. 29, Table 4 · p. 31, Table 5 · p. 31
  • Minimalist, bohemian, modern, and rustic prompts pair their styles with clean monochrome lines, eclectic patterns and plants, geometric forms and technology, or natural materials and warm tones.The minimalist prompt emphasizes natural light and low clutter; the rustic prompt adds chunky knits and a fireplace.Source: §11.2 · p. 36
  • Nautical, glamorous, industrial, and traditional prompts evoke their styles with seaside colors, opulent finishes, exposed raw materials, or classic furniture and symmetry.The prompts respectively specify maritime accessories, plush textiles, vibrant art, and rich textures.Source: §11.2 · p. 36
  • Scandinavian and contemporary prompts emphasize simplicity and functionality or dynamic mixes of textures, clean lines, color, and adaptable furnishings.The Scandinavian prompt specifies pale woods, muted colors, and hygge accents; the contemporary prompt calls for versatile pieces that adapt to trends.Source: §11.2 · p. 36
Loading 2404.02838v1…