PaperlayerView exact revision on arXiv

Source-linked AI summary

ReRoom: Blending Virtual and Physical Contexts for In Situ Room Planning in Mixed Reality

Hongliang Yang, Yanjing Xu, Anhang Zhang, Hui Ye, Pengfei Xu

arXiv:2609.03596v1cs.GRcs.HC

TL;DR

Existing room-planning tools offer limited support for evaluating and refining whole-room proposals in the physical room. ReRoom combines a registered virtual room proxy, shared layout state, and a skill-guided agent for iterative in situ authoring. Its layouts score best across six reported quality metrics, and its in situ workflow improves the planning experience over off-site VR; generation currently takes around three minutes per round.

  • Problem

    Existing approaches separate layout editing from the physical room or offer limited support for evaluating and refining whole-room proposals in situ.

    Source: §1 · p. 1

  • Method

    ReRoom uses a spatially registered virtual room proxy and shared layout state for in situ editing, with a skill-guided agent generating updates while preserving accepted placements.

    Source: §1 · pp. 1–2

  • Results

    ReRoom achieved the best result on all six layout-quality metrics across 43 rooms, and in situ authoring improved the planning experience over an otherwise equivalent off-site VR workflow.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ReRoom supports iterative real-room planning by connecting proposal generation, inspection, editing, and preservation of accepted decisions in one in situ workflow.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

    A layout generation round currently takes around three minutes on average, limiting how rapidly users can inspect a proposal and receive a revision.

    Source: §7 · p. 9

Abstract

from arXiv · show

Planning a real domestic space is an in situ authoring process: users evaluate candidate layouts at true scale, refine their intent, and carry accepted decisions into later iterations. Existing approaches either separate layout editing from the physical room or provide limited support for evaluating and refining whole-room proposals in situ. We present ReRoom, a mixed-reality system for in situ room-layout authoring. ReRoom presents a shared layout state through a virtual room proxy spatially registered to the target room, allowing interaction and layout generation to remain grounded in the physical context. Users refine the current proposal through direct manipulation or language and preserve accepted placements, allowing each generated update to continue the same evolving design. To balance layout quality with generation efficiency, ReRoom uses a skill-guided layout agent whose room-layout design skill operationalizes three principles that we formulate by synthesizing established interior-design guidance for real-room layout generation. The skill grounds these principles in a normalized representation of the scanned room and reusable geometric checks. Evaluations show that ReRoom produces high-quality layouts for non-rectangular rooms, while its in situ workflow improves the room-planning experience over an otherwise equivalent off-site VR workflow. Code will be released upon acceptance of the paper.

1 INTRODUCTION

ReRoom supports iterative room-layout authoring in the target physical room, where users can inspect proposals at true scale, refine intent, and preserve accepted decisions. Its shared layout state and skill-guided agent connect that workflow to layout generation for scanned rooms.

1 INTRODUCTION

  • In situ planning lets users judge candidates against room features and discover constraints that are difficult to specify beforehand, then refine proposals while preserving accepted decisions.A candidate may reveal doorway obstruction, blocked sightlines, or an area that should remain unchanged in later iterations.Source: §1 · p. 1
  • ReRoom links a shared evolving layout state to a spatially registered virtual room proxy and a skill-guided agent, so edits and generated updates remain grounded in the target room.Users can inspect proposals at true scale, manipulate furniture directly, or request language-guided edits and lock accepted placements.Source: §1 · pp. 1–2
  • The room-layout skill translates established interior-design guidance into three principles, using an axis-normalized scanned-room representation and reusable geometric checks to focus agent reasoning and improve generation efficiency.The principles are functional grouping, anchor furniture selection, and geometry-guided refinement.Source: §1 · p. 2
  • Across 43 rooms, ReRoom achieved the best result on all six layout-quality metrics; 45 raters also preferred its layouts, and 14 participants reported a better planning experience in situ than off-site VR.The user study found improved spatial perception and decision confidence, lower task load, and higher usability relative to an off-site VR workflow with the same interaction functions and layout-generation capability.Source: §1 · p. 2

2.1 Immersive Spatial Authoring and Editing

  • Prior immersive authoring systems support direct interaction or AI-assisted refinement, but existing systems do not combine iterative furniture planning with blended real and virtual views for evaluating proposals against the actual room.The related work spans fully virtual authoring systems [3] [45] [10] [12] and mixed-reality editing systems [46] [4] [39] [34].Source: §2.1 · p. 2

2.2 Automated Indoor Layout Synthesis

  • Existing layout methods use one-shot generation or rearrangement, but cited approaches lack some combination of language-based spatial preferences, selectively preserved placements, and support for irregular room boundaries.Training datasets are overwhelmingly composed of rectangular rooms, limiting the generalizability of data-driven methods to diverse non-rectangular homes.Source: §2.2 · pp. 2–3
  • Across prior layout-method families, users evaluate layouts only after generation and interact through non-immersive channels; ReRoom instead places generation, evaluation, and revision in a physically grounded planning loop.The loop supports preserving approved placements and planning over irregular polygonal boundaries.Source: §2.2 · p. 3

2.3 General-Purpose Agents for Domain-Specific Tasks

  • Prior domain-specific agents use execution feedback to revise outputs, and ReRoom applies this pattern to room planning through a design skill and geometric-validation feedback.The skill encodes design principles and spatial knowledge for interpreting the target room.Source: §2.3 · p. 3

3 REROOM

ReRoom organizes real-room layout authoring around a shared state that connects the proxy, direct manipulation, language edits, and the layout agent. The system represents scanned room geometry and furniture poses for the agent, which uses a room-layout design skill to generate and refine proposals.

3.1 In Situ Authoring Interface

  • The authoring loop presents a scanned room’s layout at physical scale, supports direct manipulation and language-based changes, and returns generated updates to the proxy for further in situ evaluation.All three modalities act on the same evolving layout state.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1.1 Virtual Room Proxy

  • The spatially registered virtual room proxy places candidate furniture in the actual room context and offers full-real, blended, and full-virtual views with adjustable blending.Existing furniture can be reconstructed as 3D meshes from captured images for more faithful representations.Source: §3.1.1 · p. 4

3.1.2 Direct Manipulation and Visual Feedback

  • Direct manipulation supports furniture translation, rotation, deletion, locking, and adding assets, while visual feedback and recoverable layout snapshots support editing and experimentation.Users can undo changes or restore complete prior states, including furniture, dimensions, poses, and lock status.Source: §3.1.2 · pp. 4–5

3.1.3 Language-Guided Editing and Intent Locking

  • Language-guided editing conveys higher-level spatial preferences, while intent locking makes approved furniture poses hard constraints and leaves unlocked items eligible for rearrangement.The agent receives furniture identities, dimensions, continuous poses, and lock status for each generation request.Source: §3.1.3 · p. 5

3.2 Spatial Scene Representation

  • The scene representation encodes polygonal room boundaries and furniture identities, dimensions, continuous poses, and lock status; axis normalization aligns the longest wall to a room axis while preserving metric scale and polygon geometry.The shared layout state carries the geometry, arrangement, and preservation decisions used by both the interface and agent.Source: §3.2 · p. 5

3.3 Skill-Guided Layout Agent

  • The layout agent combines visual and structured scene inputs with user requirements, using the visual view for spatial relations and structured data for precise continuous furniture poses.Locked poses encode user-confirmed decisions as hard constraints.Source: §3.3 · p. 5

3.3.1 Room-layout Design Skill

  • The room-layout skill operationalizes three principles from interior-design guidance: functional grouping, anchor furniture selection, and geometry-guided refinement.These principles guide region and furniture relations, place function-defining items before associated furniture, and revise layouts using geometric checks while preserving locked poses.Source: §3.3.1 · pp. 5–6

3.3.2 Adaptive Layout Generation and Refinement

  • The agent chooses its own sequence of interpretation, layout generation, checking, revision, and selection, using a common coarse-to-fine path as guidance rather than a fixed procedure.It can revise individual items, regions, or the full composition, and select a candidate once geometric conditions and design principles are satisfied.Source: §3.3.2 · p. 6

4 IMPLEMENTATION DETAILS

ReRoom was implemented on a Meta Quest 3 with a local PC, using built-in scene understanding and a language-model agent for layout generation. Candidate layouts undergo deterministic validation before being written to the shared state and displayed.

4 IMPLEMENTATION DETAILS

  • ReRoom used Meta Quest 3 with built-in SLAM tracking and depth sensing; its interface was developed on a local PC (Intel Core i9-14900K CPU, 128GB RAM).The interface was developed using Unity 6000.1.10f1.Source: §4 · p. 6
  • Room capture uses the headset’s built-in scene understanding without external scanning devices, and Hunyuan3D Studio [18] reconstructs textured furniture models from single images.Captured scene data are converted into the spatial representation used by ReRoom.Source: §4 · p. 6
  • Layout generation runs Codex 0.147.0 with GPT-5.6 Sol at high reasoning effort; deterministic checks validate candidate completeness and geometry before selected layouts update the shared state.The agent runs in a temporary Docker environment with file-system access limited to the current task’s inputs.Source: §4 · p. 6

5 TECHNICAL EVALUATION

The technical evaluation compares ReRoom’s generated layouts with automated methods on non-rectangular rooms, then uses human ratings to assess physical plausibility, functional coherence, and preference. The perceptual study finds ReRoom rated above both baselines on all three dimensions.

5.1.1 Baselines

  • LayoutVLM-PolyRepair adapts LayoutVLM with true polygon boundaries and a full-footprint polygon constraint, while GLTScene uses a global-to-local transformer for polygonal rooms.The evaluation compares these two method families as representative baselines.Source: §5.1.1 · p. 6

5.1.2 Evaluation Metrics

  • The six metrics assess collision avoidance, boundary compliance, semantic placement and orientation coherence, combined semantic and physical quality, and alignment with the nearest wall.CF and IB measure physical constraints; PC, RC, and PSA assess semantic quality, while AE measures wall-relative orientation consistency.Source: §5.1.2 · p. 6

5.1.3 Experimental Setup

  • The benchmark uses 43 3D-FRONT rooms, nearly all non-rectangular, with each method generating one layout from the original furniture categories and dimensions.ReRoom and LayoutVLM-PolyRepair use GPT-5.6 Sol with high reasoning effort; GLTScene receives polygonal floor-plan masks and ground-truth furniture categories and sizes.Source: §5.1.3 · pp. 6–7

5.1.4 Results

  • ReRoom raises CF from 0.8548 to 0.9731, improves PC and RC by 3.04 and 3.40 points, lowers AE from 0.0740 to 0.0083, and reaches PSA 55.78 versus the strongest baseline’s 39.53.It also achieves complete boundary compliance, exceeding GLTScene on that metric.Source: §5.1.4 · p. 7, Table 1 · p. 7

5.2.1 Study Design

  • A perceptual study presented 27 layouts from nine representative non-rectangular rooms as paired top-down and perspective images, rated on three 7-point dimensions.The dimensions were physical plausibility, functional coherence, and overall preference.Source: §5.2.1 · p. 7, Figure 4 · p. 7

5.2.2 Participants and Procedure

  • Forty-five participants each rated all 27 layouts after a tutorial, using anonymized and randomized method labels.The study used a within-subjects design.Source: §5.2.2 · p. 7, §5.2.1 · p. 7

5.2.3 Results

  • ReRoom exceeded both baselines on physical plausibility, functional coherence, and overall preference (all p < .001); 34 of 45 raters preferred it on all three dimensions.It had the highest overall-preference mean in eight of nine rooms and tied for highest in the remaining room.Source: §5.2.3 · p. 7, Figure 4 · p. 7

6 SYSTEM USABILITY STUDY

The system usability study compares in-situ MR planning with an otherwise equivalent off-site VR baseline. Participants reported stronger spatial perception, lower workload, higher usability, and qualitative benefits from physical grounding, alongside challenges and requested improvements.

6.1.1 Conditions and Participants

  • The within-subjects comparison held generation and interaction capabilities constant, contrasting spatially registered MR in the target room with off-site VR lacking physical room context.Fourteen participants completed two planning tasks per system: redesigning a furnished office and planning an activity space in an irregular hall.Source: §6 · p. 7, §6.1.1 · p. 8, §6.1.2 · p. 8

6.1.2 Tasks and Procedure

  • The tasks were counterbalanced, followed by a questionnaire and interview; measures included NASA-TLX, SUS, and custom items on spatial perception and in-situ evaluation.The study used paired Wilcoxon signed-rank tests and matched-pairs rank-biserial correlations for ReRoom–Baseline comparisons.Source: §6.1.2 · p. 8, §6.1.3 · p. 8

6.2 Quantitative Findings

  • All four shared spatial-perception measures favored ReRoom (p ≤ .0022, rrb ≥ .95), while Raw TLX was lower (Mdn = 20.75 vs. 30.42) and SUS higher (Mdn = 83.75 vs. 67.50).The reported paired-test results were p = .0017 for Raw TLX and p = .0022 for SUS.Source: §6.2 · p. 9
  • For ReRoom alone, participants rated visual comparison 6.64 (SD = 0.50) and environmental awareness 6.43 (SD = 0.65) on the custom 7-point items.These items asked about comparing layouts with the real room and maintaining awareness of physical surroundings.Source: §6.1.3 · p. 8, §6.2 · p. 9

6.3.1 In Situ Spatial Awareness and Physical Grounding

  • Participants described physical presence and real-world reference as helpful for judging furniture placement, moving through the layout, and refining proposals.Six participants reported greater confidence navigating when the physical environment remained visible.Source: §6.3.1 · p. 9

6.3.2 Efficient Layout Comparison and Evaluation

  • Participants valued rapid comparison between proposed layouts and the existing room, with some describing in-situ feedback as making layouts more precise.Visual state switching also helped participants judge whether furniture removal was reasonable.Source: §6.3.2 · p. 9

6.3.3 Multimodal Controls in Iterative Layout Refinement

  • Participants mixed generation, language, direct manipulation, and intent locking as needed, accepting about 75.2% of proposed placements and locking accepted furniture before regenerating other items.One in-situ session reorganized 38 items through generation and required one subsequent manual adjustment.Source: §6.3.3 · p. 9

6.3.4 Challenges of Virtual–Physical Coexistence

  • Participants noted that realistic virtual furniture could be mistaken for physical objects or distract them from real furniture, and requested multi-select and copy-and-paste capabilities.Three requested multi-select; two of those also suggested copy-and-paste.Source: §6.3.4 · p. 9, §6.3.5 · p. 9

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

ReRoom supports in-situ room authoring by keeping an evolving layout registered to the target room and shared across editing modes. Its current implementation remains constrained by generation latency and the need to balance proxy realism with clear virtual–physical differentiation.

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • A shared layout state connects direct manipulation, language-guided editing, and intent locking to the evolving proposal shown at true scale.The virtual room proxy is spatially registered to the target room.Source: §7 · p. 9
  • A room-layout design skill guides an LLM agent through functional grouping, anchor furniture selection, and geometry-guided refinement.The skill uses an axis-normalized scanned-room representation and reusable geometric checks.Source: §7 · p. 9
  • A generation round takes around 3 minutes on average, limiting how rapidly users can inspect a proposal and receive a revision.Future work could focus agent execution on the parts of the scene involved in each modification and use faster foundation models.Source: §7 · p. 9
  • The proxy requires balancing visual realism with clear virtual–physical differentiation during movement and manipulation.Adaptive visual cues could help distinguish proposed furniture from physical objects while preserving in-situ comparison.Source: §7 · p. 10
  • ReRoom places the intended physical environment in the loop throughout spatial-design generation, inspection, and revision.The paper frames this as a shift from generating spatial designs in isolation to authoring them with physical context.Source: §7 · p. 10

S1 SUMMARY OF SUPPLEMENTARY MATERIALS

The supplement collects additional evaluation, usability-study, and agent-instruction materials. Its metric definitions cover physical plausibility, semantic coherence, and wall alignment.

S1 SUMMARY OF SUPPLEMENTARY MATERIALS

  • Supplementary materials include benchmark metrics and per-room-type results, LayoutVLM-PolyRepair details, latency analysis, qualitative layouts, usability workflows, and the room-layout design skill.The usability materials include representative final layouts and detailed questionnaire results.Source: S1 SUMMARY OF SUPPLEMENTARY MATERIALS · p. 12

S2.1 Metric Definitions

  • Collision-Free Score measures the percentage of furniture items whose oriented bounding boxes do not intersect; In-Boundary Score measures the percentage entirely inside the room boundary.These metrics assess collision and boundary compliance, respectively.Source: S2.1 Metric Definitions · p. 12
  • Positional Coherency rates proximity of semantically related items, while Rotational Coherency rates whether their orientations support joint use.Both use LLM judgments of top-down and side-view renderings on a 0–100 scale; Gemini 3 Flash is the evaluator.Source: S2.1 Metric Definitions · p. 12
  • Physically-Grounded Semantic Alignment combines semantic coherence with physical plausibility as PSA = CF × IB × LayoutMatch.The product penalizes layouts that are semantically plausible but physically infeasible.Source: S2.1 Metric Definitions · p. 12
  • Alignment Error measures each furniture item's orientation relative to its nearest wall, averaging across items; 0 indicates perfect alignment and out-of-room items receive the maximum penalty of 1.The angle is measured between the item's orientation and its nearest wall segment.Source: S2.1 Metric Definitions · p. 12

S2.2 LayoutVLM Polygonal-Boundary Adaptation

The adaptation supplies LayoutVLM with a reconstructed true room polygon and constrains optimization using complete furniture footprints. Its projection step moves furniture back inside the polygon while preserving orientation and dimensions.

S2.2 LayoutVLM Polygonal-Boundary Adaptation

  • Ordered wall endpoints are joined into a validated floor polygon, then transformed into LayoutVLM’s local task frame using a minimum-area enclosing rectangle.The rectangle’s longer edge defines the local x-axis and the orthogonal direction defines y.Source: S2.2 LayoutVLM Polygonal-Boundary… · p. 12
  • The true polygon is supplied to LayoutVLM’s wall representation, scene renderer, and optimizer.Source: S2.2 LayoutVLM Polygonal-Boundary… · p. 12
  • Optimization requires each furniture item's full oriented bounding box to lie within the room polygon with a 5 mm outward tolerance.When a footprint exceeds the boundary, the optimizer searches for the closest feasible center while keeping yaw and dimensions fixed; this also detects edges crossing concave cut-outs.Source: S2.2 LayoutVLM Polygonal-Boundary… · p. 12
  • Full-footprint projection runs during joint position optimization, repeats after constraints that can move furniture, and runs once more before the layout is returned.Source: S2.2 LayoutVLM Polygonal-Boundary… · p. 12

S2.3 Skill-Guidance Generation Efficiency

The skill-guided agent generates layouts faster and with fewer output tokens than the plain agent in the 43-room evaluation. Room-type results show strong performance across categories, with LayoutVLM-PolyRepair leading library positional coherence.

S2.3 Skill-Guidance Generation Efficiency

  • 48.07% lower mean generation latency (1.93× speedup): skill-guided 170.29 seconds vs. plain agent 327.93 seconds per room.The paired 43-room experiment held room and furniture inputs, task request, and model constant.Source: S2.3 Skill-Guidance Generation… · p. 12
  • Skill-guided generation uses 43.50% fewer output tokens (5,397 vs. 9,552) and costs $0.334 vs. $0.410 per room, despite 0.51% more input tokens (208,499 vs. 207,449).Source: S2.3 Skill-Guidance Generation… · p. 12

S2.4 Per-Room-Type Quantitative Analysis

S3 QUALITATIVE RESULTS USED IN PERCEPTUAL EVALU-

S4 REPRESENTATIVE WORKFLOWS

Three illustrative sequences cover two assigned study tasks and a supplementary living-room scenario, tracing room preparation, iterative generation and refinement, and saving.

S4 REPRESENTATIVE WORKFLOWS

  • The three sequences cover two assigned study tasks and one supplementary living-room demonstration.The study tasks are described as an office redesign and open-hall planning.Source: S4 REPRESENTATIVE WORKFLOWS · p. 13
  • Each sequence includes room and asset preparation, an initial language-guided generation round, in situ inspection, and manual refinement.Source: S4 REPRESENTATIVE WORKFLOWS · p. 13
  • Users lock intent, issue follow-up generation rounds from refined instructions, then compare or adjust layouts before saving.Source: S4 REPRESENTATIVE WORKFLOWS · p. 13

S5 LAYOUT RESULTS FROM THE SYSTEM USABILITY STUDY

Across both tasks, MR and off-site VR produced qualitatively comparable final layouts. The reported advantage of in situ MR was an improved planning experience rather than measurably superior final designs.

S5 LAYOUT RESULTS FROM THE SYSTEM USABILITY STUDY

  • Final layouts were qualitatively comparable across MR and VR: both formed coherent zones, placed items within room boundaries, and reflected reasonable design intent.The conditions shared the same agent and interaction modalities, and participants could refine proposals without time limits.Source: S5 LAYOUT RESULTS FROM THE SYSTEM… · p. 13
  • The reported benefit was an improved planning experience, not measurably superior final layouts.Source: S5 LAYOUT RESULTS FROM THE SYSTEM… · p. 13
  • In situ MR users reported better spatial perception, higher layout confidence, and lower task load than VR users.Significant reductions were reported in temporal demand, effort, performance, and frustration.Source: S5 LAYOUT RESULTS FROM THE SYSTEM… · p. 13

S6 SKILL-GUIDED AGENT INSTRUCTIONS

The agent is instructed to use room and furniture evidence with the room-layout design skill, then produce a continuous-coordinate layout. The accompanying materials specify an iterative authoring workflow and output conventions.

S6 SKILL-GUIDED AGENT INSTRUCTIONS

Loading 2609.03596v1…