PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction

Zhening Huang, Yueyan Li, Johnathan Chiu, Xiaoyang Lyu, Matt Zhou, Yuxin Yao, Joan Lasenby, Shangzhe Wu

arXiv:2610.01863v1cs.CVcs.GRcs.RO

TL;DR

Reconstructing structured indoor scenes from real scans for interaction and physical simulation remains a challenging system-building problem. LiteReality-Agent treats reconstruction as executable scene coding and refines scenes through evidence-driven agentic authoring; across evaluated scenes, it reports gains over frontier-model baselines in visual fidelity, geometric consistency, and perceptual quality.

  • Problem

    Robustly turning real-world indoor scans into structured environments suitable for interaction and physical simulation remains a longstanding challenge.

    Source: §1 · p. 2

  • Method

    LiteReality-Agent represents a room as editable executable Room.py code and uses a coding agent, specialised evidence tools, and an observe–edit–verify harness to refine the scene.

    Source: §1 · p. 2

  • Results

    Across ten scenes, LiteReality-Agent reports gains in visual fidelity, geometric consistency, and perceptual quality over frontier-model agent baselines.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors present the system as a practical foundation for robust real-to-sim systems and downstream embodied AI.

    Source: §5 · p. 9

    Fidelity gains do not consistently extend to object-based layout measures, and the comparisons do not isolate the harness from other system components.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We present LiteReality-Agent, an agentic system for reconstructing real indoor environments as realistic, articulated, and simulation-ready 3D scenes from RGB-D scans. At its core, LiteReality-Agent formulates 3D reconstruction as a coding problem, in which a coding agent gathers evidence using specialised tools and iteratively edits a Python script, Room.py, which can be executed to produce a 3D digital twin of the room. With this formulation, we develop a robust observe-edit-verify harness that supports evidence gathering, measurement, verification, layout optimisation, simulation readiness, and quality control throughout the reconstruction process. LiteReality-Agent produces high-quality reconstructions suitable for simulation and downstream embodied AI tasks. Furthermore, as agent capabilities continue to improve rapidly, the system introduced by LiteReality-Agent remains a strong orchestration framework for future agents: it equips them with specialised tools, structured workflows, and robust verification mechanisms that substantially improve reconstruction quality and reliability. We demonstrate that LiteReality-Agent produces reconstructions that are more geometrically accurate, visually realistic, and simulation-compatible than those generated by recent frontier models, such as Astra and Fable. We therefore view LiteReality-Agent as a practical and important building block for robust real-to-sim systems. Both the source code and the data-capture application are publicly available. Code:https://github.com/LiteReality/LiteReality-Agent/

1 INTRODUCTION

Functional digital twins require recovering structured, articulated scenes suitable for interaction and physical simulation, but robust systems that ground agent outputs in real observations remain lacking. LiteReality-Agent addresses this gap with an executable scene program and agentic authoring, and reports the highest mean scores across its evaluated visual and perceptual criteria.

1 INTRODUCTION

  • The paper targets structured environments for interaction and physical simulation, a longstanding challenge for robust, scalable reconstruction from real-world scans.Prior work spans CAD retrieval, generative object composition, inverse rendering, and scan-to-scene pipelines, but the cited LiteReality pipeline is limited beyond predefined scenarios.Source: §1 · p. 2
  • LiteReality-Agent represents each room as an executable Room.py workspace that an evidence-gathering coding agent can modify and compile into a readable, reproducible, editable 3D scene.Its two stages initialize objects and repair layout before an observe–edit–verify authoring harness refines the scene with task-specific tools and quality checks.Source: §1 · p. 2
  • The system also demonstrates natural-language editing for scene decoration, object replacement, and layout adjustment.The authors describe these edits as enabled by representing the environment as an executable Python program.Source: §1 · pp. 2–3
  • LiteReality-Agent achieves the highest mean scores across reported visual metrics and perceptual criteria against the deterministic LiteReality pipeline and general-purpose coding agents.The evaluation combines image-based visual-fidelity measures, geometric accuracy and physical-plausibility measures, and human and agent-based perceptual assessments.Source: §1 · p. 3

2 RELATED WORK

Prior work reconstructs scenes through asset retrieval, generative object composition, inverse rendering, and language- or agent-based authoring. LiteReality-Agent is positioned as a metric RGB-D room-reconstruction system combining articulated assets and a persistent scene program with multi-view verification and explicit physical checks.

Indoor Reconstruction and Real-to-Sim

  • Scan-to-scene systems such as LiteReality and MetaScenes build asset-based environments, while video methods reconstruct interactive worlds and can recover articulation.The related work also covers CAD alignment, neural scene representations, and procedural environments for robot learning.Source: §2 · p. 3

Object Generation and Articulation

  • Object-generation work ranges from multi-view synthesis and generative models to methods that align objects, infer articulation, or validate programmatically generated assets.Articraft couples programmatic asset generation with a specialised SDK and validation harness.Source: §2 · p. 3

Language-Guided Scene Authoring

  • Language-guided scene work spans layout generation, feedback-based placement refinement, programmatic authoring, and agentic scene-generation pipelines.Other systems explore visually guided editing and benchmarks for code-based graphics editing.Source: §2 · p. 3

Agentic Inverse Graphics

  • LiteReality-Agent targets metric RGB-D room reconstruction with articulated assets, a persistent scene program, multi-view verification, and explicit collision and support checks.Its end-to-end pipeline progresses from scan-guided layout repair and object reconstruction through Room.py authoring to MuJoCo export.Source: §2 · p. 3, Figure 2 · p. 4

3 LITEREALITY-AGENT

LiteReality-Agent initializes a room from RGB-D scans, detections, and reconstructed objects, then refines its executable scene program through evidence-driven authoring and validation. Its tools support multi-scale inspection and measurement, while bounded control, geometry and support checks, and physical metadata prepare scenes for simulation export.

3 LITEREALITY-AGENT

  • The system has two stages: scene initialization establishes layout and major objects, then agentic authoring refines the executable scene program through an observe–edit–verify loop.The end-to-end pipeline exports the editable scene and physical metadata to MuJoCo.Source: §3 · p. 4

3.1 INPUT: RGB–D SCANS AND DETECTION RESULTS

  • Inputs are RGB-D scans and detection results from the LiteReality Scan application, including room layout and object bounding boxes.Source: §3.1 · p. 4

3.2 SCENE INITIALISATION

  • The Layout Agent models support, adjacency, grouping, and potential clashes, then repairs noisy boxes using geometric checks and image-based reasoning under cumulative limits.Corrections guide asset generation and placement so later assembly does not restore the original errors.Source: §3.2 · p. 4
  • Object reconstruction generates reference images, routes articulated assets to procedural generation and complex static objects to TRELLIS.2, and checks generated meshes and simulation properties.Failed chair checks trigger bounded regeneration; Room.py then retains the assembled scene as editable, rebuildable code.Source: §3.2 · pp. 4–5

3.3 AGENTIC AUTHORING

  • During authoring, the agent observes the capture, edits the scene program, and verifies changes through rendered comparisons and geometric checks.The loop uses defined tools and records actions and results until its turn limit or quality-control pass.Source: §3.3 · p. 5, §3.2 · p. 5

3.3.1 EVIDENCE TOOLS

  • Evidence tools select complementary room, wall, or object views and compare calibrated render–photograph pairs; wall stitching and measurement provide metric placement references.The harness orders material matching, fixture and lighting addition, small-object placement, and procedural-asset refinement; PBR texture maps support colour matching.Source: §3.3.1 · pp. 5–6

3.3.2 CONTROL AND QUALITY GATES

  • Authoring is bounded by turn and tool-call budgets, while geometry and support gates validate the rebuilt scene and return unresolved findings for correction.Mesh intersections guide bounded object translations; separate checks detect missing or invalid support contacts and ensure articulated parts remain distinct.Source: §3.3.2 · p. 6

3.4 SIMULATION-READY EXPORT

  • MuJoCo export represents objects with mass, inertia, collision geometry, and joints, keeping walls and attached fixtures fixed and assigning movable objects free joints.Separate collision shapes preserve space under tables and inside furniture; the exporter accounts for asset scale and placement and estimates physical properties when needed.Source: §3.4 · p. 6

4 RESULTS

Across ten indoor scenes, the evaluation compares visual fidelity, geometric consistency, and perceived similarity against LiteReality and general-purpose agents. LiteReality-Agent leads on the reported visual metrics, geometric criteria, and human and agent perceptual ratings.

4.1 EXPERIMENTAL SETUP

  • The benchmark covers ten indoor scenes and 620 capture views, comparing LiteReality-Agent with LiteReality and GPT-6 and Fable 5.1 agents run with RGB-only or scan inputs.Reconstructions use recorded camera poses and intrinsics at 960 × 720 and are evaluated at captured viewpoints; lighting is shared with the corresponding LiteReality-Agent scene but is not independently calibrated to reference photographs.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 VISUAL FIDELITY

  • LiteReality-Agent achieves the best mean on all four visual metrics; versus GPT-6 (+Scan), PSNR improves by 1.162 dB and LPIPS falls by 13.7%.Against LiteReality, the corresponding gains are 2.248 dB in PSNR and 19.6% in LPIPS.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 GEOMETRIC CONSISTENCY AND PHYSICAL PLAUSIBILITY

  • LiteReality-Agent outperforms both scan-informed agent baselines on all three geometric criteria, with 10.50 cm depth MAE, 100% bottom near-contact, and zero detected furniture intersections.Against LiteReality, depth MAE falls from 15.03 to 10.50 cm and detected intersection pairs from 43 to zero, while bottom near-contact remains 100%.Source: Table 3 · p. 7, §4.3 · p. 8

4.4 EFFECT OF SCAN DATA ON GENERAL-PURPOSE AGENTS

  • Adding LiteReality Scan data improves all four visual metrics for both general-purpose agents and reduces depth MAE by 67.1% for GPT-6 and 72.2% for Fable 5.1.The fidelity gains do not consistently extend to object-based layout measures: intersection counts rise for both agents, and GPT-6’s bottom near-contact rate falls from 69.1% to 63.0%.Source: §4.4 · p. 8

4.5 QUALITATIVE RESULTS

  • Representative kitchen and meeting-room views show capture photographs, reconstructions, and point-cloud or scene-plan views; edits demonstrate material, furniture, and fixture changes, while a MuJoCo kitchen is shown under shaking and in a Franka pick-and-place example.The demonstrations do not measure robot-task success or identify real-world dynamics.Source: §4.5 · p. 8, Figure 4 · p. 8, Figure 5 · p. 9

4.6 HUMAN AND AGENT-BASED PERCEPTUAL EVALUATION

  • LiteReality-Agent has the highest human means: layout 4.60, object similarity 4.58, and overall similarity 4.54, and both agent evaluators rank it first.For overall similarity, it wins 53 of 65 paired participant–scene comparisons against GPT-6 (+Scan), with eleven ties and one loss; ratings do not establish statistical significance or physical validity.Source: §4.6 · p. 9, Table 4 · p. 9

5 CONCLUSION

LiteReality-Agent is an open-source system that turns RGB-D scans into realistic, articulated, simulation-ready scenes through a coding-based reconstruction workflow. The appendix adds comparisons across reconstructed rooms and describes implementation details.

5 CONCLUSION

  • LiteReality-Agent treats 3D reconstruction as a coding problem and produces realistic, articulated, simulation-ready scenes; across ten scenes, it improves visual fidelity, geometric consistency, and perceptual quality over frontier-model agent baselines.The conclusion presents its observe-edit-verify harness as a practical foundation for robust real-to-sim systems and downstream embodied AI.Source: §5 · p. 9

A ADDITIONAL RECONSTRUCTION COMPARISONS

B.1 CAPTURE DATA AND LAYOUT REPAIR

The capture data provide a point cloud and initial layout estimates, while layout repair uses a scene graph and bounded corrections before room assembly.

B.1 CAPTURE DATA AND LAYOUT REPAIR

  • The point cloud is derived from RGB-D observations, and detections provide initial layout estimates rather than final object geometry.Figure A3 traces the kitchen example from point cloud and detected boxes through layout and reconstructed shell to an assembled initial scene; fine appearance and small objects are refined during authoring.Source: Table 5 · p. 16, Figure A3 · p. 16
  • The scene graph represents the room, structural surfaces, openings, and objects, with relations for support, wall adjacency, functional groups, and opening hosts.Height and footprint overlap identify support candidates; expected containment is distinguished from conflicting footprints, and unsupported objects are flagged.Source: §B.1 · p. 16
  • Layout repair prioritizes candidate layouts by error count and summed error magnitude, while limiting horizontal movement to 0.6 m, vertical movement to 0.08 m, and per-axis size changes to 15% or 35% by category.Repairs remain cumulatively bounded against the original scan, cannot delete measured objects to remove violations, and stop on validation, repeated lack of progress, or six agent-assisted rounds.Source: §B.1 · p. 16

B.2 LAYOUT AGENT: RELATIONS, REPAIR, AND BEFORE–AFTER RESULTS

The Layout Agent repairs scan-derived boxes through geometric checks and bounded edits; saved runs pass the layout checker after repair.

B.2 LAYOUT AGENT: RELATIONS, REPAIR, AND BEFORE–AFTER RESULTS

  • A scene graph separates support from functional grouping, while deterministic search and optional image-guided proposals repair boxes before they set crops, dimensions, and placements.Proposals must improve the geometric score while preserving anchors and cumulative scan-fidelity bounds.Source: §B.2 · p. 17
  • 30 to 0 checker errors across saved runs; eight scenes changed, two valid scenes stayed unchanged, and 29 of 107 boxes moved while four resized.The runs used deterministic search, so they do not measure the added benefit of image-guided proposals; passing the checker does not validate later meshes or authored objects.Source: §B.2 · p. 17

B.3 OBJECT RECONSTRUCTION AND PHYSICAL METADATA

  • Articulated parts use procedural geometry and the sofa uses image-to-3D generation; raw captures remain fidelity evidence because occluded asset regions are estimates.The figure pairs captured views, cleaned references, and reconstructed assets.Source: Figure A5 · p. 19, §B.3 · p. 19
  • Mesh checks flag malformed geometry and can trigger chair regeneration; physical-property estimates and per-object MuJoCo tests cover mass, inertia, collisions, joints, and motion.If retries are exhausted, the original asset is retained and the unresolved failure recorded.Source: §B.3 · p. 19

B.4 EVIDENCE TOOLS AND AUTHORING STATE

  • Room, wall, and object queries select evidence views; calibrated render–photo pairs expose discrepancies, while Room.py and object.py preserve editable builds and reusable material recipes.Rebuilding the programs hands saved edits between modules.Source: §B.4 · p. 20

B.5 STITCHED WALL VISUALIZATIONS

  • Rectified wall views include an observed-pixel mask and metric grid for fixture placement, but planar projection can distort protruding objects.The grid measures distance along the wall and height above the floor.Source: §B.5 · p. 21, Figure A8 · p. 21

B.6 QUALITY CONTROL AND MUJOCO INTEGRATION

  • Quality control separately checks mesh intersections and support geometry; bounded local corrections rebuild and retest, while unresolved cases are reported and session termination is not a QC verdict.Declared-support checks flag gaps above 1 cm and penetration exceeding 2 cm.Source: §B.6 · p. 22
  • The exporter combines shell geometry, poses and supports, appearance, and physics records; transforms carry collision geometry and joint frames, and Figure 6 shows perturbation and robot interaction.Attached fixtures are fixed, movable objects receive free joints, and articulated parts use their specified joints.Source: §B.6 · p. 22

B.7 ADDITIONAL SCENE-EDITING DEMONSTRATIONS

The additional demonstrations show controlled edits to a saved office reconstruction and natural-language requests for material, layout, and decoration changes.

B.7 ADDITIONAL SCENE-EDITING DEMONSTRATIONS

  • Natural-language requests demonstrate exposing brickwork, clearing the floor for a meeting, and decorating the room for World Cup viewing.Each instruction is applied as an edit to Room.py.Source: §B.7 · p. 23, Figure A9 · p. 23
  • Each controlled variant starts independently from the same saved office scene and uses an identical camera, resolution, and render configuration.The examples are Blender renders from a reproducible scene-editing script, not measurements of autonomous success or physical validity.Source: §B.7 · p. 23
  • The carpet edit changes colour while retaining its spatial texture, room geometry, and furniture placement.This isolates an appearance change from structural reconstruction.Source: §B.7 · p. 23
  • The layout edit removes the meeting-table group and four guest-chair groups, exposing the floor while leaving walls, shelves, and other fixtures in place.Source: §B.7 · p. 23
  • The lighting edit adds three pendant assemblies above the table, changing both scene geometry and illumination.Each assembly has a shade, suspension cord, light source, and ceiling-attachment declaration.Source: §B.7 · p. 23

C ADDITIONAL EXPERIMENTAL DETAILS

These experimental details specify the scan inputs, camera-aligned evaluation comparisons, rendering and preprocessing choices, and aggregation used for image metrics.

C ADDITIONAL EXPERIMENTAL DETAILS

  • The +Scan condition adds aligned LiDAR depth and confidence maps, camera calibration and poses, timestamps, a derived point cloud, and a RoomPlan layout.LiteReality and LiteReality-Agent receive the same scan information as the +Scan baselines.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 24
  • Ten evaluation scenes contribute 27–91 capture views each; six method–input configurations share 3,720 scene–frame comparisons, with aligned scenes rendered from recorded camera poses.Alignment preserves scene scale and individual object placements.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 24
  • Blender renders use shared per-scene lighting and colour management, while references are downsampled and paired without further warping, masking, or colour correction.All capture frames, including dark renders, are retained.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 24
  • SSIM, PSNR, RMSE, and LPIPS use specified RGB normalization and model settings; reported metric means average per-frame scores over 620 views with equal view weights.Scenes with more capture views therefore contribute more to each aggregate.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 24

D GEOMETRIC EVALUATION DETAILS

The geometric evaluation details define depth agreement, bottom near-contact, and furniture intersection measures, along with their scope.

D GEOMETRIC EVALUATION DETAILS

  • Depth agreement is mean absolute error between rendered and captured depth on shared valid pixels, averaged over 611 frames and reported in centimeters.The comparison includes evaluated outputs such as RoomPlan.Source: D GEOMETRIC EVALUATION DETAILS · p. 24
  • Bottom near-contact counts furniture with at least one sampled bottom point within 1 cm of an external support, reporting counts and percentages.It does not require every leg or base component to contact a supporting surface.Source: D GEOMETRIC EVALUATION DETAILS · p. 24
  • Furniture intersections count strictly intersecting triangle surfaces across ten scenes, excluding within-object and architecture pairs; counts require semantic interpretation.Method-specific furniture populations differ, so intersection counts and near-contact rates should be read alongside furniture counts.Source: D GEOMETRIC EVALUATION DETAILS · p. 24

E ADDITIONAL PERCEPTUAL EVALUATION DETAILS

LiteReality-Agent leads the reported human overall-scene ratings and has higher scores than GPT-6 (+Scan) on that criterion. The evaluation reports three criteria separately, using human and agent ratings.

Rating criteria

  • The three five-point criteria are layout accuracy, object similarity, and overall scene similarity, reported separately without a composite score.Higher ratings indicate closer agreement with the reference scene.Source: E ADDITIONAL PERCEPTUAL EVALUATION… · p. 25

Questionnaire design

  • Participants compared reference RGB images and reconstruction renders from matching viewpoints, alongside orthographic plan-view comparisons, with methods anonymized and positions randomized per scene.Each participant rated five of ten scenes and all four methods on all three criteria.Source: E ADDITIONAL PERCEPTUAL EVALUATION… · p. 25

Sample inclusion and aggregation

  • Human results aggregate 13 completed submissions into 65 participant–scene evaluations and 780 ratings, with each method receiving 65 ratings per criterion.Participant–scene evaluations receive equal weight, but scene coverage ranges from three to eleven participants; three submissions with unresolved method identities are excluded pending verification.Source: E ADDITIONAL PERCEPTUAL EVALUATION… · p. 25

Agent-based evaluation

  • Two sets of agent-generated ratings, Agent-1 and Agent-2, are reported separately from human ratings for the same four methods and three criteria.The agent ratings are aggregated.Source: E ADDITIONAL PERCEPTUAL EVALUATION… · p. 25

Additional comparisons

  • LiteReality-Agent has the highest human mean overall scene similarity in all ten scenes, with a 1.54-point advantage over GPT-6 (+Scan); each agent evaluator's advantage is 1.30 points.Agent-1 scores LiteReality-Agent above the human mean on all three criteria, while Agent-2 gives it a higher layout score but lower object and overall similarity scores; the evaluators agree on the two leading methods but differ in absolute scores.Source: E ADDITIONAL PERCEPTUAL EVALUATION… · p. 25

F LIMITATIONS AND FUTURE WORK

The authors identify unvalidated physical properties and uncertain layout accuracy as limitations, and call for evaluation in larger, more structurally diverse environments.

Limitations

  • Physical properties rely largely on language-model inference and category or material priors, with geometry-based estimates for some quantities; they lack ground-truth validation.Simulation export and consistency checks do not establish that reconstructed scenes reproduce real-world dynamics; measured properties and validation against real interactions remain future work.Source: F LIMITATIONS AND FUTURE WORK · p. 26
  • RoomPlan errors in object dimensions and poses can persist, and collision or support corrections and visual agreement do not guarantee recovery of the true layout.Future work should integrate image evidence with collision and support constraints more tightly and evaluate layouts against measured ground truth.Source: F LIMITATIONS AND FUTURE WORK · p. 26

Future work

Loading 2610.01863v1…