PaperlayerView exact revision on arXiv

Source-linked AI summary

LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

Peijun Xu, Chuansen Nie, Yiyang He, Yinuo Bai, Jingyang Liu, Kuixiang Shao, Yuyang Jiao, Kuanhao Xia, Jiayi Zhu, Zitian Yang, Yanqi Zhang, Tianye Tan, Shuwei Di, Junyi Xu, Jingyi Yu, Jiayuan Gu

arXiv:2610.12069v1cs.CV

TL;DR

Realistic household simulation must account for lived-in arrangements and spatial constraints, yet constructing faithful, interactive replicas of diverse homes remains challenging. LIVIN builds digital twins of lived-in homes through a human-reviewed reconstruction workflow and evaluates four spatial and embodied tasks. The evaluations report that current methods still struggle with clutter, occlusion, and constrained interaction in realistic household scenes.

  • Problem

    Constructing realistic, interactive replicas of diverse lived-in homes remains challenging, despite the need to represent the arrangements and spatial constraints shaping robot movement and interaction.

    Source: §1 · p. 1

  • Method

    LIVIN constructs digital twins of lived-in homes through instance recognition, architectural reconstruction, and object generation and placement, with human review against source observations at each stage.

    Source: §1 · p. 2

  • Results

    Current methods still struggle in LIVIN's complex household scenes, with 3D methods missing objects or recovering inaccurate geometry and embodied systems failing at navigation and manipulation subtasks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    LIVIN provides a testbed for evaluating spatial understanding and embodied behavior in realistic household environments.

    Source: §7 · p. 9

Abstract

from arXiv · show

Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction. Existing resources often trade off scale, real-world correspondence, and interaction readiness, leaving a gap in faithful, interactive replicas of how real homes are actually arranged. To this end, we introduce LIVIN, a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes. These replicas preserve observed room layouts, furniture configurations, and everyday belongings. To construct them, we design a human-in-the-loop workflow comprising instance recognition, architectural reconstruction, and object generation and placement, with intermediate results reviewed and corrected by humans against the source observations at each stage. We evaluate four tasks in LIVIN: 3D detection, 3D reconstruction, navigation, and loco-manipulation. Our evaluations show that current methods remain challenged by the dense object arrangements, occlusions, limited free space, and constrained interaction regions found in realistic lived-in homes. We hope LIVIN will help advance embodied AI in real-world homes, from spatial understanding to robotic interaction, and ultimately bring embodied intelligence into everyday home environments.

1 INTRODUCTION

LIVIN addresses the challenge of simulating lived-in homes, where clutter, occlusion, and constrained space shape robot movement and interaction. It provides digital twins and a four-task benchmark, built through a human-reviewed reconstruction workflow, and finds that current methods still struggle in these environments.

Motivation

  • Furniture, belongings, and surrounding objects constrain traversable space, create clutter and occlusion, and affect how robots reach or manipulate targets.The introduction argues that household robots need both spatial understanding and the ability to act within these constraints.Source: §1 · p. 1

LIVIN-Scenes

  • LIVIN provides digital twins of 30 lived-in homes that preserve observed layouts, furniture, and everyday belongings, including small objects that create occlusions and constrain navigation and interaction.The benchmark contains 319 rooms and 12,808 object instances, including 1,559 articulated objects.Source: §1 · pp. 2–3

Benchmark and evaluation

  • Current 3D and embodied methods still struggle in complex household scenes, with failures ranging from missed objects and inaccurate geometry to navigation and long-horizon manipulation problems.Reported failures include difficulty localizing destinations, escaping obstacles, establishing stable grasps, and completing manipulation sequences.Source: §1 · p. 2

Construction workflow

  • A human-in-the-loop workflow reconstructs architecture and objects from captured observations, with stage-wise review and correction against the source homes.The workflow combines instance recognition, architectural reconstruction, and object generation and placement.Source: §1 · pp. 2–3

LIVIN-Bench

  • LIVIN-Bench evaluates 3D detection, 3D reconstruction, navigation, and loco-manipulation in realistic household environments.The tasks cover scene understanding, robot mobility, and physical interaction.Source: §1 · p. 3

2 RELATED WORK

Related work spans generated interiors, observation-grounded scene reconstruction, and embodied benchmarks. LIVIN extends these directions to complete lived-in homes with preserved layouts, object arrangements, and articulated fixtures.

Real-to-Sim Generation

  • Real-to-sim generation ranges from object-level asset recovery to scene-level generation and reconstruction; LIVIN extends this direction to complete lived-in homes through a human-in-the-loop workflow.Prior methods address detailed geometry and appearance, articulated assets, compositional environments, or scene reconstruction from visual observations.Source: §2 · p. 3

Indoor Scene Datasets

  • Indoor scene resources trade off manual effort, scalability, and correspondence to real homes; capture-grounded datasets may still omit instance-specific structure or complete object arrangements.LIVIN is grounded in individual lived-in homes and aligns agent-initialized reconstructions against real-world evidence.Source: §2 · p. 3

Navigation and Loco-manipulation Benchmarks

  • Embodied benchmarks use scanned, curated, constructed, or generated settings, while LIVIN preserves observed layouts, belongings, and articulated fixtures that jointly shape navigation and manipulation.The preserved constraints come from individual lived-in homes.Source: §2 · p. 3

3 LIVIN BENCHMARK

LIVIN comprises 30 digital twins of lived-in homes, assessed for correspondence to captured homes and preservation of household complexity. The authors report stronger user-study ratings and greater object richness than the compared datasets on the stated measures.

Dataset overview

  • The collection contains 30 home digital twins, 319 rooms, and 12,808 object instances, including 1,559 articulated objects.Scenes span household scales, room functions, furnishing styles, and object compositions.Source: §3 · p. 4

Real-world correspondence

  • LIVIN receives higher user-study ratings than compared datasets for physical plausibility, perceived realism, and reconstructed mesh quality.The scenes preserve architectural details, instance-specific meshes, and modeled movable parts and kinematic relationships.Source: §3 · p. 4

Reproducing household complexity

  • LIVIN has the highest object count and density among datasets with available statistics, and a larger articulated-object proportion than datasets reporting that metric.The authors interpret these results as evidence that LIVIN captures object richness and interaction-relevant structure in lived-in homes.Source: §3 · p. 4

4 REPLICA CONSTRUCTION WITH HUMAN VERIFICATION

The replica-construction pipeline starts from floorplans, posed multi-view images, and metric scans, and uses an agentic workflow with human inspection after each stage. Its stages cover object inventory, architectural reconstruction, object generation and placement, and review-driven correction.

4 REPLICA CONSTRUCTION WITH HUMAN VERIFICATION

  • The pipeline uses floorplans, posed multi-view images, and metric scans to build each digital replica, with human inspection and refinement after every stage.The three stages are instance recognition, architectural reconstruction, and object generation and placement.Source: §4 · p. 4

4.1 INSTANCE RECOGNITION

  • Instance recognition segments objects across posed views, matches detections across views to remove duplicates, and records labels, materials, rooms, articulation, and reference views.The agent revisits dense regions and uses complementary views to recover objects partly or fully occluded in individual images.Source: §4.1 · pp. 4–5

4.2 ARCHITECTURAL CONSTRUCTION

  • Architectural reconstruction begins with a coarse shell from floorplans, scans, and room metadata, then aligns it to images and refines openings and observed architectural details.The refinement includes features such as niches, beams, ceiling details, and configured window frames and glass.Source: §4.2 · p. 5

4.3 OBJECT GENERATION AND PLACEMENT

  • Articulated objects are jointly modeled and placed with geometry, materials, and kinematics, while non-articulated objects use Rodin-generated meshes placed to preserve observed spatial relations and avoid collisions.The articulated-object workflow represents functional components and movable parts; the non-articulated workflow estimates scale, pose, and placement from scene context and observations.Source: §4.3 · p. 5

4.4 HUMAN REVIEW AND REFINEMENT

  • Human reviewers inspect and refine each stage before proceeding, correcting recognition, geometry, articulation, appearance, and placement errors to limit error propagation.Defective non-articulated objects can be regenerated, while object poses and placements can be adjusted through the review interface.Source: §4.4 · p. 5

5 3D DETECTION AND RECONSTRUCTION

LIVIN evaluates 3D detection and reconstruction in dense household scenes, where results show that object recovery and geometric or appearance fidelity remain challenging.

5 3D DETECTION AND RECONSTRUCTION

  • Figure 2 compares four detection methods and five reconstruction methods with ground truth, marking correct, missed, and incorrect detections in green, yellow, and red.The figure’s lower panel shows the reconstructions alongside ground truth.Source: Figure 2 · p. 6

5.1 3D OBJECT DETECTION

  • Detection is evaluated on 268 rooms from 30 homes using released checkpoints without fine-tuning, with method-specific inputs derived from rendered RGB-D observations.The detection task assesses object localization under clutter, occlusion, and dense household arrangements.Source: §5 · p. 5, §5.1 · p. 6
  • SpatialLM has the highest precision and matched-box IoU, but only 0.111 recall at IoU 0.25; Boxer has the highest F1, yet object recovery remains incomplete.The evaluation compares class-agnostic room-level predictions against visible-object oriented boxes.Source: §5.1 · p. 7

5.2 3D SCENE RECONSTRUCTION

  • ShapeR and Fire3D achieve higher recall and geometric accuracy overall, while SAM3D, RecGen, and Gen3DSR trade off geometry and appearance differently.ShapeR and Fire3D receive additional ground-truth inputs; Gen3DSR has the strongest PSNR, SSIM, and LPIPS among the other three methods.Source: §5.2 · p. 7
  • Reconstruction is evaluated on 128 views with RGB, ground-truth instance masks, depth, and camera calibration, while some methods also receive ground-truth boxes or other priors.ShapeR additionally uses depth-derived point clouds and object descriptions.Source: §5.2 · p. 7

6 NAVIGATION AND LOCO-MANIPULATION

LIVIN evaluates navigation and loco-manipulation in faithful replicas of lived-in homes with naturally occurring spatial constraints. Results show that current methods still struggle with route completion and physical interaction.

6 NAVIGATION AND LOCO-MANIPULATION

  • LIVIN tests navigation and interaction in complete lived-in homes, preserving naturally occurring spatial constraints and object arrangements.The benchmark includes narrow passages, household fixtures, and manipulation across room types.Source: §6 · p. 7

6.1 NAVIGATION

  • Navigation is framed as a long-horizon vision-language task requiring route following from visual observations and physically feasible whole-body motion.The methods are evaluated without training or fine-tuning on LIVIN.Source: §6.1 · p. 7
  • GPT-6 Astra leads navigation with a 41.48% success rate, the highest SPL, and the lowest navigation error, but can still fail in complex or tightly constrained spaces.OmniNav sometimes becomes stuck in collision regions, whereas DualVLN more often drifts from the intended route.Source: §6.1 · p. 8

6.2 LOCO-MANIPULATION

  • Loco-manipulation spans grasping and placement, pushing and pulling, articulated-object manipulation, and long-horizon tasks with dependent subgoals.Long-horizon completion remains limited: GPT-6 Astra achieves 12.5% success and a mean process score of 15.9.Source: §6.2 · pp. 8–9
  • GPT-6 Astra achieves the highest loco-manipulation success rates across all four task families, while DeepSeek-V4.1-Flash has limited success and Ψ0 completes no tasks in the evaluated zero-shot configuration.GPT-6 Astra’s success rate is 61.9% for articulated-object manipulation and 24.4% for grasping and placement.Source: §6.2 · pp. 8–9

7 CONCLUSION

LIVIN benchmarks spatial and embodied intelligence in digital twins of lived-in homes, spanning perception, reconstruction, navigation, and loco-manipulation.

7 CONCLUSION

  • LIVIN preserves real-world scene structure, object arrangements, and articulation across four evaluation areas: 3D perception, reconstruction, navigation, and loco-manipulation.Source: §7 · p. 9
  • Current methods still struggle with dense object arrangements, frequent occlusion, and physically constrained interactions in realistic household environments.Source: §7 · p. 9
  • The authors present LIVIN as a testbed for progress toward more robust and generalizable embodied systems.Source: §7 · p. 9

A DATASET STATISTICS

The dataset statistics and visualizations characterize LIVIN’s varied room types, object arrangements, and articulated household objects.

A DATASET STATISTICS

B.1 INPUT DATA AND SPATIAL REFERENCES

LIVIN homes are reconstructed from aligned floorplans, metric scans, and posed images through staged generation and human review. The workflow combines automated modeling with targeted corrections to improve reconstruction quality while controlling computational cost.

B.1 INPUT DATA AND SPATIAL REFERENCES

  • Floorplans supply room organization, scans provide coarse 3D geometry and scale, and posed calibrated images provide object appearance and spatial evidence, all aligned in a common coordinate system.Structural metadata, including room boundaries, wall geometry, openings, and connectivity, constrains the initial architecture.Source: §B.1 · p. 16, Figure 8 · p. 16

B.2 SCENE-TO-ASSETS ANNOTATION

  • The annotation stage identifies instances across views, labels articulation and composition with confidence and evidence status, and prepares reference images for asset generation.Dense or occluded regions are checked with enlarged crops or complementary views; hidden surfaces in generated references remain inferences checked against capture evidence.Source: §B.2 · pp. 16–17

B.3 RODIN GENERATION

  • Rodin generates non-articulated assets from selected reference images, with generation settings and identifiers recorded to trace later replacements.Human-guided replacement and refinement are part of the subsequent correction process.Source: §B.3 · p. 17, Table 5 · p. 17

B.4 BLENDER-BASED SCENE GENERATION

  • The Blender workflow builds architecture from room metadata and scan alignment, then models articulated objects in context with movable parts, kinematics, and motion previews.A coordinating skill assigns room-level tasks, while single feed-forward agent passes leave remaining errors for localized human correction rather than repeated self-refinement.Source: §B.4 · p. 17, Table 6 · p. 18

C.1 REVIEW AND CORRECTION WORKFLOW

  • Reviewers correct instance inventories and architectural structure before separately inspecting object-level reconstructions, using different correction strategies for articulated and non-articulated objects.Architecture and articulated objects allow at most two agent repair rounds; non-articulated objects use Rodin regeneration or manual pose adjustment without further agent rounds.Source: §C.1 · p. 18, Table 7 · p. 19

B.5 CONSTRUCTION ABLATIONS

  • Single-Round and Iterative Refinement each require over 100 million tokens, versus just over 70 million for the human-guided workflow, which improves plausibility and mesh quality while reducing penetrations.The reported gains arise from generation strategies and human verification, including Blender-based preservation of articulated structure and Rodin geometry for non-articulated objects.Source: §B.5 · p. 18, Figure 9 · p. 19

C.2 RECONSTRUCTION REVIEW INTERFACE

  • The review interface compares models with captured views and supports local collision inspection, pose editing, axis-based contact snapping, articulation previews, material inspection, and generation controls.Snapping refines contact position and offsets the final position by 2 mm to reduce mesh penetration; object selection loads nearby geometry and collision partners rather than the full scene.Source: §C.2 · pp. 20–21, Figure 10 · p. 20

D 3D SCENE RECONSTRUCTION

The section evaluates reconstruction under clutter and specifies navigation implementation and measurement protocols. Small-object reconstruction remains difficult, while navigation success is defined using a strict proximity threshold suited to subsequent interaction.

D 3D SCENE RECONSTRUCTION

  • Reconstruction uses CD-S for symmetric surface distance, a 0.05 m threshold for F-Score-S, and a 0.25 box-matching threshold for object-level F1.Appearance metrics are computed on full rendered images under common cameras and lighting; ShapeR is excluded because it produces untextured geometry.Source: D 3D SCENE RECONSTRUCTION · p. 21
  • The clutter subset filters the 128-view test set for clutter-related, physically and visibly small targets, yielding 306 object-view records for 287 objects across 76 views.The selected objects are illustrated in the small-clutter examples.Source: D 3D SCENE RECONSTRUCTION · p. 21, Figure 11 · p. 22
  • ShapeR achieves full output coverage and substantially higher box recall and surface F-score on the clutter subset, while Fire3D has lower coverage and Gen3DSR leaves more than half of selected records without output.Most methods show a marked drop in 3D box recall under cluttered small-object conditions, and all leave room for improvement.Source: D 3D SCENE RECONSTRUCTION · p. 21

E.1 IMPLEMENTATION DETAILS

  • Navigation methods use shared GR00T-based control, while OmniNav supplies waypoints and headings and DualVLN replans from visual history through one-second local subgoal execution.GPT-6 Astra outputs planar velocity, yaw rate, and execution duration, alternating action execution with new observations.Source: §E.1 · pp. 21–22

E.2 EXPERIMENTAL SETTING

  • Navigation uses 45 tasks with manually selected starts and goals, reviewed route instructions, and three trials per task, yielding 135 episodes per system.Episodes allow 240 seconds of simulation time and 900 seconds of total runtime after stabilization.Source: §E.2 · p. 22
  • A navigation execution succeeds only when the system explicitly issues STOP within 1 m of the destination, a strict threshold chosen for interaction proximity in dense homes.Navigation error measures final navigable distance to the goal on the evaluation NavMesh.Source: §E.2 · p. 23
  • SPL relates traveled trajectory length to the NavMesh reference path, while collision counts merge simultaneous-link contacts and intervals separated by less than 0.1 seconds.Collision geometry uses a voxel resolution of 2.5 cm.Source: §E.2 · p. 23

E.3 FURTHER ANALYSIS OF NAVIGATION PERFORMANCE

Failure analysis distinguishes blockage, falls, goal-completion mismatches, and other failures. A mixed obstacle-avoidance strategy can traverse a narrow passage, but its overall success rate is lower than GPT-6 Astra+GR00T.

E.3 FURTHER ANALYSIS OF NAVIGATION PERFORMANCE

  • OmniNav failures are dominated by unrecovered blockage at 69.05%, while DualVLN and GPT-6 Astra show many goal-completion mismatches; Astra’s failed episodes have lower navigation error than DualVLN’s.The lower error suggests many Astra failures occur near the destination and involve final stopping-region judgments rather than large route deviations.Source: §E.3 · p. 23
  • 34.81% is the mixed GPT-6 Astra+CAT+GR00T success rate, below GPT-6 Astra+GR00T, despite CAT enabling traversal through a constrained passage.Sideways CAT traversal can misalign body orientation and head-camera observations with motion, potentially affecting subsequent instruction following and turning decisions.Source: §E.3 · p. 24, Figure 12 · p. 24

F.1 TASK INVENTORY

The loco-manipulation evaluation spans household interaction tasks in simulated homes using a humanoid embodiment and several evaluated control approaches. It measures trial success and prerequisite-aware progress on long-horizon tasks.

F.1 TASK INVENTORY

  • The inventory contains 35 tasks across 30 homes, covering grasping and placement, pushing and pulling, articulated-object manipulation, and long-horizon dependent subgoals.Task instructions are listed in the inventory table.Source: §F.1 · p. 24, Table 10 · p. 25

F.2 SIMULATION AND EMBODIMENT

  • Tasks use a simulated Unitree G1 with 29 body joints and two 7-DoF Dex3 hands, MuJoCo physics, Isaac Sim rendering, and a 5 ms physics step.Simulation pauses during model inference and resumes for action execution.Source: §F.2 · p. 26

F.3 EVALUATED METHODS

  • DeepSeek-V4.1-Flash and GPT-6 Astra share a turn-based tool interface, instructions, and controller, while Ψ0 is evaluated with released checkpoints and native controllers without LIVIN fine-tuning.The two models receive task instructions, robot state, and head-camera images and use medium reasoning effort.Source: §F.3 · p. 26

F.4 SCENE AND OBJECT MODELLING

  • Scene meshes are exported from Blender, collision geometry uses adaptive convex decomposition, and authors review physical properties initially proposed by GPT-6 Astra.The reviewed properties include mass, centre of mass, inertia tensor, and contact friction.Source: §F.4 · p. 26

F.5 EVALUATION METRICS

  • Success rate measures trials meeting task-family criteria, while long-horizon process score measures the percentage of equally weighted physical subgoals completed with prerequisites satisfied.Mean process score is reported across trials on a 0–100 scale.Source: §F.5 · p. 26
Loading 2610.12069v1…