PaperlayerView exact revision on arXiv

Source-linked AI summary

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, Roozbeh Mottaghi

arXiv:2310.13724v1cs.HCcs.AIcs.CVcs.GRcs.MAcs.RO

TL;DR

Embodied-AI agents have seldom been studied as collaborators sharing dynamically changing environments with humans, while real-human hardware evaluation is costly and difficult to standardize. Habitat 3.0 provides humanoid and robot simulation, human-in-the-loop control, and social-task benchmarks; learned policies collaborate efficiently with unseen humanoids and human partners, while humanoid motion coverage remains limited.

  • Problem

    Embodied-AI research largely assumes solitary agents, while real-human hardware studies face safety, scalability, cost, and benchmarking challenges.

  • Method

    Habitat 3.0 combines humanoid and robot simulation, mouse/keyboard or VR human-in-the-loop control, and Social Navigation and Social Rearrangement benchmarks.

  • Results

    Learned policies complete collaborative tasks efficiently with unseen humanoid agents and human partners, exhibiting behaviors such as yielding space; automated evaluation indicates relative policy ordering with real humans.

  • Takeaways & Limitations

    Habitat 3.0 supports in-depth study and evaluation of social human-robot collaboration in home-like embodied environments.

  • Takeaways & Limitations

    Experiments use only walking and reaching, while more complex motions such as opening cabinets or sitting remain future work.

Abstract

from arXiv · show

We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.

1 INTRODUCTION

Habitat 3.0 addresses the limits of solitary embodied-AI simulations by supporting humanoid avatars, robots, and collaborative human-robot tasks in home-like environments. It combines humanoid simulation, human-in-the-loop evaluation, and standardized social-task benchmarks.

  • Existing embodied-AI environments largely assume a solitary agent causes all environmental changes, unlike assistive robots intended to share dynamically changing spaces with humans.
  • Real-human hardware evaluation is constrained by safety, scalability, cost, and standardized-benchmarking challenges, motivating simulation-based study.
  • Habitat 3.0 simulates humanoid avatars and robots in collaborative human-robot tasks within home-like environments.
  • Humanoid simulation combines articulated skeletons, skinned meshes, parameterized bodies, diverse avatar models, and programmatic motion and behavior generation.The system uses 12 base avatar models with multiple gender representations, body shapes, and appearances; reported speed is 1190 FPS with a humanoid and robot versus 1345 FPS with two robots.
  • The human-in-the-loop tool lets real collaborators control humanoid avatars through mouse/keyboard or VR while evaluating autonomous robot policies.
  • Social Navigation requires a robot to find and follow a humanoid safely, while Social Rearrangement requires coordinated pick-and-place actions toward shared goals.Automated evaluations use learned or scripted humanoid and robot policies; human-in-the-loop evaluation lets real humans control the humanoid avatar.
  • Learned policies generalize to unseen scenes and collaborators, yielding space in navigation and improving rearrangement efficiency for humanoids and real human partners.
  • Habitat 3.0 provides a platform for in-depth studies of social human-robot tasks and aims to encourage broader Embodied-AI research in this direction.

2 RELATED WORK

Related work has established embodied simulators, human-in-the-loop methods, and multi-agent environments, but Habitat 3.0 targets interactive human-robot collaboration in rich embodied settings.

  • Existing embodied-AI simulators support embodied tasks, while Habitat 3.0 differs by simulating humans alongside robots.
  • Prior human-in-the-loop Embodied-AI studies often use simplified grid or low-dimensional environments and focus on offline evaluation rather than interactive evaluation.
  • Humanoid avatars include diverse appearances and realistic walking and reaching behaviors that adapt to different indoor layouts.
  • Habitat 3.0 presents a human-in-the-loop framework for interactive evaluation and data collection.
  • Most multi-agent reinforcement-learning environments operate in low-dimensional spaces such as grid mazes, tabular tasks, or Markov games.

3 HABITAT 3.0 FRAMEWORK

Habitat 3.0 combines fast, diverse humanoid simulation with human-in-the-loop control and replayable evaluation infrastructure for collaborative embodied tasks.

  • Humanoid simulation: The framework’s humanoid models target fast, diverse, and realistic human simulation for training robots to interact with people.
  • Humanoid simulation: Articulated skeletons support pose representation and collision checking, while skinned meshes provide visual fidelity without affecting physics.
  • Humanoid simulation: Caching four male, four female, and four neutral body shapes moves model-rig and blend-shape computation offline, requiring only linear blend skinning during simulation.
  • Humanoid motion and behavior: A hierarchical behavior model combines learned policies or planners with low-level skills to generate long-range humanoid navigation and interaction behavior.
  • Humanoid motion and behavior: Navigation uses planned waypoints, rigid base rotation, and cyclic walking motion from AMASS, while picking uses VPoser-based motion generation.
  • Benchmarking: A single-environment robot runs at 245±19 FPS, compared with 188±2 FPS for the humanoid, with the difference attributed primarily to the humanoid’s larger joint count.
  • Human-in-the-loop infrastructure: The HITL platform supports online interaction evaluations and data collection by letting operators control simulated humanoids through mouse/keyboard or VR interfaces.
  • Human-in-the-loop infrastructure: The platform records and replays HITL episodes at action, trajectory, and camera-rendering levels to support data collection and reproducibility.

4 INTERACTIVE HUMAN-ROBOT TASKS

Habitat 3.0 studies collaborative human-robot interaction through Social Navigation and Social Rearrangement, evaluating heuristic and learned policies with humanoid and human partners. Results show learned policies can generalize to unseen collaborators, exhibit cooperative behaviors, and produce policy orderings in human-in-the-loop evaluation that match automated evaluation.

  • Social Rearrangement: Social Rearrangement trains population-based robot policies to coordinate with collaborators while moving objects from initial to target locations.The study compares known-partner and unseen-partner evaluation across training populations, including Learn-Single, Learn-pop, and Plan-pop variants.
  • Social Navigation: Social Navigation requires a robot to find and follow a humanoid in unseen environments while maintaining a safe distance.The task uses arm depth, a humanoid detector, and humanoid GPS; metrics include finding success, path-weighted finding success, and following rate.
  • Social Navigation: The heuristic expert reaches 100% finding success versus 97% for end-to-end RL, while RL achieves a similar collision rate without map access.Collision rate is 0.52 for the heuristic and 0.51±0.03 for RL; finding SPS and following rate remain competitive for RL.
  • Social Navigation: End-to-end RL learns to anticipate humanoid motion, back up to avoid collisions, and yield space in narrow passages while following.These behaviors support following despite the policy lacking a map of the environment.
  • Social Navigation: Social-navigation ablations show humanoid GPS matters most before finding the humanoid, whereas arm cameras matter most during collision avoidance after finding it.Removing humanoid GPS lowers finding success and SPS; removing arm cameras increases collision rate and lowers following rate.
  • Social Rearrangement: Plan-pop3 and Plan-pop4 achieve the highest unseen-collaborator success rate of 71.7%, while single-partner policies suffer large success drops on unseen partners.Learn-Single falls from 98.50% to 50.9%, Plan-Pop1 from 91.2% to 50.4%, and Learn-pop from 92.2% to 48.5% between matched and unseen evaluation.
  • Human-in-the-Loop Evaluation: With real human partners, Learn-Single and Plan-Pop3 improve relative efficiency to 134% and 123%, respectively, compared with humans working alone.Plan-Pop3 produces higher robot object contribution despite lower relative efficiency than Learn-Single.
  • Human-in-the-Loop Evaluation: Automated humanoid evaluation preserves the relative ordering of policies observed with human collaborators, although simulated zero-shot collaborators do not capture human interaction dynamics accurately.Learn-Single is more efficient for partners, while Plan-Pop3 produces higher task offloading in both evaluation settings.

5 CONCLUSION

Habitat 3.0 combines fast humanoid-and-robot simulation, human-in-the-loop evaluation, and collaborative social tasks. Its social-navigation setup trains robots to find and follow humanoids while maintaining safe distances, with reward shaping and controlled evaluation conditions.

  • 5 CONCLUSION: Habitat 3.0 supports humanoid and robot simulation, human-in-the-loop control through mouse/keyboard or VR, and automated or human evaluation of social tasks.The simulator is designed for rich indoor scenes and supports diverse humanoid appearances and motions.
  • A.1 SOCIAL NAVIGATION: Social navigation rewards the robot for approaching and following the humanoid while encouraging it to maintain a 1–2m safety distance and face the humanoid.The reward instead encourages moving away when the humanoid is closer than 1m.
  • A.1 SOCIAL NAVIGATION: Training terminates on humanoid-robot collision, while evaluation uses 1500 steps, a 5m minimum initial separation, and fixed initial conditions across baselines.These settings are intended to ensure fair baseline comparison.
  • A.1 SOCIAL NAVIGATION: The social-navigation policy uses linear and angular velocity actions scaled to −10 and +10, corresponding to 2.5m/s in the real world.Both humanoid and robot use the same maximum linear and angular velocities.
  • A.1 SOCIAL NAVIGATION: Training curves show that the end-to-end RL policy improves reward while minimizing distance to the humanoid for finding and following.Figure 5 reports average distance and reward over simulation steps using three seeds per model.

A.2 SOCIAL REARRANGEMENT

Social rearrangement uses a two-layer policy in which a learned high-level controller selects object and skill combinations. Training curves compare baselines and ablations, while learned low-level skills expose a stated robustness limitation.

  • A.2 SOCIAL REARRANGEMENT: Social rearrangement baselines use DD-PPO training across parallel environments, with results averaged over three random seeds.The reported setup trains policies for 100M environment steps.
  • A.2 SOCIAL REARRANGEMENT: The rearrangement reward combines success, subgoal, collision, and per-step terms: 10·success + 5·subgoal − 5·collision − 0.005.This reward is the final social rearrangement objective.
  • A.2 SOCIAL REARRANGEMENT: Learn-Single and Plan-pop1 learn faster than Plan-pop2,3,4, while removing training sensors slows learning and removing primitive actions has the largest effect.Figure 6 reports success and reward curves for baselines and ablations, averaged over three seeds.
  • A.2 SOCIAL REARRANGEMENT: The two-layer architecture combines a learned high-level policy with a fixed library of low-level navigation and manipulation skills.The high-level action space selects feasible skill and object or receptacle combinations.
  • A.2 SOCIAL REARRANGEMENT: Using learned low-level skills causes a considerable performance drop because the high-level policy remains frozen after training with oracle skills.The paper suggests training or fine-tuning the high-level policy with learned skills in the loop as future work.

B LOW-LEVEL SKILL TRAINING

Habitat 3.0 trains separate neural navigation, pick, and place skills that are coordinated by a high-level policy. These skills use distance-, orientation-, success-, collision-, and time-based rewards with explicit termination conditions.

  • B LOW-LEVEL SKILL TRAINING: The low-level skill module contains navigation, pick, and place controllers coordinated by a high-level policy.The controllers are neural-network policies trained for specific object-finding and manipulation subtasks.
  • B LOW-LEVEL SKILL TRAINING: Navigation rewards progress toward the target, orientation toward it, successful arrival within 1.5m, collision avoidance, and faster completion.The success condition also requires a forward-vector dot product greater than 0.5.
  • B LOW-LEVEL SKILL TRAINING: Navigation training terminates on object finding, 1500 steps, or collision with a randomly walking humanoid, with the robot initialized at least 4m from the target.The trained navigation skill reaches 90% success after about 300 million simulation steps, or roughly six days on the stated hardware.
  • B LOW-LEVEL SKILL TRAINING: The low-level controllers use neural-network policies, with “policy” and “skill” used interchangeably for neural-network-parameterized controllers.The navigation skill is distinct from social navigation because it need not continuously follow a humanoid while avoiding collision.
  • B LOW-LEVEL SKILL TRAINING: Pick rewards approaching and retracting from the object, gives +2 for successful pickup and retraction, and applies a −5 × 10^3 slack reward.Episodes terminate on wrong pickup, dropping the object, reaching 1250 steps, or successful completion.
  • B LOW-LEVEL SKILL TRAINING: Place rewards approaching and retracting from the target, gives +5 for correct placement and +10 for successful placement and retraction, and applies a −5 × 10^3 slack reward.Success requires the end-effector to be within 0.15m of the target end-effector location.

C.1.1 ADDITIONAL METRICS

The section introduces three Social Navigation metrics that quantify yielding behavior, inter-agent distance, and following distance. It also gives target interpretations for these measurements.

  • Backup-Yield Rate measures how often the robot backs up or yields to avoid collision near the humanoid.A backup motion is backward movement when the agents are less than 1.5 meters apart.
  • Total Distance measures the L2 distance between robot and humanoid across all episode steps.
  • Following Distance measures the L2 distance between robot and humanoid after the robot first finds the humanoid.
  • Ideally, Following Distance is 1-2m, while policies with higher SPS have lower Total Distance.

C.1.2 ADDITIONAL ANALYSIS

The additional analysis examines end-to-end RL components for Social Navigation. It finds that visual detection and arm depth support following and collision avoidance, while a performance gap remains against the heuristic expert.

  • Removing humanoid GPS makes the robot find the humanoid less efficiently and lose track when the humanoid enters another room.The ablation uses random walking or scanning until the humanoid is visible, producing lower SPS and following rate.
  • Removing the humanoid detector causes a 7% drop in following performance and a 13% increase in collision rate.GPS alone cannot reveal whether a wall or obstacle lies between the robot and humanoid.
  • Removing arm depth produces the highest collision rate, whereas arm depth or arm RGB lowers collision rate.Arm depth helps record empty space and obstacles for collision avoidance through LSTM memory.
  • Replacing arm depth with arm RGB slightly increases collision rate because RGB lacks direct obstacle-distance information.
  • RL and its ablations learn comparable backup-yield rates and maintain close following distances, but trail the Heuristic Expert in SPS and following rate.The remaining gap leaves room for future improvement.

C.2.1 ADDITIONAL METRICS

The section adds collision and completion-sharing metrics for Social Rearrangement and describes an ablation removing primitive navigation actions from the high-level policy.

  • Collision Rate is the proportion of episodes containing collisions between the robot and humanoid.It is measured for both training-population and zero-shot-coordination population settings.
  • Plan-Pop1’s collision rate rises from 0.09 to 0.23 when evaluated with the zero-shot population.Plan-Pop4 reaches 9% in the training population, while Plan-Pop2 has the lowest zero-shot collision rate.
  • Ratio of Completion measures the proportion of objects rearranged by the robot, with values near 0.5 indicating task sharing.A value of 1.0 means the robot completes the task, while 0.0 means the humanoid does.
  • The additional ablation removes primitive actions such as moving backward, moving forward, and turning from the high-level policy action space.This tests the effect of primitive navigation actions in Social Rearrangement.

C.2.3 ZERO-SHOT POPULATION DETAILS

The zero-shot population evaluation measures baseline coordination across ten learned and planner-based humanoid partners. Results show that specialization creates high variance, while population training reduces it.

  • The zero-shot evaluation population contains 10 collaborators: six learned checkpoints and four planner-based humanoids with varied object-picking behaviors.One planner-based humanoid stays still, while others pick up one or both objects.
  • Figures 8a–8d report success rate, efficiency rate, collision rate, and ratio of completion across baseline rows and evaluation-agent columns.Each row averages a baseline across three seeds, and the final column shows zero-shot validation results.
  • Learn-Single specializes in one object, yielding success rates near 33% or 66% depending on whether its partner’s specialization complements it.
  • Plan-Pop1 achieves success rates near 100% or 0% across partners and the highest average relative efficiency despite high variance.
  • Plan-Pop2 reduces variance across zero-shot partners by training on randomized object assignments, but adapts at lower peak efficiency.
  • Plan-Pop3 and Plan-Pop4 perform similarly across the zero-shot population, with slight success-rate improvement over Plan-Pop2 and no significant effect from adding a stationary agent.An example episode shows agents splitting objects and the robot backing up to let the humanoid pass.

F.1 BENCHMARKING

Habitat 3.0 benchmarking measures simulation speed across scene sizes, object counts, and agent configurations, including humanoid–robot settings and parallel environments. The results show strong throughput, while cross-simulator speed comparisons remain approximate because hardware and simulation settings differ.

  • Benchmark setup: Benchmarking varies scene size, object count, agent type and count, using random actions across single and 16 parallel environments on one Nvidia V100 GPU.Results report averages and standard errors across 10 runs.
  • Single-environment speed: 140–250 FPS is achieved in single-environment settings, with scene size and additional agents affecting speed more than object count.Performance decreases from 245±19 FPS to 154±14 FPS with scene-size changes, and from 245±19 to 150±1 FPS with two Spot agents.
  • Agent configuration: 150±1 FPS for Robot-Robot versus 136±1 FPS for Human-Robot shows comparable two-agent performance despite the humanoid’s greater joint count.A single humanoid is slower than a single Spot robot, but the difference narrows in the two-agent setting.
  • Parallel scaling: 1100–2290 FPS is obtained with 16 parallel environments on a single GPU, with Habitat 3.0 supporting native environment parallelization.The reported range depends on the scenario.
  • Simulator capabilities: Habitat 3.0 supports real robot models and humanoids, using SMPL-X humanoids to expand body-model diversity and motion support.The comparison with related simulators includes body models, control types, HITL support and speed.
  • Comparison caveat: Speed values across simulators are rough reference points rather than directly comparable measurements because rendering, physics, resolution and hardware differ.The comparison uses reported single-environment numbers from individual papers.

G HUMAN-IN-THE-LOOP (HITL) EVALUATION

The human-in-the-loop study evaluates robot collaboration with real users across solo and two robot-assisted conditions. Pairing users with either learned or planned agents significantly differs from solo performance, while the two robot conditions do not significantly differ from each other.

  • Study design: Thirty participants completed solo, Learn-Single, and Plan-Pop3 conditions, with each condition performed for 10 episodes in one test scene.Condition order was randomized and learning effects were addressed with Latin-square counter-balancing.
  • Measures: The study measures collision rate, task completion steps, and relative efficiency across episodes.Relative efficiency cannot be computed per episode in HITL evaluation because repeated human performance changes over time.
  • Analysis: A Poisson generalized linear mixed-effects model analyzes task completion steps while controlling for participant and scene variation with random intercepts.Post hoc pairwise comparisons are then computed across the three conditions.
  • Results: The estimated task-completion-step difference between Learn-Single and Plan-Pop3 is not significant, whereas both differ significantly from solo performance.The pairwise comparisons are reported in Table 5.
  • Interpretation: The study provides an indication that learned collaboration agents can improve human efficiency in a realistic everyday rearrangement task.The authors describe this as the first analysis of learned collaboration agents paired with real human partners in this setting.

H LIMITATIONS

The limitations concern restricted humanoid behaviors, visual artifacts in motion and skinning, and a gap between human and automated-evaluation observations.

  • Human simulation: Experiments use only walking and reaching, so the current motion approach may not suit complex behaviors such as opening cabinets or sitting down.Reaching uses interpolated static poses, while waypoint-based walking can produce rotation artifacts.
  • Human simulation: Fixed linear blend-skinning weights can produce artifacts for certain humanoid pose configurations.The limitation follows from using static weight assignments across poses.
  • Human-in-the-loop tool: The HITL tool provides humans with observations that differ from those available to humanoids in automated evaluations.The authors provide visual cues to reduce this gap.
Loading 2310.13724v1…