PaperlayerView exact revision on arXiv

Source-linked AI summary

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra

arXiv:2106.14405v2cs.LGcs.RO

TL;DR

Home assistants must manipulate and rearrange objects in interactive, physics-enabled homes, while static 3D scans cannot support such tasks. The paper introduces ReplicaCAD, Habitat 2.0, and HAB, enabling systematic study of long-horizon embodied-AI policies and showing stronger results for hierarchical than flat RL.

  • Problem

    Static 3D scans cannot support rearrangement tasks because their objects cannot be moved or manipulated, limiting study of home-assistant robots.

  • Method

    The paper builds the interactive ReplicaCAD dataset, the physics-enabled Habitat 2.0 simulator, and the Home Assistant Benchmark for mobile-manipulation rearrangement tasks.

  • Results

    Hierarchical RL performs significantly stronger than flat RL on long-horizon HAB tasks, while independent skills encounter hand-off problems.

  • Takeaways & Limitations

    ReplicaCAD and Habitat 2.0 enable efficient experimentation with embodied agents rearranging richly interactive 3D environments.

  • Takeaways & Limitations

    ReplicaCAD models apartments from one country, so regional and cultural variation in layouts, furniture, and objects is not represented.

Abstract

from arXiv · show

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.

1 Introduction

Habitat 2.0 advances embodied-AI research through interactive data, fast physics-enabled simulation, and challenging home-assistance benchmarks. These resources support comparisons of RL and SPA methods, revealing strengths and limitations of flat policies, independent hierarchies, and classical planning pipelines.

  • Motivation: Hardware training for home assistants is slow, expensive, and difficult to reproduce, motivating simulation-based research across data, simulation, and benchmark tasks.The agenda emphasizes generalization to unseen objects, receptacles, and home layouts, plus reproducible progress tracking.
  • Contributions: H2.0 is a physics-enabled simulator supporting articulated robots, piecewise-rigid objects, and rigid-body mechanics while prioritizing simulation speed over capability breadth.Its scope excludes non-rigid dynamics, physical state transformations, audio, and tactile sensing; one reported configuration reaches 1200 simulation steps per second.
  • Contributions: HAB evaluates mobile manipulation through TidyHouse, PrepareGroceries, and SetTable tasks that rearrange objects among receptacles and articulated containers.Success requires placing all target objects within 15cm of desired positions using onboard sensing without privileged state information.
  • Findings: Flat RL learns individual skills but struggles to chain them for long-horizon HAB tasks, whereas a hierarchy using a perfect STRIPS task planner performs significantly better.The comparison emphasizes systematic generalization to new objects, receptacles, and apartment layouts.
  • Findings: Independent hierarchical skills suffer hand-off problems, while SPA is more brittle than RL because clutter and imperfect navigation can produce incorrect plans.For simple skills, SPA performs as well as monolithic RL, but it requires mapping all workspace obstacles for planning.
  • Implications: H2.0 is released as free, open-source software under the MIT license and is intended to reduce reliance on commercial lock-ins and non-photorealistic simulators.The platform is presented as a benchmark for future embodied-AI research.

2 Related Work

The related work frames simulation as coupled physics and rendering, contrasts interactive and largely static environments, and motivates simulators optimized for AI workloads. It also positions this paper around visual perception, state-free operation, systematic generalization, and home-scale mobile manipulation.

  • Simulation: A simulator evolves world state through physics and generates sensor observations through rendering, with some systems tightly integrating both components.The passage distinguishes physics engines, renderers, game engines, and integrated simulators.
  • Simulation: Many prior simulators use largely static environments where agents move but cannot change states such as opening cabinets.These systems therefore emphasize rendering while providing relatively lightweight physics.
  • Object rearrangement: H2.0 retains rigid-body grasping while simplifying grasp acquisition at the task level because close-range grasping and release remain open research problems.The abstraction can be undone, but the paper defers fully integrated grasping to more mature fixed-base manipulation research.
  • Simulation: Game engines target human-oriented high-resolution display at roughly 60 FPS, whereas AI workloads may require 10k+ FPS, low resolution, and headless cluster deployment.The paper notes that game engines offer substantial engineering knowledge but limited control over AI-relevant performance characteristics.
  • Object rearrangement: Relative to prior rearrangement work, this study combines visual perception, no privileged state, systematic generalization, and visually complex home-scale environments.It uses abstracted grasping but not abstract manipulation, retaining gross motor-control challenges.

3 Replica to ReplicaCAD: Creating Interactive Digital Twins of Real Spaces

ReplicaCAD converts static, photorealistic apartment scans into interactive digital twins suitable for rearrangement research. Artist-authored assets, articulated objects, layout variation, and procedural clutter enable controlled generalization studies.

  • Motivation: Static Replica scans cannot support rearrangement because their objects cannot be moved or manipulated.ReplicaCAD addresses this limitation by recreating the apartment spaces as fully interactive environments.
  • Integration: Figure 2 contrasts the original Replica scene with the artist-recreated ReplicaCAD scene, where furniture, mugs, drawers, and fridges are physically simulated and interactive.The visual comparison highlights the transition from static reconstruction to manipulable digital twin.
  • Asset Creation: ReplicaCAD recreates 92 objects across six rooms with geometry, textures, materials, physical parameters, collision proxies, and semantic annotations.The asset pipeline uses artist-authored models based on the FRL-apartment spaces.
  • Asset Creation: Articulated objects receive segmented subparts and URDF-defined joint configurations and dynamics, while furniture receives surfaces and containment volumes for object placement.Examples include hinged refrigerator doors, cabinet joints, tabletops, and drawer spaces.
  • Layout Generation: Five macro variations and 20 micro variations produced 105 layouts with semantically meaningful changes in furniture placement and scene layout.These variations enable controlled testing of generalization.
  • Clutter Generation: ReplicaCAD scenes procedurally populate annotated surfaces and containment volumes with category-appropriate objects from ReplicaCAD or YCB.This pipeline increases clutter diversity while reusing the authored assets.
  • Integration: ReplicaCAD is fully integrated with H2.0 and supports simple import, instancing, and programmatic alteration of interactive scenes.The dataset represents more than 900 person-hours of professional artist effort and is shared under CC-BY-NC terms.

4 Habitat 2.0 (H2.0): a Lazy Simulator

H2.0 prioritizes simulation speed through localized physics and rendering, interleaving computation, and asset reuse. Its benchmarks show substantial throughput gains while supporting interactive robot scenarios and classical motion-planning comparisons.

  • 4.1–4.3 Design Principles: H2.0 prioritizes speed over simulation breadth through localized physics and rendering, interleaved computation, and simplify-and-reuse.These principles avoid simulating distant or unnecessary scene components and reduce CPU-GPU idle time.
  • 4.2 Interleaved Physics and Rendering: One-step observation latency lets H2.0 generate the next physics state on the CPU while rendering the current observation on the GPU.The policy uses π(a_t | o_t−1) rather than π(a_t | o_t), breaking the sequential dependency between sensing, action, and state transition.
  • 4.4 Benchmarking: H2.0 benchmarks use Fetch with two 128×128 RGB-D cameras, four 120 Hz physics steps per 30 FPS simulation step, and Idle versus Interact scenarios.Idle uses random actions away from objects; Interact uses a precomputed trajectory near the fridge.
  • 4.4 Benchmarking: H2.0 ablations show that removing rendering, physics, or all optimizations substantially reduces simulation performance.Table 2 reports simulation steps per second over 10 runs with 95% confidence intervals.
  • 4.4 Benchmarking: 1191 SPS makes optimized H2.0 approximately 1200% faster than iGibson’s 100 SPS in the comparable Idle setting.The paper attributes the gain to low-level C++ Bullet integration and deep rendering-physics integration, while noting that direct comparisons across simulators are not feasible.
  • 4.4 Benchmarking: H2.0 integrates OMPL, enabling comparisons between learned policies and classical sense-plan-act approaches.This integration supports the SPA baselines evaluated later in the paper.

5 Pick Task: a Base Case of Rearrangement

The Pick task serves as a controlled base case for studying manipulation and generalization before evaluation on the more difficult HAB tasks. MonolithicRL and SPA are compared across sensing, receptacles, objects, and layouts, with RL generalizing better to layouts but struggling on new objects and receptacles.

  • 5.1 Experimental Setup: The Pick task asks Fetch to retrieve a target object from a cluttered receptacle while studying generalization to layouts, objects, and receptacles.The task evaluates efficient motion while limiting excessive collisions.
  • 5.1 Experimental Setup: Fetch uses head- and arm-mounted RGB-D cameras, arm proprioception, and base egomotion, although base egomotion is unnecessary in the stationary Pick episodes.The same sensing becomes important for navigation in HAB.
  • 5.1 Experimental Setup: Pick uses end-effector control at 30Hz with 1.5cm maximum displacement per step and abstracted grasping within 15cm of an object.A positive gripper action snaps the nearest eligible object into the gripper, while release simulates falling.
  • 5.1 Experimental Setup: MonolithicRL is an end-to-end recurrent actor-critic trained with DD-PPO, whereas SPA builds a 3D point cloud and plans arm motion with Bidirectional RRT.SPA-Priv additionally uses privileged scene geometry.
  • 5.2 Systematic Generalization Analysis: MonolithicRL’s success fell from 91.7% to 74.7% on new objects and to 52.7% on new receptacles.The paper associates these drops with new visual distributions, object obstacles, and tighter or more constrained receptacle geometries.
  • 5.3 Sensor Analysis: Depth plus proprioception performed approximately 0.5% above RGBD plus proprioception and approximately 2% above RGB plus proprioception.The paper notes that depth also renders faster than RGB.
  • 5.3 Sensor Analysis: Proprioception-only blind policies achieved 78% success by slowly feeling toward goals, despite exceeding the force threshold 2x more and picking the wrong object 3x more than sighted policies.The analysis proposes that slow motion helps avoid heavy collision forces.

6 Home Assistant Benchmark (HAB)

HAB evaluates long-horizon household rearrangement through TidyHouse, PrepareGroceries, and SetTable, with increasing interaction complexity and generalization to unseen layouts. Results show that flat RL struggles to chain skills, hierarchical methods face hand-off problems, and SPA becomes brittle as perception and task complexity increase.

  • Task Definition: HAB evaluates three household rearrangement tasks: TidyHouse, PrepareGroceries, and SetTable.The tasks progress from moving objects without containers to fridge manipulation and opening or closing containers.
  • Task Definition: The robot is evaluated on unseen layouts and object configurations using onboard sensing rather than privileged maps, models, or physical properties.Each episode samples a ReplicaCAD layout with YCB objects and cluttered receptacles.
  • Methods: Task-planning methods decompose HAB into Navigate, Pick, Place, opening, and closing skills, while monolithic RL receives lists of start and goal coordinates.The hierarchical approach uses a STRIPS planner and parameterized skills; SPA is not reported for SetTable because articulated-container manipulation remains open.
  • Results and Findings: Monolithic RL learns individual skills but fails to elicit their chaining for long-horizon tasks without architectural task-structure bias.Hierarchical RL produces significantly stronger results when given a perfect STRIPS task planner.
  • Results and Findings: Learned navigation underperforms oracle navigation because manipulation requires reaching a suitable, non-unique viewpoint rather than merely navigating to a specified reachable location.This navigation challenge compounds skill-chaining difficulty.
  • Results and Findings: SPA scales poorly with task complexity: in PrepareGroceries, TP+SRL outperforms TP+SPA, while TP+SPA fails to find a goal configuration and a plan 3x more often than in TidyHouse.Cluttered fridge perception and egocentric positioning make SPA planning brittle.

7 Societal Impacts, Limitations, and Conclusion

The section concludes that ReplicaCAD, Habitat 2.0, and HAB support systematic study of embodied AI in interactive rearrangement environments, while identifying dataset and simulation limitations.

  • Limitations: ReplicaCAD’s apartment environments are modeled on spaces from one country, so furniture layouts and object types may not represent other cultures and regions.The authors note that this lack of representation can have negative social implications for developed assistants.
  • Limitations: The overall simulation-and-training loop remains bottlenecked by synchronization and asset-reloading costs despite H2.0’s fast simulator.The authors identify idle GPU time and the rendering-physics-RL interaction as open engineering problems.
  • Conclusion: H2.0 and ReplicaCAD enable comparisons between reinforcement-learning policies and classical motion-planning approaches on challenging rearrangement tasks.The authors present these resources as a platform for experimentation with embodied AI agents rearranging richly interactive 3D environments.
  • Conclusion: The MonolithicRL architecture combines visual and proprioceptive encoders with an LSTM, actor head, and value head.RGB and depth inputs are fused before recurrent processing and action-value prediction.

B.2 Training

The training setup uses a shaped reward for object manipulation and distributed PPO optimization over many concurrent environments.

  • Training: The reward function combines holding penalties, success and pickup indicators, and changes in object and arm distances.Collision force is also incorporated through the holding-related term.
  • Training: 64 concurrent processes across four GPUs are used to train the policy with DDPPO.Training runs for 100M experience steps with specified PPO coefficients, Adam optimization, and gradient clipping.

C Motion Planning

The SPA baselines decompose manipulation into goal sampling, motion planning, and execution, using RRTConnect and either privileged or egocentric perception.

  • Pipeline: SPA pipelines consist of joint-goal sampling, motion planning, and execution stages.The stages translate a high-level objective into motor torque actions.
  • Motion Planning: RRTConnect serves as the non-learning motion planner for the classical robotics pipeline.The planner is selected as a state-of-the-art, open-source sampling-based method and a stand-in for traditional robotics.
  • Perception: SPA-Priv uses privileged simulator information, whereas SPA uses egocentric depth observations to construct workspace point clouds for collision checking.The Fetch arm may lower to let the head camera sense the workspace.
  • Skill Construction: For each skill, inverse kinematics, random sampling, and collision checks determine a target arm joint state before planning.Pick samples grasp positions within a 15cm sphere and retains reachable, collision-free poses.
  • Skill Construction: Place planning samples a joint-state goal that positions the object at its target, using an exact model for SPA-Priv and a distance heuristic for SPA.Planning uses a 30-second timeout and 0.1-radian RRTConnect step size.

D.1 Blind Policy Analysis

The blind-policy analysis compares efficiency, camera placement, saliency, delay, and action spaces, finding strong sighted performance and benefits from combined head-and-arm sensing.

  • Path Efficiency: Over 80% SCT is achieved by sighted policies across collision thresholds, while blind policies are significantly less efficient.Blind-policy efficiency improves as the allowed collision threshold is reduced.
  • Camera Placement: 92% success is achieved by the combined head-and-arm camera placement on unseen layouts, versus 85% for the arm-only camera.All tested placements generally achieve 80–90% success, but arm-only sensing learns more slowly and performs worst.
  • Saliency: Grad-CAM maps show arm joints as particularly salient for both head-mounted and arm-mounted cameras across scenes.The analysis computes action gradients with respect to the visual encoder’s final convolutional activations.
  • Time Delay: Time delay has minimal impact on picking performance, although the 1-step condition has high variance across three seeds.The authors note that more seeds could reduce this variance.
  • Action Spaces: Velocity control learns just as well as end-effector control for the picking skill despite its higher-dimensional action space.The velocity-control policy outputs a seven-dimensional vector for the position target of the PD controller.

E.1 Evaluation

HAB evaluates rearrangement tasks through randomized episodes, partial task sequences, and STRIPS-generated action plans with explicit predicates, preconditions, and postconditions.

  • 100 evaluation episodes per task use unseen furniture micro-variations, randomized object positions, and random robot spawn locations.
  • Success requires placing every target object within 15cm of its goal position, without considering orientation.
  • Partial evaluation measures progressively longer rearrangement sequences for TidyHouse, PrepareGroceries, and SetTable.SetTable includes opening and closing drawers and the fridge around object transfers.
  • The hierarchical task representation defines predicates for holding, interaction distance, container state, and object containment.
  • STRIPS actions encode preconditions, postconditions, and linked skills for picking, placing, opening, and closing articulated objects.A planner transforms each task’s initial predicates into goal predicates and saves the minimum-length solution.
  • The saved STRIPS plan supplies the sequence of agent-environment interactions used for partial evaluation.

F.2 RL Skill Training

RL skill training decomposes home-assistant behavior into interaction-specific policies with structured observations, rewards, termination conditions, and randomized configurations.

  • Each skill must return the robot arm to a resting position, which supports transitions between skills and provides a termination signal.
  • Skill rewards combine task success with progress terms for arm, gripper, navigation, or articulated-object state, plus collision penalties where applicable.The collision force Ct is compared with a skill-specific threshold, and training episodes last at most 200 steps.
  • Each skill is trained on 5,000 configurations and evaluated on unseen scene configurations and object placements.A separate held-out evaluation set contains 500 configurations.
  • Pick and place skills use arm control with depth-camera observations and relative object or goal geometry.Pick episodes randomize clutter across six receptacles, while place episodes randomize goals across seven receptacles.
  • Learning navigation termination is difficult: learned termination produces a 20% drop in success rate.

G.1 SPA Failure Analysis

The SPA analysis separates failures into target planning, motion planning, and execution, showing that target-state generation and imperfect execution are the main reported weaknesses.

  • SPA failures are categorized as Target Plan, Motion Plan, and Execution errors within the sense-plan-act pipeline.
  • Target Plan failures arise when sampling cannot find a collision-free target joint configuration, especially for PrepareGroceries.PrepareGroceries requires reaching into the fridge, while privileged geometry can include more obstacles in collision checks.
  • All methods have a zero Motion Plan failure rate once a valid target joint state is found.
  • TP+SPA-Priv has zero Execution failures because it uses a perfect controller, whereas TP+SPA can fail with incomplete information and an imperfect controller.
  • SPA approaches use a robotics pipeline without learning, while TP+SRL learns its skills with reinforcement learning.
Loading 2106.14405v2…