Habitat: A Platform for Embodied AI Research

Manolis Savva; Abhishek Kadian; Oleksandr Maksymets; Yili Zhao; Erik Wijmans; Bhavana Jain; Julian Straub; Jia Liu; Vladlen Koltun; Jitendra Malik; Devi Parikh; Dhruv Batra

Source-linked AI summary

Habitat: A Platform for Embodied AI Research

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, Dhruv Batra

arXiv:1904.01201v2Apr 2, 2019cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

Embodied-AI research needs scalable, reproducible environments for training agents that act in realistic worlds, beyond the constraints of physical-robot experimentation and fragmented simulators. Habitat addresses this need with a fast 3D simulator and modular API, finding that learning can surpass SLAM with substantially more experience and that depth enables cross-dataset generalization.

Problem
Embodied-AI research is constrained by slow, unsafe, resource-intensive real-world training and fragmented simulators that impede scalable, reproducible experiments.
Method
Habitat combines Habitat-Sim, a configurable high-performance 3D simulator, with Habitat-API for defining tasks, training agents, and benchmarking across datasets and sensors.
Results
Learning outperforms SLAM when scaled to an order of magnitude more experience, while only depth-sensor agents generalize well across Matterport3D and Gibson.
Takeaways & Limitations
Habitat enables previously impractical large-scale embodied-AI experiments and provides findings intended to guide future research in the field.
Takeaways & Limitations
Existing indoor simulators commonly operate at 10-100 fps, making large-scale learning infeasible and weakening confidence in comparisons.

Abstract

from arXiv · show

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -- when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level library for end-to-end development of embodied AI algorithms -- defining tasks (e.g., navigation, instruction following, question answering), configuring, training, and benchmarking embodied agents. These large-scale engineering contributions enable us to answer scientific questions requiring experiments that were till now impracticable or 'merely' impractical. Specifically, in the context of point-goal navigation: (1) we revisit the comparison between learning and SLAM approaches from two recent works and find evidence for the opposite conclusion -- that learning outperforms SLAM if scaled to an order of magnitude more experience than previous investigations, and (2) we conduct the first cross-dataset generalization experiments {train, test} x {Matterport3D, Gibson} for multiple sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors generalize across datasets. We hope that our open-source platform and these findings will advance research in embodied AI.

1. Introduction

Habitat is a unified platform for training and benchmarking embodied agents in efficient photorealistic 3D simulation. Its simulator and API support diverse datasets, sensors, tasks, and methods, enabling larger-scale navigation experiments.

Embodied AI studies agents acting in environments through sensorimotor interaction, requiring skills including perception, language understanding, and navigation.
Real-world robot training is slow, dangerous, resource intensive, and difficult to control, motivating complementary training in rich simulators before transfer to reality.
Habitat standardizes the embodied-agent software stack across 3D datasets, high-performance simulation, task definitions, and modular training libraries.
Habitat-Sim provides configurable agents, multiple sensors, and generic 3D dataset handling, including built-in support for Matterport3D, Gibson, and Replica.
Habitat-API supports defining tasks such as navigation, instruction following, and question answering, training agents with imitation, reinforcement learning, or SLAM, and benchmarking them.
Several thousand fps single-threaded and over 10,000 fps multi-process on one GPU make Habitat-Sim substantially faster while Habitat-API supports varied datasets and methods.
The platform finds that learning can outperform SLAM with an order of magnitude more experience and that only depth-sensor agents generalize across datasets.

2. Related Work

Prior embodied-AI simulators provided valuable datasets, environments, tasks, and evaluation protocols but suffered from fragmentation and practical limitations. Habitat responds with a unifying platform designed for reproducibility, flexibility, and scalable experimentation.

Existing simulation platforms differ in scene data, tasks, and evaluation protocols, contributing to fragmentation, duplicated effort, and difficulty reproducing community progress.
Tight coupling among tasks, simulators, and datasets makes experiments spanning multiple tasks or datasets impractical.
Hard-coded agent configurations prevent supported ablations of agent parameters and sensor types, making results difficult to compare.
Most existing indoor simulators run at 10-100 fps, creating training bottlenecks that make large-scale learning infeasible and comparisons potentially unreliable.
Existing platforms provide limited programmatic control over scene objects, restricting robustness tests that modify object presence or layout.
Habitat aims to unify desirable prior capabilities while improving independent reproducibility, code reuse, consistent experimentation, and performance.

3. Habitat Platform

Habitat is organized as a modular software stack combining a high-performance simulator with APIs for configuring agents, sensors, tasks, datasets, training, and evaluation. Its implementation targets scalable, standardized embodied-AI research and supports high-throughput rendering.

Platform APIs: Habitat provides APIs for parameterized agents, arbitrary sensor suites, portable tasks, evaluation protocols, and dataset-agnostic scene ingestion.The platform also exposes C++ and Python interfaces, learning-framework interoperation, and distributed-training support.
Design rationale: The platform’s layered design mirrors the software stack because a monolithic design would not address requirements across all abstraction levels.The development effort aims to establish a common task framework supporting systematic embodied-agent research progress.
Habitat-Sim: Habitat-Sim loads standardized 3D scenes, configures agents and sensors, simulates motion, and returns sensory data through an extensible abstraction.Additional sensors such as LIDAR and IMU can be implemented as plugins, while dataset ingestion uses generic scene graphs.
Performance: Habitat-Sim achieves thousands of frames per second per simulator thread and is 2-3 orders of magnitude faster than previous realistic indoor-environment simulators.The benchmark uses Matterport3D scenes and compares Habitat-Sim with simulators operating at tens to hundreds of frames per second.
Habitat-API: Habitat-API connects task observations, simulator-provided information, and episode datasets for end-to-end embodied-AI development.Tasks define task-specific observations, actions, termination criteria, and success measures; episodes specify initial state, scene, goal, and optional shortest paths.

4. PointGoal Navigation at Scale

The paper uses PointGoal navigation to study scalable learning and cross-setting evaluation with standardized embodied agents, sensors, action spaces, and realistic collision dynamics. Experiments compare reinforcement learning and classic baselines across training experience and sensor configurations.

Task definition: PointGoal initializes an agent randomly and asks it to navigate to relative target coordinates using sensory input without a ground-truth map.Episodes allow up to 500 actions, and success requires stopping within 0.2m of the target by geodesic distance.
Embodiment and actions: The embodied agent uses four actions—turn_left, turn_right, move_forward, and stop—with 10-degree turns and 0.25m forward displacement.Experiments use a cylindrical agent with 0.2m diameter and 1.5m height in the noise-free setting.
Collision dynamics: Habitat models continuous motion with collisions that can produce partial or no progress, so a move_forward action need not yield its nominal 0.25m displacement.This collision model allows sliding along walls or obstacles and makes odometry nontrivial even without actuation noise.
Agent methods: RL agents use PPO with Blind, RGB, Depth, and RGBD sensor configurations, combining visual embeddings with the relative goal vector for recurrent actor-critic control.The visual encoder is a CNN, while the actor uses a GRU and the critic uses a linear layer.
Evaluation: The evaluation reports success and SPL on Gibson and Matterport3D test sets after RL training for 75 million steps.SPL weights success by the ratio between shortest-path geodesic distance and traversed path length.

5. Results and Findings

The experiments compare learning-based and classical navigation across increasing experience and datasets. Learning agents can surpass classical approaches with more training, while depth-equipped agents show the strongest cross-dataset generalization.

Learning versus SLAM: Learning-based agents can match and outperform classical approaches when given substantially more training experience.The study frames this comparison as representative of broad learning and SLAM families rather than an intrinsic superiority claim.
Learning versus SLAM: At 5M frames, the experiment would favor SLAM, but RL (PPO) Depth matches the classical baseline at approximately 10M Gibson frames or 30M Matterport3D frames and improves thereafter.
Point-goal navigation: RL (PPO) Depth performs best on test results, outperforming RGBD by 0.09-0.16 SPL, SLAM by 0.15-0.28 SPL, and RGB by 0.13-0.33 SPL.These comparisons were evaluated using the best validation checkpoint for each agent and dataset, with five random seeds per run.
Generalization across datasets: Cross-dataset evaluation reports average SPL for models trained on each source dataset and tested on each target dataset across four sensory configurations.
Generalization across datasets: Nearly all agents lose performance across datasets; for example, RGBD trained and tested on Gibson scores 0.70 versus 0.53 when tested on Matterport3D, a 0.17 drop.RGB and RGBD agents show significant degradation, while Blind is least affected.
Generalization across datasets: Agents trained on Gibson consistently outperform counterparts trained on Matterport3D, including when evaluated on Matterport3D, suggesting curriculum learning may benefit visual navigation.The paper attributes this pattern to Gibson’s smaller scenes and shorter, easier episodes under a fixed computation budget.

6. Habitat Challenge

Habitat Challenge addresses the difficulty of evaluating embodied AI systems that make sequential decisions by submitting executable agents rather than static predictions. It also plans more realistic localization, actuation, and sensor-noise conditions to narrow the simulation-to-reality gap.

Habitat Challenge benchmarks goal-directed visual navigation by evaluating submitted agent code in novel, unseen test environments.Embodied AI challenges require participants to upload code because sequential decision making and agent-driven control cannot be captured by a static testing dataset.
Challenge infrastructure: Participants upload Docker containers through EvalAI, which runs their agents on a live AWS GPU-enabled instance.
Future challenge changes: Future challenge iterations will replace continuously updated goal coordinates with a fixed relative coordinate to avoid assuming high-precision indoor localization.The paper notes that GPS is imprecise indoors, visual odometry may be noisy, and SLAM-based localization can fail.
Future challenge changes: Future iterations will model actuation error because commanded movements and resulting motions rarely match perfectly in real settings.
Future challenge changes: Realistic RGB and depth sensor-noise models are intended to narrow the gap between simulated and real perceptual experiences.

7. Future Work

Future work builds on Habitat’s unified platform and experimental findings by extending simulation toward physics-based interaction and distributed multi-agent settings. The platform’s modular architecture supports multiple simulators, sensors, datasets, and task configurations, while some sensor configurations remain computationally costly.

Habitat’s platform is intended to unify community efforts and accelerate embodied AI research through broader engagement.
Findings: Learned agents can match or exceed classic visual navigation methods with sufficient training, while depth-equipped agents generalize better across 3D datasets than RGB-only agents.
Roadmap: Future development will add physics simulation and physics-based interaction between mobile agents and 3D objects.The scene-graph representation is described as suitable for integrating physics engines and directly controlling object and agent state.
Roadmap: The project plans distributed simulation with many agents interacting competitively or collaboratively.
Platform design: Habitat-API can use alternative simulator backends, while task definitions specify datasets, episodes, curricula, and evaluation procedures.
Caveat: Matterport3D semantic sensing reduces performance because it requires additional geometrically complex meshes.The authors expect future versions to achieve speeds comparable to RGB plus depth.

B. Additional Dataset Statistics

Additional statistics characterize the PointGoal datasets, episode distances, and expected path horizons. The accompanying materials also include task-loading examples and the Habitat-API architecture.

Table 5 reports train, validation, and test split sizes for the three experimental datasets and their average geodesic start-to-goal distances.
Dataset statistics: Gibson episodes have shorter geodesic distances than Matterport3D episodes, making Gibson easier overall.Figure 9 compares geodesic distance, Euclidean distance, and their ratio as an approximate episode-complexity measure.
PointNav example: The PointNav example loads a preconfigured task and steps through the environment with random actions.
Habitat-API: Habitat-API provides core simulator interaction while concrete backends, datasets, and agent baselines extend the core functionality.
Path horizons: Table 5’s oracle path-length statistics contextualize the 500-action maximum-step limit for validation episodes.The oracle greedily follows the negative geodesic-distance gradient and approximates a near-perfect agent.

C. Additional Experimental Results

Scaling experiments to 800M steps preserves the observed ordering of visual inputs and shows consistent advantages for RGB over blind agents and RGBD over SLAM on Matterport3D.

800M steps preserved the visual-input ordering Depth > RGBD > RGB > Blind.
RGB exceeded Blind by 0.06/0.03 SPL on Gibson/Matterport3D.
RGBD exceeded SLAM on Matterport3D by 0.16 SPL.

C.1. Analysis of Collisions

Collision analysis distinguishes navigation behavior across sensory configurations and examines robustness to noisy depth. Blind agents collide most, while depth-equipped agents collide least; adding noise reduces depth-agent SPL, but depth-RL still exceeds SLAM on Gibson-val.

Blind agents incur many more collisions than other sensory configurations, consistent with wall-following behavior.Depth-equipped agents have the fewest collisions, while RGB agents fall between Blind and depth-equipped agents.
Depth-equipped agents have the lowest collision counts, whereas RGB agents show intermediate collision frequency.
0.13 and 0.02 SPL are the respective drops for depth-RL and SLAM on Gibson-val under noisy inverse-depth measurements.The evaluation adds iid Gaussian noise with µ = 0 and σ = 0.4 at each inverse-depth pixel without retraining.
Depth-RL still outperforms SLAM on Gibson-val after noisy-depth evaluation.The comparison uses depth-RL and SLAM agents evaluated without retraining under the same simple noise model.

D. Gibson Dataset Curation

Gibson scenes were manually curated to remove reconstruction artifacts that could disrupt navigation, using a quality threshold of 4 or higher. The associated navigation episodes characterize start-goal distances and sensory-dependent collision behavior.

D. Gibson Dataset Curation: PointGoal episodes report distributions of Euclidean start-goal distance, shortest-path geodesic distance, and their geodesic-to-Euclidean ratio.
D. Gibson Dataset Curation: Blind agents experience the most collisions, while Depth and RGBD agents have the fewest on successful Gibson and Matterport3D test episodes.
D. Gibson Dataset Curation: Scenes rated 4 or higher were retained because they have no holes, good reconstruction, and negligible texture issues.The quality scale ranges from 0 to 5, and floor holes or cracks were treated as especially problematic for navigation.
D. Gibson Dataset Curation: Floor holes can divide seemingly connected navigable areas into disconnected non-traversable components.

E. Reproducing Experimental Results

The experiments are reproducible through specified Habitat-API and Habitat-Sim repositories, training scripts, and SLAM instructions. Supplementary figures document sensory-configuration episodes, trajectory encoding, and mesh-quality criteria used in the dataset.

E. Reproducing Experimental Results: Experiments can be reproduced using Habitat-API commit ec9557a and Habitat-Sim commit d383c20.The experiment code is located under habitat-api/habitat_baselines.
E. Reproducing Experimental Results: The PPO reproduction script specifies RGB or depth sensors, blind-mode settings, optimization parameters, eight or six processes, 128 steps, and 135000 updates.The process counts are listed for Gibson and Matterport3D respectively.
E. Reproducing Experimental Results: SLAM reproduction instructions point readers to habitat-api/habitat_baselines/slambased.
E. Reproducing Experimental Results: Blind agents show the lowest performance and frequent collisions, RGB agents remain unreliable, and depth-equipped agents navigate more efficiently.The examples depict starting and goal positions, final agent position, trajectories, and progress toward the maximum allowed steps.
E. Reproducing Experimental Results: Meshes rated 4 or 5 meet the stated curation criteria, with no holes and good reconstruction; rating 5 additionally specifies uniform textures.Figure 11 presents the rating scale used for selecting Gibson meshes.