Source-linked AI summary
Navigating to Objects in the Real World
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, Devendra Singh Chaplot
TL;DR
Semantic navigation needs reliable evaluation beyond simulation because robots must find objects in unseen, uncontrolled environments. The paper conducts a large-scale comparison of classical, modular, and end-to-end methods on robots, finding strong modular transfer but poor end-to-end transfer and identifying two Sim-to-Real benchmark problems.
Problem
Semantic navigation methods have been predominantly evaluated in simulation, leaving uncertain whether simulation performance and error modes reflect performance on real robots.
Method
The study evaluates representative classical, modular learning, and end-to-end Object Goal navigation methods across six homes and a controlled simulation comparison.
Results
90% real-world success was achieved by modular learning, while end-to-end learning dropped from 77% simulation to 23% real-world success because of an RGB image domain gap.
Takeaways & Limitations
Modularity and abstraction support Sim-to-Real transfer for object navigation, while reliable evaluation requires addressing image gaps and mismatched simulation and real-world error modes.
Takeaways & Limitations
The study is restricted to six goal object categories to align with current simulation benchmarks.
Abstract
from arXiv · showhide
Semantic navigation is necessary to deploy mobile robots in uncontrolled environments like our homes, schools, and hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, while modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. But learned visual navigation policies have predominantly been evaluated in simulation. How well do different classes of methods work on a robot? We present a large-scale empirical study of semantic visual navigation methods comparing representative methods from classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We find that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% simulation to 23% real-world success rate due to a large image domain gap between simulation and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: modularity and abstraction in policy design enable Sim-to-Real transfer. For researchers, we identify two key issues that prevent today's simulators from being reliable evaluation benchmarks - (A) a large Sim-to-Real gap in images and (B) a disconnect between simulation and real-world error modes - and propose concrete steps forward.
1 Introduction
Semantic navigation aims to help robots find target objects in unseen environments, but its real-world effectiveness remains uncertain because learned policies are mostly evaluated in simulation. This study compares classical, modular, and end-to-end approaches on robots and examines the Sim-to-Real gap.
- Motivation: Object Goal navigation asks a robot to find a specified object category in a completely unseen environment.The task targets deployment in uncontrolled settings such as homes, schools, and hospitals.
- Motivation: The field has many simulated methods and benchmarks, but lacks large-scale real-world evaluation of whether simulation performance predicts robot performance.The study asks whether simulation improvements and error modes transfer to reality.
- Study design: The study compares representative classical, modular learning, and end-to-end learning methods across six visually diverse homes using 45 hours of robot experiments.It also replicates one home in simulation to decompose the Sim-to-Real performance gap.
- Results: 90% real-world success was achieved by modular learning, compared with 80% for classical approaches and 23% for end-to-end learning.Modular learning rose from 81% in simulation to 90% in the real world, whereas end-to-end learning fell from 77% to 23%.
- Sim-to-Real analysis: A large RGB image Sim-to-Real gap causes image-based policy architectures and simulation-trained segmentation models to transfer poorly.The authors recommend policy abstractions and avoiding simulation-trained segmentation when real-world replacement is difficult.
- Sim-to-Real analysis: Simulation error modes also diverge from reality: real modular failures mainly involve depth sensor errors, while simulated failures involve reconstruction and navigation-mesh errors.The authors propose realistic depth noise, better visual quality, and improved navigation meshes in simulation.
2 Results
The study evaluates representative classical, modular-learning, and end-to-end navigation policies on real robots and matched simulation settings. Modular learning transfers best to natural homes, whereas end-to-end policies show strong simulation-to-reality degradation linked to design and error-mode mismatches.
- Natural Home Environments: 60 episodes across six previously unseen homes evaluate representative classical, modular-learning, and end-to-end policies on a Hello Robot Stretch.The study covers six goal-object categories and 45 hours of robot experiments.
- Natural Home Environments: 90% success rate in reality versus 81% in simulation for modular learning, while end-to-end learning falls from 77% to 23%.Classical navigation rises from 78% to 80% success rate.
- Natural Home Environments: End-to-end failures include detecting the goal without stopping nearby and revisiting semantically unrelated locations.These observed failures are associated in the passage with limited semantic understanding, long-term memory, and exploration.
- Natural Home Environments: Modular learning reaches the couch in 84 steps with SPL = 0.74, compared with 181 steps and SPL = 0.33 for the classical policy.The end-to-end policy reaches 121 steps before exceeding the 20-collision limit.
- Controlled Experiments in a Home Replicated in Simulation: Simulation performance can overfit through camera, segmentation, and training-algorithm choices: policy 1 is best in simulation but worst in reality, while policy 4 reverses this ranking.Policy 4 uses imitation learning without reinforcement-learning fine-tuning and is selected for large-scale evaluation.
- Controlled Experiments in a Home Replicated in Simulation: The sim replica matches the benchmark closely, but low SRCC persists because policies fail different episodes in simulation and reality.For modular learning, the passage gives SRCC = 0.70 despite 0.80 simulation versus 0.90 real success rate.
3 Discussion
The study finds modular learning reliable for real-world object navigation, while simulation remains limited by mismatched visual domains and error modes. Semantic abstraction supports transfer, but the study also identifies scope and evaluation limitations.
- Error modes: Five of six real-world modular-learning errors stem from depth sensing, whereas simulation assumes perfect depth and produces largely different failures.Real failures include blocked doorways from noisy depth and reflections in mirrors or TVs; simulation failures include reconstruction errors.
- Simulation limits: Simulation is less useful for diagnosing real bottlenecks when simulated and real-world error modes barely overlap.The authors propose realistic depth noise, improved visual reconstruction, and higher-quality navigation meshes.
- Practical implications: 90% real-world success rate demonstrates that modular learning reliably navigates to objects with efficient search.The remaining errors are primarily attributed to depth sensor failures.
- Sim-to-Real transfer: Semantic map representations transfer across simulation and reality because they abstract pixels into categories and use coarser spatial resolution.The experiments use a 5 cm map voxel size.
- Sim-to-Real transfer: RGB-D end-to-end policies face a large image-domain gap because current reconstruction engines cannot generate photorealistic images.This gap causes segmentation transfer to fall from 0.50 to 0.35 mAP@0.5 for real-to-simulation transfer and from 0.45 to 0.10 for simulation-to-real transfer.
- Future directions: The study recommends modularity and abstraction for simulation training, but does not evaluate end-to-end policies that first transform RGB frames into semantic representations.The study is also restricted to six goal object categories to align with current simulation benchmarks.
4 Materials and Methods
The study deploys representative classical, modular learning, and end-to-end policies on a robot, matching simulation inputs and evaluating navigation across six real homes. The methods differ in how they represent scenes, select exploration goals, and transfer from simulation, with real-world depth noise remaining a constraint.
- Sim-to-Real Transfer Methodology: Three policies are deployed on a Hello Robot Stretch across six homes to compare Sim-to-Real transfer for object navigation.All learned components are trained in Habitat simulation, and the evaluation matches the Object Goal task’s input and output spaces on the robot.
- Sim-to-Real Transfer Methodology: The robot receives RGB-D images and sensor pose, using stereo depth estimation and LiDAR-based SLAM to approximate simulation inputs.The RealSense camera provides 640 x 480 RGB-D images with a 42-degree horizontal field of view.
- Classical and Modular Learning Approaches: Classical and modular policies build semantic maps from segmented RGB frames, projected depth, and pose, then plan low-level actions toward exploration goals.The modular policy learns semantic exploration goals from map features and the target object, whereas the classical policy selects the closest unexplored frontier independently of the target.
- End-to-end Learning Approaches: The end-to-end policy computes visual features from RGB, depth, and semantics and directly predicts low-level actions using the goal object, pose, and recurrent processing.Unlike modular methods, replacing its segmentation model creates a large domain gap and requires training a new policy.
- Semantic Map Representation: The semantic map is a binary K x M x M matrix whose channels encode object categories, obstacles, explored areas, and current or past agent locations.Each map cell represents 5 cm x 5 cm in the physical world, with K = C + 4 channels.
- Sim-to-Real Limitations: Real-world stereo depth is noisy, and training with Habitat’s indoor depth-noise model hurts performance because it does not match the robot camera’s noise.This mismatch introduces an additional Sim-to-Real domain gap.
6 Supplementary Materials
The supplementary results compare navigation outcomes, object and distance distributions, and error modes across methods and environments. They show concrete behavioral differences between modular, classical, and end-to-end policies, while separating real-world sensor failures from simulation reconstruction errors.
- Policy Comparisons: The modular policy finds a toilet goal in 74 steps, whereas frontier exploration fails to find it within 200 steps.This supplementary comparison illustrates the benefit of goal-directed semantic exploration over goal-independent frontier exploration.
- Policy Comparisons: On the same episode, classical and modular policies reach the toilet in 108 and 122 steps, while the end-to-end policy never reaches it.The end-to-end policy fails to explore beyond the kitchen and dining room.
- End-to-end Failure Modes: End-to-end failures include failing to stop after detecting the toilet and repeatedly revisiting kitchen locations while searching for a TV.These examples represent distinct stopping and exploration failure modes.
- Real-world Error Modes: Real-world depth errors include noise masking a door, mirror reflections duplicating a bed, and TV reflections sensing beyond the sensor limit.These failures affect the semantic or geometric map used for navigation.
- Simulation Error Modes: Simulation reconstruction errors produce segmentation mistakes and noisy navigation meshes with narrow paths that are difficult for discrete planners.The caption contrasts these simulation errors with the real-world depth-sensor failures.
- Evaluation Distributions: Goal-object and geodesic-distance distributions are comparable between the Habitat 2022 validation benchmark and the real-world evaluation.This comparison concerns the evaluation setting rather than policy performance.
- Aggregate Evaluation: Navigation performance is aggregated across 60 episodes, six object-goal categories, and six homes using Success Rate and Success weighted by Path Length.The supplementary tables also report consistency across homes and goal objects.
- Error Analysis: Supplementary modular-learning analyses separate multi-floor, segmentation, exploration, and manually classified residual errors on the simulation benchmark.The error analyses compare restricted episode subsets, ground-truth segmentation, and a larger time budget.