Source-linked AI summary
On Evaluation of Embodied Navigation Agents
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, Amir R. Zamir
TL;DR
Three-dimensional navigation research has grown rapidly but uses diverse, sometimes incompatible tasks and evaluation protocols. This paper synthesizes working-group recommendations covering problem definitions, generalization, evaluation measures, and benchmark scenarios. It recommends explicit reporting of sensory and localization assumptions, standardized success evaluation with SPL as the primary measure, and shared scenarios for benchmarking.
Problem
Navigation research has accumulated diverse, sometimes incompatible task definitions and evaluation protocols, creating a need for coordinated empirical methodology.
Method
The paper summarizes consensus recommendations from a working group on navigation problem statements, generalization, evaluation measures, and standard benchmarking scenarios.
Results
The recommendations specify explicit reporting of sensory and localization assumptions, define success evaluation, adopt SPL as the primary navigation measure, and provide standard scenarios.
Takeaways & Limitations
Common task definitions, evaluation protocols, and benchmark scenarios are proposed to coordinate navigation research and support comparable empirical progress.
Takeaways & Limitations
SPL is stringent, and an SPL of 0.5 is expected to indicate good performance in reasonably complex, previously unseen environments; dataset-specific calibration with humans is recommended.
Abstract
from arXiv · showhide
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.
1. Introduction
Navigation is important for mobile intelligent systems, but research has become fragmented across task definitions, testbeds, and evaluation protocols. The working group recommends coordinated empirical methodology to support progress in navigation research.
- Navigation enables mobile intelligent systems to operate in three-dimensional physical environments.
- Earlier navigation work spans motion planning and SLAM, but these approaches often assume precise localization or high-quality geometric models.
- Biological systems appear to navigate robustly using flexible representations, prior knowledge, and neither precise localization nor metric maps.
- Recent navigation research has surged across diverse perspectives, including machine learning, perception, and reasoning.
- The diversity of task definitions, experimental testbeds, and evaluation protocols creates incompatible methodologies that motivate coordinated community efforts.
- The working group presents consensus recommendations intended to coordinate empirical methodology and contribute to progress in navigation research.
2. Goal Specification and Sensory Input
Navigation tasks differ by goal type and specification modality, while agents may receive varied sensory inputs. The paper recommends explicitly reporting the sensory information and localization assumptions available to each agent.
- Navigation goals are categorized as PointGoal, ObjectGoal, or AreaGoal.PointGoal targets a location, ObjectGoal targets an object category, and AreaGoal targets an area category.
- PointGoal uses coordinates, whereas ObjectGoal and AreaGoal use categorical labels for target objects or areas.
- Goal specifications can also use perceptual inputs such as images or natural language.An ObjectGoal may be specified by an object image, while a PointGoal may be specified linguistically.
- Agents may use vision, depth, haptics, schematic maps, or idealized self-motion inputs such as perfect odometry.
- Methods should explicitly state available sensory inputs and assumptions such as perfect odometry or GPS localization.
3. Generalization and Exploration
Navigation evaluation should distinguish how much prior exposure agents receive to test environments and quantify that exposure. The paper describes generalization regimes and treats exploration as an independently measurable task.
- Navigation protocols range from no prior exposure to extensive training in the test environment, with brief-exposure regimes between them.
- The extent of an agent’s pre-test exposure should be rigorously quantified and reported, including when extensive experience is permitted.
- No-prior-exploration evaluation tests navigation in novel environments without prior exposure.
- Pre-recorded prior exploration gives the agent a third-party trajectory recording that can support an internal environmental representation.
- Time-limited exploration lets the agent choose its exploration policy while constraining and quantifying exposure by trajectory length.
- Varying exploration budgets produces a multi-objective profile of the exploration-navigation trade-off and enables Pareto-front comparisons.
- Exploration can itself be evaluated by assigning a path-length budget and measuring the fraction of a previously unseen environment covered.
4. Evaluation Measures
The recommendations define navigation success at the agent’s explicit completion signal, use environment-aware proximity, and adopt SPL as the primary performance measure with complementary metrics.
- Completion signaling: Navigation episodes should be evaluated when the agent emits a dedicated completion action, rather than at an earlier favorable configuration.Without this signal, an episode is not successful even if the agent comes close to the goal.
- Distance measures: Geodesic distance should measure proximity to goals because it reflects the environment’s structure and reachability.Euclidean distance can classify an agent separated from the goal by a wall as close.
- Primary measure: 50% success with optimal paths yields SPL 0.5, while 50% success with paths twice the shortest length yields SPL 0.25.SPL therefore penalizes both unsuccessful episodes and inefficient successful paths.
- Calibration: SPL is stringent; in reasonably complex unseen environments, SPL 0.5 is expected to indicate good navigation performance and can be calibrated using human subjects.The recommendation is to assess calibration separately for each dataset.
- Primary measure: SPL combines binary success with normalized inverse path length into a single primary summary measure for comparing navigation agents.The measure uses the agent’s actual path relative to the shortest path across test episodes.
- Auxiliary measures: SPL should be supplemented with auxiliary measures such as success-distance trade-offs, final goal distance, threshold sweeps, path-length distributions, and infractions.These measures provide complementary information beyond the primary summary score.
5. Experimental Testbeds
The paper advocates reproducible simulation for coordinated navigation research while recommending continuous spaces, SI units, documented customization, and deployment paths to physical robots.
- Simulation rationale: Simulation enables reproducible experiments and coordinated community-wide progress, but learned navigation skills should continue to be transferred to reality.The authors advocate broad adoption of simulation alongside investment in sim-to-real techniques.
- Platform design: Future simulation platforms should use continuous state spaces because they better reflect physical-world deployment conditions.Discrete environments remain convenient and can support interesting experiments, but continuous spaces are recommended.
- Platform design: Simulators should adopt SI units so that one simulator distance unit corresponds to one meter.This recommendation supports interpretability and interoperability.
- Reproducibility: Researchers customizing simulators are expected to document their modifications and release the relevant customizations as open-source code.Customizability supports controlled evaluation and new problems, but carries documentation responsibility.
- Sim-to-real transfer: Simulators should include open-source software that deploys simulation-trained agents to physical robots using standard components.The proposed deployment support helps preserve the connection between simulation research and physical systems.
6. Standard Scenarios
The paper establishes publicly shared standard navigation scenarios across multiple environment datasets to promote consistent evaluation and direct comparison of embodied agents. These scenarios specify dataset splits, start and goal states, navigability checks, and task-specific configurations and actions.
- Standard scenarios are provided publicly to promote consistency in evaluation and enable direct comparison of navigation agents.
- Scenarios are constructed by selecting scenes from SUNCG, AI2-THOR, Matterport3D, and Gibson, then splitting each dataset into training, validation, and test sets.
- Each scenario randomly samples free-space start and goal positions, verifies navigability with tile-based shortest paths, and spans geodesic distances from 1 m upward.
- SUNCG: SUNCG scenarios use 500 single-floor houses split into 300/100/100 training, validation, and test environments.
- Matterport3D: Matterport3D adopts the original dataset’s 61/11/18 training/validation/test house splits, covering 190 floors and 2,206 room regions.
- AI2-THOR: AI2-THOR assigns 80 scenes to training, 20 to validation, and 20 to testing across kitchens, living rooms, bedrooms, and bathrooms.
- PointGoal: PointGoal evaluates 125 scenarios per test scene using five targets, five starts, and five scene configurations; success requires distance below 2× the agent’s width.
- ObjectGoal: ObjectGoal defines success as bringing a target object within 1 m and into the agent’s field of view, with navigation-only and interaction-based variants.
7. Agent Architectures
The paper distinguishes reactive agents from architectures with memory or internal representations, and identifies the structure of those representations as a central research question. It recommends comprehensive, open-minded study because this structure is viewed as fundamental to AI development.
- Purely reactive agents map each timestep’s sensory input directly to an action without internal state or an environmental representation.
- Recurrent architectures add short-term vectorial memory through a recurrent update mechanism.
- Recommendation 7 calls for comprehensive, open-minded study of the internal representations embodied agents construct and maintain while navigating.
- The paper regards internal representation structure as fundamental to the development of Artificial Intelligence.