Source-linked AI summary
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, Erik Wijmans
TL;DR
ObjectNav requires a clearer framework for evaluating agents that search for labeled objects in unexplored environments. This paper develops consensus recommendations for evaluation, embodiment, and environments, and instantiates them in CVPR 2020 challenges; it also identifies important limitations of SPL for ObjectNav comparisons.
Problem
The paper addresses the need for finer-grained characterization of ObjectNav as researchers study agents that search for and navigate to objects.
Method
The working group proposes recommendations for ObjectNav evaluation, agent embodiment, and environments, and describes their challenge instantiation.
Results
The document provides a common ObjectNav task framework and detailed challenge implementations at the Embodied AI workshop at CVPR 2020.
Takeaways & Limitations
The recommendations are intended to promote consistency in future research and encourage more systematic benchmarking of ObjectNav.
Takeaways & Limitations
SPL cannot support comparisons across datasets or dataset portions with significantly different optimal path lengths.
Abstract
from arXiv · showhide
We revisit the problem of Object-Goal Navigation (ObjectNav). In its simplest form, ObjectNav is defined as the task of navigating to an object, specified by its label, in an unexplored environment. In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e.g., find a chair, by navigating to it. As the community begins to show increased interest in semantic goal specification for navigation tasks, a number of different often-inconsistent interpretations of this task are emerging. This document summarizes the consensus recommendations of this working group on ObjectNav. In particular, we make recommendations on subtle but important details of evaluation criteria (for measuring success when navigating towards a target object), the agent's embodiment parameters, and the characteristics of the environments within which the task is carried out. Finally, we provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020 http://embodied-ai.org .
1. Introduction
ObjectNav is presented as an embodied search-and-navigation capability whose evaluation requires clearer, more consistent task definitions. The working group proposes consensus recommendations and describes their implementation in CVPR 2020 challenges.
- 1. Introduction: ObjectNav focuses on an agent searching for and navigating to a labeled object in an unexplored environment.The motivating example is a robot asked to find a laptop, requiring visual perception, language understanding, memory, planning, and navigation.
- 1. Introduction: Embodied AI research increasingly requires unambiguous task definitions and systematic evaluation beyond static image or prerecorded-data tasks.Embodied agents act within physical or simulated three-dimensional environments and perform tasks including navigation and manipulation.
- 1. Introduction: ObjectGoal navigation has developed inconsistent interpretations involving success criteria, agent embodiment, and other seemingly low-level task details.The working group identifies these variations as consequential for characterizing ObjectNav.
- 1. Introduction: The document provides consensus ObjectNav recommendations that augment earlier navigation guidance.The recommendations address how ObjectNav should be characterized and evaluated.
- 1. Introduction: The recommendations are instantiated in challenges organized at the Embodied AI workshop at CVPR 2020.Figure 1 illustrates a success zone consisting of navigable, correctly oriented locations within 1m of the goal object.
2. ObjectNav Task Definition
ObjectNav evaluates an agent’s ability to navigate from a random pose in an unexplored environment to an object category, but requires precise choices about goals, success, embodiment, and environments. The recommendations separate navigation success from efficiency while acknowledging important limitations of SPL.
- Task definition: ObjectNav asks an agent to navigate from a random pose in an unexplored environment to an instance of a specified object category.The task is exemplified by finding a chair.
- Task definition: The task must specify why objects are being found and what counts as successful navigation, alongside the agent’s actions, observations, and environment scenes and categories.The working group identifies object finding and evaluation, embodiment, and environments as distinct specification choices.
- Object finding and evaluation: Evaluation should measure whether the agent navigates to the goal object and how efficiently it travels relative to an optimal path.The recommendations aim to evaluate navigation separately from object detection or final object framing.
- Object finding and evaluation: Success requires intentional stopping, a navigable location, proximity to the object, and, when enforced, object visibility within the agent’s field of view.A reasonable initial proximity shell is r = 1m, while an initial visibility threshold may be one pixel corresponding to the object.
- Object finding and evaluation: Oracle-visibility evaluates reaching a location from which the object can be viewed, without requiring the stopped pose itself to frame the object.This proxy is more suitable when the focus is path finding rather than both path finding and object framing.
- Object finding and evaluation: ObjectNav-SPL compares the path taken with the shortest path to the closest qualifying object instance, then averages episode scores.The shortest path is computed to the closest instance from the starting location; SPL is averaged over all episodes.
- Known issues with SPL: SPL treats gross failures and near misses alike, can have high variance under noisy stopping, does not penalize in-place turning, and cannot support comparisons across datasets with different path lengths.The document acknowledges these shortcomings while treating development of replacement metrics as separate from characterizing ObjectNav.
- Embodiment: The recommended embodiment uses realistic discrete navigation actions, such as a discretized differential drive, while leaving low-level torque and joint control out of scope.This design emphasizes visual and decision-making aspects without excluding fuller robot dynamics from future task versions.
3. Habitat 2020 Challenge
The Habitat 2020 ObjectNav challenge instantiates a standardized benchmark in photorealistic Matterport3D scenes, with defined object episodes, agent embodiment, actions, sensors, collision dynamics, and evaluation phases.
- Scenes: Habitat 2020 uses 90 photorealistic Matterport3D scenes with standard train, validation, and test splits that do not overlap.The scenes average about two floors and 15 rooms.
- Objects: The benchmark selects 21 visually well-defined object categories, totaling 8,349 instances across 71 environments.Architectural elements and categories such as doorways and windows are excluded.
- ObjectNav Episode Dataset: Each episode specifies a scene, agent start position and orientation, and target category, with Euclidean and geodesic distances plus valid success-zone viewpoints.The dataset also filters episodes by shortest-path length and geodesic-to-Euclidean distance ratio.
- Embodiment and Actions: The simulated agent is configured to match a LoCoBot, using a 0.18m base radius and 0.88m height, six discrete actions, RGB-D, and idealized GPS+Compass sensors.The action set includes forward motion, turns, looking up or down, and STOP; turn angles are 30 degrees.
- Collision Dynamics and Evaluation: Collision sliding can let learned agents exploit paths that appear to cross non-navigable regions, so the challenge specifies success as stopping within 0.1m geodesic distance of a valid goal viewpoint.The resulting behavior can fail in the real world, where bump sensors stop the robot on contact.
- Challenge Phases: The challenge includes sanity-checking, public-leaderboard, and winner-selection phases with daily or total submission limits.The test-standard phase is designated for reporting results in papers.
4. RoboTHOR 2020 Challenge
RoboTHOR 2020 evaluates whether agents trained in simulation generalize to real environments, unseen layouts, and unseen object instances through staged simulated and physical testing.
- Scenes: RoboTHOR contains 89 scenes: 75 simulated training scenes, 4 real/simulated test-dev scenes, and 10 real test-challenge scenes.The three sets have no overlap in interior wall layouts.
- Objects: The platform includes 731 object instances across 43 categories, with 12 categories used as test targets.The categories include furniture and small objects such as mugs and laptops.
- Agent: The modified LoCoBot is matched between simulation and reality, using an Azure Kinect camera and RGB, depth, localization, and orientation sensors.The robot is 0.88m tall to observe objects on tables and high surfaces, and control APIs are identical across platforms.
- Object Visibility: Episodes succeed when the target object is within 1m and inside the camera’s field of view.Simulation obtains this from the controller, while real experiments measure distance and manually verify visibility.
- Challenge Phases: The three challenge phases progress from simulation training to paired real/simulated testing and blind real-world evaluation.The final phase uses 10 unseen real scenes and evaluates simulation-to-real, unseen-scene, and unseen-target generalization.
5. Conclusion
The paper proposes a common ObjectNav framework covering evaluation, agent, and environment specifications to promote consistency and systematic benchmarking.
- Conclusion: The recommendations address evaluation metrics, agent specifications, and environment specifications for visual navigation toward objects.The stated aim is to promote consistency in future research and more systematic benchmarking.