Source-linked AI summary
SOON: Scenario Oriented Object Navigation with Graph-based Exploration
Fengda Zhu, Xiwen Liang, Yi Zhu, Xiaojun Chang, Xiaodan Liang
TL;DR
The paper addresses navigation toward a language-described object from an arbitrary starting position, unlike benchmarks centered on fixed trajectories and step-by-step instructions. It introduces SOON, the graph-based exploration method GBE, and the FAO benchmark with rich scene descriptions. GBE outperforms prior state-of-the-art methods on both R2R and FAO, while FAO ablations support the value of its vision-language information.
Problem
Existing navigation benchmarks commonly assume fixed starting points and detailed step-by-step instructions, leaving navigation from anywhere using target and scene descriptions insufficiently addressed.
Method
The paper introduces SOON, graph-based exploration that models navigation as a graph and learns from sub-optimal trajectories, and the FAO benchmark with rich semantic descriptions.
Results
GBE outperforms previous state-of-the-art models on both R2R and FAO datasets, and FAO ablations validate the usefulness of its vision-language information for target localization.
Takeaways & Limitations
SOON and FAO provide a benchmark for target-oriented, start-independent embodied navigation that more closely reflects the described real-world navigation setting.
Abstract
from arXiv · showhide
The ability to navigate like a human towards a language-guided target from anywhere in a 3D embodied environment is one of the 'holy grail' goals of intelligent robots. Most visual navigation benchmarks, however, focus on navigating toward a target from a fixed starting point, guided by an elaborate set of instructions that depicts step-by-step. This approach deviates from real-world problems in which human-only describes what the object and its surrounding look like and asks the robot to start navigation from anywhere. Accordingly, in this paper, we introduce a Scenario Oriented Object Navigation (SOON) task. In this task, an agent is required to navigate from an arbitrary position in a 3D embodied environment to localize a target following a scene description. To give a promising direction to solve this task, we propose a novel graph-based exploration (GBE) method, which models the navigation state as a graph and introduces a novel graph-based exploration approach to learn knowledge from the graph and stabilize training by learning sub-optimal trajectories. We also propose a new large-scale benchmark named From Anywhere to Object (FAO) dataset. To avoid target ambiguity, the descriptions in FAO provide rich semantic scene information includes: object attribute, object relationship, region description, and nearby region description. Our experiments reveal that the proposed GBE outperforms various state-of-the-arts on both FAO and R2R datasets. And the ablation studies on FAO validates the quality of the dataset.
1. Introduction
SOON reframes embodied navigation as finding a richly described object from an arbitrary starting position, rather than following fixed step-by-step instructions. The paper introduces graph-based exploration and the FAO benchmark to support this setting.
- Existing vision-language navigation benchmarks typically guide agents along fixed trajectories with detailed step-by-step instructions.
- SOON instead asks agents to find a thoroughly described target object from anywhere in a 3D house.Its instructions identify and guide toward the target without prescribing how the agent must navigate.
- SOON uses coarse-to-fine exploration, first searching larger regions and then narrowing the search according to scene descriptions and visual observations.
- GBE models navigation as a graph, provides structured understanding of observed information, and merges multiple conventional actions into one-step graph decisions.
- FAO contains 4K annotated instruction sets and 40K trajectories across 90 Matterport3D housing environments, using object, relationship, region, and nearby-region descriptions.Its instructions average 38 words and labeled trajectories average 9.6 hops, exceeding the cited R2R figures.
- Experiments report that GBE outperforms prior methods on R2R and SOON, while FAO ablations indicate that its vision-language information supports target localization.
2. Related Work
Related work has expanded embodied navigation from simple language goals to richer visual-language, dialogue, memory, and graph-based approaches. SOON builds on this progression while addressing a different navigation setting.
- Embodied navigation research has used object or room names, question-answer pairs, natural-language sentences, and multi-sentence dialogues as instructions.
- Vision-language navigation benchmarks combine visual environments with language instructions and have motivated methods using data augmentation, panoramic action spaces, imitation learning, and reinforcement learning.
- Later navigation systems incorporate SLAM-inspired memory mechanisms and graph representations for generalization and long-trajectory navigation.
3. Scenario Oriented Object Navigation
SOON defines navigation from an arbitrary position to a described object as separate navigation and localization subtasks. Its environment uses a discrete navigation graph, while object locations are represented in polar coordinates.
- SOON requires an agent to navigate from an arbitrary position in a 3D environment and localize a target object from an instruction.
- Navigation succeeds when the agent reaches within 3m of the target, while localization additionally requires correctly identifying the target in the panoramic view.
- Reachable positions are discretized into a navigation graph, and each action moves the agent to a neighboring node or stops.
- The task uses simulator-provided RGB-D, GPS coordinates, and current and candidate node indexes as navigation observations.
- Object locations are represented with polar coordinates rather than 2D bounding boxes from separate panoramic views.The paper motivates this representation by noting missing labels and distortion-related noise in the alternative labeling setup.
4. Graph-based Semantic Exploration
GBE represents explored environments as semantic navigation graphs, combines graph and language features, and uses graph-based exploration to train navigation and localization jointly.
- Graph-based Navigation: GBE builds a navigation semantic graph from visited and observed places, representing nodes with visual features and edges with connectivity.Unvisited nodes observed from multiple views use averaged features, while the graph planner updates the edge set as exploration proceeds.
- Graph-based Navigation: A graph convolutional network embeds the graph nodes into a graph representation that summarizes the current position and nearby scene.The graph planner applies GCN-based updates using node features and edges before producing the graph embedding.
- Graph-based Navigation: GBE matches graph and language features cross-modally, then predicts navigation actions over observed unvisited candidates and regresses target heading and elevation.The dynamic action space includes candidate nodes and a stop action, while localization uses heading and elevation outputs.
- Graph-based Exploration: During graph-based exploration, the agent samples predicted actions, while a teacher selects the candidate with minimum Dijkstra distance to any visible target.Training targets may include multiple target positions in SOON; target positions are visible during training but not testing.
- Training Objectives: GBE jointly optimizes navigation and localization, combining imitation learning, reinforcement learning, and graph-based exploration losses with a localization MSE objective.The navigation loss weights the three learning approaches, while localization is supervised using polar heading and elevation coordinates.
5. Experiments
Experiments evaluate GBE on R2R and FAO, using multiple dataset splits, navigation and localization metrics, baseline comparisons, and ablations of modalities and annotation granularity.
- 5.1. From Anywhere to Object (FAO) Dataset: FAO contains 3,848 instruction sets, 6,326 bounding boxes for 3,923 objects, and over 30K long-distance trajectories across 90 Matterport scenes.The training split includes 28,015 trajectories over 38 houses, with validation splits for seen instructions, seen houses, and unseen houses.
- 5.1. From Anywhere to Object (FAO) Dataset: Instructions combine object attributes, object relationships, target-region descriptions, and nearby-region descriptions through a five-stage annotation process.Annotators describe target attributes, related objects, target and nearby regions, then rewrite the descriptions into three sentences.
- 5.2. Experimental Results: Evaluation measures navigation with NE, SR, OSR, and SPL, while localization uses bounding-box success and the combined SFPL metric.SFPL combines SPL with localization success and accounts for navigation and ground-truth path lengths.
- 5.2. Experimental Results: On R2R, GBE outperforms EGP by 2.4% in SPL on unseen-house validation and exceeds RCM by 5.4% in SPL on the test set.The reported comparisons use models without pretraining or auxiliary tasks, while the baseline suite includes imitation-learning, reinforcement-learning, auxiliary-task, and memory-network methods.
- 5.2. Experimental Results: FAO results show a substantial human-machine gap, while poor random-policy performance indicates that the dataset is not biased toward easy navigation.The experiments compare baselines and GBE across validation and test sets.
- 5.2. Experimental Results: Removing both vision and language produces the worst performance, vision-only exceeds language-only, and combined vision-language input performs best.These ablations indicate that both modalities contribute to SOON navigation, with vision having greater importance in the reported comparison.
- 5.2. Experimental Results: Object names, attributes and relationships, and region descriptions each contribute to SOON navigation, while rewritten instructions achieve 0.6% higher SFPL than the first three combined.Object-name-only navigation performs worse because repeated object classes create ambiguity and omit scene and region information.
6. Conclusion
The paper introduces SOON, FAO, and GBE for object navigation from arbitrary starting positions. GBE models explored areas as a feature graph and outperforms prior state-of-the-art models on R2R and FAO.
- 6. Conclusion: SOON requires an agent to find an object in a house from an arbitrary starting position.The accompanying FAO dataset contains 3K descriptive natural-language instructions.
- 6. Conclusion: GBE models explored areas as a feature graph and uses graph-based exploration to obtain a robust navigation policy.The method is presented as a promising direction for approaching SOON.
- 6. Conclusion: GBE outperforms all previous state-of-the-art models on both R2R and FAO datasets.The authors position SOON as a task intended to help address real-world navigation problems.