PaperlayerView exact revision on arXiv

Source-linked AI summary

ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments

Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, Liang Wang

arXiv:2304.03047v3cs.CVcs.CLcs.RO

TL;DR

VLN-CE requires navigation through continuous 3D environments rather than predefined graphs, creating challenges in long-range planning and obstacle avoidance. ETPNav addresses these challenges with online topological mapping, cross-modal planning, and a trial-and-error controller, achieving state-of-the-art results on two VLN-CE benchmarks.

  • Problem

    VLN-CE removes the predefined-graph assumption, requiring agents to navigate freely on a 3D mesh with low-level actions.

  • Method

    ETPNav combines online self-organized topological mapping, high-level cross-modal planning, and low-level trial-and-error obstacle-avoiding control.

  • Results

    ETPNav achieves state-of-the-art performance on two VLN-CE benchmarks, with 55 SR and 48 SPL on R2R-CE and 51.21 SR and 41.30 SDTW on RxR-CE test unseen splits.

  • Takeaways & Limitations

    A concise depth-only waypoint design and hierarchical topological planning support robust long-range navigation, while the trial-and-error controller addresses obstacle avoidance.

  • Takeaways & Limitations

    The method assumes access to ground-truth pose and does not address pose estimation from noisy sensor readings.

Abstract

from arXiv · show

Vision-language navigation is a task that requires an agent to follow instructions to navigate in environments. It becomes increasingly crucial in the field of embodied AI, with potential applications in autonomous navigation, search and rescue, and human-robot interaction. In this paper, we propose to address a more practical yet challenging counterpart setting - vision-language navigation in continuous environments (VLN-CE). To develop a robust VLN-CE agent, we propose a new navigation framework, ETPNav, which focuses on two critical skills: 1) the capability to abstract environments and generate long-range navigation plans, and 2) the ability of obstacle-avoiding control in continuous environments. ETPNav performs online topological mapping of environments by self-organizing predicted waypoints along a traversed path, without prior environmental experience. It privileges the agent to break down the navigation procedure into high-level planning and low-level control. Concurrently, ETPNav utilizes a transformer-based cross-modal planner to generate navigation plans based on topological maps and instructions. The plan is then performed through an obstacle-avoiding controller that leverages a trial-and-error heuristic to prevent navigation from getting stuck in obstacles. Experimental results demonstrate the effectiveness of the proposed method. ETPNav yields more than 10% and 20% improvements over prior state-of-the-art on R2R-CE and RxR-CE datasets, respectively. Our code is available at https://github.com/MarSaKi/ETPNav.

1 INTRODUCTION

VLN-CE removes the predefined-graph simplification of discrete navigation, requiring free movement on a 3D mesh with low-level actions. ETPNav addresses the resulting planning and control challenges through online topological mapping, cross-modal planning, and obstacle-avoiding control.

  • Research gap: VLN-CE requires agents to navigate freely on a 3D mesh with low-level actions rather than traverse a predefined graph.This setting better reflects the challenges of deployed navigation systems.
  • Research gap: Existing waypoint methods use local representations that limit global layout modeling and long-range planning, while waypoint design choices remain insufficiently studied.Local planning can require repeated plan-control cycles for remote backtracking, accumulating instability.
  • Proposed approach: ETPNav builds topological maps online by self-organizing predicted waypoints, without predefined graphs or prior environmental exploration.The map represents places and reachability, supporting long-range plans under partial observability.
  • Design findings: Depth-only waypoint prediction supports generalization, while a trial-and-error controller nearly eliminates performance loss from sliding-forbidden obstacle interactions.The controller explicitly helps the agent escape deadlocks that can trap commonly used controllers.
  • Proposed approach: ETPNav decomposes navigation into map-based high-level planning and low-level control, using a cross-modal transformer to plan and a controller to execute actions.The controller performs obstacle-avoiding execution of the generated plan.
  • Results: 55 SR and 48 SPL on R2R-CE, and 51.21 SR and 41.30 SDTW on RxR-CE, establish state-of-the-art performance on two VLN-CE benchmarks.The system also won the CVPR 2022 RxR-Habitat Challenge.

2 RELATED WORK

Related VLN-CE work moved from end-to-end action prediction toward modular waypoint-based systems, but local maps and obstacle-unaware controllers remain limiting. ETPNav responds with an online topological map designed for long-range planning in continuous environments.

  • VLN settings: Discrete VLN assumes traversal on a predefined graph, narrowing agent and target locations while omitting low-level control challenges.Transferring such agents to real-world or continuous settings can cause severe performance drops.
  • VLN settings: VLN-CE requires free navigation on a 3D mesh using low-level actions, and its benchmark paths are simulated in Habitat.This setting removes the strong graph assumption used in discrete VLN.
  • ETPNav: ETPNav comprises topological mapping, cross-modal planning, and control modules that update an episode map, create a high-level path, and execute low-level actions.The system maintains the map as new observations arrive.
  • VLN-CE methods: End-to-end VLN-CE systems directly predict low-level actions but face a large performance gap and expensive joint learning of language grounding and control.Modular approaches instead separate waypoint generation, subgoal selection, and controller execution.
  • VLN-CE methods: Waypoint-based methods simplify policy learning but use local waypoint representations that inadequately capture global layouts and navigation dependencies.Their widely used controllers can also get stuck in obstacles when sliding along obstacles is forbidden.
  • Topological maps: Prior VLN topological maps rely on predefined graphs, while prior VLN-CE maps require offline environment pre-exploration and global topology priors.ETPNav instead constructs its map online by self-organizing waypoints during navigation.

3 METHOD

ETPNav addresses continuous-environment instruction following with a hierarchical system that builds an online topological map, plans globally over it, and executes plans through low-level control.

  • 3.1 Topological Mapping: ETPNav maintains an episode-specific topological map by organizing predicted waypoints and representing visited, current, and unexplored ghost locations as graph nodes.Edges connect directly reachable locations and store relative Euclidean distances.
  • 3.1 Topological Mapping: The waypoint predictor generates nearby candidate waypoints from panoramic depth and orientation features, using a fixed pre-trained transformer-based predictor.The resulting waypoint representations are incorporated into the graph as the agent explores.
  • 3.1 Topological Mapping: Unlike RGBD-based waypoint prediction, ETPNav uses depth-only inputs because waypoints represent spatial accessibility rather than semantic information.The paper motivates this design by noting that RGB information may be unhelpful or detrimental for accessibility inference.
  • 3.2 Cross-Modal Planning: The planning module encodes instructions and performs cross-modal graph reasoning over the topological map to predict a long-term goal and output a topological path.Its graph transformer augments node attention with all-pair shortest-path distances from graph edges.
  • 3.3 Control: The controller executes the planned path with low-level actions, while unreachable predicted ghost goals are deleted to prevent repeated selection and stalled control.Deleting the selected ghost node also reduces the candidate pool for subsequent long-term goal prediction.

4.1 Experimental Setup

Experiments evaluate ETPNav on continuous versions of R2R and RxR using standard navigation metrics and implementation configurations. The setup contrasts dataset difficulty, model components, and planner benchmarks.

  • Datasets: R2R-CE and RxR-CE convert discrete R2R and RxR paths into continuous environments through the Habitat Simulator.Both provide step-by-step language guidance but differ in path length, guidance granularity, and agent embodiment.
  • Datasets: R2R-CE contains 5,611 shortest-path trajectories, approximately three English instructions per trajectory, and 9.89m average paths.Validation includes seen scenes with novel paths and instructions, and unseen scenes, paths, and instructions.
  • Datasets: RxR-CE is more challenging than R2R-CE, with multilingual instructions averaging 120 words, longer 15.23m paths, and stricter obstacle constraints.Its agent has a larger 0.18m chassis radius and cannot slide along obstacles.
  • Metrics: Evaluation uses TL, NE, SR, OSR, SPL, and NDTW to measure path length, navigation error, success, and path fidelity.SR counts paths whose final navigation error is below 3 meters.
  • Implementation: The model uses ViT-B/32 for RGB encoding, ResNet-50 for depth encoding, and 2-, 9-, and 4-layer panorama, text, and cross-modal graph encoders.Training includes 100,000 pre-training iterations with AdamW, batch size 64, and learning rate 5e-5.
  • Comparisons: Tables 2 and 3 compare ETPNav with state-of-the-art methods on R2R-CE and with other VLN planners.The supplied captions identify the datasets and comparison scope, but not the table entries.

4.2 Comparison with State-of-the-art Methods

ETPNav outperforms prior methods across R2R-CE and RxR-CE comparisons, including substantial gains on unseen splits. Its global topological planning supports long-term navigation, while the RxR-CE results reveal a path-fidelity trade-off.

  • R2R-CE: 13 SR and 10 SPL separate ETPNav from CWP-RecBERT on R2R-CE validation unseen.On test unseen, ETPNav exceeds Sim2Sim by 11 SR and 11 SPL.
  • R2R-CE: 6 SR and 3 SPL separate ETPNav from Reborn on R2R-CE test unseen.The comparison highlights the difference between ETPNav’s global topological planning and Reborn’s local planning space.
  • RxR-CE: ETPNav records 41.30 SDTW versus Reborn’s 38.43 on RxR-CE test unseen, although Reborn records higher NDTW, 55.43 versus 54.11.The paper attributes the difference to global planning that promotes backtracking but may affect path fidelity.

4.3 Ablation Study

The ablations show that ETPNav’s performance depends on global topological planning, carefully designed waypoint maps, and obstacle-aware control, especially under sliding-forbidden conditions.

  • Waypoint Prediction: Depth-only waypoint prediction generalizes better than RGB-only prediction, whose %Open falls to 65.34 and whose navigation performance is worst.Without depth, many predicted waypoints are obstructed or off the navigation mesh.
  • Map Construction: Increasing the localization threshold reduces graph nodes and can cause an approximately 12 SR drop on the R2R-CE val-unseen split.Larger candidate-node sets can also increase planning difficulty, with 56.71 SR versus 57.21 SR in the reported comparison.
  • Planning Space: Global planning achieves 57.21 SR versus 53.92 SR for local planning on the R2R-CE val-unseen split.Global planning uses all observed ghost nodes, whereas local planning considers only adjacent ghost nodes.
  • Chassis Radius: RF consistently outperforms PointGoal across chassis radiuses and beats it by about 3 SR on RxR-CE.Success rates decline as chassis radius increases, while Tryout helps RF adapt across chassis sizes.
  • Qualitative Results: Qualitative trajectories show global planning corrects wrong turns, whereas local planning can oscillate until navigation failure at step 15.With global planning, a long-term goal redirects the agent onto the correct track and yields successful navigation.
  • Qualitative Results: Sliding-forbidden collisions can create persistent deadlocks, which Tryout is designed to prevent.Without Tryout, the agent remains blocked after chassis-wall contact through the end of the episode.

5 CONCLUSION

ETPNav introduces online topological mapping for long-range VLN-CE planning and a trial-and-error controller for obstacle avoidance. It reports over 10% and 20% absolute improvements over prior state-of-the-art on R2R-CE and RxR-CE, respectively, while identifying sensor actuation noise as future work.

  • 5 CONCLUSION: ETPNav performs online topological mapping by self-organizing predicted waypoints without prior environmental experience.The resulting map supports robust long-range planning in realistic navigation scenarios.
  • 5 CONCLUSION: A concise depth-only waypoint-prediction design is empirically shown to be optimal in the reported experiments.The paper systematically examines key topological-map design choices.
  • 5 CONCLUSION: A trial-and-error controller addresses obstacle avoidance, an often-neglected issue in VLN-CE.The controller is presented as simple and effective for the task.
  • 5 CONCLUSION: More than 10% and 20% absolute improvements over prior state-of-the-art are reported on R2R-CE and RxR-CE, respectively.The paper positions ETPNav as a strong baseline for further research on VLN-CE.
  • Future Work: Future work must address complexities introduced by sensor actuation noise in real-world topological mapping.ETPNav currently employs pose-reading sensors to facilitate mapping.
Loading 2304.03047v3…