PaperlayerView exact revision on arXiv

Source-linked AI summary

Topological Planning with Transformers for Vision-and-Language Navigation

Kevin Chen, Junshen K. Chen, Jo Chuang, Marynel Vázquez, Silvio Savarese

arXiv:2012.05292v1cs.ROcs.AIcs.CLcs.CV

TL;DR

VLN systems often struggle when agents can freely traverse environments, motivating alternatives to fully end-to-end learning. This paper uses topological maps with a cross-modal transformer for planning and a robust controller for low-level execution, and reports stronger navigation performance than an end-to-end baseline.

  • Problem

    End-to-end VLN systems struggle with freely traversable movement and can require extensive experience, while performance degrades without ground-truth odometry.

  • Method

    The system constructs topological maps, predicts navigation plans from instructions with a cross-modal transformer, and executes them through a robust low-level controller.

  • Results

    The modular topological-map approach outperforms the end-to-end VLN-CE baseline in navigation performance, with the authors reporting highest success rates and SPL for their CMTP models.

  • Takeaways & Limitations

    The modular design supports interpretable planning and robust navigation behavior, including correction of mistakes during execution.

  • Takeaways & Limitations

    Performance depends on topological-map quality, and the evaluation assumes maps built from prior exploration while the agent lacks positional information during navigation.

Abstract

from arXiv · show

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. forward, rotate) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigation plans, and exhibits intelligent behaviors such as backtracking.

1. Introduction

VLN systems often struggle in freely traversable environments, motivating a modular framework that uses topological maps, attention-based planning, and low-level control. The proposed system produces interpretable navigation plans and supports robust execution.

  • Motivation: End-to-end VLN models perform well with constrained movement but degrade when agents can move freely, while fully end-to-end mapping, planning, and control can require extensive experience and ground-truth odometry.These approaches may require millions of frames and show substantial performance degradation without ground-truth odometry.
  • Motivation: Topological maps represent places as nodes and connectivity as edges, supporting efficient planning, interpretable plans, and language-grounded navigation.Their spatial discretization can facilitate learning relationships between instructions and locations.
  • Approach: The transformer predicts one topological-map node at a time, attending to relevant instruction words and spatial regions during navigation planning.This supports associations such as relating “bedroom” in the instruction to the corresponding physical room.
  • Approach: The controller executes the predicted plan by sequentially traversing its subgoal nodes, activating the next node as each current node is reached.This figure describes the handoff from high-level planning to sequential plan execution.
  • Approach: The system decomposes VLN into planning and control: a cross-modal transformer computes a global plan from language and a topological map, then a local controller outputs low-level actions.The setup allows free traversal using discrete actions rather than movement restricted to predefined locations.
  • Results: Experiments on VLN-CE show that cross-modal attention-based planning is effective and that the modular system supports correction of navigation mistakes.The authors specifically report correction of errors such as moving in the wrong direction.

2. Related Work

Prior VLN research largely used end-to-end models and simplified action spaces, while related navigation work explored structured memory and modular systems. This paper builds on those directions with topological map construction from exploration trajectories.

  • Vision-and-language navigation: Much prior VLN progress used end-to-end models in simulators with panoramic actions between predefined locations, simplifying movement relative to freely traversable navigation.This setup also avoids directly addressing action-direction association and motion feasibility.
  • Vision-and-language navigation: Attention-based transformer approaches in VLN have used cross-modal pretraining and task-specific fine-tuning to improve generalization across navigation tasks.The related approaches draw on visual-language pairings from web data or demonstrations.
  • Topological map construction: The proposed map-construction process explores each environment multiple times, sparsifies each trajectory graph, and merges the resulting graphs into one map.The merged graph supports later localization and planning.
  • Memory for visual navigation: Navigation research progressed from reactive and unstructured-memory methods toward explicit metric and topological map representations.These representations store information about environments to support navigation and exploration.

3. Method

The method builds a topological map during exploration, then separates VLN into interpretable planning and low-level control. A cross-modal transformer predicts a node-by-node path from the map and instruction, while hierarchical controllers execute it through waypoints and discrete actions.

  • System Overview: The modular framework separates global planning from control, improving interpretability, robustness, and flexibility within the stated approach.The planner produces an interpretable path before the controller repeatedly localizes and acts toward the destination.
  • Topological Map Representation: The agent explores first to construct a topological graph whose nodes are observations and edges represent connectivity or reachability.Multiple exploration trajectories are sparsified and merged; nodes use fixed-orientation 360° panoramas to reduce redundant representations.
  • Topological Map Representation: The map representation combines node observations with edge geometry, including eight directions and three distance ranges, to support localization and language-informed planning.The edge categories are computed with ground-truth odometry during map construction, while that assumption is relaxed during navigation testing.
  • Cross-Modal Planning: The planner predicts a navigation path as a sequence of map nodes, using a GNN and cross-modal transformer to select nodes autoregressively until the plan ends.The GNN encodes visual appearance and connectivity, while the transformer conditions on the instruction and prior predicted nodes.
  • Cross-Modal Planning: The cross-modal transformer encodes map and language branches separately, fuses them through attention, and predicts one navigation-plan node at each step.Map features from the GNN and word embeddings are passed into the cross-modal transformer to produce the plan.
  • Hierarchical Control: The controller converts the planned topological path into low-level actions by predicting waypoints from RGBD observations, subgoal nodes, and heading information.A high-level controller predicts relative polar-coordinate waypoints, which are then translated into robot-specific actions.

4. Experiments

Experiments evaluate the planner, controller, and integrated system in simulated VLN environments using topological maps and low-level actions. The modular system achieves strong planning and execution performance, while exposing interpretable attention and backtracking behavior.

  • Evaluation setup: The evaluation uses Interactive Gibson, agent-generated and R2R graphs, and seen and unseen VLN-CE environments.Agents explore environments before navigation and construct agent-generated topological maps.
  • Planner evaluation: The CMTP planner outperforms the GNN baseline and performs better on R2R graphs, while performance is higher in seen than unseen environments.CMTP shows the most prominent overfitting; CMTP-BCE reduces overfitting at the cost of lower seen-environment performance.
  • Planner evaluation: CMTP-NS has significantly higher oracle success than all models, indicating accurate location selection but difficulty timing the stop action.The result isolates stopping as a major planning challenge.
  • Planner evaluation: Cross-modal attention aligns earlier instruction words with earlier planned nodes and later landmark words with later trajectory nodes.The attention visualization shows node-to-instruction correspondence across the predicted plan.
  • Controller evaluation: The controller follows multi-node plans, avoids collisions despite obstacle-cutting graph edges, and learns to backtrack after deviations.Controller performance is higher on R2R graphs than AG graphs because their node placement provides more informative observations.
  • VLN evaluation: The integrated CMTP system achieves the highest VLN success rates and SPL compared with the GNN and VLN-CE baselines.The planner produces high planning success while the controller executes plans with low-level discrete actions.
  • VLN evaluation: The modular system makes navigation intent interpretable by exposing predicted plans and executed trajectories.Qualitative examples show the controller converting planned paths into smooth trajectories between specified objects.

5. Conclusion

The paper presents a modular VLN system that builds topological maps from exploration trajectories and uses reachability to sparsify them. The approach improves navigation performance over an end-to-end baseline.

  • Conclusion: The modular topological-map approach improves navigation performance compared with an end-to-end baseline.
  • Conclusion: The system builds topological maps from exploration trajectories before navigation.Exploration uses sampled waypoints, agent navigation, observations, and odometry to create dense trajectories.
  • Conclusion: A reachability estimator sparsifies exploration trajectories into maps by retaining positions that are sufficiently reachable.Reachability depends on unobstructed line of sight and a distance threshold.

A.2.2 Sparsifying a dense trajectory

Dense exploration trajectories are converted into sparse topological graphs by greedily removing intermediate positions, merging maps, and applying a final node-merging step. The resulting maps differ from predefined R2R graphs in density and node placement.

  • A.2.2 Sparsifying a dense trajectory: Intermediate trajectory positions are discarded when a later position has sufficiently high reachability from the current node.The procedure greedily selects the next node while preserving the reachability threshold.
  • A.2.2 Sparsifying a dense trajectory: Multiple sparsified maps are merged by adding edges between nodes whose reachability exceeds the sparsification threshold.
  • A.2.2 Sparsifying a dense trajectory: Nodes are further merged when their reachability exceeds the merging threshold, while neighbors are retained only if post-merge reachability remains sufficient.
  • A.2.2 Sparsifying a dense trajectory: The setup uses d_sp = 4m, p_sparse = 0, and p_merge = 0.5, producing straight-line edges within 4m and merging nodes within 2m.
  • A.2.2 Sparsifying a dense trajectory: Agent-generated maps are generally sparser than R2R graphs, which contain dense node clusters and may place nodes near walls.Exploration figures show trajectories, traversable regions, sparse coverage, and directed edge orientations using discretized polar coordinates.

A.3. Topological Map Representation

The topological map represents visual places as graph nodes and spatial connections as discretized directional edge categories.

  • A.3. Topological Map Representation: Each topological-map node is encoded with a ResNet152 feature.
  • A.3. Topological Map Representation: Each edge is represented by an embedding corresponding to its discretized polar-coordinate category.The categories encode directional information for graph connections.

B. Method

The method processes topological maps with six sequential graph-network blocks using node, edge, and global features. Update functions are multilayer perceptrons, while aggregation functions use element-wise summation.

  • B. Method: The graph neural network contains 6 sequential graph-network blocks with node, edge, and global update and aggregation functions.
  • B. Method: All update functions are four-layer MLPs with batch normalization and ReLU, while aggregation functions are element-wise sums.
  • B. Method: The first graph-network block receives ResNet152 node embeddings, learned edge-category embeddings, and a learned global embedding.The input dimensions are 1024 for nodes, 256 for edges, and 512 for the global feature.
  • B. Method: Intermediate graph-network blocks use 1024-dimensional node features, 512-dimensional edge features, and a 512-dimensional global feature.
  • B. Method: The final graph-network block outputs 768-dimensional node features.

B.1.2 Cross-Modal Transformer

The system combines a cross-modal transformer planner with a modular controller that predicts and executes paths through topological maps. The high-level controller predicts waypoints from aligned panoramic observations, while a low-level controller converts them into movement actions.

  • Cross-modal transformer: The cross-modal transformer encodes language and topological-map nodes with self-attention, cross-attention, feed-forward layers, residual connections, and layer normalization.The architecture uses 9 language layers, 5 map layers, and 5 cross-modal layers.
  • Cross-modal transformer: The planner is trained with cross-entropy over map nodes and the [STOP] action, with CMTP variants changing whether stop prediction is included.CMTP-BCE additionally uses binary cross-entropy for the predicted stop action.
  • Controller: The controller processes RGBD panoramas for previous, current, and subgoal nodes, aligns them using heading information, and updates subgoals when predicted proximity satisfies ρ ≤ d.A bias function is applied to high-level predictions before the low-level controller translates waypoints into actions.
  • High-level controller: The high-level controller predicts a waypoint as a 24-sector direction index and numeric distance, using shared convolutional features followed by classification and regression heads.The classifier receives five concatenated feature components, while the regressor receives three.
  • High-level controller: The controller is trained offline on 2 million synthetic samples, optimizing direction with cross-entropy and distance with mean squared error.Training also uses random rotations of observation panoramas and distance values as augmentation.
  • High-level controller: Expert waypoints are obtained from lowest-cost paths on a traversability graph by selecting the farthest point within a maximum lookahead distance.The graph weights combine L2 travel distance with a penalty for proximity to walls.

C. Experiments

The experiments use iGibson and VLN-CE data, converting trajectories into paths over agent-generated and pre-defined topological graphs. Episodes whose start or goal positions were more than 2m from the nearest map node were discarded.

  • Experimental setup: Experiments use the iGibson simulator and VLN-CE data, converting trajectories into paths in agent-generated and pre-defined Room2Room topological graphs.Ground-truth 3D trajectory positions are matched to the closest nodes in the relevant maps.
  • Experimental setup: Episodes with start or goal positions more than 2m from the closest topological-map node were discarded before training and evaluation.

C.1. VLN Qualitative Results

Qualitative examples show that CMTP can generate plans near target locations and navigate through multiple rooms or past landmarks, but failures arise from overshooting and ambiguous instructions.

  • Success cases: CMTP generates plans that land near the ground-truth goal position and can traverse multiple rooms or pass landmarks named in instructions.
  • Failure cases: A planner may overshoot the ground-truth goal, while the controller can backtrack and self-correct before eventually reaching the planned destination.The illustrated episode reaches the bedroom after initially exiting it and making numerous corrections.
  • Failure cases: Ambiguous references such as “the couch” can cause the planner and controller to reach a couch different from the one intended by the instruction.The problem occurs when multiple couches are present in the environment.

C.2. Cross Modal Attention

Cross-modal attention visualizations associate instruction words with corresponding spatial regions and traversal nodes. The examples illustrate alignment between landmarks, directional language, and map locations.

  • Attention visualizations: The word “hallway” aligns with hallway nodes, while “bedroom” aligns with the bedroom node in a navigation example.
  • Attention visualizations: The word “across” correlates with nodes used to traverse a room, while “double doors” relates to the final node.These visualizations suggest that the model aligns instruction words with spatial locations.
  • Integrated examples: The integrated-system examples show successful plans whose execution follows directions and landmarks through low-level actions.
Loading 2012.05292v1…