Source-linked AI summary
Neural Topological SLAM for Visual Navigation
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, Saurabh Gupta
TL;DR
Image-goal navigation in unseen environments requires spatial representations that scale and remain usable under actuation noise while exploiting structural priors. Neural Topological SLAM builds semantic, graph-based maps with coarse geometric links and achieves over 50% relative improvement over existing methods in realistic simulation.
Problem
Navigation methods often rely on metrically precise maps that scale poorly and are brittle to actuation noise, while structural priors are usually not modeled explicitly.
Method
Neural Topological SLAM builds graph-based maps with panoramic semantic nodes, approximate relative poses, and learned geometric and semantic directional functions.
Results
NTS outperforms all baselines across difficulty levels, with overall Succ/SPL of 0.55/0.38 versus 0.35/0.23 in RGB and 0.63/0.43 versus 0.44/0.29 in RGBD.
Takeaways & Limitations
Topological representations provide robustness to actuation noise and semantic node features support efficient exploration in novel environments without prior environmental experience.
Takeaways & Limitations
The method assumes access to learned geometric and semantic functions, which are trained using supervised learning.
Abstract
from arXiv · showhide
This paper studies the problem of image-goal navigation which involves navigating to the location indicated by a goal image in a novel previously unseen environment. To tackle this problem, we design topological representations for space that effectively leverage semantics and afford approximate geometric reasoning. At the heart of our representations are nodes with associated semantic features, that are interconnected using coarse geometric information. We describe supervised learning-based algorithms that can build, maintain and use such representations under noisy actuation. Experimental study in visually and physically realistic simulation suggests that our method builds effective representations that capture structural regularities and efficiently solve long-horizon navigation problems. We observe a relative improvement of more than 50% over existing methods that study this task.
1 Carnegie Mellon University, 2 Facebook AI Research, 3 UIUC · 1. Introduction
The paper frames visual navigation as requiring both spatial representations and structural priors, then proposes Neural Topological SLAM to combine graph-based mapping with learned geometric and semantic guidance.
- 1. Introduction: Humans use structural priors to choose paths toward semantically likely destinations, such as reaching an oven through an accessible kitchen.Navigation algorithms often struggle to exploit these semantic priors and common-sense cues.
- 1. Introduction: Successful visual navigation requires building and storing spatial representations while exploiting structural priors.The introduction identifies these as two main components of a successful navigation algorithm.
- 1. Introduction: Metric maps are brittle because they scale poorly with environment size and experience and are vulnerable to real-robot actuation noise.The paper motivates a representation that avoids dependence on metrically precise free-space maps.
- 1. Introduction: The proposed topological representation uses graph-connected nodes linked by local geometric information instead of metric maps.This design directly targets brittleness to localization and noise.
- 1. Introduction: Each node is represented by a 360-degree panoramic image, and approximate relative poses connect nodes in the visual topological map.The representation combines visual observations with coarse geometric relationships.
- 1. Introduction: Learned functions Fg and Fs estimate directional free-space availability and the likelihood of encountering the target image, respectively.Fg supports geometric exploration, while Fs explicitly models semantic navigation structure.
- 1. Introduction: The representation supports efficient long-term planning, structural-prior exploitation, exploration, and online map building through graph structure and learned functions.All functions and policies can be learned in a completely supervised manner.
2. Related Work
Related work spans navigation tasks, spatial representations, learned memories, and training methodologies. This paper focuses on reaching a target image in a novel environment without prior experience from that environment.
- Navigation Tasks: Navigation tasks divide into known-goal problems emphasizing retrieval and execution, and novel-environment exploration problems requiring semantic and common-sense reasoning.Examples include collision-free wandering, object following, coordinate reaching, image- and language-guided navigation, object or room finding, and explicit exploration.
- Navigation Tasks: The paper studies reaching a target image in a novel environment with no available environmental experience.This setting is contrasted with tasks where the goal location is known and only limited exploration is necessary.
- Classical Space Representations: Classical navigation research has used metric spatial representations, sensor-specific construction methods, localization procedures, semantic associations, and non-metric topological representations.These approaches provide the historical foundation for spatial and topological representations in robot navigation.
- Learned Space Representations: For short-range locomotion, purely reactive policies can suffice, whereas target-driven navigation in novel environments has motivated memory-based policies using LSTMs, transformers, spatial memories, and topological approaches.Purely reactive strategies do not work well for the more complex target-driven setting.
- Learned Space Representations: Learned spatial approaches can acquire expressive representations but are bottlenecked by metric consistency and have mainly been demonstrated in discrete state spaces for comparatively short-horizon tasks.The passage contrasts these limitations with learned topological approaches informed by classical navigation literature.
- Training Methodology: Navigation-policy training methods include reinforcement learning, imitation learning with DAgger, and self-supervised learning for individual components.Reinforcement learning supports rich exploratory behavior but is hard and sample inefficient, while imitation learning is sample efficient.
3. Task Setup
The task is episodic image-goal navigation: an agent receives a target image and sequential visual observations with noisy pose estimates, then learns actions to reach that goal.
- Task formulation: At episode start, the agent receives a target goal image, I_G.The goal image indicates the destination to reach.
- Task formulation: At each time step, the agent observes a first-person panoramic image, I_t, and a pose estimate from a noisy motion sensor.The observation is denoted s_t and combines visual and motion information.
- Task formulation: The agent takes a navigational action a_t at each time step and learns a policy π(a_t|s_t, I_G) to reach the goal.The environment is episodic and autonomous.
4. Methods
NTS builds and maintains a semantic topological graph from panoramic observations, then uses global graph planning and local visual control for image-goal navigation. Its components rely on four learned functions and a shared supervised multi-task model.
- Topological representation: NTS represents space as a graph whose panoramic-image nodes denote visible areas and whose adjacency edges store relative poses.The graph at time t is denoted G_t, with nodes N_i, images I_Ni, edges E_i,j, and relative poses Δp_ij.
- Model components: The model has three components: Graph Update maintains the map, Global Policy selects long-term goals and subgoals, and Local Policy navigates from visual observations.The Global Policy uses path planning, while the Local Policy acts on visual observations to reach the selected subgoal.
- Learned functions and training: The method defines four functions: graph localization, 12-direction geometric exploration prediction, 12-direction semantic scoring, and relative pose prediction.A shared ResNet18 encoder is trained in a single supervised multi-task model to learn all four functions.
- Graph Update: Graph Update localizes observations, creates or connects nodes, and adds ghost nodes for predicted unexplored areas using geometric explorable-area estimates.Ghost nodes represent predicted explorable directions and are connected to the newly observed node.
- Global Policy: Global Policy localizes the goal when possible; otherwise, it scores ghost nodes semantically, selects the highest-scoring one, and plans a shortest path with Dijkstra’s algorithm.The next node on that path becomes the subgoal, whose relative pose is passed to the Local Policy.
5. Experimental Setup
Experiments use visually realistic Gibson scenes in Habitat with physically realistic motion noise, evaluating RGB and RGBD image-goal navigation under defined action, sensor, training, and episode protocols. The study compares NTS against imitation-learning and reinforcement-learning baselines trained under specified data and optimization conditions.
- Environment: Experiments use Habitat with the visually realistic Gibson dataset and simulated physically realistic motion-sensor and actuation noise.Actuation noise produces stochastic transitions and includes translational noise during rotations and rotational noise during translations.
- Task setup: 128 × 512 panoramic RGB or RGBD images, 3×1 odometry, and four actions define the navigation task.Actions are move forward approximately 25cm, turn right or left approximately 10 degrees, and stop.
- Training data: 86 scenes are split into 68/4/14 train/validation/test sets, with 300 sampled images per training scene yielding approximately 6.12 million labeled pairs.Labels are generated automatically from each scene’s available ground-truth map.
- Test episodes: Test episodes span Easy (1.5 − 3m), Medium (3 − 5m), and Hard (5 − 10m) goal distances, with 500 maximum steps per difficulty.Difficulty is determined by the distance between starting and goal locations.
- Baselines: Baselines include ResNet + GRU + IL, Target-driven RL, and Metric Spatial Map + RL, alongside NTS in RGB and RGBD settings.The baselines use imitation learning or reinforcement learning, with Target-driven RL adapted from Zhu et al. and Metric Spatial Map + RL projecting depth geometrically.
- Baseline training: RL baselines train for 25 million frames with PPO and dense rewards, while imitation learning uses behavioral cloning on ground-truth trajectories.All baselines use the ground-truth map for training; their maximum RL sample count is 10.2 million from 300 episodes across 68 scenes and 500 steps.
6. Results
Across 1000-episode evaluations, NTS outperforms baselines in both RGB and RGBD settings, while sequential-goal ablations show that its graph and semantic score components are important. NTS also exceeds end-to-end RL and metric spatial-map methods, whose results are sensitive to stop-action design and accumulated pose error.
- Comparison with end-to-end RL and the effect of stop action: NTS performs better than both end-to-end RL baselines and methods using metric spatial maps across the evaluated settings.The paper attributes weaker RL performance to the complexity of the exploration search space.
- Comparison with end-to-end RL and the effect of stop action: Without the stop action, RL baselines perform much better, while NTS performance also increases.The paper evaluates all models without the stop action in Table 2(left).
- Comparison with spatial map-based methods and the effect of motion noise: Metric spatial-map baselines deteriorate quickly as goal distance increases, likely because accumulating pose errors make maps noisy and path planning incorrect.The models are also evaluated without motion actuation and sensor noise in Table 2(right).
- Sequential goals: Sequential-goal tests sample goals 1.5m to 5m from the previous goal, provide 500 timesteps per goal, and measure success over 1000 episodes.Success is defined as the ratio of goals reached across the test set.
- Sequential goals: As sequential goals increase, all models’ performance decreases; NTS’s gap over the score-function ablation shrinks, while its gap over the graph ablation grows.NTS remains considerably higher than both ablations, indicating the importance of both components and increasing importance of the topological map over time.
7. Discussion
The paper introduces semantic topological representations that support coarse geometric reasoning for image-goal navigation. These representations can be built autonomously, tolerate actuation noise, and enable efficient exploration in novel environments.
- Representations: The proposed topological representations leverage semantic features alongside coarse geometric reasoning.Nodes store semantic features, while their interconnections encode coarse geometric information.
- Navigation: The method autonomously builds and uses these representations for image-goal navigation.The paper demonstrates autonomous construction and deployment of the representations for the navigation task.
- Robustness and exploration: The topological representations provide robustness to actuation noise and support efficient exploration in novel environments.Stored semantic features allow the method to exploit statistical regularities during exploration.
A. NTS Multi-task Learning Model Architecture and Training Details
The NTS Multi-task Learning model combines four modules to reason over panoramic source and goal images through patch-based representations. It is jointly trained with supervised classification and regression losses using Adam.
- Architecture: The model comprises a ResNet18 encoder, Connection model, Inter-node Predictions model, and Intra-node Predictions model.These four modules form the NTS Multi-task Learning architecture.
- Panoramic Representation: Each 128 × 512 panoramic image is divided into nθ = 12 overlapping 128 × 128 square patches centered at evenly spaced angles.Adjacent patches overlap by one-third.
- Prediction Tasks: The Connection model predicts whether source and goal images belong to the same node, while the Intra-Node Predictions model estimates goal direction and score when they do.The score is equivalently the distance of the goal image relative to the source image.
- Training: The model is jointly trained with all the 5 losses using supervised learning, Cross-Entropy Loss for connection and direction labels, and MSE Loss for score labels.Loss coefficients are 10 for score labels and 1 for connection and direction labels.
- Training: The Adam optimizer trains the model with a learning rate of 5e-4 and a batch size of 64.These settings are used for the joint multi-loss training procedure.
B. Dataset collection and automated labeling
The dataset samples 300 points per training scene and generates labels for image pairs using visibility and coarse geometric reasoning. Labels distinguish node connectivity, intra-node direction and score, and inter-node direction and score.
- Data collection: 300 points are randomly sampled in each training scene, and labels are collected for every ordered source-goal image pair.The source image is I_S and the goal image is I_G.
- Connection labels: The connection label is 1 when a 5-degree depth patch shows the goal location is visible from the source, and 0 otherwise.The label compares the patch’s maximum depth with the source-goal distance.
- Intra-node labels: For connected image pairs, the intra-node direction uses a 30-degree bin of the relative angle, while the score uses the relative distance.The relative position is Δp = (d, θ), and direction is discretized while score remains continuous.
- Inter-node labels: For disconnected pairs, inter-node directions are computed from an obstacle map using 12 directions at radius r = 3m, ignoring obstacles beyond 3m.Shortest path distances determine whether candidate directional points are traversable.
- Inter-node labels: Inter-node scores use the farthest explored traversable point in each direction and its geodesic distance to the goal, with dmax = 20m yielding score 0 above that distance.The scores are denoted by s_i for i ∈ [1, 2, . . . , 12].