Source-linked AI summary
Learning Exploration Policies for Navigation
Tao Chen, Saurabh Gupta, Abhinav Gupta
TL;DR
Task-independent exploration for new environments is under-studied despite its relevance to downstream navigation. The paper learns exploration policies using spatial memory, sensor-derived coverage rewards, and human-demonstration bootstrapping, and reports improved exploration and downstream-task performance.
Problem
Task-independent exploration of new environments has received less attention than task-driven navigation, despite supporting generic spatial understanding for downstream tasks.
Method
The paper learns exploration policies while studying policy architectures, intrinsic rewards, and optimization with human demonstrations and on-board sensor signals.
Results
Learned exploration policies explore better than classical geometry-based and generic learning-based approaches, and their experience improves downstream navigation-task performance.
Takeaways & Limitations
Spatial-memory policies bootstrapped by imitation learning and fine-tuned with sensor-derived coverage rewards can effectively explore novel environments.
Takeaways & Limitations
The method assumes an RGB-D camera with depth clipped at 3m and a bump sensor, and uses estimated rather than necessarily true position.
Abstract
from arXiv · showhide
Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that the use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Code and Videos are available at: https://sites.google.com/view/exploration-for-nav.
1 INTRODUCTION
The paper addresses task-independent exploration for navigation, a step needed to build generic spatial understanding but often overlooked. It proposes learning exploration policies through architecture, reward, and training design, combining human supervision with intrinsic coverage rewards.
- Task-independent exploration helps agents become familiar with new environments before performing navigation tasks, yet current navigation approaches often overlook it.
- An exploration policy is a bottleneck for both geometric and learning-based navigation because it must build maps or collect trajectories efficiently without task-specific goals.
- Heuristic exploration can be brittle under localization noise, restrictive about interactions and semantics, and dependent on specialized range sensors.
- Intrinsic-reward reinforcement learning can remain sample inefficient when black-box rewards are difficult to differentiate for effective gradient computation.
- The proposed approach studies policy architectures, reward functions, and optimization by combining human exploration supervision with intrinsic rewards, including coverage rewards that outperform curiosity.
2 RELATED WORK
The paper connects learned exploration to active SLAM, learning-based navigation, intrinsic-reward exploration, and system identification. Its focus is task-agnostic exploration in complex, realistic 3D environments with long spatio-temporal dependencies.
- Classical navigation builds maps, localizes agents, and plans paths, but most prior SLAM research assumes human-operated traversals rather than automatic exploration.
- Learning-based navigation research investigates policy architectures for representing space, including feed-forward networks, neural memory, spatial memory, and topological maps.
- The paper’s sensor-derived exploration signals support real-world deployment, while learned collision-avoidance methods target open space rather than exploring an entire environment.
- Intrinsic-reward exploration methods learn policies from novelty-based rewards, whereas this work studies that idea specifically for navigation in realistic 3D environments.
- System-identification approaches explore before goal-driven behavior, but this paper addresses complex realistic 3D environments with very long spatio-temporal dependencies.
3 APPROACH
The approach maintains an estimated-position map from RGB-D observations and uses spatially informed recurrent policies to explore. It trains with sensor-derived coverage and collision rewards, using human demonstrations to reduce reinforcement-learning sample complexity.
- Map Construction: The agent updates an allocentric map by projecting depth-camera observations into 3D and transforming them with its estimated position.
- Map Construction: Because wheel slippage can make estimated and true positions differ, the constructed map is only approximate.
- Policy Design: A good exploration policy must avoid obstacles, recognize semantic cues such as doors, and track explored and unexplored regions.
- Policy Design: The architecture fuses RGB observations with egocentric occupancy-map inputs and passes the combined information through a recurrent policy.
- Reward Design: Coverage reward equals the increase in mapped traversable or non-traversable area, while collisions receive a bump-sensor penalty.
- Training Procedure: Human exploration demonstrations are used to pre-train the policy before reinforcement-learning optimization, addressing the sample complexity of long-term behavior.
4 EXPERIMENTS
The experiments evaluate task-agnostic exploration in novel 3D environments, testing observation inputs, training strategies, intrinsic rewards, robustness, and downstream navigation utility. The learned policy combines human-trajectory imitation with reinforcement learning and is compared with geometric and curiosity-based baselines.
- Experimental Setup: Experiments use House3D with separate training and testing houses, RGB-D observations, bump sensing, and six discrete motion primitives.Testing houses are unseen during training; exploration uses no externally specified environmental reward.
- Training and Reward Design: The proposed training pipeline imitates human exploration trajectories and then optimizes sensor-derived intrinsic rewards with PPO.The intrinsic reward combines coverage and collision terms, with hyperparameters controlling aggressiveness.
- Coverage Quality: Without estimation noise, frontier-based exploration performs best, but the full Map + RGB system outperforms random, curiosity-based, straight-line, and RGB-only alternatives.The frontier baseline benefits from perfect registered maps and optimal path planning, whereas the learned policy can operate from RGB images.
- Coverage Quality: With noisy state estimation, frontier-based coverage drops rapidly, while the learned agent remains effective and exceeds the frontier baseline at 4% noise.With 10% noise affecting intrinsic-reward computation and map construction, performance is 98.9m2 versus 117.4m2 when noise affects test-time map construction only.
- Coverage Quality: Under geometry-affordance mismatch, frontier-based performance drops substantially, whereas the learned agent has a relatively minor decline by learning that doors are traversable.The mismatch renders doors but excludes them from collision checking.
- Ablation Study: Ablations show that RGB images, map inputs, imitation learning, and the intrinsic coverage reward improve exploration, while imitation learning also reduces variance.The downstream evaluation reports improved path efficiency when using exploration data from the learned policy.
5 DISCUSSION
The paper argues that learning exploration policies is needed for novel 3D navigation environments and that gathered exploration experience improves downstream task performance.
- Learned exploration policies are designed and trained for navigation in novel 3D environments.
- Experience gathered by these policies enables better performance on downstream tasks.
- The authors identify semantic exploration with more expressive architectures, reward functions, and training techniques as future research directions.
APPENDIX A SPL METRIC
The appendix defines SPL and illustrates the multi-scale visual and map inputs used by the exploration policy as the agent builds a map.
- SPL measures navigation efficiency using success and the ratio between shortest-path and executed-path lengths over N trials.Higher SPL indicates better performance.
- Policy inputs include RGB images, coarse maps, and fine maps observed at times t and t + 10.
- Snapshots show how the built map changes as the agent explores the house.
C.1 ENVIRONMENT DETAILS
The experiments use House3D with separate training and testing houses, whose layouts and traversable areas are summarized in the appendix figures.
- House3D experiments use 20 training houses and 20 new testing houses with no overlap.The split evaluates generalization to previously unseen houses.
- Training and testing houses have average movable areas of 289.7 m2 and 327.9 m2, respectively.
- Top-down layout examples encode traversable regions in white and occupied regions in black.
C.2 MAP RECONSTRUCTION
The map-reconstruction procedure converts depth-image pixels into world-coordinate points using camera poses and intrinsics, then merges the reconstructed points.
- Given camera poses, depth images, and camera intrinsics, the method reconstructs a scene point cloud using multiple-view geometry.
- Equation (2) relates each depth-image pixel to its corresponding world-coordinate point through the camera intrinsics and pose.
- The reconstructed world points are merged to form the scene representation.
C.3 TRAINING DETAILS
The policy combines coarse and detailed occupancy maps with RGB inputs, processes them through ResNet18 encoders and a GRU, and uses separate policy and value heads.
- Policy inputs: The policy receives coarse and detailed occupancy maps covering 40m×40m and 4m×4m areas at 0.5m and 0.05m resolutions, respectively.The agent-generated occupancy map has 0.05m resolution; the coarse map is down-sampled from 800 × 800 to 80 × 80.
- Visual and spatial encoding: Three ResNet18 outputs are concatenated into a 384-dimensional vector and transformed into a 128-dimensional representation.Each ResNet18’s final fully connected layer is modified to size 128 before concatenation.
- Recurrent policy: A single-layer GRU with 128 hidden units processes the representation before separate policy and value heads.The policy head uses a 128 −32 −6 fully connected architecture.
- Optimization and reward: Training uses 500-step episodes, sequence length 20, and coverage area measured by the number of covered occupancy-map grids.The coverage-reward coefficient α for R_cov is introduced in the training details.
- Optimization and reward: The network is optimized with Adam at learning rate 0.00001, with PPO entropy loss coefficient 0.01.The intrinsic-reward coefficient is 0.006.
C.4 NOISE MODEL
The noise model recursively estimates pose by adding truncated Gaussian perturbations to action primitives, starting from the origin and iterating until the step limit.
- Initialization: The estimated initial pose is set to the origin, ˆx0 = x0 = 0.This initialization is stated without loss of generality.
- Pose estimation: At each step, the agent adds truncated Gaussian noise to the executed action primitive to estimate the next pose.The update is ˆxt+1 = ˆxt + ˜at, where ˜at is the noisy action primitive.
- Iteration: The procedure repeats the noisy action-and-estimation step until the maximum number of steps is reached.The model therefore propagates pose estimates recursively across the episode.
- Assumptions and limitations: The model does not use oracle odometry and produces compounding errors over time, although it may not perfectly match real-world noise.The estimate depends on the previous estimated pose and executed action.
C.5 IMITATION LEARNING DETAILS
Imitation learning bootstraps exploration by cloning human behavior from exploration trajectories, after filtering and segmenting the demonstrations.
- Demonstration data: Behavioral cloning is used to imitate human exploration behavior from trajectories collected by Das et al. (2018).Only the exploration trajectories were retained; the question-answering portion was ignored.
- Data preprocessing: Trajectories shorter than 100 time steps were removed before conversion into short trajectory-segment sequences.The cleaned demonstrations are represented as sequences containing states and actions.