Learning to Navigate in Complex Environments

Piotr Mirowski; Razvan Pascanu; Fabio Viola; Hubert Soyer; Andrew J. Ballard; Andrea Banino; Misha Denil; Ross Goroshin; Laurent Sifre; Koray Kavukcuoglu; Dharshan Kumaran; Raia Hadsell

Source-linked AI summary

Learning to Navigate in Complex Environments

Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell

arXiv:1611.03673v3Nov 11, 2016cs.AIcs.CVcs.LGcs.RO

TL;DR

The paper asks how agents can navigate complex, dynamic environments despite sparse rewards, partial observability, and memory demands. It combines goal-driven reinforcement learning with recurrent memory and auxiliary depth-prediction and loop-closure tasks. The resulting agents learn efficiently in 3D mazes, reach human-level performance on static mazes, and approach human scores when goals change.

Problem
Navigation in dynamic, partially observable environments is difficult because rewards are sparse and agents need memory for goals, observations, velocity, and environmental structure.
Method
The method combines A3C with a stacked LSTM and auxiliary depth-prediction and loop-closure objectives trained from multimodal sensory inputs.
Results
The agents reach human-level performance on Static 1 and 2 and attain about 91% and 59% of human scores on Random Goal 1 and 2.
Takeaways & Limitations
Depth prediction and loop closure provide richer training signals that bootstrap learning and enhance data efficiency while supporting navigation and localization analysis.
Takeaways & Limitations
The stacked LSTM has limited capacity for rapid memory, stretching navigation abilities in procedurally generated mazes and motivating external-memory architectures.

Abstract

from arXiv · show

Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks leveraging multimodal sensory inputs. In particular we consider jointly learning the goal-driven reinforcement learning problem with auxiliary depth prediction and loop closure classification tasks. This approach can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour, its ability to localise, and its network activity dynamics, showing that the agent implicitly learns key navigation abilities.

1 INTRODUCTION

The paper frames navigation as end-to-end reinforcement learning, while addressing sparse rewards, dynamic environments, and memory demands through auxiliary sensory-learning tasks.

Motivation: Navigation can emerge from a reward-maximizing policy learned jointly with task-relevant representations rather than explicit position inference and mapping.The approach keeps actions and representations coupled in an end-to-end framework.
Motivation: Sparse rewards and dynamic elements require memory for goal locations, temporal integration of velocity and vision, and persistent environmental structure.The paper identifies multiple memory timescales needed for navigation in partially observable environments.
Proposed direction: Auxiliary depth prediction supplies denser training signals and encourages representations supporting obstacle avoidance and short-term navigation.Depth is predicted from colour channels to exploit 3D geometric information.
Evaluation: A stacked LSTM addresses memory requirements in five 3D mazes with complex geometry, random starts, dynamic goals, and long episodes.The evaluation includes episodes requiring thousands of agent steps.
Evaluation: The analysis examines whether agents localize and resolve ambiguous observations despite position inference and mapping being absent from the training loss.Localization is reported as correlated with higher task reward.

2 APPROACH

The approach combines A3C navigation with recurrent memory and auxiliary depth and loop-closure objectives, using multimodal signals to accelerate representation learning.

Core framework: The end-to-end framework maximizes cumulative reward while minimizing auxiliary losses for depth inference and loop-closure detection.The objectives jointly train navigation and navigation-relevant representations.
Core framework: A3C learns both a policy and value function from state observations, with shared intermediate representations.The policy and value outputs use separate linear layers above the shared representation.
Memory and inputs: Nav A3C uses a two-layer stacked LSTM and augments observations with agent-relative velocity, the previous action, and the previous reward.These inputs are distributed across recurrent layers to support temporal processing.
Auxiliary objectives: The architecture adds outputs for depth and loop-closure prediction, with auxiliary gradients combined with A3C gradients using weights β_d1, β_d2, and β_l.Depth may be predicted from the convolutional or top LSTM layer, while loop closure is predicted as a separate auxiliary task.
Depth prediction: Depth prediction uses RGB observations to learn features for the environment’s 3D structure rather than directly supplying depth as an input.The auxiliary loss shares representation with the navigation policy.
Loop closure prediction: Loop closure is labeled when the current position is near an earlier position while an intermediate position is sufficiently far away.The binary target is learned with a Bernoulli loss from the last hidden representation.

3 RELATED WORK

The related work spans robotics and deep reinforcement learning, while this paper focuses on jointly learning spatial, geometric, and movement representations with reward maximization.

Navigation research: Navigation research has primarily developed within robotics, including explicit localization and mapping approaches.The paper situates its contribution against this broader literature while focusing on deep RL.
Deep RL navigation: Deep RL navigation studies have used successor representations, feedforward actor-critic models, and pretrained visual encoders in gridworld and 3D environments.These examples address reward flexibility, bottleneck detection, and target navigation.
Auxiliary learning: Prior deep RL work had examined auxiliary tasks, but the paper targets representation learning for space, geometry, and movement during reward maximization.The contribution is validated in maze domains with random start and goal locations.

4 EXPERIMENTS

The experiments evaluate navigation agents across five visually rich 3D mazes, including static, dynamic-goal, and memory-demanding settings. Auxiliary depth and loop-closure tasks accelerate learning and improve performance, while the best agents approach human-level scores on several tasks.

Environments: Five DeepMind Lab mazes test agents under static goals, randomly changing goals, and an I-maze requiring memory of hidden goal locations.The evaluation includes random starts and orientations, dynamic goals, and long episodes.
Performance: Nav A3C+D2 reaches human-level performance on Static 1 and 2 and achieves about 91% and 59% of human scores on Random Goal 1 and 2.Results are averaged over the five top-performing hyperparameter settings.
Design choices: Reward clipping can yield slightly suboptimal policies, while depth prediction helps mediate the resulting instability; depth classification also outperforms depth regression.Predicting depth from the last LSTM layer performs better than predicting it earlier.
Baselines: Feedforward A3C performs competitively on static mazes, motivating dynamic environments that encourage memory and more general navigation strategies.Static layouts can support reactive or wall-following strategies despite navigation often requiring memory.
Auxiliary tasks: Nav A3C+D1D2L speeds learning dramatically on most mazes and produces a substantial, lasting performance increase on random-goal mazes.The strongest effect occurs on static mazes, while random-goal tasks retain a lasting performance gain.
Auxiliary-task analysis: The Nav A3C*+D1L agent achieves an F-1 score of 0.83 for loop detection over 100 test episodes in a large random-goal maze.Each test episode contains 2250 steps.

5 ANALYSIS

The analysis examines navigation trajectories, localization, internal representations, and auxiliary-task combinations. The agents explore to discover changing goals, then can return efficiently while their hidden states encode task-relevant spatial information.

Position decoding: Position decoding uses a linear classifier over LSTM or convolutional representations to predict discretized maze locations.Small and large mazes contain 50 and 135 locations, respectively.
Position decoding: 85.5% accuracy: Nav A3C+D2 achieves the best position decoding in Random Goal 1, versus approximately 50% for FF A3C and LSTM A3C.Position uncertainty decreases as the agent acquires more observations and spikes after respawn.
Position decoding: 68.5% accuracy: Nav A3C*+D1L’s I-maze position decoder is less accurate than the plain LSTM A3C’s 87.8%, despite direct returns to the correct branch.The authors hypothesize that maze symmetry induces a policy that need not track exact position.
Goal-directed behavior: Nav A3C+D2 agents achieve the lowest post-discovery goal latency in Random Goal tasks, while Random Goal 2 shows no such improvement.Figure 5 depicts initial exploration followed by consistent returns to the goal after respawns.
Network activity: After exploration, Nav A3C agents consistently return to each goal location, with tSNE revealing distinct activation structure associated with goal locations.The LSTM A3C has four distinct goal-location clusters, whereas the Nav A3C has two main clusters for diagonally opposite arms.
Auxiliary-task combinations: Reward prediction improves over plain Nav A3C*, but depth prediction from the policy LSTM performs better; combining reward and depth is comparable to depth alone.Normalised average AUC values are 0.995 for Nav A3C+RD2 and 0.981 for Nav A3C+D2.

6 CONCLUSION

The paper proposes end-to-end deep reinforcement learning with memory and auxiliary learning targets for navigation in visually rich environments with changing starts and goals. It argues that depth prediction and loop closure provide richer training signals, while noting limits from stacked-LSTM memory capacity.

Conclusion: The proposed method augments deep reinforcement learning with memory and auxiliary learning targets for environments with frequently changing start and goal locations.The analysis covers agent behavior, localization, and network activity dynamics.
Conclusion: Online auxiliary losses provide richer training signals that bootstrap learning and enhance data efficiency.The auxiliary objectives are depth prediction and loop closure.
Limitations and future work: Stacked-LSTM capacity limits rapid-memory demands in settings such as procedurally generated mazes.The authors identify external memory as a future direction and suggest comparison with SLAM-based approaches.

Supplementary Material

The supplementary material presents videos of the navigation agent across five maze environments, pairing visual behavior with value, reward, maze-layout, and trajectory information.

Supplementary Material: Five videos cover I-maze, small and large static mazes, and small and large random-goal mazes.Each video combines high-resolution visualization, value-function dynamics, rewards, goal acquisitions, maze layouts, and trajectories.

B NETWORK ARCHITECTURE AND TRAINING

The agent uses a modular multitask architecture in which separate task networks share building blocks and are jointly optimized. Asynchronous threads contribute gradients from navigation, depth prediction, and loop-closure tasks on their own schedules.

Network architecture: The architecture builds one network per task from shared modules, including visual convolutional processing and LSTMs.Inputs span vision, depth, past rewards, and past actions.
Training: Each asynchronous training thread runs its own episode and adds gradients from navigation, depth, and loop-closure tasks as they arrive.The tasks can be trained at their own schedules within each thread.

B.2 NETWORK AND TRAINING DETAILS

The agents use convolutional and recurrent modules with auxiliary prediction heads, trained through asynchronous A3C-style updates and hyperparameter search. Training details include depth and loop-closure outputs, multiple workers, action repeats, and reward-clipping comparisons.

Architecture: The encoder uses convolutional layers followed by fully connected or recurrent layers that produce policy and value outputs alongside auxiliary predictions.The architecture includes depth predictors and loop-closure detection modules.
Auxiliary heads: Depth prediction is implemented from convolutional or LSTM representations, while loop closure uses a separate two-class prediction head.The depth outputs use multiple softmax predictions, and the loop-closure module uses a 2-dimensional softmax.
Optimization: Training follows asynchronous A3C with 16 workers, RMSProp, and gradients computed over non-overlapping episode chunks.Experiments run for up to 1e8 environment steps, with action repeat 4 and reported agent-perceived steps capped at 2.5e7.
Hyperparameters: Hyperparameters vary across learning rate, entropy regularization, reward handling, gradient-chunk length, and auxiliary-loss coefficients.The reported grid includes reward clipping in earlier experiments and sampled coefficients for depth and loop-closure losses.
Training stability: Without auxiliary tasks, baseline and Nav A3C agents learn worse without reward clipping, motivating clipped baselines in the main comparisons.The paper describes removing reward clipping as making learning unstable in the absence of auxiliary tasks.

C.2 DEPTH PREDICTION AS REGRESSION OR CLASSIFICATION TASKS

The comparison evaluates whether depth should be supplied directly as input or learned as an auxiliary target. Predicting depth performs better than providing RGBD inputs, with classification outperforming regression overall.

Input versus target: RGBD inputs perform worse than predicting depth as an auxiliary regression task for the Nav A3C agent.The comparison treats depth as either an input modality or a prediction target.
Regression: Depth prediction as a regression task outperforms direct RGBD input in the reported comparison.The passage explicitly compares using depth as an input with predicting it as a target.
Classification: Depth classification generally performs better than depth regression in the evaluated Nav A3C configurations.The passage reports classification as preferable to regression in general.

C.3 NON-NAVIGATION TASKS IN 3D MAZE ENVIRONMENTS

The paper tests its agents on 3D maze tasks that do not require navigation-specific planning, using reward-identification and persistent-exploration challenges. Auxiliary-task Nav architectures perform well across varied replicas and task settings.

Task design: Seek-Avoid Arena and Stairway to Melon test reward identification and persistent exploration rather than shortest-path navigation.The tasks involve distinguishing apples, lemons, or melons in environments with different layouts and episode structures.
Evaluation: Figure 11 compares reward AUC across fixed-goal static mazes, dynamic-goal mazes, and the I-maze using 64 replicas per experiment.Replica AUC values are sorted in decreasing order for each experiment.
Results: Nav architectures with auxiliary tasks achieve higher results for a comparatively larger number of replicas.This pattern suggests greater robustness to hyperparameter choice across the evaluated experiments.

C.5 ASYMPTOTIC PERFORMANCE OF THE AGENTS

The asymptotic analysis compares navigation performance and position decoding after different amounts of training. Auxiliary depth prediction reaches stronger performance and localization than extended baseline training, indicating effects beyond merely accelerating updates.

Baseline scaling: 57 points and 33.4% position decoding for the baseline rise to 90 points and 66.5% after twice as many training steps.These figures describe baseline changes from 120M to 240M training frames.
Auxiliary-task effect: After half the training frames, the Nav A3C+D2 agent still exceeds the twice-trained baseline in performance and position decoding accuracy.The authors therefore conclude that the auxiliary task does more than simply accelerate training.