Source-linked AI summary
Learning to Navigate in Cities Without a Map
Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, Raia Hadsell
TL;DR
The paper asks whether end-to-end deep reinforcement learning can navigate large-scale real-world cities from visual inputs despite sparse rewards and limited robustness outside simulated environments. It introduces StreetLearn and a dual-pathway, goal-conditioned architecture that separates general navigation from locale-specific knowledge. Agents learn courier navigation across multiple cities and demonstrate transfer to new regions, while the environment remains constrained by static imagery, discrete panorama actions, and fixed street topology.
Problem
Large-scale visual navigation in real-world environments remains uncertain because deep reinforcement learning is data inefficient, sensitive to environmental perturbations, and mainly demonstrated in games or simulations.
Method
The paper builds StreetLearn from Google Street View and trains a modular, goal-conditioned recurrent reinforcement-learning agent with shared general pathways and locale-specific goal pathways.
Results
The agent learns courier navigation in New York, London, and Paris, and the multi-city architecture demonstrates transfer to new environments.
Takeaways & Limitations
City-scale real-world navigation can be studied with deep reinforcement learning using a Street View environment and an architecture that combines general and locale-specific processing.
Takeaways & Limitations
The environment lacks dynamic elements, uses discrete actions that jump between panoramas, and cannot arbitrarily alter street topology.
Abstract
from arXiv · showhide
Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, grounded by recognisable landmarks and robust visual processing, that can simultaneously support continuous self-localisation ("I am here") and a representation of the goal ("I am going there"). Building upon recent research that applies deep reinforcement learning to maze navigation problems, we present an end-to-end deep reinforcement learning approach that can be applied on a city scale. Recognising that successful navigation relies on integration of general policies with locale-specific knowledge, we propose a dual pathway architecture that allows locale-specific features to be encapsulated, while still enabling transfer to multiple cities. We present an interactive navigation environment that uses Google StreetView for its photographic content and worldwide coverage, and demonstrate that our learning method allows agents to learn to navigate multiple cities and to traverse to target destinations that may be kilometres away. The project webpage http://streetlearn.cc contains a video summarising our research and showing the trained agent in diverse city environments and on the transfer task, the form to request the StreetLearn dataset and links to further resources. The StreetLearn environment code is available at https://github.com/deepmind/streetlearn
1 Introduction
The paper investigates whether end-to-end deep reinforcement learning can support large-scale visual navigation in real-world city environments. It introduces a city-scale challenge and a modular goal-conditional approach intended to combine locale-specific learning with transferable navigation behavior.
- Motivation: The study extends prior visual-navigation work from house scenes, mazes, and 3D games toward large-scale real-world environments.These earlier domains demonstrate feasibility but do not establish city-scale navigation from real-world images.
- Motivation: Real-world city navigation is difficult because deep reinforcement learning is data inefficient, sensitive to environmental perturbations, and largely validated in games or simulations.The paper therefore tests whether it can operate on large-scale real-world imagery.
- Contributions: StreetLearn provides an interactive reinforcement-learning challenge using real-world Google Street View images across London, Paris, and New York City.The traversal task requires agents to navigate from goal to goal within these cities.
- Contributions: The proposed agent uses a dual-pathway, modular architecture to support locale-specific features while retaining general navigation behavior transferable across cities.The architecture is presented as a strong baseline for future research.
2 Related Work
Prior navigation research spans neuroscience, robotics, supervised visual localization, deep reinforcement learning, and neural mapping. The paper distinguishes its real-world, geographically diverse environment from simpler simulators and smaller photographic settings.
- Navigation and localization: Many navigation algorithms explicitly build a map during exploration and then plan and act using that representation.This work instead focuses on end-to-end deep reinforcement learning in large-scale real-world environments.
- Learning from real-world imagery: Supervised approaches use Street View or OpenStreetMap data for navigation-related tasks, while other work studies visual geolocation and active localization.These methods provide related precedents for learning from real-world imagery.
- Deep reinforcement learning: Many deep reinforcement learning navigation systems rely on simulators that offer procedural variation but often use visually simple or unrealistic observations.Auxiliary tasks and predictive objectives are commonly used to address sparse rewards.
- Simulation versus reality: Higher-fidelity simulators reduce the simulation-reality gap but remain limited by environmental diversity and the antiseptic quality of observations.Photographic environments have also been used for shorter indoor navigation problems with limited scale.
- Simulation versus reality: StreetLearn offers diverse, visually realistic scenes containing vegetation, pedestrians, vehicles, varied weather, and large geographic areas.Its stated limitations are the absence of dynamic elements, discrete panorama-to-panorama actions, and fixed street topology.
3 Environment
StreetLearn represents city regions as graph-connected Google Street View panoramas and gives agents cropped visual observations, landmark-based goals, discrete movement actions, and courier-task rewards. The task requires repeated navigation to randomly sampled destinations over long distances.
- Environment: StreetLearn uses geolocated 360° panoramas as nodes in an undirected graph covering regions of New York City, Paris, and London.Selected regions contain 7,000–65,500 nodes, 7,200–128,600 edges, mean node spacing of 10m, and coverage up to 5km.
- Agent interface: The agent observes an 84 × 84 RGB crop from a panorama and chooses among five discrete actions: slow or fast rotation in either direction, or forward movement.Forward becomes a no-op when no visible graph edge is available; among multiple visible edges, the most central is selected.
- Goal representation: Goals are represented through softmax-normalised distances to a fixed set of geographic landmarks.The landmark code is intended to remain locally continuous, coordinate-system independent, and scalable to new regions.
- Architectures: Figure 2 contrasts GoalNav, CityNav, and MultiCityNav, moving from a single policy LSTM to separate locale-specific goal pathways for single- and multi-city learning.CityNav also includes an optional heading-prediction pathway.
- Courier task: The courier task repeatedly assigns random city locations, advances to a new goal within 100m, and ends each episode after 1000 agent steps.Goal reward is proportional to the shortest path from the agent’s position when the goal was assigned, rewarding longer journeys without rewarding detours.
4 Methods
The method formalizes goal-conditioned navigation as a recurrent actor-critic problem and separates general visual-policy processing from locale-specific goal processing. MultiCityNav adds city-specific pathways, while heading prediction and curriculum learning address geometric learning and sparse long-distance rewards.
- Formalization: The navigation objective maximizes discounted expected return under a policy conditioned on the current state and sampled goal.The value function is likewise defined for state-goal pairs.
- Architectural motivation: The architecture is motivated by combining general representations for scene understanding with locale-specific memory of place-dependent features and topology.Multiple neural pathways are used to support both forms of learning.
- Architectures: GoalNav feeds goal descriptions into a policy LSTM, whereas CityNav adds a goal LSTM and optional auxiliary heading prediction.The goal LSTM receives visual features and the goal description.
- Architectures: MultiCityNav assigns an individual goal-LSTM pathway to each city or region while retaining shared convolutional and policy-LSTM components.After multi-city training, the paper reports that only a new goal LSTM needs training for a new city.
- Auxiliary learning: Heading prediction is an auxiliary multinomial classification task over binned orientations that supplies additional gradients and may help the convolutional network learn environmental geometry.The agent can navigate without this auxiliary task.
- Training: Curriculum learning gradually increases destination distance to address the courier task’s sparse rewards.Training uses IMPALA with decoupled acting and learning, while the experiments compare recurrent architectures and training configurations.
5 Results
The experiments show that CityNav learns courier navigation across large cities, generalizes imperfectly to held-out goals, and transfers effectively when trained across multiple regions.
- 5.1 Courier Navigation in Large, Diverse City Environments: CityNav, trained with curriculum learning, succeeds on the courier task in New York, London, and Paris.
- 5.1 Courier Navigation in Large, Diverse City Environments: The agent learns navigation policies across city-scale environments, with steps to reach goals growing linearly with straight-line goal distance.
- 5.2 Reward and Curriculum: Coins alone are ineffective for exploration, so the final reward design combines 200m early reward shaping with curriculum learning.
- 5.3 Generalization on Held-out Goals: Held-out-goal performance decreases as the held-out area grows, partly because larger areas can contain unseen landmark-based goal specifications.
- 5.3 Generalization on Held-out Goals: Despite missing more goals on larger held-out grids, the agent reaches halfway to goals in similar time, indicating an approximate representation of unseen goals.
- 5.4 Transfer in Multi-city Experiments: Transfer effectiveness increases with the number of training regions; pre-training on four regions yields performance comparable to joint training on all regions.
6 Conclusion
The paper presents city-scale real-world navigation with deep reinforcement learning and a multi-city architecture that transfers to new environments. It releases the StreetLearn environment and resources for further research.
- City-scale StreetLearn enables training reinforcement-learning agents on real-world Google Street View environments.
- StreetLearn and its training resources were released for Manhattan and Pittsburgh through the project webpage and code repository.
- MultiCityNav agents trained jointly on four environments transfer to a fifth environment, Lower Manhattan.
B.1 Architecture Ablation Analysis
Architecture ablations show that heading prediction and skip connections improve navigation, while CityNav outperforms the main GoalNav baseline and preserves transfer advantages through locale-specific pathways.
- Figure 7 compares learning curves across 2LSTM, skip-connection, heading-prediction, and GoalNav ablations on NYU.
- CityNav significantly outperforms the main GoalNav LSTM baseline in the NYU navigation analysis.Experiments used a two-day curriculum, 200m early rewards, five random seeds, and mean final reward with standard deviations.
- Heading prediction improves GoalNav, and combined with a convnet-to-policy-LSTM skip connection benefits the 2-LSTM architecture.
- MultiCityNav uses locale-specific LSTM pathways so new cities can be supported without retraining the full architecture or adding city-specific components to the general pathways.
B.2 Goal Representation
The study compares landmark and coordinate-based goal representations. Scalar latitude and longitude perform best, while landmark density can be reduced substantially without significant performance loss and goals remain necessary.
- Scalar latitude-and-longitude goal representations perform best among the representations tested on NYU.The alternatives include landmark-based, scalar coordinate, and binned coordinate representations.
- Reducing landmark density to 50%, 25%, or 12.5% does not significantly reduce performance.
- The canonical landmark representation uses normalized distances to fixed landmarks and is independent of the coordinate system, making it more scalable.
- Goal-less CityNav performs poorly, indicating that navigation performance depends on the supplied goal representation rather than street-graph exploration alone.
B.3 Allocentric and Egocentric Goal Representation
The paper decodes agent and goal positions from region-specific LSTM activations to examine allocentric and egocentric representations during navigation. The evaluation uses trajectory visualizations and distance-based reward shaping.
- Decoders predict the agent’s allocentric position, goal position, and egocentric direction toward the goal from 256 region-specific LSTM activations.The decoders are two-layer MLPs; allocentric position uses multinomial latitude-and-longitude classifiers.
- Figure 9 overlays decoded agent positions, decoded goal positions, and three trajectories for goals at St Paul’s Cathedral in London.
- Goal rewards are reported at the destination because reward shaping can make longer paths yield slightly more intermediate rewards.
C.1 Neural Network Architecture
The agents combine convolutional visual processing, recurrent policy and goal pathways, auxiliary heading prediction, and distributed IMPALA training. Experiments use StreetView graph environments spanning selected areas of New York, London, and Paris.
- Base architecture: The baseline GoalNav agent uses a two-layer convolutional vision model, a 256-dimensional visual feature output, and a recurrent LSTM predicting policy and value.The convolutional stack uses 16 and 32 feature maps, followed by a 256-unit fully connected layer.
- Inputs and goal representation: The policy LSTM receives visual features, past reward, previous action, and goal information, with five discrete actions represented as one-hot vectors.Goal inputs can encode 644 landmarks or normalized two-dimensional latitude and longitude coordinates.
- Dual pathway architecture: CityNav and MultiCityNav use goal LSTMs whose 64-dimensional bottleneck outputs connect to the policy LSTM, constraining transmitted information and encouraging robustness to noise.Each goal LSTM includes 256 hidden units, dropout with probability 0.5, a 256-to-64 linear layer, and a tanh nonlinearity.
- Outputs and auxiliary task: An auxiliary heading-prediction task classifies 16 binned directions, while the policy and value outputs use separate linear layers for five actions and one scalar value.The heading task is attached to goal-LSTM hidden units and optimized with a multinomial loss.
- Optimization and training: Training uses IMPALA with distributed actors, matching A3C on single-city tasks and preferred for multi-city and transfer experiments because of its demonstrated multi-task capability.CityNav uses 256 actors and MultiCityNav uses 512 actors; curriculum phases are timed by actor and learner wall-clock duration.
- Experimental environments: The evaluation environments are StreetView graphs covering five Manhattan sub-areas plus bounded Central London and Paris Rive Gauche regions, with 644 study landmarks.The Manhattan sub-areas are defined by breadth-first searches from selected coordinates, while London and Paris are specified by latitude/longitude bounding boxes.