PaperlayerView exact revision on arXiv

Source-linked AI summary

DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames

Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, Dhruv Batra

arXiv:1911.00357v2cs.CVcs.AIcs.LG

TL;DR

The paper addresses the need to scale reinforcement learning in resource-intensive simulated environments and introduces DD-PPO, a synchronous, decentralized, distributed approach. DD-PPO scales near-linearly, enabling massive training that essentially solves PointGoalNav with RGB-D and GPS+Compass and transfers to other navigation tasks, while RGB-only navigation remains an open frontier.

  • Problem

    Large-scale distributed parallelization is needed because deep reinforcement learning relies on significant numbers of training samples, and prior PointGoalNav studies had not reached performance saturation.

  • Method

    DD-PPO synchronously distributes PPO across multiple machines without a parameter server, coordinating gradient updates and preempting straggling experience collection.

  • Results

    DD-PPO scales near-linearly and enables near-perfect PointGoalNav performance with RGB-D and GPS+Compass, while learned policies transfer to other navigation tasks and outperform ImageNet-pre-trained CNNs there.

  • Takeaways & Limitations

    DD-PPO provides a scalable training method and the learned navigation policies can serve as a reusable resource for other embodied-AI navigation tasks.

  • Takeaways & Limitations

    RGB-only navigation without GPS+Compass remains an open research problem: training to 2.5 billion steps improves SPL to 0.15 from 0 at 100 million steps.

Abstract

from arXiv · show

We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever stale), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim, DD-PPO exhibits near-linear scaling -- achieving a speedup of 107x on 128 GPUs over a serial implementation. We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs. This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially solves the task --near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs). Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks -- the analog of ImageNet pre-training + task-specific fine-tuning for embodied AI. Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models and code are publicly available).

1 INTRODUCTION

DD-PPO addresses the difficulty of scaling reinforcement learning in GPU-intensive 3D environments by replacing parameter-server distribution with synchronous, decentralized multi-machine training. Near-linear scaling enables massive experience collection, near-perfect PointGoal navigation with RGB-D and GPS+Compass, and transfer to other navigation tasks.

  • 1 INTRODUCTION: GPU-accelerated 3D simulators and deep visual networks make conventional parameter-server architectures inadequate, motivating a new distributed architecture.Compared with CPU-simulated environments, 3D simulators limit worker counts and place greater strain on the parameter server.
  • 1 INTRODUCTION: DD-PPO is a synchronous, decentralized distributed RL method that removes the parameter server while scaling PPO across machines.Workers synchronize gradients after experience collection and optimization, with preemption limiting delays from stragglers.
  • 1 INTRODUCTION: Near-linear scaling is observed across both homogeneous and heterogeneous simulation workloads when experience throughput is measured relative to one worker.The authors characterize scaling using steps of experience per second and report that the method may extend beyond PPO.
  • 1 INTRODUCTION: PointGoalNav policies transfer to Flee and Explore, rapidly outperform ImageNet-pretrained CNNs, and are released with code and trained models.The paper presents this as an embodied-AI analogue of ImageNet pre-training followed by task-specific fine-tuning.

2 PRELIMINARIES: RL AND PPO

The paper frames RL as decision making under partial observability and introduces PPO as an on-policy policy-gradient method that updates policies from collected trajectories using a clipped objective. The agent architecture receives visual observations and GPS+Compass information to produce actions and value estimates.

  • 2 PRELIMINARIES: RL AND PPO: In a POMDP, the agent receives an observation, takes an action, and receives a reward while the objective is to maximize cumulative episode reward.The observation may be an egocentric RGB image and does not fully specify the environment state.
  • 2 PRELIMINARIES: RL AND PPO: PPO is an on-policy policy-gradient algorithm that updates a parameterized policy using trajectories collected under that policy.The rollout supplies the experience used for subsequent parameter updates.
  • 2 PRELIMINARIES: RL AND PPO: The advantage estimate subtracts the expected value from the return, while the probability ratio compares the current policy with the rollout-collecting policy.These quantities determine how collected actions contribute to the PPO update.
  • 2 PRELIMINARIES: RL AND PPO: PPO clips the probability ratio to a bounded interval, functioning as a trust-region method while allowing multiple gradient updates from one rollout.The clipped objective is described as improving sample efficiency.
  • 2 PRELIMINARIES: RL AND PPO: At each time-step, the agent combines egocentric RGB or depth input with GPS+Compass updates to output an action and estimate the value function.The sensor updates the target position relative to the agent’s current position.

3 DECENTRALIZED DISTRIBUTED PROXIMAL POLICY OPTIMIZATION

DD-PPO adapts synchronous data-parallel training to on-policy reinforcement learning without a parameter server. Workers synchronize gradients and use preemption to reduce delays from slow experience collection.

  • Decentralized distributed PPO: DD-PPO is distributed across machines, decentralized without a parameter server, and synchronous during model updates.It adapts Proximal Policy Optimization to distributed reinforcement learning.
  • Design rationale: DD-PPO addresses asynchronous RL’s complexity by using synchronously running workers with explicit communication and shared update timing.The method retains synchronization while avoiding the separate parameter-server and rollout-worker programs described for common asynchronous systems.
  • Synchronous optimization: Each worker collects rollout experience, computes policy gradients, synchronizes them, and updates the model.This follows the synchronous data-parallel pattern while using a policy-gradient method such as PPO.
  • Straggler handling: A preemption threshold ends straggling rollouts once a specified percentage of other workers finish, allowing all workers to optimize together.The mechanism targets synchronization overhead caused by variable environment simulation times.
  • Algorithm scope: The method is directly demonstrated with PPO, while the authors state that it can extend to other on-policy and potentially off-policy RL algorithms.For off-policy methods, the proposed adaptations add synchronization to optimization and preemption to experience collection.

4 EXPERIMENTAL SETUP: POINTGOAL NAVIGATION, AGENTS, SIMULATOR

The experiments evaluate embodied agents performing PointGoal Navigation in Habitat, where they must reach relative target coordinates in unseen environments using onboard sensors. The setup combines visual processing, recurrent control, PPO training, and SPL-based evaluation.

  • PointGoal Navigation: PointGoal Navigation initializes an agent in a new environment and asks it to reach target coordinates relative to its starting position.No map is available; navigation uses RGB-D or RGB observations together with GPS+Compass.
  • Evaluation: SPL combines success with the ratio of shortest-path length to the agent’s actual path length, making inefficient successful navigation score lower.Success requires stopping within 0.2 meters of the target, and SPL = 1 requires matching the shortest path.
  • Agent interface: The agent uses four actions and receives 256x256 images while GPS+Compass supplies target coordinates relative to its current state.The RGB-D agent is limited to depth because earlier work found depth performed best.
  • Agent architecture: The architecture pairs a visual encoder with a 2-layer LSTM policy whose inputs are the previous action, relative target, and visual features.The LSTM outputs both an action distribution and a value-function estimate.
  • Training: Training uses PPO with Generalized Advantage Estimation, parallel rollouts, minibatch optimization, Adam, and terminal plus geodesic-distance-shaped rewards.Each worker collects up to 128 frames from four parallel agents before PPO optimization.
  • Simulator and datasets: Habitat provides a performant, stable 3D simulator suitable for generating billions of embodied-AI experience steps.Training uses Gibson scenes and augments them with Matterport3D scenes to form a larger dataset.

5 BENCHMARKING: HOW DOES DD-PPO SCALE?

DD-PPO scales near-linearly for homogeneous workloads and can recover similar scaling on heterogeneous workloads when stragglers are preempted. Without preemption, heterogeneous simulation suffers substantial straggler-induced degradation.

  • Benchmark design: The benchmark measures experience steps per second with N workers relative to one worker across homogeneous and heterogeneous simulation workloads.It also compares multiple preemption-threshold values.
  • Figure interpretation: Figure 4 plots scaling relative to one GPU for different preemption thresholds, with shading denoting 95% confidence intervals.The figure compares threshold-dependent scaling in the two workload regimes.
  • Homogeneous workloads: 196x speedup with 256 GPUs and 7.3x with 8 GPUs demonstrate near-linear scaling for homogeneous workloads when the preemption threshold exceeds 50%.The ideal linear-scaling reference is used for comparison.
  • Heterogeneous workloads: 100% preemption threshold scales poorly for heterogeneous workloads because one substantially slower rollout creates a straggler effect.Gibson and Matterport3D scenes differ greatly in simulation complexity and size.
  • Heterogeneous workloads: 80% or 60% preemption thresholds produce near-identical scaling to the homogeneous workload under heterogeneous simulation times.The authors report no degradation in model performance for these threshold values despite less frequent learning in large scenes.

6 MASTERING POINTGOAL NAVIGATION WITH GPS+COMPASS

DD-PPO agents achieve near-perfect PointGoalNav performance with RGB-D and GPS+Compass, while extensive training, broader data, and larger models progressively improve results. The learned representations and policies also transfer to Flee and Exploration, whereas navigation without GPS+Compass remains unsolved.

  • Training scale: 2.5 billion steps of training produce continued improvement, with 90% of peak performance reached at 100 million steps.Training used 64 Tesla V100 GPUs for 2.75 days; 90% of peak performance required 0.1 day with 64 GPUs or 1 day with 8 GPUs.
  • Training data: Adding Matterport3D scenes improves SPL by approximately 0.03, from 0.922 val/0.917 test to 0.956 val/0.941 test.Adding poor Gibson reconstructions also improves performance on good reconstructions, from 0.922 val/0.917 test to 0.956 val/0.944 test.
  • Model capacity: Larger visual encoders and policies improve SPL by 0.010, while replacing ResNet50 with SE-ResNeXt50 improves SPL by 0.003.The reported 0.010 gain comes from increasing encoder capacity and policy size; the 0.003 gain comes from the stronger visual encoder.
  • GPS+Compass navigation: The best RGB-D agent reaches SPL 0.969 val/0.948 test, within 3–5% of the shortest-path oracle.The SE-ResNeXt101 plus 1024-d LSTM agent trained on Gibson-2+ also sets the state of the art on the Habitat Challenge 2019 RGB-D track.
  • RGB navigation: RGB training reaches success of 0.991 val/0.977 test and SPL of 0.929 val/0.920 test, improving substantially over prior RGB results.The RGB agent achieves near-perfect success but lower SPL than the depth agent.
  • Open problems: Without GPS+Compass, navigation remains an open problem: SPL rises from 0 at 100 million steps to only 0.15 at 2.5 billion steps.The paper identifies navigation from RGB without GPS+Compass as an open frontier for embodied AI.
  • Transfer learning: PointGoalNav-trained agents rapidly transfer to Flee and Exploration, outperforming ImageNet-pretrained CNNs and supporting neural-controller use by a high-level planner.PointGoalNav visual encoders outperform ImageNet-pretrained encoders, while fine-tuned policies rapidly learn the new tasks.

7 RELATED WORK

Distributed RL systems differ in how simulation, learning, and parameter synchronization are organized. DD-PPO adapts synchronous data parallelism to multi-GPU reinforcement learning while addressing variable rollout times.

  • Synchronous RL performs simulation, inference, learning, and parameter management in one process, limiting it to one node’s GPUs.
  • DD-PPO replicates simulation, inference, and learning across GPUs and synchronizes gradients with AllReduce.
  • Asynchronous distributed RL separates CPU-based simulation and inference from learner and parameter-server processes, while making multi-learner scaling unclear.
  • DD-PPO adapts distributed synchronous data parallelism to reinforcement learning, where variable environment runtimes create synchronization overhead.

A ADDITIONAL ANALYSIS AND DISCUSSION

The analysis examines navigation behavior, SPL’s limitations, failure modes, and differences from a classical map-and-plan agent. Near-perfect average performance coexists with substantial deviations and identifiable environmental weaknesses.

  • Intricacies of SPL: SPL can obscure qualitative navigation errors because percentage path deviations become less informative on long trajectories.A one-meter deviation on a 10-meter episode remains within 10% despite potentially reflecting a significant mistake.
  • Error recovery: 54% of non-perfect episodes have average SPL 0.85 and 99.0% success, showing that many successful episodes substantially deviate from shortest paths.Across all episodes, the reported average is 0.92 SPL with 99.5% success.
  • When does the agent fail?: The agent often recovers from the failure mode associated with an unfavorable geodesic-to-Euclidean goal-distance ratio and still reaches the goal.
  • When does the agent fail?: Slight vertical stair traversal causes poor performance, although such cases are rare and GPS provides only two-dimensional location.
  • Differences from a classical agent: Unlike the map-and-plan baseline’s fixed 1.25m collision retreat, the learned agent navigates around unseen obstacles without a large SPL impact.

C AGENT DESIGN

The agent combines visual and goal encoders with a recurrent navigation policy, while throughput experiments evaluate DD-PPO across workload conditions and rollout settings.

  • Agent components: The agent consists of a visual encoder, a goal encoder, and a navigation policy.
  • Visual Encoder: The visual encoder uses modified ResNet50, SE-ResNeXt50, or SE-ResNeXt101 backbones and produces a 128x4x4 feature map.
  • Goal encoder: The goal encoder converts polar coordinates [d, θ] into [d, cos(θ), sin(θ)] and maps them to a 32-dimensional representation.
  • Navigation Policy: The navigation policy combines visual features, goal features, and previous-action embeddings before processing them with a 2-layer LSTM.

E DD-PPO IMPLEMENTATION

DD-PPO implementation adds decentralized gradient synchronization and straggler preemption to a single-GPU PPO implementation, using distributed storage and standard framework primitives.

  • A decentralized distributed PPO variant is created by adding gradient synchronization and synchronizing training or rollout statistics across workers.
  • Workers use distributed key-value storage to track completion of experience collection and identify stragglers for preemption.
  • DD-PPO’s visual depiction is provided in Figure 10, while Table 2 reports validation and test performance across sensors and agent methods.

F TRANSFER EXPERIMENTS ADDITIONAL DETAILS

The transfer experiments reuse the PointGoalNav PPO settings and train with DD-PPO across eight workers and eight GPUs. Agents are trained on Gibson-4+ and evaluated in Habitat Challenge 2019 validation scenes and starting locations.

  • The transfer experiments use the same PPO hyper-parameters as the PointGoalNav experiments.
  • Agents are trained with DD-PPO using 8 workers on 8 GPUs.
  • Training uses Gibson-4+, while evaluation uses Habitat Challenge 2019 validation scenes and starting locations with goal locations discarded.
  • The ImageNet encoder follows Xie et al. (2017) training hyper-parameters and procedure without data augmentation.

G NEURAL CONTROLLER ADDITIONAL DETAILS

The neural-controller details describe a planner that shares the policy architecture and visual encoder while producing value and PointGoal estimates. The accompanying figures document DD-PPO implementation, scene-quality examples, and performance by architecture, distance, and validation difficulty.

  • G NEURAL CONTROLLER ADDITIONAL DETAILS: The neural-controller planner shares the agent policy architecture and visual encoder, using a 512-dimensional hidden state.
  • G NEURAL CONTROLLER ADDITIONAL DETAILS: The planner takes the previous controller action or a start token, then predicts a value function and a three-dimensional PointGoal representation.
  • F IMPLEMENTATION: Figure 9 depicts DD-PPO implemented with PyTorch v1.1 and the NCCL backend, using SLURM to populate distributed-rank fields.
  • F IMPLEMENTATION: Figure 10 illustrates preempting experience collection by straggling processes so other processes do not wait for costly simulations.
  • F EVALUATION: Figure 11 shows Gibson meshes associated with different quality ratings.
  • F EVALUATION: Figures 12–14 compare architecture performance across scene subsets and performance against geodesic distance on standard and harder validation episodes.
Loading 1911.00357v2…