Source-linked AI summary
Distributed Distributional Deterministic Policy Gradients
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, Timothy Lillicrap
TL;DR
Continuous-control tasks require methods that handle real-valued actions without relying on costly optimization or coarse discretization. The paper develops D4PG by combining DDPG with distributional critic learning, distributed actors, N-step returns, and prioritized replay, and reports state-of-the-art performance across diverse continuous-control domains.
Problem
Continuous control commonly uses real-valued actions, whereas discrete-action methods can require costly optimization or poor discretization in high-dimensional spaces.
Method
D4PG extends off-policy DDPG with distributional critic updates, parallel actors sharing replay, N-step returns, and prioritized experience replay.
Results
D4PG achieves state-of-the-art performance across simple control, difficult manipulation, and hard obstacle-based locomotion tasks.
Takeaways & Limitations
The combined modifications produce a high-performing continuous-control algorithm across a wide variety of tasks, with N-step returns identified as the biggest performance gain among the simple changes.
Takeaways & Limitations
The considered environments assume real-valued action spaces, while no general assumptions are made about the input space.
Abstract
from arXiv · showhide
This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of $N$-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.
1 INTRODUCTION
The paper targets continuous control, where discrete-action methods face costly optimization or poor discretization in high-dimensional spaces. It extends DDPG with distributional critic updates and distributed experience collection, yielding D4PG and state-of-the-art performance across varied control tasks.
- Motivation: Continuous action spaces are central to robotics control, but DQN-style methods require costly optimization or potentially poor discretization.The limitation is especially relevant when action spaces are high-dimensional or require fine-grained control.
- Contribution: D4PG augments off-policy DDPG with a distributional critic update that models return randomness and provides a more stable learning signal.Because actor updates depend only on the learned critic, critic improvements directly affect actor gradients.
- Contribution: D4PG distributes experience gathering across many parallel actors that write to a shared replay table using the ApeX framework.The paper reports significant wall-clock savings for difficult control tasks.
- Contribution: The algorithm combines distributional learning and distributed off-policy training with additional improvements, including N-step returns and prioritized replay.The experiments examine both individual component contributions and their interactions.
- Outcome: D4PG achieves state-of-the-art performance across simple control, difficult manipulation, and hard locomotion tasks.This conclusion is reported across a wide variety of continuous-control domains.
2 BACKGROUND
The background formulates continuous-control reinforcement learning with a parameterized policy and critic, using deterministic policy gradients that can be estimated off-policy. DDPG implements these updates with target networks and replayed experience.
- Problem setup: The setting assumes real-valued actions, with a policy mapping observations to actions and a value function measuring expected return.The paper does not generally restrict the observation space but assumes continuous action spaces.
- Deterministic policy gradients: A parameterized policy is optimized by maximizing its expected value rather than directly maximizing the value function over continuous actions.This avoids the complications of action maximization in continuous spaces.
- Deterministic policy gradients: The deterministic policy gradient can be evaluated using data collected by a behavior policy different from the target policy.This enables off-policy learning and experience reuse.
- DDPG: DDPG approximates the value function with a parameterized critic and trains it by minimizing temporal-difference error.Separate target policy and value networks are periodically replaced with copies of current weights to stabilize learning.
- DDPG: DDPG trains the actor with the deterministic policy gradient and the critic with the TD loss using samples from a replay table.The algorithm uses sample-based approximations to both gradients.
3 DISTRIBUTED DISTRIBUTIONAL DDPG
D4PG extends DDPG with distributional critic learning, distributed actors, N-step returns, and prioritized replay. The distributional critic maps state-action pairs to return distributions, while distributed actors populate shared replay for learner updates.
- 3 DISTRIBUTED DISTRIBUTIONAL DDPG: D4PG adds a distributional critic, distributed parallel actors, N-step returns, and prioritized experience replay to DDPG.These extensions are evaluated as components of the overall algorithm.
- Distributional critic: The distributional Bellman operator maps state-action pairs to return distributions rather than scalar values.The return distribution is parameterized and trained with a distributional loss; experiments use a categorical distribution.
- Distributional critic: The actor update incorporates the action-value distribution by taking an expectation with respect to that distribution.The expectation is estimated empirically with samples.
- Architectures: Figure 1 distinguishes architectures for standard control and manipulation domains from those used by parkour domains.The critic torso output is fed into a parameterized distribution to complete the critic architecture.
- N-step returns: N-step returns replace the standard Bellman operator with an N-step variant and can be applied analogously to the distributional critic.This changes the TD-error target using N-step transition dynamics.
- Distributed learning: Parallel actors write experience to a shared replay table, while the learner samples it, optionally using non-uniform priorities with importance-sampling weights.The implementation uses K independent actors and the ApeX framework.
4 RESULTS
Experiments compare D4PG with ablations across standard control, manipulation, and parkour tasks, emphasizing distributional updates, N-step returns, parallel actors, and replay prioritization. The full D4PG generally performs best, while N = 5 improves results and prioritization contributes little to D4PG.
- Experimental setup: The experiments evaluate D4PG and component ablations using wall-clock performance, sample efficiency, and contribution analysis across continuous-control domains.The study includes standard control, manipulation, and two- and three-dimensional parkour tasks.
- Standard control suite: Across the standard control suite, the full D4PG variant achieves the best performance, while N = 5 is uniformly better than N = 1.N = 1 can occasionally become unstable, especially on Cheetah (Walk) and Cartpole (Swingup Sparse).
- Standard control suite: The distributional critic provides the next largest gain, particularly on difficult Humanoid (Run) and Acrobot tasks.On the manipulator, distributional and non-distributional variants obtain approximately the same performance.
- Manipulation: In manipulation, N = 5 is uniformly better, and D4PG performs at least as well as other ablations; prioritization offers limited value and can harm D3PG with N = 1.D3PG can become unstable or completely fail to learn on Pickup and Orient when N = 1 with prioritization.
- Parkour: In parkour, distributional updates produce the largest gain, D4PG outperforms PPO, and prioritization adds little or no benefit to D4PG.For the two-dimensional walker, all algorithms outperform PPO when N = 5; in the three-dimensional humanoid task, prioritized and non-prioritized D4PG curves are nearly identical.
5 DISCUSSION
D4PG combines distributional updates, distributed workers, and smaller algorithmic changes for continuous control. The authors report state-of-the-art performance on difficult continuous-control problems.
- D4PG combines distributional updates to DDPG with multiple distributed workers writing to one replay table.
- N-step returns provide the biggest performance gain among the smaller algorithmic changes considered.
- Prioritization is less crucial on harder problems and can produce unstable updates, especially in manipulation tasks.
- D4PG achieves state-of-the-art performance on difficult continuous-control problems.
A DISTRIBUTIONS AND LOSSES
D4PG represents action-value distributions with neural-network output layers, considering categorical and mixture-of-Gaussians parameterizations alongside the standard scalar value function.
- Parameterized distributions map the critic torso to distribution parameters through a neural-network layer.The parameters include quantities such as means and variances.
- Figure 7 contrasts categorical, mixture-of-Gaussians, and standard scalar value-function output layers.
- The categorical parameterization uses logits over a fixed set of atoms with bounded support defined by Vmin and Vmax.The atom spacing is determined by the support bounds and the number of atoms.
- The categorical layer applies a linear mapping from the critic torso to logits followed by softmax.
- Because categorical support is not closed under the Bellman operator, D4PG uses a projected distributional Bellman operator.
- The mixture-of-Gaussians layer outputs mixture weights, means, and variances for each component.
B CATEGORICAL PROJECTION OPERATOR
The categorical distribution requires projection because Bellman-transformed returns generally fall outside its fixed atom support. The projection redistributes probability onto the supported atoms.
- Bellman updates generally produce values that do not coincide with the categorical distribution’s finite atom support.
- Figure 8 compares mixture-of-Gaussians results across two learning rates and against the Categorical parameterization.
- The categorical projection redistributes target probabilities across atoms using a piecewise-linear hat function.
C MIXTURES OF GAUSSIANS CONTROL SUITE RESULTS
On selected control-suite tasks, the mixture-of-Gaussians output distribution underperformed the Categorical distribution by a fair margin, despite being identified as worthy of further exploration.
- The mixture-of-Gaussians distribution underperformed the Categorical distribution by a fair margin in initial control-suite experiments.The experiments used a sample-based KL loss and two learning rates.
D CONTROL SUITE DETAILS
This section details the Control Suite benchmark domains and specifies the physical state, action, and observation dimensions for each task.
- The Control Suite domains are used to structure the benchmark tasks.
- The section provides further details about the benchmark’s control-suite tasks.
- Table 1 reports the dimensionalities of each task’s physical state, action, and observation spaces.
E MANIPULATION DETAILS
The manipulation experiments use a simulated, underactuated prosthetic hand to perform three cylindrical-object tasks with structured observations, actuator-target actions, and task-specific rewards. Episodes terminate on table contact in catch and rotate-in-hand, while pick-up-and-orient uses reward gating to encourage pickup before orientation.
- Experimental setup: The simulated Johns Hopkins Modular Prosthetic Limb hand has 22 degrees of freedom driven by 13 position actuators, with finger-joint coupling causing underactuation.The hand has 19 finger and 3 wrist degrees of freedom, controlled by PD controllers.
- Tasks: The three manipulation tasks are catch, pick-up-and-orient, and rotate-in-hand, each involving a cylindrical object and randomized task conditions.Catch involves intercepting a falling object; pick-up-and-orient requires matching target position and orientation; rotate-in-hand requires axial rotation toward a moving target.
- Observations and actions: Observations include hand joint states, actuator position targets, object pose, and translational and rotational velocities; actions increment actuator position targets.The observation components and their dimensionalities are summarized in Table 2.
- Reward design: Pick-up-and-orient combines position and orientation rewards, with distance multiplicatively gating orientation to encourage pickup before alignment.Initial object and target positions and orientations are randomized, and episodes last 500 steps.
- Reward design: Rotate-in-hand uses multiplicative rotation and axis-alignment rewards, runs for up to 1000 steps, and terminates early if the object contacts the table.The target angle changes each step using temporally correlated Ornstein-Uhlenbeck noise; catch instead has a 500-step limit and terminates on table contact without reward.