Source-linked AI summary
Continuous control with deep reinforcement learning
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra
TL;DR
Deep Q-Learning does not straightforwardly extend to high-dimensional continuous action spaces. The paper introduces DDPG, an actor-critic algorithm that robustly learns competitive policies across challenging physical-control tasks, often directly from pixels.
Problem
Deep Q-Learning cannot straightforwardly handle high-dimensional continuous actions, while discretization suffers from exponentially expanding action spaces that are difficult to explore efficiently.
Method
DDPG is a model-free, off-policy actor-critic algorithm based on deterministic policy gradients and deep function approximators for continuous action spaces.
Results
DDPG robustly learns competitive policies across varied physical-control tasks using fixed hyperparameters and network structure, often directly from pixel observations.
Takeaways & Limitations
The experiments demonstrate stable learning across continuous-control environments without environment-specific modifications, including challenging tasks with raw-pixel observations.
Takeaways & Limitations
DDPG requires a large number of training episodes to find solutions, as is common for model-free reinforcement-learning approaches.
Abstract
from arXiv · showhide
We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
1 INTRODUCTION
The paper introduces DDPG, a model-free, off-policy actor-critic algorithm for high-dimensional continuous action spaces, addressing limitations of DQN and action-space discretization. It evaluates DDPG on challenging physical-control tasks, finding competitive policies with fixed architectures and hyperparameters, including from pixel inputs.
- Motivation: DQN cannot straightforwardly handle continuous, high-dimensional actions because maximizing its action-value function requires iterative optimization at every state.DQN addresses high-dimensional observations but only discrete, low-dimensional action spaces.
- Motivation: A 7-degree-of-freedom system with three discrete actions per joint produces an action space of 3^7 = 2187 actions.The passage identifies this exponential growth as the curse of dimensionality in discretized continuous control.
- Method: The work presents DDPG, a model-free, off-policy actor-critic algorithm using deep function approximators for high-dimensional, continuous action spaces.The method is based on the deterministic policy gradient algorithm.
- Method: DDPG combines an actor-critic approach with DQN-inspired techniques intended to make learning with large nonlinear value-function approximators stable and robust.The cited DQN innovations include off-policy replay-buffer training, though the supplied passage truncates the remaining details.
- Evaluation: DDPG learns competitive policies across the physical-control tasks using identical network structures and hyperparameters, and often learns good policies directly from pixels.The evaluation includes complex multi-joint movements, contact dynamics, gait behavior, and cartpole swing-up.
- Evaluation: The approach uses a straightforward actor-critic architecture with few moving parts and is compared against a planner with full access to simulated dynamics and their derivatives.The paper presents simplicity as a key feature for implementation, scaling, and comparison on physical-control problems.
2 BACKGROUND
The paper formulates reinforcement learning as an agent interacting with an environment over discrete timesteps, with real-valued actions, observations, and rewards. Policies map states to action distributions, and learning seeks to maximize expected discounted return, often using action-value functions and Bellman-style recursion.
- Reinforcement learning setup: At each discrete timestep, the agent receives an observation, takes a real-valued action in IR^N, and receives a scalar reward.When environments are partially observed, the relevant state may be the full history of observation-action pairs.
- Reinforcement learning setup: The environment is modeled as a possibly stochastic Markov decision process with state space S, action space A = IR^N, initial-state distribution, transition dynamics, and reward function.An agent’s policy maps states to probability distributions over actions.
- Objective and value functions: The return is the sum of discounted future rewards, and the learning objective is to maximize expected return from the start-state distribution.The discount factor satisfies γ ∈ [0, 1], and the return depends on the policy’s actions and may be stochastic.
- Objective and value functions: Action-value functions describe expected return after taking an action and then following a policy, while Bellman equations provide a recursive relationship used by many reinforcement-learning algorithms.The discounted state-visitation distribution for policy π is denoted ρ^π.
3 ALGORITHM
DDPG adapts deterministic policy gradients into an actor-critic method for continuous action spaces, avoiding the impractical per-timestep optimization required by Q-learning. Its design combines neural-network function approximation with replay-based minibatch learning, stabilized target networks, batch normalization, and independent exploration noise.
- Core algorithm: DDPG uses a deterministic actor-critic architecture because continuous-space Q-learning requires an impractically slow greedy-action optimization at every timestep.The actor deterministically maps states to actions, while the critic is learned with the Bellman equation.
- Function approximation: Neural-network function approximators enable learning and generalization over large state spaces, but remove convergence guarantees and make batch-learning alternatives intractable for large networks.The paper motivates a minibatch-compatible approach while noting the stability trade-off introduced by nonlinear approximators.
- Experience replay: DDPG uses a finite replay buffer to decorrelate sequential experience and support efficient minibatch updates, discarding the oldest transitions when full.Transitions are sampled under the exploration policy and minibatches are drawn uniformly from the buffer; the off-policy formulation permits a large buffer.
- Target networks: Softly updated copies of the actor and critic serve as target networks to reduce divergence caused by using the changing Q network to compute target values.This modifies the DQN target-network idea for the actor-critic setting by replacing direct weight copying with soft updates.
- Observation normalization: Batch normalization standardizes each observation dimension across minibatches and maintains running statistics for normalization during testing or exploration.The technique addresses differing physical units and value ranges across features and environments, which can hinder learning and hyper-parameter transfer.
- Exploration: DDPG separates exploration from learning by adding noise to the actor policy, using an Ornstein-Uhlenbeck process for temporally correlated exploration in inertial physical-control tasks.The exploration noise process can be chosen to suit the environment.
4 RESULTS
The results evaluate DDPG across diverse simulated control environments using both state and pixel inputs. DDPG learns strong policies across many tasks, with target networks and batch normalization important for robust performance.
- 4 RESULTS: DDPG was tested in simulated environments spanning cartpole, high-dimensional manipulation, contact-rich puck striking, locomotion, and Torcs driving.Most tasks used joint torques as actions and were simulated with MuJoCo; Torcs used acceleration, braking, and steering.
- 4 RESULTS: All tasks were evaluated with both low-dimensional state descriptions and high-dimensional renderings, using three simulation steps per agent timestep for action repeats.The action-repeat procedure was intended to make high-dimensional environments approximately fully observable.
- 4 RESULTS: Target networks and batch normalization were necessary for strong performance across all tasks, while removing the target network made learning very poor in many environments.The comparison included original DPG, batch normalization, target networks, and pixel-only target-network variants.
- 4 RESULTS: In simpler tasks, pixel-based policies learned as quickly as policies using low-dimensional state descriptors.The authors suggest action repeats or easily separable convolutional representations may explain this result.
- 4 RESULTS: DDPG learned good policies on many tasks, and some replicas exceeded iLQG even when learning directly from pixels.Scores were averaged over 5 replicas and normalized so the naive policy scored 0 and iLQG, a full-model planning solver, scored 1.
- 4 RESULTS: Q estimates were accurate without systematic bias in simple tasks, degraded on harder tasks, yet still supported good policies.The comparison used post-training Q estimates against true returns from test episodes.
5 RELATED WORK
Prior work established deterministic policy gradients on toy or low-dimensional tasks, while this work addresses scaling to difficult, high-dimensional observations. Related approaches improve stability, data efficiency, or model-based control, but impose limitations such as stochastic policies, local dynamics models, computational cost, or restricted task assumptions.
- Deterministic policy gradients: The original DPG study used tile-coding and linear function approximators, showing data-efficiency advantages and solving an octopus-arm task, but not scaling to large, high-dimensional observations.It compared off-policy DPG with on- and off-policy stochastic actor-critic methods.
- Policy-search scalability: Standard policy search has been considered too fragile for difficult problems because complex dynamics and policies can cause instability or impractically slow learning.Reported failure modes include later updates destroying progress or learning proceeding too slowly to be practical.
- Model-free policy search: Recent model-free policy-search work challenged this fragility using stochastic actor-critic policies with replay buffers, while related DPG extensions remained limited to two low-dimensional domains.SVG(0) also uses a Q-critic but learns a stochastic policy, with DPG described as its deterministic limit.
- Alternative policy optimization: TRPO constrains policy updates to prevent large divergences and produces near-monotonic return improvements without learning an action-value function.It directly constructs stochastic neural-network policies rather than decomposing the problem into optimal-control and supervised-learning phases.
- Model-based and guided methods: GPS improves data efficiency by combining local dynamics approximations, optimal control, and supervised policy learning, while PILCO offers analytic policy gradients but is computationally impractical for high-dimensional problems.GPS has been applied to vision-based robotic manipulation, whereas PILCO uses Gaussian processes to model dynamics.
6 CONCLUSION
The work yields a robust algorithm for challenging continuous-action problems across domains, including settings with raw-pixel observations. Experiments also show relatively efficient learning, while large training requirements remain a limitation.
- 6 CONCLUSION: DDPG robustly solves challenging problems across varied continuous-action domains, including tasks using raw pixels for observations.The approach combines insights from deep learning and reinforcement learning.
- 6 CONCLUSION: 2.5 million steps of experience solved nearly all problems, usually with far fewer steps.This was substantially fewer experience steps than DQN used to find solutions in Atari.
- 6 CONCLUSION: 20 fewer steps than DQN required for good Atari solutions suggests DDPG may solve more difficult problems with additional simulation time.The conclusion frames this as an implication of the observed experience efficiency, not as a demonstrated result on harder problems.
- 6 CONCLUSION: DDPG requires a large number of training episodes to find solutions, remaining a notable limitation of the model-free approach.The authors suggest robust model-free methods may contribute to larger systems that address such limitations.
Supplementary Information: Continuous control with
This section is identified as supplementary information on continuous control with deep reinforcement learning.
- It concerns deep reinforcement learning.
7 EXPERIMENT DETAILS
The experiments used Adam with separate actor and critic learning rates, stabilized Q-learning with weight decay and soft target updates, and bounded actor outputs with tanh. Exploration used temporally correlated Ornstein–Uhlenbeck noise suited to momentum-rich physical environments.
- Optimization and network architecture: Adam used learning rates of 10−4 for the actor and 10−3 for the critic, with Q weight decay of 10−2, γ = 0.99, and τ = 0.001.Hidden layers used rectified nonlinearities, while the actor’s final tanh layer bounded actions.
- Exploration noise: Exploration used an Ornstein–Uhlenbeck process with θ = 0.15 and σ = 0.2 to produce temporally correlated noise for physical environments with momentum.The process models a frictional Brownian particle, yielding temporally correlated values centered around zero.
8 PLANNING ALGORITHM
The planner is a receding-horizon model-predictive controller that performs one iLQG trajectory-optimization iteration per timestep from the system’s true state.
- 8 PLANNING ALGORITHM: At every timestep, the planner runs one iLQG trajectory-optimization iteration from the true system state using a receding 250ms–600ms horizon.The horizon recedes as the simulated world unfolds.
- 8 PLANNING ALGORITHM: Each iLQG iteration initializes its nominal trajectory with a rollout of the previous policy, then locally approximates dynamics linearly and costs quadratically.Repeated simulated-dynamics samples support the local model expansions.
9 ENVIRONMENT DETAILS
The environments use dense, task-specific rewards for physical control and Torcs, and span balancing, locomotion, reaching, grasping, manipulation, and ball-hitting tasks. Several highlighted tasks vary the number of poles, require fast movement or stabilization, or involve gripper-based object transport.
- Reward functions: Torcs rewards velocity projected along the track and penalizes collisions by −1, terminating episodes after 500 frames without progress.The reward is applied at each step, and termination depends on failure to progress along the track.
- Reward functions: Physical-control rewards provide feedback at every step, include a small action cost, and encode distance to static goals or movement toward targets.Some static-goal tasks also add a positive reward within a small target radius, while grasping and manipulation rewards encourage movement toward the target.
- Balancing tasks: The benchmark includes cart-pole variants requiring one, two, or three poles to remain upright, including parallel and serial attachments.The cartpoleBalance task begins with the pole upright, while the parallel-double, serial-double, and serial-triple variants extend the balancing challenge.
- Locomotion and manipulation: Highlighted locomotion tasks require a cheetah to move forward quickly, a gripper arm to transport an object to a fixed target, and other agents to maintain balance.The cheetah is constrained to the plane, the gripper manipulates an object toward a fixed target, and the listed benchmark also includes cart-pole balancing.