Continuous control with deep reinforcement learning

1 INTRODUCTION

• DQN handles high-dimensional observations but only discrete, low-dimensional actions, while discretizing continuous actions causes exponential growth and discards action-domain structure.For example, a 7-degree-of-freedom system discretized to three choices per joint yields 2187 actions, making exploration difficult.Source: §1 · p. 1
• DDPG is a model-free, off-policy actor-critic for high-dimensional continuous actions, based on deterministic policy gradients; replay, target Q networks, and batch normalization support stable learning.A naive neural-function-approximation application of the actor-critic method is unstable on challenging problems.Source: §1 · pp. 1–2
• The evaluation spans physical-control tasks with multi-joint movement, rich contact dynamics, and gait, including cartpole swing-up, tested with low-dimensional observations and raw pixels.The authors used a fixed-viewpoint simulator camera to evaluate learning directly from pixels.Source: §1 · p. 2
• DDPG learns competitive policies across all tasks with the same hyperparameters and network structure, including policies learned directly from pixels in many cases.Its simple actor-critic design has few moving parts, and it sometimes exceeds a planner with full access to simulated dynamics and derivatives, even when learning from pixels.Source: §1 · p. 2

• The agent receives observation x_t, chooses real-valued action a_t, and receives scalar reward r_t; this paper assumes fully observed states, s_t = x_t.Source: §2 · p. 2
• The environment is modeled as an MDP with initial-state distribution, transition dynamics, reward function, and policies mapping states to action distributions.Source: §2 · p. 2
• The objective is to learn a policy that maximizes expected return from the start distribution, using discounted future rewards with γ ∈ [0, 1].Source: §2 · p. 2
• For deterministic target policies, Qµ can be learned off-policy from transitions generated by a different stochastic behavior policy β.Source: §2 · p. 3
• Mnih et al. (2013; 2015) scaled neural Q-learning with replay buffers and separate target networks, techniques this work employs for DDPG.Source: §2 · p. 3

3 ALGORITHM

• DDPG adapts DPG with neural-network modifications for online learning in large state and action spaces, where batch-learning NFQCA is intractable at scale.The actor deterministically maps states to actions, while the critic learns Q by the Bellman equation and the actor follows the chain-rule gradient of expected return.Source: §3 · p. 3
• Uniform minibatches from a finite replay buffer address sequentially correlated experience and enable minibatch learning; older transitions are discarded when the buffer fills.Because DDPG is off-policy, the buffer can be large and support learning across uncorrelated transitions.Source: §3 · p. 4
• Slowly tracking target actor and critic networks stabilize critic targets and reduce divergence, using θ′ ← τθ + (1 − τ)θ′ with τ ≪ 1.Both target networks were required for stable targets; their slower value propagation was outweighed in practice by improved learning stability.Source: §3 · p. 4
• Batch normalization normalizes minibatch dimensions to unit mean and variance, enabling learning across tasks with differing units and scales without manual feature scaling.For low-dimensional inputs it was applied to the state input, all actor layers, and critic layers before the action input; running statistics were used during evaluation and exploration.Source: §3 · p. 4
• DDPG separates exploration from learning by adding noise to the actor policy; an Ornstein–Uhlenbeck process supplied temporally correlated noise for physical control with inertia.The noise process can be chosen to suit the environment.Source: §3 · p. 4

4 RESULTS

• The evaluation covered simulated tasks including cartpole, high-dimensional manipulation, contact tasks, and locomotion, using both low-dimensional state descriptions and rendered pixel inputs.For pixel inputs, each observation combined three repeated-action renderings, downsampled to 64x64 pixels.Source: §4 · pp. 4–5
• Target networks and batch normalization were necessary for strong performance across tasks; removing target networks made learning very poor in many environments.Figure 2 compares DPG variants with these components and pixel-only inputs.Source: §4 · p. 5, Figure 2 · p. 6
• In some simpler tasks, policies learned from pixels as fast as those learned from low-dimensional state descriptions.The authors suggest action repeats or convolutional layers may help explain this result.Source: §4 · p. 5
• DDPG learned good policies on many tasks, and some replicas outperformed iLQG, even when learning directly from pixels.Scores were averaged over 5 replicas and normalized so the naive policy scored 0 and iLQG scored 1.Source: §4 · pp. 5–6
• DDPG’s value estimates were accurate without systematic bias on simple tasks, but worse on harder tasks where it still learned good policies.Source: §4 · p. 6
• On Torcs, some replicas learned reasonable policies that completed a circuit from either low-dimensional state or pixels, while others failed to learn sensible policies.The authors kept the physics-task architecture and hyperparameters but changed the exploration noise process for Torcs’ different time scales.Source: §4 · p. 6

5 RELATED WORK

• The original DPG study showed data-efficiency advantages and solved an octopus-arm task, but did not demonstrate scaling to large, high-dimensional observations.It used tile-coding and linear function approximators, and compared off-policy DPG with on- and off-policy stochastic actor-critic.Source: §5 · p. 6
• Policy search was considered fragile on difficult problems, with actor-critic methods often unstable or too slow to scale practically (Levine et al., 2015).Learning progress could be destroyed by later updates or proceed too slowly to be practical.Source: §5 · pp. 6–7
• Wawrzyński et al. trained stochastic policies with replay, while Balduzzi & Ghifary (2015) tested a DPG deviator network only on two low-dimensional domains.Heess et al. (2015) proposed stochastic-policy SVG(0), of which DPG is the deterministic limit; DDPG scaling techniques can also extend to stochastic policies via reparametrization (Heess et al., 2015; Schulman et al., 2015a).Source: §5 · pp. 7–8
• TRPO (Schulman et al., 2015b) constrains stochastic-policy updates for near-monotonic return improvement without learning an action-value function, but appears less data-efficient.It directly constructs stochastic neural-network policies rather than decomposing the problem into optimal-control and supervised-learning phases.Source: §5 · p. 8
• Guided policy search uses local dynamics approximations, optimal control, and supervised policy training, and has succeeded on vision-based robotic manipulation tasks (Levine et al., 2015).Its spatial softmax and direct robot-state inputs could increase policy power and data efficiency and could be used within DDPG.Source: §5 · p. 8
• PILCO uses Gaussian-process dynamics models for data-efficient analytic policy gradients, but its computational demand makes it impractical for high-dimensional problems.Wahlström et al. (2015) used a learned deep dynamics model and model-predictive control to solve pixel-based pendulum swing-up, but required demonstrable goal states.Source: §5 · p. 8
• Evolutionary methods learned competitive Torcs policies from pixels using compressed weight parametrizations or unsupervised learning, but their generalization to other problems was unclear (Koutník et al., 2014a; Koutník et al., 2014b).Source: §5 · p. 8

• DDPG robustly solves challenging continuous-action problems, including from raw pixels, and learns stably across environments without modifications despite lacking convergence guarantees.Source: §6 · p. 8
• Nearly all problems were solved within 2.5 million experience steps, a factor of 20 fewer than DQN required for good Atari solutions.The authors suggest that more simulation time may let DDPG solve harder problems than those considered.Source: §6 · pp. 8–9
• DDPG still requires many training episodes, but a robust model-free approach may help larger systems address this limitation (Gl¨ascher et al., 2010).Source: §6 · p. 9

7 EXPERIMENT DETAILS

• Adam used learning rates of 10−4 for the actor and 10−3 for the critic, with Q weight decay of 10−2, γ = 0.99, and τ = 0.001.All hidden layers used rectified nonlinearities; the actor’s final layer used tanh to bound actions.Source: §7 · p. 11
• Low-dimensional networks used two hidden layers of 400 and 300 units, totaling approximately 130,000 parameters.Actions were added to Q at its second hidden layer.Source: §7 · p. 11
• Pixel-input networks used three convolutional layers with 32 filters each, followed by two fully connected layers of 200 units, totaling approximately 430,000 parameters.The convolutional layers used no pooling, and actions were added at the fully connected layers.Source: §7 · p. 11
• Final actor and critic layer weights and biases were initialized uniformly in [−3 × 10−3, 3 × 10−3] for low-dimensional inputs and [3 × 10−4, 3 × 10−4] for pixels.This was intended to keep initial policy outputs and value estimates near zero; other layers used [−1/√f, 1/√f], where f is fan-in.Source: §7 · p. 11
• Minibatches contained 64 samples for low-dimensional problems and 16 for pixel inputs, with a replay buffer size of 10^6.Source: §7 · p. 11
• Exploration used Ornstein–Uhlenbeck noise with θ = 0.15 and σ = 0.2 to provide temporally correlated noise for momentum-rich physical environments.The process models a Brownian particle’s velocity with friction and produces values centered around zero.Source: §7 · p. 11

8 PLANNING ALGORITHM

• At each time step, the model-predictive controller runs one iLQG trajectory-optimization iteration from the true system state over a 250–600 ms receding horizon.The planner follows Tassa et al. (2012) and uses iLQG.Source: §8 · p. 11
• iLQG linearizes dynamics and quadratically approximates cost along a nominal rollout, then uses a backward value-function pass and forward line search to select a lower-cost trajectory.The nominal trajectory comes from an initial rollout of the previous policy; repeated simulated-dynamics samples approximate the local models.Source: §8 · p. 11
• The planner stores the selected action sequence to warm-start the next iteration, executes its first action, and uses the resulting state to initialize the next optimization.Source: §8 · p. 11

9.1 TORCS ENVIRONMENT

• Torcs rewards track-direction velocity and penalizes collisions; episodes end when the car makes no track progress for 500 frames.Source: §9.1 · p. 11

9.2 MUJOCO ENVIRONMENTS

• MuJoCo rewards include a small action cost, goal-distance or manipulation terms, and locomotion incentives for forward motion, smooth impacts, and avoiding falls.Falls trigger negative reward and early termination based on height and torso-angle thresholds in walker2d.Source: §9.2 · p. 12
• The MuJoCo suite includes tasks involving arm reaching and manipulation, pole balancing, and forward locomotion.Examples include a fixed-target 3-DOF reaching arm, cart-pole balancing, and a planar bipedal walker.Source: §9.2 · pp. 13–14