Learning Continuous Control Policies by Stochastic Value Gradients

Nicolas Heess; Greg Wayne; David Silver; Timothy Lillicrap; Yuval Tassa; Tom Erez

Source-linked AI summary

Learning Continuous Control Policies by Stochastic Value Gradients

Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, Tom Erez

arXiv:1510.09142v1Oct 30, 2015cs.LGcs.NE

TL;DR

Continuous-control policy gradients often suffer from high-variance likelihood-ratio estimates and value-gradient methods are restricted by deterministic policies, models, or planning. The paper introduces Stochastic Value Gradients, which re-parameterize stochastic Bellman equations and use learned models, critics, and real environment observations across model-based and model-free algorithms. SVG(1) performs well across the reported continuous-control experiments, while experience replay gives SVG(1)-ER the strongest tested results.

Problem
Existing policy-gradient estimates can have high variance, while prior value-gradient methods are limited to deterministic policies and deterministic models.
Method
The framework re-parameterizes policy and environment stochasticity as functions of exogenous noise and computes value gradients using learned models, value functions, and policies trained from environment interaction.
Results
SVG(1) performs well across the experiments, and SVG(1)-ER demonstrates a significant advantage over all other tested algorithms.
Takeaways & Limitations
SVG broadens value-gradient methods to stochastic policies and environments while combining model-based gradient computation with real-environment data and value-function robustness.
Takeaways & Limitations
The framework assumes a learned differentiable dynamics model when model-based variants are used, although model accuracy can constrain performance.

Abstract

from arXiv · show

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

1 Introduction

The paper introduces Stochastic Value Gradient methods, extending backpropagation-based policy gradients to stochastic policies and environments while reducing reliance on model-predicted trajectories.

Likelihood-ratio policy-gradient estimates often have high variance, requiring many samples or low-dimensional policy parameterizations.
Value-gradient methods compute policy gradients through backpropagation using environment models, value functions, or Q-functions.
Stochastic Value Gradient methods address prior restrictions to deterministic policies and deterministic modeled environments through re-parameterization.
The framework learns dynamics models, value functions, and policies jointly from environment interaction, using models for gradient computation rather than trajectory prediction.This reduces the impact of model error while combining model-based and model-free elements.
SVG algorithms span model-based and model-free variants and are evaluated on stochastic control and physics-based continuous-control problems.The experiments include a toy stochastic problem and several simulated physical control tasks.

2 Background

The paper formulates continuous-control policy optimization as maximizing expected discounted rewards in continuous-state, continuous-action Markov decision processes.

The setting is a discrete-time MDP with continuous states and actions, stochastic transitions, an initial-state distribution, and potentially time-varying rewards.
The objective may use either finite-horizon or infinite-horizon discounted reward sums.
The analysis uses state-value and state-action-value functions, with time-dependent values for finite horizons and stationary values for infinite horizons.

3 Deterministic value gradients

Deterministic value gradients differentiate Bellman recursions through deterministic policies and dynamics to compute policy-gradient contributions along trajectories.

For deterministic models and policies, the Bellman equation expresses current value as reward plus discounted value at the modeled next state.
Differentiating the deterministic Bellman equation with respect to state and policy parameters yields value-gradient expressions.
The resulting coupled recursions can proceed backward from trajectory termination to compute gradients of value with respect to state and policy parameters.
Summing per-time-step contributions produces the total policy gradient, with Q-functions providing action-based contributions.

4 Stochastic value gradients

Stochastic Value Gradients re-parameterize policy and environment noise so Bellman-value information can be backpropagated, including along real trajectories, while supporting model, critic, replay, and model-free variants.

4.1 Differentiating the stochastic Bellman equation: Re-parameterization expresses stochastic variables as deterministic functions of exogenous noise, allowing stochastic and deterministic derivatives to be computed similarly.A conditional Gaussian can be written as a mean plus scale multiplied by standard Gaussian noise.
4.1 Differentiating the stochastic Bellman equation: The resulting Monte Carlo derivative estimator uses the Jacobian of the transformed function rather than a likelihood-ratio term.
4.1 Differentiating the stochastic Bellman equation: SVG represents stochastic actions and transitions with policy noise η and environment noise ξ, then differentiates the re-parameterized Bellman equation.
SVG(∞): SVG(∞) computes value gradients by backward recursion on finite-horizon trajectories and trains the learned model and policy after episodes.
4.1 Differentiating the stochastic Bellman equation: Bayes-rule conditioning allows gradient expectations to use observations from real-environment interaction instead of model-generated trajectories.For additive noise, a deterministic learned model can still provide the relevant derivatives, while richer noise can use a re-parameterized generative model.
4.3 SVG(1) and SVG(0): Critics replace single-trajectory gradient estimates with estimated state-value derivatives, reducing variance and supporting infinite-horizon gradients.
4.3 SVG(1) and SVG(0): SVG(1)-ER adds off-policy experience replay with importance weighting, improving data efficiency and enabling infinite-horizon computation.
4.3 SVG(1) and SVG(0): SVG(0) is a model-free stochastic value-gradient method that estimates derivatives around policy noise and can learn policy noise variance.

5 Model and value learning

The framework supports flexible learning of differentiable models and value functions, including supervised model learning from observed transitions and joint model-policy training.

Model learning: Differentiable generative models can represent nonlinear, state- and action-dependent noise in the environment dynamics.The paper parameterizes these models as neural networks, including forms such as ˆf(s, a, ξ) = ˆµ(s, a) + ˆσ(s, a)ξ.
Model learning: Model learning is purely supervised, using observed state transitions rather than simulated trajectories.The model and policy are trained jointly, and the paper does not require an additional motorbabbling phase in the supplied passage.
Value learning: Value functions can be learned with temporal-difference learning, regression to empirical episode returns, or Bellman residual minimization.In practice, the experiments used a version of fitted policy evaluation.

6 Experiments

Experiments test SVG on stochastic control and physics-based environments, emphasizing real-trajectory gradients, robustness to model and value-function degradation, and performance across complex tasks.

6.1 Analyzing SVG: Real-trajectory backpropagation optimized a stochastic policy, whereas planning with a learned stochastic model made little progress because prediction errors compounded over 1,000 steps.The hand task used a 0.01-second simulation step and required backpropagating distal reward over a long horizon.
6.1 Analyzing SVG: SVG(∞) learned the toy control task, while both model-based planning and REINFORCE performed poorly on the long-horizon problem.The planner’s model roll-outs became inaccurate as prediction error accumulated; REINFORCE achieved very poor results due to the long horizon.
6.1 Analyzing SVG: SVG(1) remained effective as the swimmer model shrank from 20 to 5 hidden units per state-dimension subnetwork, while SVG(∞) deteriorated significantly.SVG(1) also scaled to 5- and 7-link swimmers.
6.1 Analyzing SVG: SVG(1) outperformed the tested model-free actor-critic approach on 3-, 5-, and 7-link swimmer tasks.The actor-critic method used a V-function and TD-error policy updates, with the same value-learning code as SVG(1).
6.1 Analyzing SVG: SVG(1) degraded less than Q-function-based DPG when value-function capacity was reduced on cart-pole.The paper attributes this difference presumptively to auxiliary information about the Q function contained in the dynamics model.
6.2 SVG in complex environments: SVG(1)-ER reliably optimized policies across several complex physical environments with stochastic, nonlinear, and contact-driven discontinuous dynamics.The evaluated domains included Reacher, Gripper, Monoped, Half-Cheetah, and Walker.
6.2 SVG in complex environments: Policies learned gripping and walking behavior under adversarial noise levels up to 25%, although similar walking rewards did not always yield equally good walking phenotypes.The noise level exceeded that used during training.

7 Related work

The paper situates its approach among methods using models and value functions to compute policy gradients, emphasizing its use of global neural-network models with value-function approximators.

Prior work used Gaussian process models or locally linear models to compute policy gradients or optimize policies.These approaches include uncertainty-sensitive gradients and trajectory optimization with locally linear models.
This work instead combines global neural-network models with value-function approximators.

8 Discussion

The discussion concludes that SVG broadens value-gradient methods beyond planning and deterministic models, while identifying extensions and modeling improvements for future work.

SVG removes reliance on planning and restriction to deterministic models, broadening value-gradient methods’ relevance to reinforcement learning.
SVG can robustly train neural-network policies for continuous-control problems.
SVG(1) with experience replay produced the best results among the tested variants.
Future extensions include SVG(k) variants with k-step model backpropagation and more sophisticated generative models of stochastic dynamics.

A Derivation of recursive gradient of the deterministic value function

The derivation explains that policy-parameter changes propagate through immediate actions and future states and actions, motivating a recursive total-derivative operator for the value function.

Changing policy parameters affects the immediate action as well as every future state and action.
The derivation expands the total derivative into recursive contributions from action and state sensitivities over future time steps.
The deterministic Bellman equation expresses the value as immediate reward plus discounted next-state value before applying the gradient operator.
The recursive derivation corresponds to equation 4 in the main text’s tick notation.

B Gradient calculation for noise models

The noise-model gradient procedure handles latent policy and environment noise through forward sampling or posterior inference, while the appendix also records implementation details for replay-based SVG(0).

Gradient calculation for noise models: Forward-sampled noise variables permit Jacobian evaluation by generating actions and next states with the policy and learned system model.
Gradient calculation for noise models: Real-environment trajectories conceal the noise variables, which may need to be inferred to evaluate Jacobians at the correct arguments.
Gradient calculation for noise models: Bayes’ rule reverses the forward sampling process by inferring noise variables conditioned on observed states, actions, and next states.
Gradient calculation for noise models: The appendix illustrates substituting a specific reward-and-value expression for g when evaluating equation 11.
Gradient calculation for noise models: The learned dynamics model used parallel subnetworks to predict one mean-and-scale pair per state dimension, with scale components parameterized as constant biases.
Gradient calculation for noise models: SVG(0) with replay stores observed transitions, trains a value function from the experience database, infers policy noise, and updates the policy from replayed samples.

D Experimental details

The experiments specify algorithm comparisons, replay and gradient-stability procedures, hyperparameter tuning, task rewards, discount factors, and network-layer sizes.

Policy updates: SVG(0), SVG(1), SVG(∞), and DPG omit experience replay for policy updates, while value-function updates still use replay.SVG(∞) cannot use replay for policy updates; SVG(1)-ER instead updates the policy at episode ends after value-function updates.
Gradient stability: SVG(∞) uses backpropagation through a recurrent-network-like chain, so the experiments clip policy-gradient norms to improve stability.Gradient clipping was used for all algorithms but was primarily necessary for SVG(∞).
Hyperparameter tuning: Hyperparameters were optimized separately for each algorithm using range identification followed by systematic grid search.Tuned parameters included policy, value-function, and model learning rates and update counts, plus SVG(1)-ER regularization and policy-noise standard deviation.
Task rewards: The Hand reward penalizes action magnitude before 10 seconds and uses hand-ball and ball-target distances at 10 seconds.The reward terms are weighted by α1 and α2 and use Euclidean distance.
Task rewards: The Swimmer reward combines center-of-mass velocity toward the goal with an action-magnitude penalty.The goal direction is represented by a unit vector from the nose to the goal.
Discounting: Discount factors vary by task, including 1.0 for Hand, 0.995 for Swimmer, 0.98 for Reacher, and 0.95 for Hopper.The listed factors also assign 0.98 to Gripper, Cheetah, and Walker.