PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Dimensional Continuous Control Using Generalized Advantage Estimation

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel

arXiv:1506.02438v6cs.LGcs.ROeess.SY

TL;DR

Policy gradient methods face high sample complexity and noisy advantage estimates when applied to difficult control problems. This paper introduces generalized advantage estimation with trust-region optimization, enabling neural-network policies to learn challenging simulated locomotion tasks, with intermediate λ values of 0.9–0.99 usually performing best.

  • Problem

    Policy gradient methods have limited success on difficult control problems because high-variance advantage estimates contribute to high sample complexity.

  • Method

    The paper combines generalized advantage estimation with trust-region optimization for neural-network policy and value-function training.

  • Results

    The approach learns complex gaits and difficult control behaviors for simulated 3D robots using high-dimensional neural-network policies and value functions.

  • Takeaways & Limitations

    Choosing λ between 0.9 and 0.99 usually gives the best performance in the paper’s simulated robotic locomotion experiments.

  • Takeaways & Limitations

    The relationship between value-function estimation error and policy-gradient estimation error remains unresolved.

Abstract

from arXiv · show

Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.

1 INTRODUCTION

The paper introduces generalized advantage estimation (GAE), a policy-gradient estimator designed to reduce variance while tolerating limited bias. It combines this analysis with trust-region optimization and evaluates neural-network policies and value functions on challenging high-dimensional 3D locomotion tasks.

  • Contributions: GAE reduces policy-gradient variance while maintaining a tolerable level of bias through parameters γ and λ.The estimator family is parameterized by γ ∈ [0, 1] and λ ∈ [0, 1].
  • Contributions: The paper provides novel analysis and intuition for GAE, enabling its application to online, batch, and batch trust-region algorithms.The analysis also interprets GAE as reward shaping using an approximate value function.
  • Experiments: The experiments use general-purpose neural networks for both policy and value functions, each with over 10^4 parameters, on challenging 3D locomotion tasks.The policies control simulated robots directly at the torque level, with up to 33 state dimensions and 10 actuators.
  • Contributions: Trust-region optimization is proposed as part of the approach for stabilizing learning with neural-network policy and value-function representations.The paper specifically applies a batch trust-region algorithm in its experiments.

2 PRELIMINARIES

The section formulates policy optimization as maximizing finite expected total reward in an undiscounted, episodic process and introduces policy-gradient estimators based on value and advantage functions. It treats γ as a variance-reduction parameter that introduces bias and defines γ-just estimators for the resulting discounted policy gradient.

  • Problem formulation: The objective is to maximize finite expected total reward over trajectories sampled from an initial-state distribution, policy, and environment dynamics until termination.The formulation is undiscounted, with rewards defined at each timestep.
  • Discounting and bias: γ reduces variance by downweighting rewards from delayed effects, at the cost of introducing bias, while serving as an algorithm parameter in the undiscounted problem.The discounted problem can also be represented as undiscounted by absorbing the discount into a time-dependent reward function.
  • Policy gradients: Policy gradients repeatedly estimate the gradient of expected total reward, using state-action values, advantages, and baselines to construct related estimators.The advantage is defined as Aπ(st, at) := Qπ(st, at) −V π(st).
  • Policy gradients: Using Ψt = Aπ(st, at) yields almost the lowest possible variance, but the advantage function is generally unknown and must be estimated.The advantage measures whether an action is better or worse than average at a state.
  • Discounting and bias: A γ-just advantage estimator produces an unbiased estimate of the γ-discounted policy gradient when substituted for the unknown discounted advantage.This unbiasedness concerns gγ; using Aπ,γ instead of Aπ has already introduced bias relative to the undiscounted policy gradient.
  • Estimator conditions: A sufficient condition for γ-justness is decomposing the estimator into Qt minus bt, where Qt is unbiased for the γ-discounted Q-function and bt depends only on prior trajectory variables.The baseline bt may be an arbitrary function of states and actions sampled before at.

3 ADVANTAGE FUNCTION ESTIMATION

This section develops generalized advantage estimation from TD residuals and multi-step return estimates. GAE uses an exponentially weighted combination controlled by λ to trade off estimator bias and variance when the value function is approximate.

  • TD residual estimator: The one-step TD residual δ_t^V = r_t + γV(s_t+1) − V(s_t) estimates the action advantage and is unbiased when V equals the true discounted value function.With an inaccurate value function, it produces biased policy-gradient estimates.
  • Multi-step estimators: The k-step estimator combines discounted rewards with a terminal value estimate and subtracts the baseline V(s_t); as k increases, its bias generally decreases.As k approaches infinity, the estimator becomes empirical returns minus the value-function baseline.
  • Generalized advantage estimator: GAE(γ, λ) is an exponentially weighted average of k-step advantage estimators, equivalently a discounted sum of Bellman residuals analogous to TD(λ).Unlike TD(λ), which estimates the value function, GAE estimates the advantage function.
  • Special cases: GAE(γ, 0) equals the TD residual, whereas GAE(γ, 1) is γ-just for any V but has high variance; intermediate λ values compromise between bias and variance.The tradeoff is controlled by λ: λ = 0 typically has lower variance but can be biased with an inaccurate value function.
  • Roles of γ and λ: The parameters γ and λ affect the bias-variance tradeoff differently: γ sets the scale of Vπ,γ, while γ < 1 introduces policy-gradient bias and λ < 1 controls estimator bias from value approximation.The supplied passage explicitly distinguishes their purposes and preferred ranges, though its discussion of λ is truncated.

4 INTERPRETATION AS REWARD SHAPING

This section interprets GAE as applying an additional λ discount after reward shaping, while using the response function to explain bias from γ and λ. Reward shaping preserves the discounted advantage, and γ and λ control temporal credit assignment and estimator bias.

  • Reward shaping: Reward shaping transforms rewards using a state potential while leaving the discounted advantage function Aπ,γ unchanged.The transformed action-value and state-value functions each subtract Φ(s), so their difference remains Aπ,γ.
  • GAE interpretation: Using a steeper discount γλ on shaped rewards exactly recovers the generalized advantage estimator.λ = 1 gives an unbiased estimate of gγ, whereas λ < 1 gives a biased estimate.
  • Response function: The response function χ(l; st, at) decomposes advantage across timesteps and measures long-range dependencies between actions and rewards.Nonzero χ values for l ≫ 0 indicate temporally extended action effects on rewards.
  • Temporal credit assignment: Using γ < 1 drops response terms with l ≫ 1/(1 −γ), causing little error when action effects are forgotten within that timescale.The approximation is accurate when χ rapidly decays as l increases.
  • Temporal credit assignment: Choosing Φ = Vπ,γ makes the response immediate by reducing the value of the shaped reward to zero after the current timestep.An approximation V ≈ Vπ,γ can therefore partially reduce the temporal spread of the response function.

5 VALUE FUNCTION ESTIMATION

The value function can be estimated by nonlinear regression using discounted reward returns, including the Monte Carlo or TD(1) approach. In these experiments, a trust-region update limited overfitting to recent data and was approximately solved with conjugate gradients.

  • Value-function estimation: Nonlinear function approximation turns value-function estimation into a regression problem using discounted sums of rewards, known as the Monte Carlo or TD(1) approach.The discounted return is formed from rewards across timesteps in a batch of trajectories.
  • Trust-region optimization: A trust-region method optimized the value function in each batch iteration, helping avoid overfitting to the most recent data.The update uses the parameter vector before optimization as the previous value-function reference.
  • Trust-region optimization: The constraint limits the average KL divergence between the previous and new value functions to less than ϵ under a conditional Gaussian parameterization.The Gaussian has mean Vφ(s) and variance σ2.
  • Trust-region optimization: Conjugate gradients approximately solve the trust-region quadratic program using matrix-vector products with a Gauss-Newton Hessian approximation.The resulting step is rescaled to satisfy the trust-region constraint before updating φ.

6 EXPERIMENTS

The experiments evaluate generalized advantage estimation across γ and λ settings and test its combination with trust-region optimization on cart-pole and challenging 3D robot-control tasks. Intermediate parameter values produce the strongest results across the evaluated domains.

  • Experimental questions and method: The experiments vary γ and λ to assess generalized advantage estimation and use TRPO for policy updates while keeping the underlying algorithm fixed.The study also examines whether this combination can optimize large neural-network policies for challenging control problems.
  • Tasks: The evaluation covers cart-pole balancing, bipedal locomotion, quadrupedal locomotion, and dynamically standing up from a supine start.The 3D robot tasks use MuJoCo simulations.
  • Task setup: The 3D robot policy and value function use identical feedforward networks with three hidden layers of 100, 50, and 25 tanh units, followed by a linear output layer.The value network has one scalar output; cart-pole instead uses a linear policy and a one-hidden-layer value network with 20 units.
  • Parameter study: 21 experiments show the best overall results at γ ∈[0.96, 0.99] and λ ∈[0.92, 0.99].For cart-pole, the fastest policy improvement occurs at intermediate λ values in [0.92, 0.98] when γ = 0.99.
  • 3D locomotion results: 9 trials on 3D bipedal locomotion find best performance at γ ∈[0.99, 0.995] and λ ∈[0.96, 0.99].After 1000 iterations, the learned gait is described as fast, smooth, and stable; quadrupedal locomotion and standing-up experiments compare λ = {0, 0.96} with γ = 0.995.

7 DISCUSSION

The discussion identifies advantage estimation as central to reducing policy-gradient variance and combines generalized advantage estimation with trust-region optimization for difficult control tasks. It highlights λ selection, value-function error, shared representations, and continuous-action methods as directions for further investigation or comparison.

  • Motivation: Advantage estimates are identified as the key to reducing policy-gradient variance and addressing the high sample complexity limiting difficult control problems.Policy gradients provide unbiased gradient estimates, but their success on difficult control problems has been limited by high sample complexity.
  • Method: Generalized advantage estimation uses γ and λ to adjust the bias-variance tradeoff and can be combined with trust-region optimization for neural-network policies and value functions.The combined techniques enabled solving difficult control tasks that had previously been challenging.
  • Hyperparameter selection: λ values in the range [0.9, 0.99] usually produced the best simulated robotic locomotion performance.The discussion identifies adaptive or automatic adjustment of γ and λ as future work.
  • Open questions: The relationship between value-function estimation error and policy-gradient estimation error remains unresolved, motivating error metrics matched to policy-gradient accuracy.Candidate metrics include the Bellman error and projected Bellman error.
  • Open questions: Sharing function-approximation architectures between the policy and value function could enable faster learning by reusing useful input features.Formulating the approach for numerical optimization with convergence guarantees remains an open question.
  • Related work: Although one-step returns with λ = 0 caused excessive bias and poor performance here, concurrent continuous-action methods worked when tuned appropriately on lower-dimensional control problems.The cited concurrent work differentiates with respect to continuous-valued actions and considers substantially lower-dimensional state and action spaces.

A FREQUENTLY ASKED QUESTIONS · A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?

The paper relates compatible features to policy-gradient methods using value functions and explains their role in natural policy-gradient computation. It also notes that the compatible-feature projection can use the paper’s advantage estimators, while experiments use a more efficient numerical procedure.

  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: Compatible features arise in policy-gradient algorithms that use a value function.The idea is attributed to Konda & Tsitsiklis (2003).
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: Limited policy representation means the policy gradient depends only on a subspace of advantage functions.This subspace is spanned by ∇θ_i log πθ(a_t|s_t).
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: The compatible-feature idea motivates an elegant method for computing the natural policy gradient.The passage cites Kakade (2001a) and Peters & Schaal (2008).
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: An empirical advantage estimate can be projected onto the compatible-feature subspace by solving a least squares problem.The estimate is denoted ˆA_t at each timestep.
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: When ˆA is γ-just, the least squares solution equals the natural policy gradient.This result is attributed to Kakade (2001a).
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: Any advantage estimator can be substituted into the compatible-feature projection formula, including the estimators derived in this paper.The paper states that its experiments also compute natural policy-gradient steps.
  • A.1 WHAT’S THE RELATIONSHIP WITH COMPATIBLE FEATURES?: For experiments, the paper computes natural policy-gradient steps with a more computationally efficient numerical procedure.The procedure is from Schulman et al. (2015) and is discussed in Section 6.

A.2 WHY DON’T YOU JUST USE A Q-FUNCTION?

The paper acknowledges Q-functions as a route to potentially low-variance policy-gradient estimates, but reports prohibitively large bias for one-step returns and leaves bias-reduction methods using parameterized Q-functions unexplored.

  • Q-function alternatives: Prior actor-critic methods use Q-functions to obtain potentially low-variance policy-gradient estimates.Recent work also demonstrated effective use of neural-network Q-function approximators in policy-gradient methods.
  • Bias concern: They expect similar difficulty for advantage estimators using a parameterized Q-function, while leaving bias-reduction approaches beyond this work’s scope.The discussed form is ˆAt = Q(s, a) − V(s).

B PROOFS

The proof of Proposition 1 splits the expectation into terms involving Q and b, then considers those terms in turn.

  • The proof begins by splitting the expectation into terms involving Q and b.
  • It then considers the Q and b terms separately.
Loading 1506.02438v6…