PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans

arXiv:1707.01891v3cs.AI

TL;DR

Trust-region methods can stabilize policy optimization, but require substantial on-policy interaction, while off-policy methods can have unstable optimization. Trust-PCL combines relative-entropy regularization with multi-step pathwise consistency to use off-policy data; on continuous-control tasks, it improves on TRPO in solution quality and sample efficiency.

  • Problem

    Trust-region methods can stabilize policy optimization but require substantial on-policy interaction, while off-policy methods such as DDPG can be sensitive to hyperparameters and initialization.

    Source: §1 · p. 1

  • Method

    Trust-PCL uses relative-entropy regularization and multi-step pathwise consistency to train policy and value estimates with off-policy data.

    Source: §1 · p. 2

  • Results

    Trust-PCL improves on TRPO in final reward and sample efficiency on continuous-control tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors report that Trust-PCL maintains TRPO’s stability and solution quality while approaching the sample efficiency of value-based methods.

    Source: §6 · p. 9

    The episode-sampling protocol is not exactly sampling from the prior policy, though the authors report that it works well in practice.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a prohibitively large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL. The algorithm is the result of observing that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. Thus, Trust-PCL is able to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.

1 INTRODUCTION

Trust-PCL combines off-policy learning with a trust-region approach and improves on TRPO in final reward and sample efficiency. It addresses the tension between off-policy methods’ data efficiency and trust-region methods’ stability.

1 INTRODUCTION

  • Off-policy methods such as DDPG can improve data efficiency, but their optimization stability depends strongly on hyperparameter selection and initialization.Even with optimal hyperparameters, individual runs can have highly varying outcomes.Source: §1 · p. 1
  • Trust-region methods such as TRPO perform strongly on difficult continuous-control tasks but require extensive on-policy interaction.This requirement makes them impractical for challenging real-world problems, according to the authors.Source: §1 · p. 1
  • Trust-PCL uses pathwise consistencies under relative-entropy regularization to train policy and value estimates with off-policy data.The resulting objective is equivalent to maximizing expected reward subject to a penalty-based constraint on divergence from a previous policy.Source: §1 · p. 2
  • Trust-PCL improves on TRPO in both final reward and sample efficiency on continuous-control tasks.The authors report that it solves difficult tasks and that its results can match or beat the state of the art.Source: §1 · p. 2
  • Trust-PCL includes a method for setting the relative-entropy coefficient to remain agnostic to reward scale and reduce hyperparameter-tuning demands.The authors report that this regularizer is crucial for good and stable performance.Source: §1 · p. 2

2 RELATED WORK

Prior trust-region approaches use local parameter-space geometry or policy divergence to constrain updates, while entropy regularization motivates a reverse-KL formulation. Trust-PCL builds on multi-step consistency relations and applies them to challenging continuous-control benchmarks.

Trust Region Methods

  • Natural-gradient methods use the Fisher information matrix to define local geometry, and TRPO uses approximations to make nonlinear policy optimization feasible.TRPO has achieved state-of-the-art results in continuous control, while proximal-gradient approaches replace hard trust-region constraints with objective penalties.Source: §2 · p. 2
  • With entropy regularization, the paper argues that a reverse-KL trust-region constraint is more natural than the usual approximation and characterizes its optimal update.Using the Fisher matrix at the current parameters is an approximation that is not ideal, particularly for large updates.Source: §2 · p. 3

Softmax Consistency

  • Earlier softmax-consistency work used single-step rollouts, whereas this work leverages consistency relations across multi-step rollouts.Multi-step consistencies with entropy regularization and relative entropy had been noted by Nachum et al. (2017) and Schulman et al. (2017a), respectively.Source: §2 · p. 3
  • This work combines relative-entropy and entropy regularization in a hybrid objective and automatically determines the relative-entropy coefficient.The authors say coefficient tuning is especially crucial when the reward distribution changes dramatically during training.Source: §2 · p. 3
  • Prior softmax-consistency evaluations mostly used simple tasks; this paper reports competitive or state-of-the-art performance on difficult, standard continuous-control benchmarks.Earlier examples included grid-world and discrete algorithmic environments, and simpler CartPole and Pendulum variants.Source: §2 · p. 3

3 NOTATION & BACKGROUND

The section defines the interaction and objective setting for model-free RL, then introduces entropy-regularized path consistency and TRPO’s trust-region procedure. PCL uses path consistency to optimize policy and value estimates from on- or off-policy trajectories.

3 NOTATION & BACKGROUND

  • An agent samples actions from its policy, receives scalar rewards, and transitions between states in the environment.The policy may be over discrete or continuous actions.Source: §3 · p. 3
  • The standard RL objective is expected future discounted reward, optimized per state and averaged over states sampled through environment interaction.Policy-based methods such as REINFORCE and actor-critic optimize expected reward for a parameterized policy.Source: §3 · p. 3
  • PCL augments expected reward with discounted entropy, with a user-specified temperature τ controlling the degree of entropy regularization.The entropy-regularized objective is defined recursively in terms of reward, policy entropy, and future objective.Source: §3 · pp. 3–4
  • The optimal policy and value function for the entropy-regularized objective satisfy a softmax temporal consistency along sampled state-action sequences.PCL minimizes squared error between the two sides of the consistency relation to optimize policy and value parameters, and supports on- and off-policy trajectories.Source: §3 · p. 4
  • TRPO samples trajectories from a prior policy, optimizes expected reward under an average per-state KL-divergence constraint, then replaces the prior with the new policy.It repeats this procedure at each iteration.Source: §3 · p. 4

4 METHOD

Trust-PCL combines entropy-regularized reward with a relative-entropy trust region around a prior policy, using the two regularizers for exploration and stability. It derives multi-step pathwise consistencies, trains policy and value estimates against them, and tunes the relative-entropy penalty from a desired divergence bound.

4 METHOD

  • Entropy regularization supports exploration, while relative entropy to a prior policy improves stability and allows a faster learning rate.The method combines both regularizers in its objective.Source: §4 · p. 4

4.1 PATH CONSISTENCY WITH RELATIVE ENTROPY

  • Relative-entropy regularization yields a transformed-reward objective whose optimal policy and values satisfy multi-step softmax temporal consistencies.The transformed reward is r(s,a) + λ log π˜(a|s).Source: §4.1 · pp. 4–5

4.2 TRUST-PCL

  • Trust-PCL trains a parameterized policy and value estimate by minimizing squared consistency error on sub-trajectories.The loss is defined over batches of episodes or sub-episodes.Source: §4.2 · p. 5
  • Training can use on- or off-policy trajectories; the implementation samples episodes from a recency-prioritized replay buffer and uses a lagged geometric-mean prior policy.The prior parameters are updated as a moving average of the online policy parameters.Source: §4.2 · p. 5

4.3 AUTOMATIC TUNING OF THE LAGRANGE MULTIPLIER λ

  • Trust-PCL redirects tuning from λ to a desired divergence ϵ, then uses a line search to choose λ(ϵ) from prior-policy episodes.The chosen coefficient makes KL(π*||π˜θ) as close as possible to ϵ.Source: §4.3 · p. 6
  • The tuning analysis assumes an undiscounted, deterministic finite-horizon setting, while its practical estimate uses recent episodes that only approximate prior-policy samples.The authors report that the protocol works well in practice and note importance weighting could make it more accurate.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Across benchmark tasks, Trust-PCL matches or improves on TRPO in average reward and sample efficiency. Experiments examine its setup, trust-region size, off-policy training, and reward results against external implementations.

5.1 SETUP

  • The experiments use Acrobot and five MuJoCo continuous-control tasks; TRPO trains on batches of 25,000 steps, while Trust-PCL trains from replay-buffer batches.Trust-PCL collects 10 environment steps between gradient steps and uses batches of 64 sub-episodes, with stability measures including Huber loss on consistency error.Source: §5.1 · p. 7

5.2 RESULTS

  • Trust-PCL matches or improves on TRPO in average reward and sample efficiency across benchmark tasks.The reported comparison covers reward during training and final reward; gains are especially significant on Walker2d and Ant.Source: §5.2 · p. 7, Figure 1 · p. 8

5.2.1 HYPERPARAMETER ANALYSIS

  • As ϵ increases, training becomes less stable, eventually harming the agent’s ability to achieve optimal reward.ϵ sets the size of the trust region; Figure 2 compares average greedy reward across values of ϵ over millions of environment steps.Source: §5.2.1 · p. 7, Figure 2 · p. 8
  • Off-policy Trust-PCL improves sample efficiency over its more on-policy setting while retaining competitive reward.The more on-policy setting uses α = 0.95, β = 0.1, and P = 1,000; it can achieve state-of-the-art reward but requires an exorbitant amount of experience.Source: §5.2.1 · pp. 8–9, Figure 3 · p. 9
  • External implementation results are overall competitive with state-of-the-art, but differing setups and evaluation protocols prevent definitive comparisons.Table 1 reports best average reward in the first 10M training steps.Source: Table 1 · p. 9
  • The exploration parameter τ had little effect in the evaluated tasks, and many of the best results used τ = 0.Including τ > 0 had at most a marginal effect; the authors suggest this may reflect the tasks themselves.Source: §5.2.1 · p. 9

6 CONCLUSION

Trust-PCL combines an off-policy algorithm with a relative-entropy trust-region penalty on a maximum reward objective. On standard control tasks, it improves on TRPO in average reward and sample efficiency.

6 CONCLUSION

  • Trust-PCL imposes a trust region on a maximum reward objective with a relative-entropy penalty.The conclusion presents it as an off-policy algorithm.Source: §6 · p. 9
  • Trust-PCL improves on TRPO in average reward and sample efficiency on a set of standard control tasks.The authors report maintaining TRPO’s stability and solution quality while approaching the sample efficiency of value-based methods.Source: §6 · p. 9
  • The reported results support the possibility of achieving stability and sample efficiency without trading one for the other in a single RL algorithm.Source: §6 · p. 9

A IMPLEMENTATION BENEFITS OF TRUST-PCL

Trust-PCL uses simple gradient descent rather than TRPO’s second-order calculations and line search. Its policy need not have a closed-form KL divergence, provided its log-density is analytically computable.

A IMPLEMENTATION BENEFITS OF TRUST-PCL

  • Trust-PCL uses simple gradient descent, whereas TRPO uses second-order KL-divergence calculations and a line search.The authors describe Trust-PCL as easier to implement in standard deep-learning frameworks.Source: A IMPLEMENTATION BENEFITS OF TRUST-PCL · p. 12
  • Trust-PCL does not require an analytically computable KL-divergence or its gradient; it requires only an analytically computable log-density.The authors say this permits more expressive policies, including Gaussian mixtures and autoregressive policies.Source: A IMPLEMENTATION BENEFITS OF TRUST-PCL · p. 12
  • The paper has not fully explored Trust-PCL’s implementation flexibility and ease in this work.The authors identify these benefits as subjects for future investigation.Source: A IMPLEMENTATION BENEFITS OF TRUST-PCL · p. 12

B EXPERIMENTAL SETUP

The appendix describes the implementation and hyperparameter search in detail.

B EXPERIMENTAL SETUP

B.1 ENVIRONMENTS

The environments differ in episode termination and batch composition, and Humanoid is excluded after both implementations made negligible progress on it.

B.1 ENVIRONMENTS

  • Acrobot episodes ended at step 500, while episodes in the remaining environments ended at step 1,000.Source: §B.1 · p. 12
  • Acrobot, HalfCheetah, and Swimmer are non-terminating, so batches contain equal-length episodes and the same number of episodes.Source: §B.1 · p. 12
  • Hopper, Walker2d, and Ant can terminate, so batches stay constant in steps but vary in episode count.Source: §B.1 · p. 12
  • Humanoid is omitted because both implementations made negligible progress; TRPO with GAE can perform well there, but GAE was excluded for fair variant comparisons.Trust-PCL's analogous varying-time-scale consistency approach is left for future work.Source: §B.1 · p. 12

B.2 IMPLEMENTATION DETAILS

The implementation uses neural networks for policy and value, with distinct update procedures for TRPO and the on- and off-policy Trust-PCL variants.

B.2 IMPLEMENTATION DETAILS

  • The policy uses two 64-unit tanh hidden layers and samples actions from a Gaussian whose mean depends on the observation and whose standard deviation is exp(ξ).The value network also has two 64-unit tanh layers; it receives the observation and its component-wise square and outputs one scalar.Source: §B.2 · pp. 12–13

B.2.1 TRPO LEARNING

  • TRPO updates the policy with a trust-region step and fits values with LBFGS to a 0.9-weighted mixture of empirical and expected values.The value fit also uses the previous value parameterization.Source: §B.2.1 · p. 13

B.2.2 TRUST-PCL LEARNING

  • On-policy Trust-PCL uses one Adam policy step and five value-gradient steps; off-policy Trust-PCL updates both networks in one Adam step and uses a lagged target value network.The on-policy value fit mixes empirical and expected values with κ = 0.95; the off-policy variant does not mix them.Source: §B.2.2 · p. 13

B.3 HYPERPARAMETER SEARCH

The section describes a grid search over key learning parameters and pseudocode that collects experience, trains from replay, and updates parameters and auxiliary variables.

B.3 HYPERPARAMETER SEARCH

  • The search varied trust-region size ϵ and rollout d for TRPO and Trust-PCL, and tested constant or decaying exploration τ for Trust-PCL.TRPO used ϵ ∈ {0.01, 0.02, 0.05, 0.1}; Trust-PCL used ϵ ∈ {0.001, 0.002, 0.005, 0.01}; both tested d ∈ {10, 50}.Source: §B.3 · p. 13

C PSEUDOCODE

  • The pseudocode takes environment, trust-region, learning-rate, discount, rollout, batch, collection, replay-buffer, and training-step settings as inputs.Source: C PSEUDOCODE · pp. 13–14
  • It initializes policy, value, and auxiliary parameters, collects P-step trajectories under the current policy, and inserts them into the replay buffer.Source: C PSEUDOCODE · p. 14
  • The iteration updates policy θ as θ←θ−ηπΔθ and value φ as φ←φ−ηvΔφ, then updates lagged policy θ̃=αθ̃+(1−α)θ and λ in terms of ϵ.Source: C PSEUDOCODE · p. 14
Loading 1707.01891v3…