PaperlayerView exact revision on arXiv

Source-linked AI summary

Constrained Policy Optimization

Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel

arXiv:1705.10528v1cs.LG

TL;DR

Deep reinforcement learning lacks general policy-search methods that enforce safety constraints during learning, despite the need for constrained behavior in realistic settings. The paper introduces CPO, whose theoretically justified updates improve reward while approximately satisfying constraints, and demonstrates this on high-dimensional neural-network control tasks.

  • Problem

    Deep reinforcement learning policy search does not generally guarantee constraint satisfaction throughout training, although safety constraints matter in realistic applications.

  • Method

    CPO derives and approximately implements policy updates from a bound relating differences in policy rewards or costs to divergence between policies.

  • Results

    CPO trained neural-network policies with thousands of parameters on high-dimensional constrained robot-locomotion tasks while maximizing reward and approximately satisfying constraints.

  • Takeaways & Limitations

    CPO provides a principled step toward applying reinforcement learning to real-world settings where behavioral constraints are necessary for safety.

  • Takeaways & Limitations

    Approximation errors can produce infeasible iterates, requiring a recovery method; the reported recovery experiments use only one constraint.

Abstract

from arXiv · show

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

1. Introduction

Deep reinforcement learning performs well in high-dimensional control, but realistic applications require safety constraints during exploration. CPO addresses this gap with theoretically justified updates and demonstrates constrained neural-network control.

  • High-dimensional deep reinforcement learning has achieved state-of-the-art performance across games, locomotion, manipulation, and Go.
  • Safety constraints are important when exploration could damage equipment or harm nearby people.
  • Existing policy-search methods lacked guarantees that every policy during continuous-CMDP training would satisfy constraints.
  • CPO derives a policy-improvement step from a bound on reward or cost differences between policies.
  • CPO approximates the theoretically justified update while training neural-network policies with thousands of parameters on constrained locomotion tasks.

2. Related Work

Prior safe policy-search methods used heuristics, restrictive assumptions, or expensive optimization. CPO is presented as the first policy-search algorithm for CMDPs that guarantees constraint satisfaction throughout training across arbitrary policy classes.

  • Earlier methods used gradient projection but could not prevent a policy from becoming unsafe initially.
  • A theoretically motivated alternative required an expensive semidefinite-program inner loop and restrictive safety assumptions.
  • Other deep-RL approaches relied on restrictive robotic assumptions, intrinsic fear heuristics, or separate trajectory planning.
  • CPO guarantees constraint satisfaction throughout training and works with arbitrary policy classes, including neural networks.

3. Preliminaries

The preliminaries define MDPs, policies, returns, value functions, and state distributions used to analyze policy performance. These quantities support expressing differences between policies through discounted state visitation and advantage terms.

  • An MDP specifies states, actions, rewards, transition probabilities, and a starting-state distribution.
  • A stationary policy maps each state to a probability distribution over actions.
  • RL typically selects a policy that maximizes infinite-horizon discounted total return.
  • The value, action-value, and advantage functions quantify expected returns from states, state-action pairs, and their difference.
  • The discounted future state distribution enables compact expressions for performance differences between two policies.

4. Constrained Markov Decision Processes

A CMDP augments an MDP with auxiliary cost functions and limits, restricting the allowable policies. The constrained RL objective optimizes over feasible policies, with stationary-policy optimization justified under mild technical conditions.

  • A CMDP adds auxiliary cost functions and limits to an MDP, thereby restricting the set of allowable policies.
  • Each constraint return is the expected discounted return under its corresponding auxiliary cost function.
  • The CMDP reinforcement-learning problem optimizes reward while requiring every selected policy to remain feasible.
  • Optimizing only over stationary policies is justified because optimal CMDP policies include stationary policies under mild technical conditions.
  • CMDP theory is supported by established treatments of constrained decision processes.

5. Constrained Policy Optimization

CPO formulates constrained policy search as a trust-region update supported by bounds relating policy-return differences to average divergence. The resulting method addresses off-policy evaluation and provides guarantees for monotonic performance and approximate constraint satisfaction.

  • Local policy search for CMDPs restricts optimization to parametrized policies that satisfy the CMDP constraints.
  • Off-policy evaluation makes direct feasibility checks difficult when policy updates rely on samples from high-dimensional control tasks.
  • Policy Performance Bounds: Theoretical bounds relate returns or constraint returns between arbitrary policies to average divergence, yielding tight bounds when the policies coincide.
  • Trust Region Methods: Trust-region updates provide a principled approximation with a worst-case performance-degradation guarantee that depends on the step size δ.
  • Trust Region Optimization for Constrained MDPs: The surrogate update uses samples from the current policy for both objectives and constraints, producing monotonically nondecreasing returns while satisfying original constraints under the stated conditions.
  • Trust Region Optimization for Constrained MDPs: CPO uses trust regions to enable larger steps while inheriting performance guarantees and bounds on worst-case constraint violation.

6. Practical Implementation

The practical implementation approximates the theoretically justified CPO update for high-dimensional policies. It combines local approximations, efficient dual optimization, line search, recovery steps, and tightened cost constraints to address computational and approximation errors.

  • CPO replaces the exact update with an efficiently computable approximation for policies having thousands of parameters.
  • Linearized objectives and constraints plus a second-order KL approximation make the update tractable for small step sizes δ.
  • The convex local problem uses the Fisher information matrix and can be reduced to a lower-dimensional dual when the number of constraints is small.
  • The algorithm samples trajectories, estimates gradients, constraint gradients, curvature, and costs, then applies backtracking line search to enforce sample-based constraint satisfaction.
  • Conjugate gradients approximately compute inverse-Fisher products because directly inverting the Fisher information matrix is impractical for high-dimensional policies.
  • Feasibility: Approximation errors can produce infeasible iterates, so the single-constraint implementation uses a recovery update that purely decreases the constraint value.
  • CPO constrains upper bounds on original costs through cost shaping, using a nonnegative correction term Δ correlated with the cost.

7. Connections to Prior Work

CPO differs from primal-dual approaches in how it determines constraint multipliers. It recomputes dual variables at every update to enforce constraints on intermediary policies rather than only at convergence.

  • Primal-dual methods learn stateful dual variables concurrently with the policy variables.
  • CPO recomputes dual variables from scratch at each update, unlike primal-dual methods whose intermediary policies are not guaranteed to satisfy constraints.

8. Experiments

The experiments evaluate CPO on safety-motivated constrained control tasks and compare it with primal-dual, unconstrained, cost-shaping, and fixed-penalty approaches. Across environments, CPO approximately enforces constraints while preserving reward performance.

  • Experimental setup: CPO is evaluated on Circle and Gather tasks using point-mass, quadruped, and humanoid agents with neural-network policies.The experiments train all agent-task combinations except Humanoid-Gather, using two hidden layers of sizes 64 and 32.
  • CPO versus PDO and TRPO: CPO tracks the constraint return almost exactly to the limit in the Point-Circle and Point-Gather environments.The broader experiments find that CPO approximately enforces constraints in all environments.
  • CPO versus PDO and TRPO: CPO generally outperforms PDO on constraint enforcement without compromising return performance.PDO can violate constraints during training, including a spike in Ant-Circle, and its behavior depends on dual-variable initialization.
  • Cost shaping: Cost shaping generally improves adherence to the true C-return constraint, while CPO remains nearly constraint-satisfying without it.The experiments evaluate performance using C+ return because that is the quantity constrained by the algorithm, except in Point-Gather.
  • Fixed penalties: Fixed-penalty methods are highly sensitive to the penalty coefficient, whereas CPO automatically selects coefficients for the reward–constraint-cost trade-off.In Ant-Circle, changing the coefficient from 1 to 5 shifts behavior from massive constraint costs to no learned rewards.

9. Discussion

The discussion presents CPO as a principled approximation to policy updates that improve return while approximately satisfying constraints. The experiments demonstrate this capability for neural-network policies in high-dimensional constrained control tasks.

  • Discussion: CPO combines policy updates that are guaranteed to improve return and satisfy constraints with a practical approximation for CMDPs.The method is designed for neural-network policies and approximates the theoretically guaranteed algorithm using trust-region methods.
  • Discussion: CPO trains neural-network policies with thousands of parameters while simultaneously maximizing reward and approximately satisfying constraints.The demonstrated tasks are high-dimensional constrained control problems motivated by safety.

10. Appendix

The appendix develops identities and bounds for discounted visitation distributions and policy returns, culminating in a new policy improvement bound based on policy divergence. The bound supports updates that improve reward while satisfying constraints.

  • Distribution identities: The appendix defines discounted future state distributions and represents them using finite-state transition matrices.These identities support subsequent comparisons between policies.
  • Return identities: A general identity relates discounted returns, value-function estimates, and on-policy average TD-error.It also shows that reward shaping by γf(s′) − f(s) shifts return by a policy-independent constant.
  • Interpretation: Choosing f = Vπ′ recovers the exact equality J(π′) − J(π), but evaluating Vπ′ for every candidate is too expensive for practical optimization.The associated divergence penalty reflects mismatch between f and Vπ′.
  • Divergence bound: The new visitation-divergence result bounds ∥dπ′ − dπ∥1 by 2γ/(1 − γ) times expected total variational policy divergence.The expectation is taken over states distributed according to dπ.
  • Policy improvement theorem: The policy improvement theorem bounds return differences for arbitrary policies using a function f and the quantity δf(s, a, s′) = R(s, a, s′) + γf(s′) − f(s).The proof combines the earlier return identity with the visitation-divergence bound.
  • Tightness: The resulting bounds are tight when π′ = π, because all three expressions become identically zero.

10.2. Proof of Analytical Solution to LQCLP

This appendix solves a linear objective with linear and quadratic constraints by exploiting convexity and strong duality. It analyzes trust-region feasibility and obtains candidate solutions through a piecewise dual problem.

  • Problem formulation: The optimization problem has vectors g, b, x, scalars c and δ, and a positive-definite matrix H defining the quadratic trust region.The analysis assumes δ > 0 and at least one strictly feasible point for the stated optimality result.
  • Dual coefficients: The coefficients q = gᵀH⁻¹g, r = gᵀH⁻¹b, and s = bᵀH⁻¹b parameterize the dual analysis.
  • Dual solution: Strong duality follows from Slater’s theorem when a strictly feasible point exists, enabling an analytical solution.
  • Dual cases: The sign-based sets Λa and Λb partition nonnegative λ according to whether λc − r is positive or nonpositive.Their interval forms depend on the sign of c.
  • Feasibility: The quantity c²/s − δ determines whether the linear-constraint plane intersects the quadratic trust region.If it is positive, the sign of c distinguishes a redundant constraint from an infeasible problem; otherwise the intersection exists.
  • Solution construction: The optimal dual solution is selected from the two piecewise branches and projected onto the relevant convex segment.

10.3. Experimental Parameters

The experiments use common neural-network policy settings across Circle, Point-Gather, and Ant-Gather environments, with task-specific rewards and safety costs. Constraint-advantage estimation and primal-dual implementation choices materially affect safe updates.

  • Gather environments: Point-Gather rewards collecting apples and charges cost for collecting bombs, while Ant-Gather adds a −10 penalty when the agent falls.Point-Gather uses two apples and eight bombs; Ant-Gather uses eight apples and eight bombs.
  • Circle: Circle rewards motion around a wide target circle while constraining the agent to a smaller safe region.The optimal constrained path follows line segments AD and BC because blue regions are forbidden.
  • Policy architecture: All experiments use Gaussian policies whose neural-network means have two tanh hidden layers of sizes 64 and 32.Policy variances are separate learnable parameters.
  • Advantage estimation: GAE-λ estimates both regular and constraint advantages, using neural-network value functions with separate λGAE settings.
  • Constraint estimation: Failing to discount constraint advantages substantially overestimates constraint-gradient magnitude and causes unsafe algorithmic steps.The reported search selected λGAE^C = 0.5, while 0.92 worked nearly as well.
  • Primal-dual comparison: The primal-dual implementation learns stateful constraint dual variables, whereas CPO solves those variables from scratch at each update.
  • Learning-rate caveat: Selecting the correct dual learning rate is challenging, and CPO removes the need for that choice.
Loading 1705.10528v1…