PaperlayerView exact revision on arXiv

Source-linked AI summary

Lyapunov-based Safe Policy Optimization for Continuous Control

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, Mohammad Ghavamzadeh

arXiv:1901.10031v2cs.LGcs.AIstat.ML

TL;DR

Safe reinforcement learning must optimize long-term performance without exposing agents or environments to unsafe behavior during training. The paper develops Lyapunov-based policy-gradient algorithms for continuous-action CMDPs, using policy or action projection with on- and off-policy methods. Across simulated control and real navigation tasks, the algorithms balance performance and constraint satisfaction, although function approximation, stochasticity, and conservative thresholds can leave constraints above target.

  • Problem

    Continuous-action reinforcement learning needs methods that maintain safe policies during training and deployment, because unrestricted exploration can harm the agent or environment.

  • Method

    The paper combines Lyapunov-based constraints with policy-gradient and actor-critic methods, projecting policy parameters or actions to support continuous actions and both on-policy and off-policy training.

  • Results

    The algorithms outperform CPO and Lagrangian baselines in balancing performance and constraint satisfaction during training across MuJoCo tasks and real-world robot navigation, with improved generalization to more complex environments.

  • Takeaways & Limitations

    Lyapunov-based safe policy optimization provides a practical framework for continuous control that supports PPO or DDPG, data reuse, and end-to-end policy training.

  • Takeaways & Limitations

    Function approximation and stochasticity caused premature convergence with constraints above threshold, while real-world constraint satisfaction generally requires more training data.

Abstract

from arXiv · show

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction. Videos of the experiments can be found in the following link: https://drive.google.com/file/d/1pzuzFqWIE710bE2U6DmS59AfRzqK2Kek/view?usp=sharing.

1. Introduction

Safe reinforcement learning is needed when unrestricted exploration could harm the agent or environment. The paper extends Lyapunov-based constrained policy optimization to continuous actions while supporting safe training and convergence.

  • Unrestricted exploration can damage robots, environments, or nearby people, making safety necessary during both training and deployment.
  • CMDPs encode safety by constraining expected cumulative auxiliary costs alongside the standard reward or cost objective.
  • Existing Lyapunov-based methods provide safety guarantees but are not straightforward to apply to continuous-action problems because they are value-function-based.
  • The paper extends Lyapunov-based CMDP methods to large or infinite action spaces using policy-gradient and actor-critic algorithms, addressing continuous-action policy updates and state-dependent integrated constraints.
  • The proposed methods work with on-policy and off-policy policy gradients, support end-to-end training, and outperform baselines in balancing performance and constraint satisfaction across simulated and real-world tasks.

2. Preliminaries

The preliminaries formulate safety-constrained continuous control as a CMDP and review policy-gradient, Lagrangian, and Lyapunov-based optimization. Lyapunov constraints guarantee feasibility but become difficult to enforce directly in large or continuous state and action spaces.

  • A CMDP augments an MDP with state-dependent constraint costs d(x) and an upper bound d0 on expected cumulative constraint cost.
  • Policy-gradient methods update parameterized policies using sample estimates of gradients of expected cumulative costs; DDPG is off-policy, whereas PPO is a trust-region-related method.
  • The Lagrangian method jointly optimizes policy parameters and a nonnegative multiplier after adding constraint costs to task costs, but its current multiplier can yield unsafe policies during training.
  • Lyapunov functions define state-wise constraints whose induced policies are feasible, while a suitable function can preserve an optimal policy within the feasible set.
  • Directly solving the Lyapunov policy-improvement problem is difficult because it has one action-integrating constraint per state, becoming numerically intractable for large or continuous spaces.

3. Safe Lyapunov-based Policy Gradient

The paper develops Lyapunov-based safe policy-gradient methods for continuous-action CMDPs, combining standard policy-gradient algorithms with policy-parameter or action projection. These methods approximate state-dependent Lyapunov constraints so safety can be maintained during training and at convergence.

  • Framework: The algorithms combine a baseline policy-gradient method, such as DDPG or PPO, with a method for solving the Lyapunov-based policy-optimization problem.The framework is designed for continuous or large action spaces and supports both on-policy and off-policy policy-gradient algorithms.
  • θ-projection: θ-projection updates the policy parameter by projecting it onto the feasible set induced by linearized Lyapunov constraints.The approach uses first-order and second-order Taylor approximations together with conservative policy-gradient bounds.
  • θ-projection: The θ-projection constraint becomes equivalent to the constraint used in CPO, establishing a connection between the methods apart from CPO’s line search.The DDPG and PPO versions are called SDDPG and SPPO, respectively.
  • a-projection: The a-projection approach embeds Lyapunov constraints in a safety layer that projects each unconstrained action onto a feasible action set.This reformulates the CMDP as unconstrained policy optimization over an augmented policy network and can be extended to stochastic policies.
  • a-projection: The action projection is a convex quadratic program based on a first-order action approximation and can be solved with an in-graph QP solver.Its closed-form solution linearly projects the unconstrained action onto a Lyapunov-safe hyperplane; multiple constraints are possible when at most one is active.
  • Empirical illustration: Figure 1 compares DDPG, DDPG-Lagrangian, SDDPG, and DDPG a-projection on HalfCheetah-Safe and Point-Gather across episodes.Shaded regions show 1-SD confidence intervals over 10 random seeds, and the dashed purple line marks the constraint limit.

4. Experiments on MuJoCo Benchmarks

The MuJoCo experiments evaluate safe policy-gradient methods on continuous-control tasks with physically motivated constraints against unconstrained, Lagrangian, and CPO-based baselines. Results show stable learning and rapid constraint stabilization, with a-projection often improving speed and constraint control.

  • Experimental setup: The experiments assess training performance, safety, and robustness to constraint violations caused by function-approximation errors across MuJoCo locomotion and navigation tasks.Tasks include HalfCheetah-Safe, Point-Circle, Point-Gather, and Ant-Gather.
  • Baselines: The comparison includes DDPG and PPO, a tuned Lagrangian method, and an on-policy CPO baseline implemented through its PPO alternative.The study evaluates both unconstrained and constrained policy-gradient methods.
  • Constraint satisfaction: Lyapunov-based methods quickly stabilize constraint cost below the threshold, whereas unconstrained DDPG and PPO violate constraints and Lagrangian learning oscillates around the threshold.All methods converge to feasible policies with reasonable performance, but their training-time constraint behavior differs substantially.
  • Projection comparison: In many experiments, a-projection converges faster and has lower constraint violation than its θ-projection counterpart.The authors associate this pattern with less conservative updates and smoother gradient updates during end-to-end training.
  • DDPG versus PPO: DDPG generally learns faster than PPO, while PPO provides tighter control of constraint violations and satisfies lower constraint thresholds.The authors suggest off-policy replay data may improve data efficiency, while covariate shift makes tight constraint control harder.

5. Safe Policy Gradient for Robot Navigation

The robot-navigation experiments compare Lyapunov-based policy-gradient methods with unconstrained and Lagrangian baselines across standard and more difficult environments. The Lyapunov methods achieve stronger safety-performance trade-offs, safer routes, and transfer to a real Fetch robot.

  • Robot navigation: Lyapunov-based policy-gradient methods achieve higher navigation success by controlling total collision energy while minimizing travel distance.Success is averaged over 100 evaluation episodes with random initialization.
  • Robot navigation: The Lagrangian method is less robust to parameter initialization, producing lower success rates and higher variability than Lyapunov-based methods.
  • Generalization: As evaluation goals move farther away in a larger environment, all methods lose success rate, but safety methods outperform DDPG and LA-DDPG on harder tasks.Training uses a 22 by 18 meter environment, while evaluation uses a 60 by 47 meter environment with goals up to 15 meters away.
  • Generalization: Lyapunov-based methods retain lower constraint costs as navigation becomes difficult and choose safer routes with fewer collisions than the Lagrangian method.
  • Real-robot transfer: The SL-DDPG policy completed 500 meters across five repetitions of 12 office-navigation tasks while avoiding static and dynamic obstacles.When no clear path was available, the robot stayed put rather than narrowly passing the obstacle.

6. Conclusions

The paper formulates safe reinforcement learning for continuous actions as a CMDP and develops Lyapunov-based policy optimization methods. Across simulated and real tasks, the methods support safe learning, data efficiency, differentiable integration, and real-world scalability.

  • Contributions: The authors develop θ-projection and a-projection algorithms that combine Lyapunov functions with on-policy or off-policy policy-gradient optimization.The algorithms use DDPG or PPO together with a critic that evaluates the policy and computes its Lyapunov function.
  • Evaluation: The algorithms are evaluated on four simulated robot-locomotion tasks and indoor robot navigation, including deployment on a real Fetch robot.
  • Conclusions: The reported contributions are safe learning, better data efficiency, natural integration into differentiable policy-gradient training, and scalability to real-world problems.
  • Future work: Future work includes further exploration of Lyapunov constraints in constrained policy optimization and extensions to model-based reinforcement learning.

A. The Lyapunov Approach to Solve CMDPs

The Lyapunov approach constructs policy-feasibility sets whose induced policies satisfy the CMDP constraint. Safe policy iteration updates a feasible policy within these sets while seeking policy improvement and, with a suitable Lyapunov function, preserving access to an optimum.

  • Lyapunov construction: A baseline feasible policy πB is used to define Lyapunov functions relative to the initial state x0 and constraint threshold d0.
  • Feasibility: Lyapunov-induced policies satisfy Dπ(x) ≤ L(x) for every state, and L(x0) ≤ d0 implies feasibility of the induced policy.
  • Optimality: Because an arbitrary Lyapunov-induced feasible set may exclude an optimal policy, the Lyapunov function must be designed to preserve optimality.
  • Optimality: Theorem 1 gives a Lyapunov-function construction whose induced feasible policy set contains an optimal policy under the stated conditions.
  • Safe policy iteration: Safe policy iteration recomputes the Lyapunov function, evaluates the current cost value function, and updates the policy by minimizing cost over the Lyapunov-feasible set.
  • Safe policy iteration: The iteration is designed to provide consistent feasibility and monotonic policy improvement.

B. Lagrangian Approach to Safe RL

The Lagrangian-based policy-gradient framework converts CMDP constraints into a saddle-point problem and supports trajectory-based and incremental actor-critic updates. Convergence relies on feasibility, differentiability, and multiple stochastic-approximation time scales.

  • Assumptions: The framework assumes continuously differentiable policies, a strictly feasible policy, and step-size schedules satisfying stochastic-approximation conditions.
  • Lagrangian formulation: The Lagrangian relaxation converts the constrained problem into minimax optimization over policy parameters θ and multiplier λ.The policy can be optimized with DDPG or PPO using augmented cost c(x,a) + λd(x), while λ is updated by stochastic gradient descent.
  • Policy gradient: The policy-gradient algorithm estimates gradients from N trajectories generated by the current policy and updates θ and λ after observing those trajectories.
  • Convergence: Projection of λ onto [0, λmax] and step-size schedules place updates on separate time scales to support convergence.The cited schedules assign fast, intermediate, and slow time scales to different updates.
  • Convergence: Under the stated assumptions, the two-time-scale policy-gradient procedure converges to a local saddle point of L(θ, λ).The analysis uses the corresponding ordinary differential equation in the sufficiently small step-size limit.
  • Actor-critic methods: Actor-critic variants reduce trajectory-gradient variance by using value-function approximation and updating parameters incrementally after each state-action transition.

C. Experimental Setup in MuJoCo Tasks

The MuJoCo evaluation covers four safety-augmented continuous-control domains, using simulated point-mass, ant, and HalfCheetah agents with both policy-gradient variants and systematically tuned settings.

  • The experiments use safety-augmented MuJoCo domains: PointGather, AntGather, PointCircle, and HalfCheetahSafe.
  • Task domains: HalfCheetahSafe constrains a speed-rewarded two-legged robot to speed below 1, with episodes of length 200 and a constraint threshold of 50.
  • Task domains: PointGather rewards collecting apples and penalizes collecting bombs, with episode length 15 and algorithm-dependent constraint thresholds of 4 for DDPG and 2 for PPO.
  • Task domains: AntGather uses an ant robot with apples and bombs, penalizes falls, allows up to 500 steps, and sets thresholds of 10 for DDPG and 5 for PPO.
  • Agents and implementation: The experiments include point-mass, ant, and HalfCheetah agents with action dimensions 2, 8, and 6, respectively, and use neural-network actor and critic models.
  • Agents and implementation: Each algorithm is evaluated across grid-searched learning rates, batch sizes, entropy-regularization settings, and natural-policy-gradient choices.

C.1. More Explanations on MuJoCo Results

The MuJoCo comparison tests θ- and action-projection safe methods with both DDPG and PPO against unconstrained and Lagrangian baselines, showing stronger constraint control overall.

  • The comparison includes θ-projection and action-projection safe algorithms, unconstrained and Lagrangian baselines, across PointGather, AntGather, PointCircle, and HalfCheetahSafe.
  • Results: In PointCircle DDPG, Lagrangian achieves higher return but violates the constraint more often than the safe algorithms.
  • Results: In the other reported domains and settings, Lagrangian either has lower performance than a safe algorithm or violates constraints during training, while safe algorithms do not.

D. Experimental Setup in Robot Navigation

The robot-navigation task evaluates collision-free point-to-point control from onboard observations, with a CMDP objective balancing short paths and collision-impact energy under limited-data deployment.

  • P2P requires navigating to arbitrary goal positions collision-free without memorizing workspace topology.
  • Task and observations: The robot observes relative goal position and velocity plus 64-bin noisy Lidar measurements over a 220° field of view and acts through linear and angular velocities.
  • CMDP formulation: The finite-horizon, non-discounting CMDP uses T = 100, minimizes relative goal distance, and limits average collision impact energy through a threshold d0.
  • CMDP formulation: Because intermediate policies are deployed to collect additional samples under limited data, safety during training is critical.
  • Environments: Training environments are generated from real office plans, while the evaluation environment is an order of magnitude bigger.
  • Environments: Figure 9 presents navigation routes for two policies on similar setups and logs from on-robot experiments.

E. Pseudo-code of the Safe Policy Gradient Algorithms

The pseudocode section introduces a Lagrangian trajectory-based policy-gradient procedure with initialized policy and multiplier parameters followed by iterative updates.

  • Algorithm 2 is a Lagrangian trajectory-based policy-gradient algorithm.
  • The procedure initializes policy parameters θ to θ0 and the Lagrangian parameter λ to λ0 before iterating over updates.

E.1. Practical Implementations of Safe PG

The practical implementations combine policy-gradient updates with Lyapunov-based feasibility mechanisms, supporting both parameter projection and state-wise action projection. They also include safeguards for violations caused by function-approximation errors.

  • Safeguards: Function-approximation errors can produce infeasible policy updates that the safe policy-gradient method cannot automatically undo.The implementation therefore adds a safeguard update that decreases constraint cost and can target the worst violated constraint.
  • Parameter projection: The θ-projection implementation initializes an initially feasible policy and iteratively updates its policy parameters under Lyapunov-based constraints.The supplied algorithm specification identifies this procedure as Lyapunov-based Policy Gradient with θ-projection.
  • Policy-gradient components: The implementations estimate learning quantities from trajectories, using Bellman-residual training and replay-buffer samples for DDPG or generalized advantage estimation for PPO.These choices allow the practical procedures to accommodate both off-policy and on-policy policy-gradient methods.
  • Policy-gradient components: The parameter-update procedures use an adaptive penalty weight for a DKL regularizer and then update policy parameters by following the objective gradient.The described workflow calculates λ* from a quadratic-program solution before applying the policy update.
  • Action projection: The action-projection implementation computes a feasible action probability for each state using the Lyapunov gradient and a state-dependent feasibility margin.The safety layer takes ∇aQL(x, a) and ϵ(x) as inputs before returning the projected action probability.
Loading 1901.10031v2…