Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Adam Stooke; Joshua Achiam; Pieter Abbeel

Source-linked AI summary

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Adam Stooke, Joshua Achiam, Pieter Abbeel

arXiv:2007.03964v1Jul 8, 2020math.OCcs.AIcs.LG

TL;DR

Safe reinforcement learning needs to control constraint violations during training, since standard Lagrangian updates can oscillate and overshoot. This paper adds proportional and derivative terms to the multiplier update, applies the resulting PID methods in deep RL, and reports improved cost performance, robustness, and Safety-Gym performance while preserving simplicity.

Problem
Standard Lagrangian methods for safe reinforcement learning can exhibit oscillations and cost overshoot, causing intermediate constraint violations during training.
Method
The paper interprets constrained reinforcement learning as a dynamical system and adds proportional and derivative constraint terms to the traditional integral multiplier update.
Results
The PID Lagrangian methods improve cost performance and hyperparameter robustness in deep reinforcement learning and achieve state-of-the-art performance in Safety-Gym.
Takeaways & Limitations
The resulting constrained-RL algorithms address the baseline's primary shortcoming while preserving simplicity and making controller use easier.
Takeaways & Limitations
The paper leaves more rigorous analysis of the modified method and theoretical selection of its coefficients for future work.

Abstract

from arXiv · show

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as \emph{integral} control; our terms introduce \emph{proportional} and \emph{derivative} control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

1. Introduction

Safe reinforcement learning must limit hazardous outcomes during exploration, but standard Lagrangian updates can oscillate and overshoot constraints. The paper introduces PID multiplier updates to make safety responses more responsive while retaining simple implementation and improving deep-RL performance.

Motivation: Safe reinforcement learning must satisfy cost constraints during exploration, not only after convergence, because hazardous outcomes can occur while the agent learns.The constrained Markov decision process framework represents hazards through accumulated cost signals and seeks reward maximization subject to cost limits.
Problem: Standard Lagrangian methods use a dual multiplier as a learned penalty coefficient, but intermediate training iterates often violate constraints through overshoot and oscillations.The multiplier update behaves as integral control, with a 90-degree phase shift between constraint and multiplier values.
PID Lagrangian method: PID Lagrangian methods add proportional and derivative terms based on constraint derivatives, damping oscillations and enabling anticipation of violations.The proportional term hastens response, while derivative control can prevent cost overshoot and limit cost increases within the feasible region.
Evaluation: The paper applies PID control to deep RL by adapting PPO and reports state-of-the-art performance in the OpenAI Safety-Gym environments.The contribution description identifies the Safety-Gym evaluation as a state-of-the-art result.
Evaluation: The proposed algorithms improve cost performance and hyperparameter robustness while remaining intuitive and simple to implement.The authors present these properties as outcomes of their deep-RL experiments.

2. Related Work

Related work establishes Lagrangian methods as simple constrained-RL baselines, while prior alternatives often add computational complexity or use related but narrower control interpretations. The paper extends this line by introducing derivative-based multiplier updates for safe RL.

Constrained Deep RL: Constrained-RL convergence results can require updating the multiplier more slowly than policy parameters, allowing many constraint-violating policy iterations before penalties take effect.This motivates methods that respond more directly during training.
Constrained Deep RL: Constrained Policy Optimization provides near-constraint satisfaction guarantees at every iteration but requires a projection step with a time-consuming backtracking line search.Recent empirical comparisons nevertheless found simple Lagrangian-based algorithms performed as well or better.
Dynamical Systems View of Optimization: Prior dynamical-systems work related gradient descent to integral control and connected momentum methods with proportional-integral control or PID interpretations.These viewpoints concerned optimization dynamics rather than the learned Lagrange multiplier in constrained RL.
Constrained Optimization: Classical constrained-optimization treatments generally update multipliers using constraint values, whereas this paper introduces derivatives of the constraint function.Related work includes quadratic penalty methods, proportional-control rules, and an intricate adaptive weighting scheme for continuous-time objectives.

3. Preliminaries

The paper formulates safe reinforcement learning as a constrained Markov decision process whose policy maximizes discounted reward subject to discounted cost limits. Deep policy-gradient learning then seeks high performance while ideally satisfying constraints at every iterate, and control theory supplies a feedback formulation.

Constrained Reinforcement Learning: A constrained Markov decision process extends an MDP with cost functions and corresponding limits, using a single encompassing cost here for notational simplicity.The CMDP tuple includes states, actions, rewards, transitions, an initial distribution, cost functions, and cost limits.
Constrained Reinforcement Learning: The constrained RL objective is to find the best feasible policy by maximizing expected discounted rewards subject to expected discounted cost constraints.Reward and cost are defined as expectations over trajectories induced by the policy.
Deep Reinforcement Learning: Deep RL parameterizes the policy with a neural network and iteratively estimates the reward-gradient update from task experience.The practical goal is high score at an iterate while ideally obeying constraints at each iteration.
Dynamical Systems: A discrete-time feedback-control system consists of state dynamics, measured outputs, an applied control, and a feedback rule using past and present measurements.The control problem is to design the feedback rule so the resulting trajectory performs well under a cost function.
Dynamical Systems: Control-affine systems are especially amenable to analysis because their input dependence is simpler, even when state dependence is nonlinear or uncertain.The paper seeks this form to support control and future analysis.

4. Modified Lagrangian Methods for Constrained Optimization

The paper extends Lagrangian multiplier updates with proportional and derivative terms, interpreting constrained optimization as a dynamical system to improve damping and responsiveness. The resulting PID combination preserves convergence conditions while addressing oscillatory learning dynamics and penalty-method complications.

Convergence and dynamics: Under a positive-definite damping matrix, the analyzed system converges to a constraint-satisfying solution, although its oscillation frequency and settling time depend on α.The paper notes that oscillations occur as the system converges into the feasible region.
Modified multiplier dynamics: The traditional multiplier integrates the constraint, while proportional and derivative terms add damping and predictive responses to constraint violations.The proportional term modifies damping through the constraint derivative; derivative control uses the second derivative and curvature to act predictively.
Proportional-integral method: Adding the proportional term contributes a positive-semidefinite outer product to the damping matrix, which can increase damping eigenvalues and boost convergence.The stated convergence conditions remain unaffected because the solution still satisfies ẋ = 0, g(x) = 0, and ġ(x) = 0.
Comparison with penalty methods: Compared with the quadratic penalty method, the proposed approach avoids modifying the Lagrangian derivative and avoids an additional potentially non-positive-semidefinite Hessian term.Both approaches are described as producing similar effects on the damping matrix.
Integral-derivative method: The derivative method weakens and rotates damping and forcing terms while adding a curvature-dependent drag that can decrease the constraint along the direction of travel.The paper describes this predictive behavior as potentially prone to instability.
PID method: The PID multiplier method combines the proportional and derivative developments, producing independent changes in the learning dynamics.The paper carries these dynamics intuitions forward to practical enhancements for Lagrangian-based constrained reinforcement learning.

5. Feedback Control for Constrained RL

The paper recasts constrained RL as a dynamical system in which the adaptive penalty coefficient is feedback control and the cost threshold is a setpoint. It then builds constraint-controlled RL around sampled cost feedback, objective rescaling, and PID multiplier updates.

Dynamical-system formulation: Constrained RL is modeled as a first-order dynamical system with the policy update as an unknown nonlinear function and the multiplier update as feedback control.The cost objective is the system measure, while the cost limit supplies the setpoint information to the controller.
Constraint-controlled RL: The controller receives sampled estimates of the cost criterion and updates a nonnegative Lagrange multiplier before the policy and critics are updated.Algorithm 1 retains a typical minibatch-RL loop and uses separate value and cost-value approximators because the multiplier may change rapidly.
Constraint-controlled RL: Objective rescaling maintains a consistent policy-parameter step size when a large multiplier would otherwise destabilize learning.The rescaled objective is a convex combination of reward and cost objectives and is also used for traditional Lagrangian baselines.
Alternative parameterization: Direct control over a reparameterized variable u ∈ [0, 1] is left for future work because its behavior under inequality-constraint nonlinearity may differ.The alternative parameterization weights reward and cost gradients directly and may provide superior performance in some cases.
PID Lagrangian method: PID control replaces integral-only multiplier updates with proportional and derivative terms based on constraint violations and learning-iteration cost changes.The derivative term is projected to oppose cost increases without impeding decreases, while the integral term remains for eliminating steady-state violations.
PID Lagrangian method: Proportional control accelerates response and dampens oscillations, while derivative control can anticipate violations, prevent cost overshoot, and limit feasible-region cost increases.Setting KP = KD = 0 recovers the traditional Lagrangian method.

6. PID Control Experiments

Safety Gym experiments show that PI and derivative control reduce the oscillations, overshoot, and slow settling observed with the baseline Lagrangian method while preserving reward performance and improving hyperparameter robustness.

6.3.1. Robust Safety with PI Control: PI-control eliminated cost oscillations in a challenging DOGGOBUTTON1 environment while maintaining good reward performance across cost limits.The comparison used individual runs at different cost limits.
6.3.1. Robust Safety with PI Control: Increasing KI reduced the period and amplitude of integral-control oscillations but also diminished returns, whereas PI-control stabilized costs while retaining high returns over a broad KI range.With PI-control, most oscillations were reduced to the noise floor for KI > 10^-4, while returns remained relatively high for KI < 10^-1.
6.3.1. Robust Safety with PI Control: PI-control expanded the return-versus-cost FOM Pareto frontier into a region of high rewards at relatively low cost inaccessible to the Lagrangian method.Each frontier point represented a different KI setting averaged over four runs.
6.3.1. Robust Safety with PI Control: Across four Safety Gym environments, PI-control improved cost FOM for KI < 10^-1 and significantly increased robustness to KI selection.Lower cost FOM indicates better performance in this comparison.
6.3.2. Control Efficiency: At high KI, rapidly fluctuating control reduced long-term returns, while PI-control remained relatively steady and allowed reward learning at each iteration.The high-KI setting used KI = 10^-1, whereas the PI comparison used moderate KI = 10^-3.
6.3.2. Control Efficiency: Derivative cost control removed overshoot from I- and PI-controlled baselines and slowed the cost curve’s approach toward the constraint limit.The result was demonstrated in a noisy deep RL setting.

7. Reward-Scale Invariance

Reward scaling changes the numerical scale required by Lagrange controller parameters even when the optimal policy is unchanged. Objective-weighting addresses this sensitivity by balancing reward and cost gradient contributions, producing nearly identical learning dynamics across reward scales.

7. Reward-Scale Invariance: Scaling rewards by ρ leaves the optimal policy unchanged but requires controller settings, including λ0, KI, KP, and KD, to scale by ρ.Changes in reward and cost magnitudes can also occur during learning within one CMDP.
7. Reward-Scale Invariance: Objective-weighting introduces βk into the policy gradient to maintain a fixed interpretation of λ as the relative influence of reward and cost.The method targets robustness within and across environments with different objective scales.
7. Reward-Scale Invariance: Choosing β∇ as the ratio of unscaled policy gradients balances reward and cost contributions at λ = 1 and encourages λ∗ = 1.β∇ is computed using existing algorithm components.
7. Reward-Scale Invariance: With objective-weighting, learning dynamics were nearly identical for reward scales ρ ∈ {0.1, 1, 10}, unlike the substantially different dynamics without scaling.The representative experiment used POINTGOAL1 with PI-control, KI = 1e-3, and KP = 0.1.

8. Conclusion

The paper introduces constrained RL solutions based on modified Lagrangian methods and shows they are straightforward to implement and effective with deep learning. The approach improves a common constrained RL baseline while preserving simplicity and easing use.

The paper presents constrained RL solutions derived from modified Lagrangian methods.
The proposed methods are straightforward to understand and implement when paired with deep learning.
The contributions improve a widely used constrained RL workhorse baseline while preserving its simplicity.
The methods are intended to make the baseline easier to use across a wide range of applications.

Supplementary Materials for: Responsive Safety in Reinforcement Learning by

This supplementary section derives the effects of proportional and derivative terms by starting from continuous Lagrangian dynamics and differentiating the resulting equations. The derivation expresses the dynamics in vector form and identifies the Hessian contribution.

The derivation starts from continuous Lagrangian dynamics for an objective f and constraint g.
Differentiating the dynamics for x_i and substituting for the multiplier derivative produces higher-order equations.
The derivation defines a damping matrix to characterize the resulting dynamics.
The resulting oscillator equation is written first for the i-th component and then in vector form.
The vector formulation explicitly includes ∇2f, the Hessian of f.

A.2. Proportional-Integral Method

This section derives proportional-integral and derivative multiplier dynamics by modifying the differential equation for λ and substituting it into the system equations. The derivative term adds damping-related and acceleration-coupling effects, while the full PID method combines proportional and derivative effects.

A.2. Proportional-Integral Method: The multiplier differential equation is amended to introduce the proportional-integral update.
A.2. Proportional-Integral Method: Substitution into the second derivative of x yields modified system dynamics.
A.2. Proportional-Integral Method: The modified dynamics differ from the traditional method through addition of a positive-semidefinite term to the damping matrix.
A.2. Proportional-Integral Method: Derivative control introduces second-order velocity terms and additional mixing terms in the acceleration.
A.2. Proportional-Integral Method: Acceleration terms are coupled by a positive definite matrix involving the identity and the outer product of ∇g.
A.2. Proportional-Integral Method: The full PID-Lagrangian dynamics combine the independent effects of proportional and derivative methods.

B. Training Details

The supplementary materials describe the training setup, experimental figures, alternative objective balancing, and additional learning curves. They report that unconstrained PPO violates cost constraints, while gradient-based balancing makes dynamics roughly invariant across reward scales in the shown cases.

B. Training Details: Experiments used randomly initialized agents with shared policy, reward-value, and cost-value estimator parameters.
B. Training Details: Generalized Advantage Estimation was used for both reward and cost advantages, with control updates once per training iteration.
B. Training Details: The supplementary figures examine cost and return behavior across environments while varying Lagrange multiplier learning rate and proportional control coefficient.
B. Training Details: Derivative control slows cost increases and reduces overshoot after the cost limit changes in POINTGOAL1 and CARBUTTON1.
B. Training Details: Unconstrained PPO does not satisfy the cost constraints and achieves higher rewards as a result.
B. Training Details: The alternative KL-based estimator worked but performed less well than the gradient-norm estimator in the reported cases.
B. Training Details: Gradient-based objective balancing produced roughly similar learning dynamics across reward scales for fixed controller settings.