PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions

Jason Choi, Fernando Castañeda, Claire J. Tomlin, Koushil Sreenath

arXiv:2004.07584v2eess.SYcs.LGcs.RO

TL;DR

Model uncertainty makes safety-critical CLF- and CBF-based control difficult when nominal dynamics do not match the plant. The paper learns uncertainty terms with reinforcement learning and incorporates them into an online CBF-CLF-QP, achieving safe bipedal walking on randomly spaced stepping stones under model uncertainty.

  • Problem

    CLF- and CBF-based controllers rely on accurate system models, while existing adaptive or robust approaches may not account for correct uncertainty magnitudes.

  • Method

    An RL agent learns uncertainty in CLF, CBF, and other dynamic constraints, which a quadratic program uses to compute safety-critical control online.

  • Results

    The framework demonstrates stable walking on randomly spaced stepping stones with one-step preview under high model uncertainty.

  • Takeaways & Limitations

    RL-CBF-CLF-QP can compensate trained model uncertainty and shows some robustness to additional uncertainty during evaluation.

  • Takeaways & Limitations

    The nominal controller must avoid rapid failure before reinforcement learning can learn the uncertainty, and this may not hold at high uncertainty levels.

Abstract

from arXiv · show

In this paper, the issue of model uncertainty in safety-critical control is addressed with a data-driven approach. For this purpose, we utilize the structure of an input-ouput linearization controller based on a nominal model along with a Control Barrier Function and Control Lyapunov Function based Quadratic Program (CBF-CLF-QP). Specifically, we propose a novel reinforcement learning framework which learns the model uncertainty present in the CBF and CLF constraints, as well as other control-affine dynamic constraints in the quadratic program. The trained policy is combined with the nominal model-based CBF-CLF-QP, resulting in the Reinforcement Learning-based CBF-CLF-QP (RL-CBF-CLF-QP), which addresses the problem of model uncertainty in the safety constraints. The performance of the proposed method is validated by testing it on an underactuated nonlinear bipedal robot walking on randomly spaced stepping stones with one step preview, obtaining stable and safe walking under model uncertainty.

I. INTRODUCTION

The paper combines model-based safety-critical control with reinforcement learning to address uncertainty in CLF, CBF, and other control constraints. It proposes RL-CBF-CLF-QP and validates the framework on an uncertain bipedal robot.

  • I. INTRODUCTION: Model uncertainty limits CLF- and CBF-based methods, while existing adaptive and robust approaches may misestimate uncertainty magnitudes.The paper motivates a data-driven alternative that preserves formal safety and stability structures.
  • I. INTRODUCTION: RL-CBF-CLF-QP uses an RL agent to learn uncertainty in multiple dynamic constraints and a quadratic program to compute safety-critical control online.The learned uncertainty terms augment constraints derived from the nominal model.
  • I. INTRODUCTION: The framework learns CLF, CBF, and other control-affine constraint uncertainties in a single learning process.The stated contributions also include high relative-degree outputs and parameterized CBFs.
  • I. INTRODUCTION: The method is numerically validated on an underactuated nonlinear hybrid bipedal system walking across stepping stones under significant model uncertainty.The experiment uses one-step preview.
  • I. INTRODUCTION: Input-output linearization transforms the control-affine nonlinear system into output dynamics driven by an auxiliary input, with transverse and zero-dynamics coordinates.The formulation assumes Lipschitz vector fields and matching numbers of inputs and outputs.

B. Control Lyapunov Function Based Quadratic Programs

The CLF-based quadratic program encodes exponential stabilization of transverse dynamics as an online optimization constraint. A quadratic CLF is constructed from the linearized closed-loop dynamics and a Lyapunov equation.

  • B. Control Lyapunov Function Based Quadratic Programs: A rapidly exponentially stabilizing CLF provides a condition for exponential convergence of the transverse coordinates η.The RES-CLF is defined as a parameterized family Vε with positive constants satisfying the stated stability condition.
  • B. Control Lyapunov Function Based Quadratic Programs: The quadratic CLF candidate is Vε(η) = ηT Pεη, where Pε solves a Lyapunov equation for the closed-loop matrix A = F + GK.Q is any symmetric positive-definite matrix.
  • B. Control Lyapunov Function Based Quadratic Programs: At each time step, the CLF condition becomes a linear constraint on the auxiliary input µ within a quadratic program.The objective can minimize the norm of the control inputs.

C. Control Barrier Function and Control Lyapunov Function Based Quadratic Programs

The CBF-CLF-QP extends CLF-based optimization with safety constraints derived from an exponential control barrier function. Safety is prioritized over stability by relaxing the CLF constraint, while additional affine constraints can be included.

  • C. Control Barrier Function and Control Lyapunov Function Based Quadratic Programs: An exponential control barrier function B with relative degree rb admits a gain Kb that characterizes its safety condition.The ECBF dynamics are input-output linearized using a virtual input µb.
  • C. Control Barrier Function and Control Lyapunov Function Based Quadratic Programs: The virtual input µb is selected so the ECBF condition becomes a constraint in the CBF-CLF-QP.The physical input u is defined through the input-output linearization controller.
  • C. Control Barrier Function and Control Lyapunov Function Based Quadratic Programs: Safety is prioritized over stability in the QP by relaxing the CLF constraint.This permits the optimization to preserve the barrier condition when both objectives cannot be satisfied exactly.
  • C. Control Barrier Function and Control Lyapunov Function Based Quadratic Programs: The QP can incorporate additional control-affine constraints, including input saturation and contact-force constraints.

III. REINFORCEMENT LEARNING FOR CLF-QP BASED CONTROLLERS UNDER UNCERTAIN DYNAMICS

This section examines model–plant mismatch in CLF-QP control and uses reinforcement learning to estimate or compensate for uncertainty while retaining nominal input-output linearization structure.

  • The section analyzes how unknown plant vector fields affect CLF dynamics and other constraints when the controller uses an inaccurate model.
  • The analysis assumes Lipschitz-continuous nominal vector fields and matching model and plant vector relative degrees.
  • The nominal input-output linearization controller produces transverse dynamics that differ from the nominal model through uncertainty terms ∆1 and ∆2.When ∆1 = ∆2 = 0, the uncertain dynamics reduce to the nominal transverse dynamics.
  • The first approach adds a reinforcement-learning input intended to cancel uncertainty in the transverse dynamics and recover nominal CLF behavior.Exact cancellation would remove uncertain terms from the CLF dynamics because the CLF derivative then depends only on F and G.
  • The learned policy uses αθ and βθ to make the output derivative y(r) approach the virtual input µ, with reward based on their mismatch.After training, µ is obtained by solving the CLF-QP in real time, producing the IO-RL + CLF-QP approach.

B. Reinforcement Learning for CLF-QP Based Controllers: Second Approach

The second approach estimates uncertainty directly in the CLF derivative rather than correcting the transverse dynamics, then uses that estimate in the optimization problem.

  • The second approach directly analyzes how transverse-dynamics uncertainty enters the CLF dynamics instead of compensating it first.
  • This approach assumes that the CLF designed for the nominal transverse dynamics is also a CLF for the true plant dynamics.
  • The uncertain CLF derivative is decomposed into nominal Lie-derivative terms plus uncertainty contributions from ∆1 and ∆2.
  • Reinforcement learning estimates the uncertainty terms ∆v in the CLF derivative.
  • The learned policy αVθ is trained so that the estimated derivative b˙Vε,θ is as close as possible to the true derivative ˙Vε.The reward penalizes the absolute difference between the estimated and actual terms.
  • The CLF is applied to input-output linearized dynamics because this provides a systematic way to construct a CLF candidate.The approach is also applicable beyond input-output linearization to general nonlinear control-affine systems.

IV. REINFORCEMENT LEARNING FOR CBF-CLF-QP BASED CONTROLLERS UNDER UNCERTAIN DYNAMICS

The framework is extended from CLF-based control to safety-critical CBF-CLF-QPs by learning uncertainty in CBF dynamics and incorporating the estimates into a new quadratic program.

  • The CBF extension follows the CLF uncertainty-compensation strategy to construct a safety-critical CBF-CLF-QP under model mismatch.
  • The uncertain CBF derivative contains mismatch terms ∆b1 and ∆b2, whose analytic forms can be derived from the model–plant discrepancy.
  • For high-relative-degree CBFs in robotic systems with x = [q, ˙q]T, model uncertainty affects only the rth time derivative of B.B^(r) is the only term depending on the plant dynamics through f and g.
  • The learned CBF terms are combined through a new virtual CBF input and corresponding QP constraints.
  • Parameterized CBFs incorporate their parameters ψ as additional neural-network inputs when the barrier depends on more than the system state.

B. Reinforcement Learning for Additional Control-Affine Dynamic Constraints

The method also learns uncertainty in additional control-affine dynamic constraints and incorporates multiple learned CBF and constraint estimates into a final optimization problem.

  • Additional linear constraints depending on the system dynamics are represented as affine functions of the control input.
  • Model mismatch introduces uncertainty terms ∆c1 and ∆c2 into the actual value of each dynamic constraint.
  • Reinforcement learning constructs an estimator ζ̂θ that is trained to approach the true constraint value ζ.
  • The final optimization problem can include learned estimates for multiple CBFs and multiple control-affine dynamic constraints.
  • The learned CBF constraints take the form µb,i + Kb,iηb,i ≥ 0 for i = 1 · · · nb.
  • The resulting learned affine constraints are imposed as Ac j,θ(x)µ + bc j,θ(x) ≤ 0 for j = 1 · · · nc.

V. REINFORCEMENT LEARNING-BASED FRAMEWORK

The framework uses reinforcement learning to estimate uncertainty in CLF, CBF, and other dynamic constraints, then incorporates those estimates into a nominal quadratic program. Training minimizes estimation errors using numerical derivatives and exploration, while evaluation can use a plant differing from the training environment.

  • The RL agent learns uncertainty terms for CLF, CBF, and other dynamic constraints in a unified process.These learned terms are combined into a policy representing uncertainty corrections across the constraints.
  • The learned uncertainty terms are added to nominal-model QP constraints to estimate the true plant constraints before solving for control.The resulting RL-CBF-CLF-QP problem is solved point-wise in time.
  • The reward minimizes each estimation error through weighted negative loss functions and a user-specific failure-case penalty.
  • Numerical differentiation computes the needed time derivatives without using true plant dynamics information in the loss values.The method uses explicit expressions for Vε, B, and ζ; the CBF derivatives require only one numerical differentiation regardless of rb.
  • A discretized learning problem can be solved with conventional reinforcement learning algorithms, using white noise to encourage exploration.The noise is sampled from a zero-mean Gaussian distribution with variance σ2.
  • Training assumes the robot operates under true plant dynamics, while later evaluation tests whether the trained policy works when the plant differs from training.

VI. APPLICATION TO BIPEDAL ROBOTS

The framework is validated on RABBIT, a planar five-link underactuated bipedal robot walking across discrete stepping stones with one-step preview.

  • The method is tested on RABBIT, a planar five-link bipedal robot navigating a discrete stepping-stone terrain.The task uses one-step preview of the terrain.

A. Simulation Settings

The simulations evaluate RL-based CLF and CBF-CLF-QP controllers on RABBIT under scaled model parameters, added torso weight, and randomly spaced stepping stones. Results assess stability, foot-placement safety, and contact-force constraints against prior controllers and nominal model-based CBF-CLF-QP.

  • Simulation scenarios: RABBIT is tested on flat terrain and randomly spaced stepping stones with one-step preview, where the robot must maintain stability and satisfy contact-force constraints.The stepping-stone task requires placing the swing foot on the next stone while managing stability and contact forces.
  • Uncertainty and evaluation: The primary model uncertainty scales every link’s mass and inertia by 2, making the nominal model’s parameters half those of the actual plant.Evaluation also adds motor torque-rate dynamics and, separately, torso-weight uncertainty.
  • Controller and training setup: The RL agent is trained with DDPG using RABBIT’s full state and the minimum step-length parameter, while two CBFs and two dynamic constraints encode foot placement and contact forces.Training uses ten-step episodes, a 0.01-second timestep, and a 200Nm torque saturation.
  • Flat-ground evaluation: Under mass-inertia scaling by 2, IO-RL + CLF-QP and RL-CLF-QP maintain stable walking, whereas L1-CLF-QP fails; only RL-CLF-QP satisfies |F_T /F_N| ≤ k_f = 0.8.IO-RL + CLF-QP exceeds the friction limit during the first two steps, while L1-CLF-QP violates it for multiple steps.
  • Additional torso-weight evaluation: With an added 32kg torso weight, both RL-CLF-QP and IO-RL + CLF-QP adapt despite not encountering this uncertainty during training.RL-CLF-QP stabilizes walking up to 72kg additional weight, while IO-RL + CLF-QP adapts up to 53kg.
  • Stepping-stone evaluation: RL-CBF-CLF-QP keeps foot placement on 20 randomly spaced stepping stones and maintains the friction limit, while nominal CBF-CLF-QP violates step-length safety after 5.6±4.64 steps on average.The comparison uses ten random executions of 20-step simulations; RL-CBF-CLF-QP also remains safe under 43–72kg added torso weight.

VIII. DISCUSSION

The framework compensates for trained model uncertainty and shows some robustness to additional uncertainty, while its success depends on the nominal controller remaining viable during learning. The discussion also identifies noisy numerical differentiation and non-guaranteed QP feasibility as practical limitations.

  • The method compensates well for trained model uncertainty and shows some robustness to additional uncertainty during evaluation.
  • Successful results extend beyond mass and inertia scaling to joint friction, joint damping, and link bending uncertainties.The tested cases include a 15% reduction in motor power from joint friction, damping up to 1 (rad/s2)/(rad/s), and link bending up to 5% of link length.
  • The approach requires the nominal controller not to fail rapidly before RL learns the uncertainty, a condition that may not hold at high uncertainty levels.
  • Numerical differentiation of ˙Vε and B(rb) may produce noisy measurements in experiments.
  • RL is used to gradually improve quadratic-program feasibility and performance while learning multiple safety-constraint uncertainties in a unified process.
  • Feasibility of CBF-CLF-QPs with additional constraints such as friction is not guaranteed, although the trained RL-CBF-CLF-QP showed drastically improved feasibility over the nominal method.
Loading 2004.07584v2…