Source-linked AI summary
Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control
Chieh Tsai, Muhammad Junayed Hasan Zahed, Jinzhi Shen, Yi Xie, Ruoshan Lan, Majed Obaid, Salim Hariri, Hossein Rastgoftar
TL;DR
Safety-critical adaptive control must improve performance without violating hard constraints, while common methods treat safety as an external correction. This paper embeds safety in a policy class by restricting learning to a finite library of controllers sharing a common Lyapunov certificate. In nominal tests, the scheduler reduced hover RMSE by 17.93% relative to the median/nominal fixed controller while all three policies met the deadline and remained safe.
Problem
Safety-critical reinforcement learning needs to improve adaptive-control performance without violating hard constraints, but common approaches handle safety through external penalties, shields, or filters.
Source: §I · p. 1Method
The method restricts actions to a finite library of feedback controllers with a common Lyapunov certificate, making the prescribed set forward invariant under arbitrary switching.
Source: §I · p. 1, §II · p. 3, A. Stability and Forward Invariance · p. 4Results
17.93% reduction in hover RMSE, from .013326 to .010937 m, relative to the median/nominal fixed controller; all three policies met the deadline in every nominal test rollout.
Source: B. Main comparison: safety,… · p. 6Takeaways & Limitations
The quadcopter case study demonstrates state-dependent DQN gain scheduling within a certified policy class, separating nominal safety certification from policy optimization.
Source: §VI · pp. 7–8The common-certificate guarantee is conservative, the price of remaining valid under arbitrary switching.
Source: A. Stability and Forward Invariance · p. 4
Abstract
from arXiv · showhide
This paper proposes a reinforcement learning (RL) framework for safe gain scheduling based on forwardinvariance-induced action-space design. Under stated nominalmodel and inversion-domain assumptions, rather than enforcing safety through runtime shielding or penalty-based constraints, safety is embedded directly into the policy class. Specifically, we construct a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set under arbitrary switching. Consequently, any policy whose actions are restricted to this library, including policies encountered during RL exploration, inherits the same certificate. Policy optimization can therefore focus on closed-loop performance without runtime safety filtering or action projection. The framework is instantiated for quadcopter hover regulation, where a DQN schedules among certified feedback controllers. Nonlinear MuJoCo simulations demonstrate state-dependent gain scheduling and empirically evaluate robustness to wind, model mismatch, sensor noise, and sensing delay. The results illustrate how safety certification can be separated from policy optimization by learning over a forward-invariant policy class.
I. INTRODUCTION
Safety-critical adaptive control must improve performance without violating hard constraints, yet common safety mechanisms act as external corrections and fixed controller gains can be conservative. This paper instead restricts learning to a library of certified controllers, embedding safety in the policy class.
I. INTRODUCTION
- Safety-critical learning must respect hard constraints, while nonlinear quadcopter dynamics, underactuation, actuator limits, and attitude constraints complicate adaptive control.Fixed offline gains must balance transient rejection, hover precision, and disturbance robustness, often conservatively.Source: §I · p. 1
- The framework restricts the action space to pre-certified stabilizing feedback laws, so every policy over those actions preserves the prescribed safe set.Offline certification constructs the controller library; online DQN decisions select only from that library.Source: §I · p. 1
A. Related Work
- Existing quadrotor gain scheduling often relies on heuristics or manually tuned switching, generally without formal guarantees that updates preserve the nonlinear system’s safety envelope.Prior trajectory-tracking methods also typically fix gains offline, which can be conservative across operating regimes.Source: A. Related Work · p. 1
- Forward invariance means trajectories starting in an admissible set remain there, a relevant property because constraint violations can compromise feasibility and invalidate controller assumptions.Barrier certificates, online control barrier functions, and safe-learning methods provide related verification and enforcement approaches.Source: A. Related Work · pp. 1–2
- Unlike approaches that learn control commands or parameters without embedding invariance in gain selection, this work selects among certified controllers and avoids runtime safety correction.The resulting framework is presented as interpretable and verification-friendly.Source: A. Related Work · p. 2
B. Contributions
The paper contributes an action space whose feedback laws share a common Lyapunov certificate, then optimizes policies within the resulting invariant class. It demonstrates this approach for DQN-based quadcopter hover control and evaluates robustness in nonlinear simulation.
Contribution 1: Forward-Invariant Action Design for Safe Learning
- A finite action space of feedback laws shares a common Lyapunov certificate, establishing forward invariance of the admissible set under arbitrary switching.The resulting safety guarantee is independent of the action-selection policy and applies during exploration and deployment.Source: B. Contributions · p. 2
Contribution 2: Policy Optimization over an Invariant Policy Class
- Policy optimization stays within the forward-invariant policy class, so learning affects performance but not the underlying safety certificate.Under the nominal closed-loop model, preserving the certified set requires no runtime filter, action projection, dwell-time constraint, or safety penalty.Source: B. Contributions · p. 2
Contribution 3: Safety-Certified Gain Scheduling for Quadcopter Control
- For quadcopter hover control, a DQN schedules among certified feedback controllers, and nonlinear MuJoCo simulations examine nominal performance and robustness.The robustness tests consider wind, model mismatch, sensor noise, and sensing delay.Source: B. Contributions · p. 2
II. PROBLEM STATEMENT
The problem statement models safe learning as an MDP and designs its admissible actions so that every action sequence preserves a prescribed set. The quadcopter case instantiates the state with tracking errors and actions with certified feedback controllers.
II. PROBLEM STATEMENT
- The learning problem uses an MDP with continuous state space, finite actions, a stage reward, and a discount factor.The MDP state is denoted ξ_k at decision instant k.Source: §II · p. 2
- Invariance-induced action-space design constructs a feedback parameterization and finite action set so the prescribed set remains invariant under every admissible action sequence.Policy learning then optimizes within the forward-invariant class rather than handling safety through online filtering.Source: §II · pp. 2–3
- For quadcopter hover regulation, the MDP state is the sampled tracking-error state, and each action selects a controller from a finite certified library.Source: §II · p. 3
- The quadcopter model uses position, velocity, Euler angles, thrust, and thrust rate, with a rotation matrix mapping body coordinates to inertial coordinates.The stated model parameters include gravitational constant and vehicle mass.Source: §III · p. 3
- Differential flatness and dynamic inversion express the external tracking-error dynamics, with three inputs acting on translational snap and a fourth on yaw acceleration.Source: §III · p. 3
A. Stability and Forward Invariance
A common quadratic Lyapunov certificate establishes stability and forward invariance for the controller library under arbitrary switching. The certified invariant sublevel set lies within the prescribed tracking-error bounds.
A. Stability and Forward Invariance
- A shared P satisfying the matrix inequality for every controller establishes uniform input-to-state stability under arbitrary switching.The certificate is independent of the switching sequence, including the ε-greedy action sequences encountered during DQN training.Source: A. Stability and Forward Invariance · pp. 3–4
- For bounded input, the stability result also establishes uniform ultimate boundedness.Selecting an appropriate common Lyapunov sublevel set yields forward invariance of the prescribed admissible set.Source: A. Stability and Forward Invariance · p. 4
- Safety here means satisfying the prescribed tracking-error bounds; constraints outside Zsafe are not implied by the certificate.A common P is more restrictive than mode-dependent Lyapunov functions, with that conservatism providing validity under arbitrary switching.Source: A. Stability and Forward Invariance · p. 4
- The common Lyapunov sublevel set Ωρ is forward invariant under every admissible switching signal and remains inside Zsafe.The bound on ρ ensures the sublevel set is contained in the prescribed tracking-error admissible set.Source: A. Stability and Forward Invariance · pp. 4–5
B. Nonlinear Closed-loop Dynamics
Dynamic inversion maps the external input to the physical control input and yields nonlinear closed-loop dynamics parameterized by the controller gains. The certified result transfers to nominal trajectories only under the stated inversion-domain and model-matching assumptions.
B. Nonlinear Closed-loop Dynamics
- Dynamic inversion maps the external input s to the physical control input u and defines nonlinear closed-loop dynamics parameterized by k.The policy holds its selected controller constant between decision instants.Source: B. Nonlinear Closed-loop Dynamics · p. 5
- Transfer of the stability result requires nonsingular inversion, no control saturation, and exact nominal-model matching.Delay, mismatch, saturation, and singularity are outside this implication; their MuJoCo tests are empirical.Source: B. Nonlinear Closed-loop Dynamics · p. 5
- At RL decision instants, every admissible action preserves the certified state space X := Ωρ.The policy selects an action from the certified controller library at each decision instant.Source: B. Nonlinear Closed-loop Dynamics · p. 5
IV. DQN-BASED SAFE LEARNING
The DQN policy is trained as a discrete-action RL problem over the certified action set, using tracking and control objectives without a safety penalty.
IV. DQN-BASED SAFE LEARNING
- The quadcopter MDP uses the external tracking-error state z_k as its state and the certified controller set as its discrete action space.Source: §IV · p. 5
- The stage reward penalizes tracking error, residual motion, aggressive attitude and control, and frequent controller switching, but includes no safety penalty.Forward invariance is guaranteed by the certified action set independently of the reward and learned policy.Source: §IV · p. 5
- A neural network Q(z, a; θ) approximates the action-value function, and actions are selected using an ϵ-greedy policy.θ denotes the trainable parameters.Source: §IV · p. 5
V. EXPERIMENTAL EVALUATION
The experiments assess common certification, state-dependent scheduling, the distinction between individual-mode stability and common certification, and behavior beyond the nominal model.
V. EXPERIMENTAL EVALUATION
- Four questions guide evaluation: certification over the initial set, scheduling benefits, common certification versus individual-mode stability, and behavior beyond the nominal model.Figure 2 previews proposed-DQN executions under nominal, wind, and sensing-delay conditions.Source: §V · p. 5
A. MuJoCo plants, task, and frozen protocol
The evaluation uses nonlinear MuJoCo simulations of a prescribed arrival-and-hover task, with a frozen learning protocol and additional tests near the certificate boundary.
A. MuJoCo plants, task, and frozen protocol
- The 10-s episode tracks a ninth-order smoothstep reference during its first 5 s before holding terminal position; the task specifies entering |r−rf| ≤ 0.10 m and |v| ≤ 0.20 m/s by 3.66 s, then maintaining precision hover.The nominal simulated mass and inertia match the controller model; RK4 integration uses a 2-ms step, and the controller updates every 20 ms.Source: A. MuJoCo plants, task, and frozen… · p. 5
- The proposed library's initial-error box lies inside its certified sublevel set: the maximum corner value is V(z0)/ρ = .001141 < 1.The library has 81 gain vectors and residual-adjusted common-certificate margin q = .00606588.Source: A. MuJoCo plants, task, and frozen… · p. 5
- Across 700 nominal rollouts near the certificate boundary, all tested policies remained physically safe with max_t V/ρ ≤ 1 and zero saturation.The test samples five V(z0)/ρ levels from .05 to .90 and includes the median fixed controller, five DQN checkpoints, and random certified switching.Source: A. MuJoCo plants, task, and frozen… · p. 5
- Training, checkpoint selection, and hyperparameter choices were completed before evaluation on 40 paired test scenarios.Figure 2 depicts matched proposed-DQN rollouts for nominal, 1.5-m/s wind, and 60-ms sensing-delay conditions, with time-coded trajectories.Source: A. MuJoCo plants, task, and frozen… · p. 6, Figure 2 · p. 6
B. Main comparison: safety, performance, and scheduling
The proposed DQN improves nominal tracking over a pre-specified fixed controller using the same certified library; fresh-seed tests show a tracking–switching trade-off as the minimum hold increases.
B. Main comparison: safety, performance, and scheduling
- The matched comparison uses the same common-certified 81-action library for the DQN and fixed action 40, isolating state-dependent scheduling against non-adaptive nominal tuning.Action 40 is the central row, with translational root scales of one and yaw roots (2, 6).Source: B. Main comparison: safety,… · p. 6
- Although all 81 ablation modes are Hurwitz, their common-P calculation has margin −1.56 × 10−5; individual-mode stability therefore does not establish the proposed library's positive common quadratic margin.The calculation does not rule out other possible switching certificates; the uncertified-library DQN also succeeds nominally, but only the proposed library has the stated arbitrary-switching guarantee among the two tested libraries.Source: B. Main comparison: safety,… · pp. 6–7
- The proposed DQN reduces hover RMSE from .013326 to .010937 m, a 17.93% reduction, and advances sustained arrival by .0091 s versus the median/nominal fixed controller.All three policies were safe and met the deadline in every nominal test rollout.Source: B. Main comparison: safety,… · p. 6
- In the fresh-seed evaluation, the .10-s DQN reduces hover RMSE by 18.65% versus the fixed controller across 40 previously unused paired scenarios.Each DQN scenario metric was averaged across five frozen checkpoints before paired comparison.Source: B. Main comparison: safety,… · p. 6
- At a .20-s hold, switching falls by 46% while RMSE reduction remains 16.59%; at .50 s, further switching reduction accompanies a substantially smaller tracking margin.The .20-s hold retains a tracking advantage over the median/nominal fixed controller while reducing switching relative to the .10-s hold.Source: B. Main comparison: safety,… · p. 7, Figure 3 · p. 7
C. Robustness and empirical applicability boundary
Robustness tests compare certified-library policies with an uncertified DQN under wind, model mismatch, noise, and sensing delay. Delay and combined perturbations separate their empirical safety outcomes, while higher wind levels reveal a task-performance boundary.
Robustness tests
- The evaluation used 40 paired scenario seeds across nominal operation, wind, plant mismatch, sensing noise, 60-ms delay, and a joint condition.Each deterministic baseline contributed 40 rollouts per condition; each DQN row pooled five frozen checkpoints for 200 rollouts. The joint condition combined 1.5-m/s wind, noise, and 60-ms delay.Source: C. Robustness and empirical… · p. 7
- Under 60-ms delay, both common-certified-library policies stayed safe in every rollout; under the joint condition, the median fixed and proposed DQN also stayed safe in every run, unlike the uncertified DQN.The uncertified DQN was safe in 2% and met the deadline in 81.5% under delay; under the joint condition it had zero safety and deadline satisfaction.Source: C. Robustness and empirical… · p. 7
- In one nominal near-boundary rollout starting at V(z0)/ρ = .90, fixed, learned, and random certified switching stayed below V/ρ = 1 while showing distinct decay transients.This is an illustrative rollout; the 700-rollout containment summary is reported in the text.Source: Figure 5 · p. 7
- Wind caused no physical-safety termination through 3 m/s, but deadline satisfaction degraded at 2.5 m/s and no policy met the deadline at 3 m/s.The 2.5- and 3-m/s cases are treated as an empirical task-performance boundary; Fig. 4 covers deadline satisfaction at 0–2 m/s and safety and joint completion across nonideal conditions.Source: C. Robustness and empirical… · p. 7, Figure 4 · p. 7
- Under a separate 100-ms-delay stress test, no policy remained physically safe, and rollouts showed saturation and large excursions in nominal V/ρ.This marks an empirical failure boundary.Source: C. Robustness and empirical… · p. 7
VI. CONCLUSION
The conclusion presents a common-Lyapunov-certified controller library that makes safety a property of the admissible policy class under nominal assumptions. Quadcopter simulations demonstrate scheduling within that class, while identifying certificate conservatism and directions for extending the framework.
VI. CONCLUSION
- A finite controller library sharing a common Lyapunov certificate establishes forward invariance of the admissible set under arbitrary switching, under the stated nominal assumptions.Policy optimization can proceed within the certified class without runtime filters, action projection, or dwell-time constraints.Source: §VI · pp. 7–8
- Table II covers held-out nominal and nonideal safety and arrival performance, favoring higher safety and deadline rates and lower conditional post-arrival RMSE.The caption specifies the metrics and preferred directions but gives no outcome values.Source: Table II · p. 8
- The quadcopter DQN schedules among certified controllers, improving tracking over the pre-specified nominal fixed controller in the considered arrival task.Nonlinear MuJoCo simulations also empirically examined robustness beyond the certificate assumptions; a dwell-time ablation exposed an execution-level switching trade-off.Source: §VI · p. 8
- The common quadratic certificate may be conservative because individually stabilizing controllers may not admit a common P.Future work will investigate less conservative invariance certificates, larger or continuous action spaces, and autonomous-driving extensions.Source: §VI · p. 8