Source-linked AI summary
CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors
Fabrizio Cesareo, Sebastiano Mengozzi, Nicola Mimmo, Andrea Acquaviva
TL;DR
Quadrotor DRL policies lack guarantees of respecting safety constraints, and runtime enforcement must be coordinated with learning. CALOS combines tilt bounds and a Lyapunov condition in a runtime torque-correction QP. It reports 55–74% lower lateral tracking error than unconstrained PPO, with zero attitude violations on the training trajectory.
Problem
Quadrotor DRL policies can reach the edge of the flight envelope without classical stability guarantees, while existing safe-DRL approaches leave joint tilt and Lyapunov enforcement unresolved.
Source: §I · p. 1Method
CALOS is a runtime safety layer that combines attitude-tilt constraints and a Lyapunov condition in a single QP producing a minimum-norm torque correction.
Source: §I · p. 1, A. Tilt constraints and Predictive… · p. 2Results
55–74% lower mean lateral error than PPO, with zero attitude-constraint violations on the training trajectory.
Source: D. Internalized Safe Behavior · p. 4, §V · p. 4Takeaways & Limitations
Policies trained with CALOS retain safer, more accurate behavior without the safety layer at test time, while constrained exploration accelerates convergence and improves data efficiency.
Source: D. Internalized Safe Behavior · p. 4, §V · p. 4The current quadratic certificate drives toward zero tilt, causing unnecessary interventions during feasible non-zero-attitude tracking.
Source: §V · p. 4
Abstract
from arXiv · showhide
Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment. We present CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that enforces attitude constraints on a quadrotor without modifying the underlying learning algorithm. CALOS formulates four tilt-angle inequalities and a Lyapunov descent condition as a single quadratic program whose solution is the minimum-norm correction to the nominal torque output of the policy. The quadratic program is solved exactly via active-set enumeration over the three-dimensional torque space, with a computational cost low enough to enforce constraints in real time across thousands of parallel simulation environments, as required by modern massively parallel Deep Reinforcement Learning training. Evaluated on trajectory-tracking tasks in NVIDIA Isaac Lab, CALOS reduces lateral tracking error by 55-60% relative to an unconstrained Proximal Policy Optimization baseline while achieving zero attitude-constraint violations on the training trajectory. By restricting exploration to safe regions of the state space, the safety layer also accelerates training convergence and improves data efficiency without producing suboptimal policies.
I. INTRODUCTION
Safe quadrotor DRL must reconcile runtime constraint enforcement with learning, while existing approaches leave tilt and Lyapunov constraints uncoordinated. CALOS combines them in a transparent runtime layer that corrects policy torque.
I. INTRODUCTION
- Safe DRL approaches modify the optimization criterion through CMDPs or modify actions at runtime, but each can complicate learning.CMDP solvers can destabilize learning, while aggressive runtime corrections during training can degrade gradient signals and impede policy improvement.Source: §I · p. 1
- Prior runtime methods lack a joint single-step optimization of tilt and Lyapunov constraints, leaving sequential composition and its feasibility issues.Existing approaches enforce a single certificate type, treat constraints independently, or couple enforcement to constrained actor updates and a backup controller.Source: §I · p. 1
- CALOS unifies attitude-tilt and Lyapunov stability constraints in one QP, returning a minimum-norm torque correction without modifying the learning algorithm.It is used with PPO and is compatible with policy-gradient methods.Source: §I · p. 1
II. PROBLEM FORMULATION
- The trajectory-tracking problem requires a learning controller to follow a time-varying position reference while respecting attitude constraints.The quadrotor uses a Combined Thrust and Body Torques representation with input-affine dynamics, enabling linear input constraints.Source: §II · p. 1
- The policy supplies collective thrust and nominal body torque; CALOS receives the torque and state, then returns a corrected torque while leaving thrust unchanged.Leaving thrust unmodified preserves vertical authority and avoids coupling with altitude regulation.Source: §II · p. 1
III. SAFETY LAYER DESIGN
The tilt-constraint design represents attitude using body-frame gravity and predicts its next value as an affine function of torque. Four linear inequalities encode the roll and pitch limits, with PTP applying iterative corrections.
A. Tilt constraints and Predictive Tilt Projection
- Body-frame gravity represents attitude tilt without Euler-angle singularities, and its predicted value depends affinely on torque.The prediction uses a symplectic Euler discretization of the gravity-direction dynamics and rotational dynamics.Source: A. Tilt constraints and Predictive… · pp. 1–2
- Four linear inequalities enforce the roll and pitch bounds by constraining the predicted body-frame gravity components.The bounds are expressed as |gb,x| ≤ sin θmax and |gb,y| ≤ sin ϕmax.Source: A. Tilt constraints and Predictive… · p. 2
- The torque-to-gravity sensitivity matrix maps torque changes to predicted body-gravity changes, while gconst collects torque-independent terms.The passage defines M(xt) = Δt²[gb]×J−1 as the sensitivity matrix.Source: A. Tilt constraints and Predictive… · p. 2
- Predictive Tilt Projection enforces the four bounds through an ATACOM-inspired iterative correction to torque.The correction uses a tunable gain and a damped pseudoinverse, with damping used to ensure numerical conditioning.Source: A. Tilt constraints and Predictive… · p. 2
B. Lyapunov constraint and scaling
The Lyapunov condition requires a candidate measuring tilt error and angular velocity to decrease at a specified rate. Its derivative is affine in torque, while standalone scaling enforces the condition by shrinking torque magnitude.
B. Lyapunov constraint and scaling
- The Lyapunov candidate combines weighted tilt error and angular velocity, and its derivative must satisfy a negative decay-rate bound.The condition is expressed as V̇(x, τ) ≤ −cV(x), with c > 0.Source: B. Lyapunov constraint and scaling · p. 2
- Because the rotational dynamics are control-affine, the Lyapunov derivative is affine in torque.The constraint coefficients include rotational-energy sensitivity and terms for tilt-error coupling and gyroscopic effects.Source: B. Lyapunov constraint and scaling · p. 2
- The standalone Lyapunov scaling operator enforces the condition by multiplying nominal torque by a factor s between zero and one.The scaling reduces torque magnitude.Source: B. Lyapunov constraint and scaling · p. 2
C. Sequential composition
Sequentially composing tilt projection, Lyapunov scaling, and a tilt check does not guarantee joint feasibility. Large tracking error can drive scaling toward zero, nullifying torque when recovery requires it.
C. Sequential composition
- Sequential tilt projection, Lyapunov scaling, and a final tilt check do not guarantee joint feasibility.When tracking error is large, the growing Lyapunov value can drive the scale toward zero and nullify torque needed for recovery.Source: C. Sequential composition · p. 2
- Large tracking error can make Lyapunov scaling nullify torque precisely when the controller needs maximum torque to recover.The passage identifies this interaction as motivation for the unified formulation.Source: C. Sequential composition · p. 2
- The proposed unified formulation is motivated by the sequential composition’s feasibility problem and loss of recovery torque under large error.Source: C. Sequential composition · p. 2
D. CALOS: unified formulation
CALOS combines tilt and Lyapunov constraints in one QP, with approximate and exact solver options for the unified system.
D. CALOS: unified formulation
- CALOS stacks the tilt and Lyapunov conditions into five linear constraints and solves a QP with a unique solution when the feasible set is non-empty.Source: D. CALOS: unified formulation · p. 2
- CALOS-P applies a damped pseudo-inverse correction to enforce all five constraints jointly, trading exact minimum-norm optimality for speed in large batches.Source: D. CALOS: unified formulation · p. 2
- CALOS-QP exploits the three-dimensional torque space to enumerate 26 candidate active sets, checking dual-variable and constraint feasibility before selecting a solution.Source: D. CALOS: unified formulation · p. 2
- Among feasible candidates, CALOS-QP selects the torque minimizing its distance from the nominal policy torque.Source: D. CALOS: unified formulation · p. 2
- If no feasible candidate exists, the solver applies the uncorrected policy torque and clamps actuators to physical limits; enumeration is parallelizable across thousands of environments.Source: D. CALOS: unified formulation · p. 2
A. Setup
The evaluation trains PPO agents in parallel Isaac Lab environments, then compares six variants across training and unseen trajectory conditions.
A. Setup
- Agents train with PPO in 4096 parallel Isaac Lab environments for approximately 10^9 timesteps, using a curriculum that tightens tracking precision from 0.50 m to 0.15 m.Source: A. Setup · p. 2
- Six variants—PPO, PTP, Lyapunov scaling, Cascade, CALOS-P, and CALOS-QP—are evaluated over 200 episodes on two Lissajous trajectories and a circle.Source: A. Setup · p. 2
- The harder Lissajous L2 and circle are unseen during training and tested with a large initial position offset; mean lateral error above 0.20 m counts as tracking failure.Source: A. Setup · pp. 2–3
B. Tracking Performance
CALOS variants maintain tracking errors below 0.08 m where Lyapunov and Cascade fail, while safety outcomes differ by formulation and perturbation severity. Under a large initial offset, CALOS-P violations are brief; CALOS-QP has zero roll-violation episodes on nominal L1.
B. Tracking Performance
- Both CALOS variants keep tracking errors below 0.08 m, while Lyapunov and Cascade fail to track the reference.The failures are attributed to torque nullification.Source: B. Tracking Performance · p. 3
- CALOS-QP tracks circular trajectory C closely and settles roll and pitch faster than PTP, which has larger oscillations.The figure marks the ±60° attitude constraint; PPO shows visible tracking drift.Source: B. Tracking Performance · p. 3, Figure 1 · p. 3
C. Safety Analysis
- On L1 with a large initial offset and a 60° limit, CALOS-P violations last at most 3 consecutive steps and occur during initial correction.Table II reports roll-violation statistics for this stress-test setting.Source: C. Safety Analysis · p. 3, Table II · p. 3
- Lyapunov violations reach a maximum tilt of 180° and last up to 27 consecutive steps.Source: C. Safety Analysis · p. 3
- On L1 without an initial offset, CALOS-QP achieves zero roll-violation episodes.Source: C. Safety Analysis · p. 3
- Violations under large initial offsets could be further reduced by including high-error conditions in the training distribution.Source: C. Safety Analysis · p. 4
D. Internalized Safe Behavior
Policies trained with CALOS retain safer, more accurate behavior when the safety layer is disabled at test time. The reported tracking improvement accompanies constrained exploration and policy internalization of constraint boundaries.
D. Internalized Safe Behavior
- With the safety layer disabled at test time, mean lateral error is 55–74% lower for the CALOS-trained policy than PPO.Table III reports tracking with the safety layer disabled.Source: D. Internalized Safe Behavior · p. 4, Table III · p. 4
- The projection keeps training samples in states where meaningful learning occurs, restricting exploration to safe regions and accelerating convergence and data efficiency.Source: D. Internalized Safe Behavior · p. 4
- CALOS preserves freedom within the feasible set, allowing policies to internalize constraint boundaries without becoming suboptimal.The resulting policies match or exceed the unconstrained baseline's tracking performance.Source: D. Internalized Safe Behavior · p. 4
E. Why Sequential Methods Fail
CALOS’s unified torque QP avoids the recovery failure of sequential scaling and remained feasible across tests, while the conclusion reports tracking and training benefits. The paper identifies certificate and dynamics scope boundaries for future work.
E. Why Sequential Methods Fail
- With large tracking errors, scaling drives s toward zero and nullifies torque; CALOS’s affine Lyapunov constraint preserves a large torque component along the recovery direction.Source: E. Why Sequential Methods Fail · p. 4
- The CALOS-QP solver found a feasible solution across all tested agents, trajectories, and initial-offset conditions, so the fallback to the uncorrected policy action was never triggered.Source: E. Why Sequential Methods Fail · p. 4
V. CONCLUSIONS
- 55–74% lower lateral tracking error than unconstrained PPO, with zero training-trajectory attitude violations and at most 3 consecutive violation steps under large perturbations.Source: §V · p. 4
- Policies trained with CALOS retain comparable tracking improvement with the safety layer disabled, and the authors report better convergence and data efficiency without sacrificing optimality.Source: §V · p. 4
- The current certificate can prompt unnecessary interventions at feasible nonzero attitudes; rotor-speed or blade-flapping dynamics break the affine assumption, and hardware transfer remains future work under model uncertainty.Source: §V · p. 4