Source-linked AI summary
Towards Safe Reinforcement Learning with Reduced Conservativeness: A Case Study on Drone Flight Control
Loizos Hadjiloizou, Michael C. Welle, Hang Yin, Danica Kragic
TL;DR
Reinforcement learning on real hardware needs safety without unnecessarily restricting exploration. The paper combines online disturbance learning with reachability-based shielding, and reports less restrictive controller training in a real-world canyon flight without compromising safety.
Problem
Safety and reliability for reinforcement learning on real hardware remain open problems, while restrictive safety methods can limit exploration.
Method
The framework refines disturbance models from online data and evaluates controller actions with zonotopic reachability analysis and shielding.
Results
In a real-world drone canyon flight under unknown disturbances, the framework enabled less restrictive training without compromising system safety.
Takeaways & Limitations
Online disturbance learning and efficient reachability analysis can support less restrictive training of learning-based controllers in the demonstrated drone scenario.
Abstract
from arXiv · showhide
Incorporating formal methods into reinforcement learning (RL) has the potential to result in the best of both worlds, combining the robustness of formal guarantees with the adaptability and learning capabilities of RL, though careful design is needed to balance safety and exploration. In this work, we propose a framework to mitigate this loss of exploration while still allowing for the safety of the system to be ensured. Specifically, we introduce a less restrictive method that can reduce the conservativeness of formal methods by refining a disturbance model using online collected data and it evaluates the safety of a learning-based controller, using computationally efficient zonotopic reachability analysis for the safety analysis to facilitate a real-time implementation. We validate the framework in a real-world drone flight through a canyon, where the drone is subjected to unknown external disturbances and the framework is tasked with learning those disturbances online and adjusting the safety guarantees accordingly. The results show that the framework enables a less restrictive online training of learning-based controllers without compromising the safety of the system.
I. INTRODUCTION & RELATED WORK
Safety remains an open problem for reinforcement learning on real hardware, motivating provably safe methods that evaluate future outcomes but can restrict exploration. This work targets adaptive refinement of disturbance assumptions and efficient reachability analysis to reduce that restrictiveness.
- Safety and reliability for reinforcement learning on real hardware remain open research problems.
- Provably Safe RL methods use shielding to predict future outcomes and replace unsafe actions with provably safe alternatives.
- Restrictive safety assumptions can cause sub-optimal performance or failures on task completion.
- The framework refines disturbance assumptions online using collected data and efficient reachability analysis to support less restrictive exploration.
- Larger disturbance bounds can enlarge forward reachable sets and tighten backward reachable sets, changing the resulting safety guarantees.
B. Zonotopes & Zonotopic Reachability Analysis
Zonotopes provide a computationally efficient representation for approximating reachable sets. The approach linearizes system dynamics and computes predecessor and successor sets using Minkowski arithmetic.
- Zonotopes are centrally symmetric convex polytopes represented as affine images of unit hypercubes.A zonotope has the form Z = {c + Gξ | ||ξ||∞≤1}.
- For zonotopic reachability analysis, system dynamics are linearized as xt+1 = Axt + But + wt.
- Predecessor and successor reachable sets are computed using formulas derived from Minkowski arithmetic.The sets T and I are taken as infinitesimally small zonotopes around the current state xk.
- Minkowski sum combines pairwise element sums, while Minkowski difference identifies translations whose addition with an element of one set lies in another.
C. Action safety evaluation and replacement
The framework evaluates whether a controller action is safe by examining reachable states under disturbances. Unsafe actions are replaced with alternatives that are provably safe or empirically well behaved.
- A binary safety function maps each state-action pair to 1 when safe and 0 otherwise.
- When a controller action is unsafe, the mechanism replaces it with an alternative that is provably safe or well behaved.
III. METHODOLOGY
The methodology combines an RL agent and environment with a disturbance learning unit and a safety mechanism. Online disturbance predictions are combined with policy actions and spatial information to evaluate and, when needed, replace unsafe actions.
- The proposed method integrates SGD regression models with reachability analysis for less restrictive yet safe controller training.
- The disturbance learning unit predicts expected disturbance levels from online state-action-disturbance data.
- The safety mechanism combines predicted disturbances, policy actions, and spatial awareness to evaluate action safety.
- Unsafe actions activate replacement, and the agent skips its training step for that time step.
A. Learning-Based Control
The learning-based controller uses a Gaussian MLP trained by policy gradients, while an online disturbance-learning unit refines future disturbance predictions from logged state-action-disturbance experiences.
- Controller: The Gaussian MLP maps the state error vector to probability distributions for the continuous control inputs.Its input is (x_t − g_t), and it outputs N_i(μ_i, σ) for each control dimension.
- Controller: Policy-gradient optimization updates the network parameters using a learning rate and the gradient of the loss function.The update rule is θ_t = θ_t−1 + α∇_θL(θ).
- Disturbance Learning Unit: The disturbance-learning unit replaces worst-case a priori disturbance estimates with online predictions based on disturbances observed during system operation.For additive disturbances, the observed disturbance is computed as w_t = x_t+1 − f(x_t, u_t, 0).
- Disturbance Learning Unit: Logged state, action, and disturbance histories train an SGD regressor to predict the disturbance distribution over the next N time steps.Each experience includes the system state, selected control input, and disturbance experienced under that state-input pair.
- Disturbance Learning Unit: The regressor uses a linear score function and predicts a mean and standard deviation for the disturbance sequence.Its score is f(S_I) = w^T S_I + b, and the predicted bounds are [μ̂_d − 3σ̂_d, μ̂_d + 3σ̂_d].
C. Shielding Mechanism
The shielding mechanism evaluates candidate actions through reachable-set safety checks and replaces actions that fail with actions from a replacement controller.
- Safety Evaluation: An action is safe when it belongs to the locally safe action space defined for the current state.The local action space is used to evaluate the safety function for the action under consideration.
- Safety Evaluation: The forward reachable set is computed with the action space restricted to the action being evaluated, and obstacles define nonsafe regions.The safety function equals 1 when the evaluated action satisfies the local safety conditions.
- Safety Evaluation: Safety combines avoiding immediate entry into unsafe regions with checking whether the system can recover to its current state.The recovery check verifies that the backward reachable set from the current state contains the forward reachable set.
- Action Replacement: The action replacement unit passes the controller’s action when φ(x_t, u_t) = 1 and otherwise substitutes an action from the replacement controller.This connects the safety decision to the fallback action mechanism shown in the framework.
IV. DRONE FLIGHT THROUGH CANYON SCENARIO
The framework is evaluated by flying a drone through a progressively narrowing canyon, where obstacles and wall-induced air disturbances make control increasingly challenging.
- Scenario: The experiment uses a narrowing canyon to test the framework in a confined real-world flight environment.The setup includes a laboratory canyon and a rendered top view of the flight scenario.
- Scenario: As the drone advances, reduced maneuvering space, closer obstacles, and stronger wall-related air disturbances increase the flight difficulty.The canyon becomes progressively narrower along the flight path.
- Evaluation Questions: The experiment asks whether the learning unit can predict variation in the disturbance pattern.This tests the disturbance model’s ability to capture changes encountered during the flight.
- Evaluation Questions: It also tests whether predicted disturbances produce less restrictive safe sets through zonotope-based analysis.The comparison concerns disturbance prediction’s effect on the safety analysis.
- Evaluation Questions: The study measures whether safe sets reduce unnecessary unsafe decisions and increase the agent’s opportunities for data exploration.Training exposure is linked to how often the agent’s actions are deemed safe in the same episodes, compared with a restrictive disturbance model.
A. Drone Model
The drone model describes a Crazyflie 2.1 quadrotor whose outer-loop position dynamics are analyzed using linearized state-space equations with bounded disturbances.
- Drone Platform: The experimental platform is a Crazyflie 2.1 quadrotor with an onboard controller and a cascaded control architecture.The onboard inner loop regulates the drone’s orientation.
- Drone Platform: The framework focuses on the outer-loop position controller rather than the onboard orientation-regulation loop.The drone dynamics are represented in state-space form for this control layer.
- State-Space Model: The state comprises global position and translational velocity, while the input includes collective thrust and roll and pitch angles.The state uses (x, y, z, v_x, v_y, v_z), with mass m and gravity g defining physical parameters.
- Linearization: Zonotopic reachability analysis uses linearized dynamics, but linearization errors can invalidate guarantees outside limited angular motions.The framework accounts for pruned nonlinearities by adjusting control-input and state-space constraint bounds.
- Linearization: Under a small-angle approximation, the drone dynamics are expressed as ẋ = Ax + Bu + w.The continuous-time A and B matrices define the linearized model used for reachability analysis.
B. Experimental Setup
The experimental setup uses an off-board ROS controller for Crazyflie flights, with state ground truth supplied by motion capture. Results are aggregated across five flights for the restrictive setting.
- Experimental Setup: The off-board controller runs on Ubuntu 20.04 LTS with an AMD Ryzen 7 4700u processor and communicates through ROS Noetic.Motion capture provides real-time ground-truth drone state information.
- Experimental Setup: The Crazyflie Python API connects the ROS off-board controller to the drone’s onboard low-level controller.
- Experimental Setup: Figure 5 aggregates results from five flights in the restrictive setting.
- Experimental Setup: The drone follows a y-axis flight path, using 10 cm goal increments before the canyon entrance and 5 cm increments inside it.Goal updates occur every 10 seconds after the drone stabilizes around the desired state.
V. RESULTS
The results compare restrictive and non-restrictive disturbance handling during canyon flights. The non-restrictive method replaces fewer learned actions with the safety oracle and permits more safe actions while learning disturbance bounds from flight data.
- V. RESULTS: The restrictive disturbance unit drives outputs toward a priori worst-case upper and lower disturbance levels, producing conservative safety behavior.Disturbance levels increase deeper inside the canyon, especially toward its narrower end.
- V. RESULTS: Figure 6 aggregates results from five flights in the non-restrictive setting.
- V. RESULTS: The non-restrictive method replaces some actions with the LQR oracle, but replacements are more sporadic and not consecutive than in the restrictive version.This provides the learning-based controller with more opportunities to acquire new training data.
- V. RESULTS: The disturbance learning unit sufficiently captures the true dvx and dvy disturbances, while tighter predicted-disturbance bounds are associated with increased performance.
- V. RESULTS: The restrictive setting frequently deems actions near the canyon’s narrow end unsafe, whereas the non-restrictive approach triggers safety replacement much less often.
VI. CONCLUSION
The paper presents a less-restrictive controller-training framework that adapts disturbance assumptions online using real-world data and evaluates safety with zonotopic reachability analysis. A real-world canyon-flight demonstration under unknown disturbances shows the potential for less restrictive training without compromising system safety.
- VI. CONCLUSION: The framework adapts its assumed disturbance online from real-world data and adjusts safety constraints accordingly.
- VI. CONCLUSION: It combines SGD regressors for disturbance-model refinement with zonotopic reachability analysis for evaluating safety constraints.
- VI. CONCLUSION: A real-world drone flight through a canyon demonstrates the framework under unknown external disturbances.
- VI. CONCLUSION: The results show potential for less-restrictive training of learning-based controllers without compromising system safety.