Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning

• Robots operate in complex settings, often near people, where dynamics, sensor measurements, and other agents’ behavior may be uncertain or unknown.Source: §1 · p. 2
• Safety must be guaranteed both during learning and for the optimized policy, and such guarantees depend on the assumptions and structure in the problem formalization.Source: §1 · p. 2
• Control theory offers model-driven guarantees for known conditions, while data-driven reinforcement learning adapts to new contexts; combined methods pursue robustness, adaptation, and prediction.Source: §1 · p. 2
• Safe reinforcement-learning publications grew from tens to over a thousand in a few years since the prior review (4), with physics-based simulation (5) supporting progress while transfer to real systems remains an open research area (6).Source: §1 · pp. 2–3
• Earlier reviews covered particular techniques or focused on control-theoretic (16) or reinforcement-learning (17) approaches, without providing a unifying perspective.Source: §1 · p. 3
• The review surveys safety-focused learning-based control and reinforcement learning, aiming to bridge communities with shared vocabulary and benchmarks while connecting to, but not covering, perception, estimation, planning, or multi-agent systems.Source: §1 · p. 3

2.1. Problem Statement

• The safe learning control problem seeks a controller or policy that achieves a task while respecting safety constraints, even when the model, cost, or constraints are partly unknown.Its three components are the robot’s system model, a task cost, and safety requirements.Source: §2 · p. 3, §2.1 · p. 4

2.1.1. System Model.

• Robots are modeled in discrete time with state, input, dynamics, and process noise; the review assumes state measurements are available and sets aside state estimation.The model covers common platforms such as quadrotors, manipulators, and ground vehicles.Source: §2.1.1 · p. 4

2.1.2. Cost Function.

• Task performance is represented by a finite-horizon cost with stage and terminal terms; control minimizes cost, whereas reinforcement learning typically maximizes reward.The stage cost can be related to discounted reward as l_k = −γ^k r_k(x_k, u_k).Source: §2.1.2 · pp. 4–5

2.1.3. Safety Constraints.

• Safety requirements include state and input limits and stability, and range from guaranteed hard constraints to probabilistic satisfaction or encouraged compliance; practical systems may mix levels.For example, input limits are typically hard while state constraints may be soft.Source: §2.1.3 · p. 5

2.1.4. Formulation of the Safe Learning Control Problem.

• Safe learning control combines prior knowledge and system data to learn unknown components and find a policy that performs the task while respecting constraints and approximating the true optimal policy.The true dynamics, constraints, and cost are decomposed into nominal prior components and unknown components learned from data.Source: §2.1.4 · p. 6

2.2. A Control Theory Perspective

• Adaptive control adjusts parameters online for parametric uncertainty but requires a known uncertainty structure and may overfit recent observations without converging to true parameters.Robust control instead designs for all bounded disturbances, but is largely limited to linear time-invariant systems.Source: §2.2 · pp. 6–7
• Robust MPC guarantees state and input constraints for bounded disturbances, but worst-case conservatism can hurt performance; learning can reduce conservatism by refining uncertainty sets or dynamics models.Tube-based MPC tightens constraints around a nominal prediction so the true state remains within a bounded tube.Source: §2.2 · pp. 7–8

2.3. A Reinforcement Learning Perspective

• Reinforcement learning can learn policies through interaction without an a priori dynamics model, but exploration may require unsafe actions and deployment faces challenges including provable safety, robustness, interpretability, and convergence.Model-based methods learn explicit dynamics, while model-free approaches learn value functions, policies, or both; Bayesian uncertainty guidance is often approximated because full posterior inference is expensive.Source: §2.3 · pp. 8–9
• Constrained MDPs add discounted safety-cost constraints to MDPs, but traditional solvers scale poorly and deep RL faces costly off-policy evaluation of trajectory-level constraints (33, 34).Recent work applies deep learning to complex control tasks, seeks constraint satisfaction during exploration, and transforms constraints to make off-policy evaluation more efficient.Source: §2.3 · p. 9
• Robust MDPs optimize worst-case costs under uncertain dynamics, but tractability-driven uncertainty sets can limit use beyond toy problems (35).Deep RL work targets uncertainty modeling and efficient min-max optimization, including through sampling or game-theoretic formulations; related approaches include adversarial RL and domain randomization (36, 37, 38).Source: §2.3 · p. 9

2.4. Bridging Control Theory and RL for Safe Learning Control

• Control approaches use prior knowledge and model assumptions for safety guarantees, whereas expressive RL models learn complex tasks from data but can impede formal guarantees.Both communities are developing safe learning control methods that leverage expressive models for closed-loop control.Source: §2.4 · p. 9
• Deterministic models such as deep neural networks abstract large datasets for real-time control, while probabilistic models such as GPs and BLRs estimate uncertainty for adaptive or robust control.Combining the two model types and quantifying uncertainty in deep learning remain active research directions.Source: §2.4 · pp. 9–10
• Safe learning control uses online data to adapt during closed-loop operation or offline batch data to update models between trials, reducing uncertainty and design conservatism.Its safety principle is to recognize what remains unknown and cautiously account for incomplete knowledge through algorithm design.Source: §2.4 · p. 10

3. SAFE LEARNING CONTROL APPROACHES

• Learning uncertain robot dynamics from data can improve performance under an a priori dynamics model, with control-theoretic safety typically at Level II or III.Source: §3 · p. 10
• RL approaches generally lack an a priori robot model or known safety constraints and encourage safe operation, such as by penalizing dangerous actions, at Safety Level I.Source: §3 · p. 11
• Certification approaches modify learning-controller outputs through policy constraints, a known safe backup, or direct output changes to achieve stability or constraint satisfaction, typically at Level II or III.Source: §3 · p. 11
• The approaches are categorized by safety level and reliance on data, with a detailed summary provided in Appendix A’s Table 1.Figure 4 presents the categorization.Source: §3 · p. 11

3.1.1. Integrating Machine Learning and Adaptive Control.

• Adaptive control integrates online learning through nonparametric dynamics models, uncertainty-weighted adaptation, and deep feature memory that can reduce the need for re-adaptation.L1 adaptation plus an NN or GP can approximate unknown nonlinear dynamics and improve performance at lower sampling rates; asynchronous DNNs update output layers quickly and inner layers more slowly, with a modeling-error bound used to guarantee Lyapunov stability.Source: §3.1.1 · pp. 11–12
• GP-based cautious adaptation scales its learned control contribution by model confidence, and stochastic stability analysis provides a Safety Level III guarantee.The approach compensates for feedback-linearization errors and has been demonstrated in quadrotor experiments.Source: §3.1.1 · p. 12

3.1.2. Learning-based Robust Control.

• GP-informed robust control reduces model uncertainty as data accumulate, improving performance while guaranteeing Safety Level II stability for stabilization tasks.Linearizing the GP near an operating point prioritizes nearby data; these approaches do not handle state or input constraints, while feedback-linearization methods extend guarantees to tracking only under a special system structure.Source: §3.1.2 · pp. 12–13

3.1.3. Reducing Conservatism in Robust MPC with Learning and Adaptation.

• Robust adaptive MPC uses data to reduce parametric uncertainty or combines inner-loop adaptation with outer-loop MPC, improving performance over standard robust MPC while satisfying hard constraints.Approaches shrink the parameter set and constraint tube over time, or use an adaptive controller to bound closed-loop dynamics for an outer MPC; iterative learning can further update terminal constraints and costs.Source: §3.1.3 · p. 13
• Learning-based robust MPC uses data to improve dynamics estimates, reduce uncertainty, or update costs; linear-model variants can preserve robust constraints using tube-based MPC.State-dependent uncertainty can tighten constraints less conservatively. For repetitive tasks, learned cost errors can penalize states whose realized costs exceeded predictions.Source: §3.1.3 · pp. 13–14
• For nonlinear nominal models, GP-based robust MPC requires uncertainty-propagation approximations that can violate probabilistic constraints, while NN-predicted quantile bounds may underestimate tube size with nonexhaustive data.Real-time implementations often require further approximations, such as fixing GP covariances over the prediction horizon.Source: §3.1.3 · p. 14

3.1.4. Safe Model-Based RL with A Priori Dynamics.

• Safe model-based RL explores uncertain states within an estimated region of attraction to reduce model uncertainty and expand that region while retaining probabilistic stability guarantees.The method assumes known Lipschitz constants, a Lyapunov function, an initial safe policy, and GP-learned dynamics; discretization preserves guarantees but makes exploration suboptimal.Source: §3.1.4 · p. 14

3.2.1. Safe Exploration and Optimization.

• Ergodicity-preserving policies and safety layers can support safe exploration, while exact ergodic-policy selection is NP-hard and the demonstrated heuristic is suboptimal (73, 75, 70).The safety layers project a learned action to the closest safe action; one approach assumes known linear constraints, while another learns approximations to constraints that can be evaluated.Source: §3.2.1 · p. 15
• SafeOpt and related Bayesian optimization methods use Gaussian-process models to sample within safety thresholds, while SafeMDP and SafeExpOpt-MDP extend the approach to MDPs (78, 69, 82).SafeOpt distinguishes inputs for expanding the safe set from inputs for optimizing the unknown function; SafeExpOpt-MDP treats the safety feature and cost as separate unknown functions.Source: §3.2.1 · pp. 15–16
• Learned safety critics detect unsafe proposed actions using model-free learned value functions and resort to fallback schemes; SQRL predicts future trajectory failure probability and filters unsafe actions, while Recovery RL learns a recovery policy.Source: §3.2.1 · p. 16

3.2.2. Risk-Averse RL and Uncertainty-Aware RL.

• Risk estimates from collision models, dynamics ensembles, or cost distributions support conservative policies, chance-constrained exploration, and offline optimization of cost risk (89, 90, 91, 92, 93).CARL pre-trains across dynamics before risk-averse adaptation, while SAVED uses predicted collision probability and a learned terminal cost estimate.Source: §3.2.2 · p. 16

3.2.3. Constrained MDPs and RL.

• CMDP methods use Lagrangian primal-dual updates and trust-region cost surrogates; the latter enable monotonic policy improvement and near constraint satisfaction at each iteration (88, 95, 34).Off-policy dual updates improve sample efficiency and convergence; trust-region surrogates use state-action data but require solving dual variables from scratch at each update.Source: §3.2.3 · p. 17
• Lyapunov constraints convert trajectory-level costs into stepwise state constraints, while backward value functions estimate accumulated costs and support efficient temporal-difference learning (97, 98, 99).Lyapunov methods require a constraint-satisfying baseline policy; later work supports continuous inputs, while backward value functions can be adapted from discrete to continuous action spaces.Source: §3.2.3 · pp. 17–18

3.2.4. Robust MDPs and RL.

• Robust adversarial RL trains a protagonist against a destabilizing policy, with extensions using risk-aware agents, adversary populations, or certified value lower bounds (36, 37, 38, 104).The protagonist and adversary update alternately; training against a population of adversaries makes the resulting protagonist less exploitable by new adversaries.Source: §3.2.4 · p. 18
• Domain randomization trains policies on perturbed systems to generalize across tasks, and simulated randomized-scene training improved a quadrotor’s real-world collision avoidance (105, 106, 107, 108).Randomization approximates a robust uncertainty set and can use standard RL; adaptive approaches include Bayesian search and discriminator-guided generation of less-explored systems.Source: §3.2.4 · p. 18

3.3.1. Stability Certification.

• Learning-based controllers can be certified for stability by bounding policy Lipschitz constants or estimating regions of attraction (ROAs) from system data.Lipschitz bounds can be enforced iteratively or during training, but often rely on specific dynamics structures; learned ROAs can guide data acquisition, and neural Lyapunov functions can provide less conservative estimates.Source: §3.3.1 · p. 19, §3.3 · p. 19

3.3.2. Constraint Set Certification.

• CBF and CLF methods certify safe-set invariance or stability, but uncertainty in their time derivatives can cause falsely certified inputs.Learning unknown dynamics from trials or RL data can refine derivative estimates; robust, adaptive, or probabilistic methods address learning error and support set-invariance guarantees.Source: §3.3.2 · p. 20
• Hamilton–Jacobi reachability estimates robust invariant safe sets under known bounded disturbances, enabling learning within the safe-set interior and a safe policy at its boundary.A Gaussian-process scheme can shrink the assumed disturbance set using observations to reduce conservativeness; extensions combine reachability with online learning, planning, and safe RL.Source: §3.3.2 · pp. 20–21
• Predictive safety filters modify learned inputs to preserve future state and input constraints, while backup control checks predicted states and falls back to a certified action when needed.Model Predictive Safety Certification filters inputs to ensure robust constraint satisfaction, and its terminal safe set can grow to include trajectories from prior iterations.Source: §3.3.2 · pp. 21–22

BENCHMARKS

• Existing evaluations vary widely: uncertain-dynamics learning often uses abstract examples, safe RL mostly uses simulations, and certification work has the largest fraction of real-world experiments.Many RL environments, including physics-based ones, are deterministic and do not represent variations in real robotic platforms.Source: §4 · p. 22
• Reproducibility remains challenging, and the section calls for simple, realistic physics-based environments with shared interfaces and useful comparison metrics.Only a minority of the reviewed research open-sourced implementations; proposed safety-aware RL environments lack a unified interface that exposes a priori dynamical-system knowledge.Source: §4 · p. 22

Cart-Pole and Quadrotor Benchmark Environments

• The open-source suite uses Bullet and Gym to benchmark cart-pole and quadrotor tasks, supporting constraints, disturbances, randomized conditions, and symbolic prior dynamics.It is designed to integrate control and RL approaches and enable comparisons of data requirements.Source: §4 · pp. 23–24

Safe Learning Control Results

• GP-MPC accounts for an inaccurate prior model and stabilizes the constrained quadrotor, while linear MPC with that model quickly violates the position constraint.The prior model used mass and moment of inertia at 150% of their true values; GP-MPC respects the boundary by a margin proportional to its 95% dynamics-prediction confidence interval.Source: Figure 6 · p. 23, §4 · p. 24
• Safe exploration with PPO achieves the same post-learning performance as cost shaping with substantially fewer constraint violations than standard PPO, and better constraint satisfaction than cost shaping.All three approaches used over 9 hours of simulation data; safe exploration requires careful tuning of its slack variable.Source: Figure 7 · p. 23, §4 · p. 24
• MPSC modifies PPO’s unsafe cart-pole inputs near constraint boundaries, keeping the system within constraints where uncertified PPO violates them.This is a proof of concept for combining safety filters with RL control.Source: §4 · pp. 24–25, Figure 8 · p. 24

5. DISCUSSION AND PERSPECTIVES FOR FUTURE DIRECTIONS

• The review synthesizes control and machine learning approaches that safely use data to improve closed-loop robot-control performance.The paper identifies many opportunities for future research in safe learning control.Source: §5 · p. 25
• Machine learning, particularly reinforcement learning, can support generalization to broader classes of systems with fewer prior model assumptions.Source: §5 · p. 25
• Control theory provides frameworks for constraint-satisfaction and closed-loop-stability guarantees during learning.Source: §5 · p. 25

1. Capturing a Broader Class of Systems.

• Existing work has focused on nonlinear systems in the form of Equation 1.Source: §1 · p. 25
• Robots can exhibit hybrid dynamics, including legged-robot and contact dynamics with the environment (150).Source: §1 · p. 25
• Robots can also have time-varying dynamics in changing environments (151, 152) and delays in actuation, sensing, or reward observation (6).Source: §1 · p. 25
• Continuum robotics can involve partial differential dependencies in the dynamics (153), and extending safe learning control to these scenarios is essential for broader applicability.Source: §1 · p. 25

• Most safe learning control approaches assume direct access to possibly noisy state measurements and neglect state estimation.Source: §2 · p. 25
• Real-world state information is difficult to obtain when sensors provide indirect measurements, estimation models are inaccurate, or state features are improperly represented.Source: §2 · p. 25
• A key open challenge is accounting for state-estimation errors and learned process and observation models while extending methods to possibly high-dimensional sensor data (143).Source: §2 · p. 25

3. Considering Scalability, and Sampling and Computational Efficiency.

• Many reviewed approaches have only been demonstrated on small toy problems, and extending them to high-dimensional robotics is nontrivial.Real-world deployment also faces data sparsity, distribution shifts, and optimality–complexity trade-offs for real-time implementations.Source: §3 · p. 25
• Efficient robot learning depends on control architecture design (154), systematic training-data collection (155), and appropriate function-class selection (156).Source: §3 · p. 25
• Formal analysis of sampling and computational complexity is indispensable for implementing safe learning control algorithms in real-world robot applications.Current approaches focus on theoretical safety guarantees.Source: §3 · p. 25

4. Verifying System and Modeling Assumptions.

• Verifying assumptions such as known Lipschitz constants and bounded disturbances, and quantifying approximation impacts with minimal online data, is crucial for real-world deployment.Implementation approximations include linearization and assuming i.i.d. Gaussian samples; interpretability of black-box models such as DNNs also merits investigation for safe closed-loop operation (6).Source: §4 · p. 26
• Open questions include fair benchmarks and metrics, simulation’s role in offline design and safe hyperparameter selection, defining safety in human-robot interaction, and safe data use in multi-agent learning.Source: §4 · p. 26
• Combining control theory and machine learning has improved control performance and system safety, motivating further theory, methods, and computation for safe robotics.The stated aim is real-world safe autonomy in complex environments.Source: §4 · p. 26

Model and learning approach

• Methods are classified as model-based or model-free, with model components grouped as Gaussian processes, neural networks, or other structures.Model-based approaches use a dynamics model to produce control inputs; model-free approaches map states or measurements directly to controls.Source: A. SUMMARY OF REVIEWED LITERATURE · p. 34

Safety and robustness

• Safety properties cover constraint satisfaction and stability during learning or after learning, while robustness covers input disturbances and model or dynamics uncertainty.Dynamics uncertainty is distinguished from input disturbances and model parameter uncertainty.Source: A. SUMMARY OF REVIEWED LITERATURE · p. 34

Tasks

• The reviewed methods are applied to stabilization, trajectory tracking, navigation, legged-robot locomotion, and robotic-arm manipulation.Manipulation includes pushing, reaching, or grasping; navigation involves reaching a goal through a sequence of actions.Source: A. SUMMARY OF REVIEWED LITERATURE · p. 34

• The paper was initially submitted on August 13, 2021.Source: B. REVISION HISTORY · p. 36