Source-linked AI summary
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, Angela P. Schoellig
TL;DR
Safe learning in robotics addresses how robots can make decisions under uncertainty while respecting safety constraints, especially in real-world settings. The paper reviews learning-based control and reinforcement-learning methods through a unified perspective, covering uncertainty learning, safety-oriented RL, and policy certification. It concludes that RL broadens generalization while control theory supplies constraint and stability guarantees, alongside a need for more realistic benchmarks.
Problem
Robots need to use learning for decision making under uncertainty while maintaining safety during learning and operation, particularly in applications near humans.
Method
The paper provides a holistic review of safe learning control, covering learning uncertain dynamics, safety-oriented reinforcement learning, and formal certification while connecting control and RL frameworks.
Results
The review concludes that reinforcement learning can generalize across larger classes of systems with fewer prior model assumptions, while control theory provides constraint-satisfaction and closed-loop stability guarantees during learning.
Takeaways & Limitations
Safe robot learning benefits from combining data-driven adaptation with control-theoretic safety guarantees, and requires realistic physics-based benchmarks for fair algorithm comparisons.
Takeaways & Limitations
Most approaches assume direct access to possibly noisy state measurements and do not yet account adequately for state-estimation errors or high-dimensional sensor data.
Abstract
from arXiv · showhide
The last half-decade has seen a steep rise in the number of contributions on safe learning methods for real-world robotic deployments from both the control and reinforcement learning communities. This article provides a concise but holistic review of the recent advances made in using machine learning to achieve safe decision making under uncertainties, with a focus on unifying the language and frameworks used in control theory and reinforcement learning research. Our review includes: learning-based control approaches that safely improve performance by learning the uncertain dynamics, reinforcement learning approaches that encourage safety or robustness, and methods that can formally certify the safety of a learned control policy. As data- and learning-based robot control methods continue to gain traction, researchers must understand when and how to best leverage them in real-world scenarios where safety is imperative, such as when operating in close proximity to humans. We highlight some of the open challenges that will drive the field of robot learning in the coming years, and emphasize the need for realistic physics-based benchmarks to facilitate fair comparisons between control and reinforcement learning approaches.
1. INTRODUCTION
Robotics increasingly uses learning to make decisions under uncertain, partially known conditions, but safe deployment requires guarantees during both learning and operation. This review unifies control-theoretic and reinforcement-learning perspectives on combining model-driven guarantees with data-driven adaptability.
- Motivation: Robots operate in complex scenarios near humans despite uncertain dynamics, noisy sensors, poorly characterized environments, and unknown other agents.Examples include self-driving vehicles, aerial delivery, and mobile manipulators.
- Safety challenge: Safe robot learning must formally guarantee behavior during policy optimization and learning, not only after deployment.Such guarantees help avoid costly hardware failures and improve convergence, but depend on the assumptions and structure of the problem formalization.
- Unifying perspectives: Control theory traditionally provides guarantees from dynamics models, whereas reinforcement learning adapts to new contexts through data but generally lacks formal guarantees.Combining both approaches is presented as a promising direction for safe learning in robotics.
- Research landscape: Safe RL research has grown from tens to over a thousand publications since its most recent review, while transfer from physics-based simulation to real systems remains unresolved.Physics-based simulation has contributed to recent RL progress, but real-system transfer remains a research area in itself.
- Review scope: Earlier reviews addressed specific learning and control techniques or treated control-theoretic and RL safety separately without providing a unifying perspective.This review instead spans learning-based control and reinforcement learning with an emphasis on safety.
2. PRELIMINARIES AND BACKGROUND OF SAFE LEARNING CONTROL
Safe learning control formulates robot decision-making under uncertainty around a task cost, dynamics model, and safety constraints, while integrating data with prior knowledge. The review distinguishes hard, probabilistic, and soft safety requirements and discusses limitations of robust control and reinforcement learning during learning and scaling.
- Safe learning control seeks a controller or policy that fulfils a task while respecting safety constraints despite uncertainty.The formulation includes a system model, cost function, and constraint functions.
- Safety requirements range from encouraged constraint satisfaction to probabilistic satisfaction and guaranteed hard-constraint satisfaction.The three levels are often mixed in practice; for example, input constraints may be hard while state constraints are soft.
- Robust control and robust MPC can provide worst-case stability and constraint guarantees but may produce poor performance because conservative uncertainty sets tighten constraints.Learning can reduce conservatism by estimating uncertainty sets or unknown dynamics.
- Reinforcement learning faces an exploration-exploitation dilemma because exploratory actions may be sub-optimal or unsafe while improving the dynamics estimate.This challenge affects safety during the learning stages.
- Constrained reinforcement learning remains computationally challenging for trajectory-level constraint evaluation, while robust MDPs may be difficult to apply beyond toy problems.Recent work addresses deep-learning integration, provable exploration safety, efficient off-policy constraint evaluation, and tractable robust optimization.
- The reviewed problem can involve online adaptation during closed-loop operation or offline batch updates between trials.Data may update controller parameters, dynamics, costs, or constraints.
3. SAFE LEARNING CONTROL APPROACHES
The review organizes safe learning control approaches by how they use data and the safety guarantees they provide. It distinguishes learning uncertain dynamics, encouraging safety in reinforcement learning, and certifying learning-based control under uncertainty.
- Learning uncertain dynamics uses an a priori robot model and improves performance while typically providing Safety Level II or III guarantees.
- Safety-oriented reinforcement learning generally lacks an a priori robot model or explicit safety constraints and encourages safe operation rather than providing hard guarantees.Penalizing dangerous actions is one example.
- Certifying learning-based control under dynamics uncertainty forms the third reviewed category.
- The approaches are categorized by safety level and reliance on data, with a detailed summary provided in Appendix Table 1.Figure 4 presents this categorization.
3.1. Learning Uncertain Dynamics to Safely Improve Performance
Learning-based control safely improves performance by learning uncertain dynamics within control frameworks that retain stability or constraint guarantees. Approaches range from adaptive learning of nonparametric dynamics to robust MPC that reduces conservatism through data.
- Integrating Machine Learning and Adaptive Control: Adaptive control integrates machine learning through black-box models, probabilistic uncertainty weighting, or deep architectures that memorize experience.These strategies address nonparametric dynamics, cautious adaptation, and reduced need for re-adaptation.
- Integrating Machine Learning and Adaptive Control: Learning-based MRAC combines L1 adaptation with neural networks or Gaussian processes to approximate unknown nonlinear dynamics while retaining fast adaptation and stability guarantees.The learned dynamics modify a controller designed for the nominal model.
- Learning-Based Robust Control: Learning-based robust control uses data to improve dynamics estimates, reduce uncertainty sets, or update costs, thereby improving classical robust-control performance.Robust MPC variants can learn unknown dynamics or costs while maintaining state and input constraints.
- Learning-Based Robust Control: Some GP-based robust-control approaches remain limited to stabilization tasks and do not account for state or input constraints.This bounds their applicability beyond stabilization settings.
- Reducing Conservatism in Robust MPC with Learning and Adaptation: Robust adaptive MPC improves standard robust MPC by reducing parameter uncertainty over time or combining an adaptive inner loop with a robust MPC outer loop while satisfying hard constraints.Tube-based MPC shrinks its uncertainty tube as the estimated parameter set becomes less conservative.
- Reducing Conservatism in Robust MPC with Learning and Adaptation: Nonlinear learning-based robust MPC commonly uses Gaussian-process means for model updates and covariance-derived uncertainty sets, but horizon approximations can violate probabilistic constraints.Real-time implementation may require sigma-point transforms, linearization, moment matching, ellipsoidal propagation, or fixed covariances.
3.2. Encouraging Safety and Robustness in Reinforcement Learning
Safety-augmented RL methods encourage constraint satisfaction or robustness during and after learning, generally without strict guarantees or an a priori nominal model. They address unsafe exploration, safe optimization, learned safety critics, risk estimation, and constrained policy updates.
- Overview: Safety-augmented RL modifies standard MDP or RL formulations to encourage constraint satisfaction or policy robustness, typically at Safety Level I rather than through strict guarantees.Some methods also learn rewards, costs, or safety constraints.
- Safe Exploration: Exploration threatens safety because learning may require selecting suboptimal actions with unpredictable or possibly unsafe consequences.This is the central exploration–exploitation tension in RL for robotics.
- Safe Exploration: Safe exploration restricts policy updates to preserve ergodicity, enabling return to arbitrary states, but exact solution is NP-hard and heuristic approximations can be suboptimal.The simplified approach produces safe exploration over only a subset of eligible policies.
- Safe Optimization: Safe optimization methods use Gaussian processes to sample inputs below safety thresholds while expanding safe regions or optimizing an unknown function.SafeOpt, SafeOpt-MC, StageOpt, GoSafe, SafeMDP, and SafeExpOpt-MDP instantiate this framework.
- Learning a Safety Critic: Safety critics learn action-value functions that predict failure risk and filter unsafe actions or select fallback inputs, but provide only Safety Level I guarantees.SQRL transfers safety knowledge across similar tasks, while Recovery RL additionally learns a recovery policy.
- Learning a Safety Critic: Conservative Safety Critic upper-bounds failure probability and ensures provably safe policy improvement at each iteration.This extends conservative value-learning ideas to safety constraints.
- Risk-Aware Learning: Risk-aware methods estimate collision or trajectory risk and use those estimates to produce more conservative policies, including collision-aware MPC and cautious adaptation.CARL pretrains across varied dynamics before risk-averse fine-tuning on the target system.
- Constrained MDPs and RL: Primal-dual RL updates with state-action surrogates can guarantee monotonic policy improvement and near constraint satisfaction at each iteration.Solving dual variables from scratch at every update can be computationally expensive.
3.3. Certifying Learning-Based Control Under Dynamics Uncertainty
Certification methods add formal or probabilistic safety guarantees to learning-based controllers under dynamics uncertainty. They certify closed-loop stability or safe constraint sets using Lipschitz bounds, regions of attraction, barrier and Lyapunov functions, Hamilton–Jacobi reachability, and predictive safety filters.
- Overview: The certification framework separates stability certification from constraint-set certification and relies on an a priori dynamics model to provide hard or probabilistic guarantees.These methods target Safety Levels II or III under dynamics uncertainty.
- Stability Certification: Lipschitz-based methods constrain neural-network policies so small-gain analysis or semidefinite programming can establish closed-loop stability.The certification depends on an upper bound for the policy’s Lipschitz constant.
- Stability Certification: Region-of-attraction methods certify states from which a learning-based controller is guaranteed to converge to a safe equilibrium.Lyapunov neural networks provide more flexible representations than typical quadratic Lyapunov functions.
- Constraint Set Certification: CBF and CLF approaches encode safe-set invariance and asymptotic stability through derivative constraints, while learning improves estimates of derivatives for unknown dynamics.Learning errors in those derivatives can still produce falsely certified constraints.
- Constraint Set Certification: Hamilton–Jacobi reachability computes robust positive control invariant safe sets under bounded disturbances and can provide a safety filter for learning-based control.The filter permits learning inside the safe set and switches to an optimally safe policy at its boundary.
- Constraint Set Certification: Predictive safety filters enforce state and input constraints by requiring a safe input sequence that returns the system to a terminal safe set or previously visited safe states.Because the filter and learning controller are usually decoupled, the resulting behavior can be suboptimal.
4. BENCHMARKS
The benchmarks compare safe learning approaches across cart-pole and quadrotor tasks, while highlighting substantial variation in evaluation realism and reproducibility. The open-source suite supports fairer comparisons between control and reinforcement-learning methods.
- Evaluation landscape: Evaluations differ substantially: uncertain-dynamics methods mostly use abstract numerical examples, safety-oriented RL mostly uses simulations, and certification methods include the largest share of real-world experiments.Many RL environments remain deterministic and do not represent existing robotic platforms or environmental variation.
- Open evaluation needs: The authors identify realistic physics-based environments, open implementations, and comparison metrics such as data requirements as needed for fairer evaluation.They also plan to use the cart-pole and quadrotor benchmarks to evaluate robustness, performance, safety, and data efficiency.
- Benchmark design: The benchmark suite simulates cart-pole and quadrotor platforms using Bullet and OpenAI’s gym interface for integration with control and RL libraries.It supports randomized inertial properties and initial conditions, disturbances, state and input constraints, and symbolic nominal models through CasADi.
- Benchmark evaluation: The benchmark evaluates approaches from uncertain-dynamics learning, safety-oriented RL, and certified learning-based control without claiming superiority of one approach.The experiments target constrained stabilization and examine control performance alongside constraint satisfaction.
- Safety-oriented RL: PPO experiments compare standard PPO, cost shaping, and a safety layer on constrained cart-pole stabilization after more than 9 hours of simulation-based data collection per approach.The task terminates after any constraint violation, and Figure 7 reports medians and quantiles over 10 seeds.
- Certified learning-based control: MPSC modifies PPO inputs near constraint boundaries, keeping the cart-pole within those boundaries while uncertified PPO violates them.This provides a proof of concept for combining safety filters with RL control.
5. DISCUSSION AND PERSPECTIVES FOR FUTURE DIRECTIONS
Safe learning control is presented as an emerging topic for robotics. The review combines control and machine-learning perspectives to explain how data can improve closed-loop performance while preserving safety and stability guarantees.
- Machine learning, particularly reinforcement learning, can generalize toward broader system classes with fewer prior model assumptions.
- Control theory provides frameworks for constraint-satisfaction and closed-loop-stability guarantees during learning.
- The review concludes that safe learning control still offers many opportunities for future research.
1. Capturing a Broader Class of Systems.
Existing safe learning control work largely focuses on nonlinear systems, leaving broader dynamical structures as an open direction. The review calls for extensions to hybrid, time-varying, delayed, and partial-differential dynamics.
- Most existing work addresses nonlinear systems, while robotic platforms may also exhibit hybrid, time-varying, delayed, or partial-differential dynamics.
- Expanding safe learning control to these broader dynamical classes is identified as future work.
2. Accounting for Imperfect State Measurements.
Most safe learning control approaches assume direct access to noisy state measurements and neglect state estimation. Broad robotic applicability therefore requires handling estimation errors, learned models, and high-dimensional sensor data.
- Most safe learning control approaches assume direct access to possibly noisy state measurements and neglect state estimation.
- Accurate state information is difficult when sensors provide images, process and observation models are inaccurate, or state features are poorly represented.
- A key open challenge is incorporating state-estimation errors and learned process and observation models into safe learning control.
- Supporting possibly high-dimensional sensor data is essential for broad applicability in robotics.
3. Considering Scalability, and Sampling and Computational Efficiency.
Scaling safe learning methods to high-dimensional robotics remains difficult because many approaches have only been demonstrated on small toy problems. Practical deployment also faces data sparsity, distribution shifts, and real-time optimality–complexity trade-offs.
- Many reviewed approaches have only been demonstrated on small toy problems, making application to high-dimensional robotics nontrivial.The passage identifies scalability as a central deployment challenge.
- Practical robot learning must address data sparsity and distribution shifts.
- Real-time implementations require managing the optimality–complexity trade-off.
- Efficient robot learning depends on control architecture design, systematic training-data collection, and appropriate function-class selection.
4. Verifying System and Modeling Assumptions.
Safety guarantees depend on assumptions about dynamics, disturbances, and data that may be difficult to verify before operation. The review therefore identifies systematic assumption verification, impact quantification, and benchmarking as open needs.
- Safety guarantees often assume Lipschitz-continuous true dynamics with a known constant or bounded disturbance sets.
- Verifying these modeling assumptions before a robot operates is difficult.
- Implementation commonly introduces approximations such as linearization and i.i.d. Gaussian data assumptions.
- Open questions include benchmarks, evaluation metrics, simulation’s role, human–robot safety definitions, and safe data use in multi-agent learning.
- Combining control theory and machine learning has been shown to improve control performance and system safety, motivating further theory and computational methods.
LITERATURE CITED
The cited literature spans foundational works and recent studies on control, reinforcement learning, adaptive and robust methods, and safe learning for robotics. It includes books, surveys, conference papers, journal articles, and preprints.
- Applications represented in the cited works include autonomous driving, aerial delivery, and mobile-manipulator service tasks.
- The references include foundational books on computer-controlled systems, adaptive control, robust control, nonlinear systems, and reinforcement learning.
- The bibliography covers surveys and reviews of model learning for robot control, iterative learning control, deep reinforcement learning, Bayesian reinforcement learning, and safe online learning.
- The references also include constrained, robust, adversarial, and risk-averse reinforcement-learning approaches.
- Recent cited work addresses safe and robust learning control using Gaussian processes, adaptive control, model predictive control, and reinforcement learning.
A. SUMMARY OF REVIEWED LITERATURE
The review classifies safe learning control methods by modeling approach, safety properties, robustness, and task type. It distinguishes model-based from model-free methods and records whether safety is enforced during learning or only by the resulting policy.
- Methods are classified as model-based when a dynamics model produces control inputs and model-free when states or measurements map directly to inputs.
- Model-based methods are further categorized by whether their a priori and learned model components use Gaussian processes, neural networks, or other structures.
- The safety-properties category records constraint satisfaction and stability guarantees during training and after learning.
- Robustness is assessed against input disturbances, model-parameter uncertainty, and other dynamics-related uncertainties.
- The task categories include stabilization, tracking, navigation, locomotion, and manipulation.
B. REVISION HISTORY
The paper's revision history records an initial submission on August 13, 2021.
- August 13, 2021 marks the paper's initial submission.