PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Reinforcement Learning Using Robust MPC

Mario Zanon, Sébastien Gros

arXiv:1906.04005v2eess.SY

TL;DR

RL controllers have difficulty guaranteeing safety and stability, while MPC performance depends on model accuracy. The paper combines RL with robust MPC to learn an efficient uncertainty-set approximation while retaining constraint guarantees, and demonstrates the approach in simulations, including nonlinear-system limitations.

  • Problem

    RL controllers have difficulty guaranteeing safety and stability, and most existing approaches do not strictly guarantee that prescribed constraints are never violated.

  • Method

    The paper combines RL with robust MPC, using RL to optimize a low-complexity uncertainty-set approximation and potentially adapt the MPC feedback matrix.

  • Results

    The proposed controller is guaranteed to satisfy prescribed safety constraints under the robust MPC assumptions, and simulations show RL adapting the uncertainty-set approximation to reduce constraint tightening.

  • Takeaways & Limitations

    RL can be used within robust MPC to improve set adaptation while preserving safety guarantees for bounded uncertainty realizations.

  • Takeaways & Limitations

    In strongly nonlinear systems, linear MPC might fail to provide satisfactory performance and even safety.

Abstract

from arXiv · show

Reinforcement Learning (RL) has recently impressed the world with stunning results in various applications. While the potential of RL is now well-established, many critical aspects still need to be tackled, including safety and stability issues. These issues, while partially neglected by the RL community, are central to the control community which has been widely investigating them. Model Predictive Control (MPC) is one of the most successful control techniques because, among others, of its ability to provide such guarantees even for uncertain constrained systems. Since MPC is an optimization-based technique, optimality has also often been claimed. Unfortunately, the performance of MPC is highly dependent on the accuracy of the model used for predictions. In this paper, we propose to combine RL and MPC in order to exploit the advantages of both and, therefore, obtain a controller which is optimal and safe. We illustrate the results with a numerical example in simulations.

I. INTRODUCTION

RL offers powerful data-driven decision making, but safety and stability remain difficult to guarantee. The paper combines RL with robust MPC to improve performance while enforcing safety through constrained updates and uncertainty-aware predictions.

  • Motivation: RL learns policies or value functions from samples, but safety and stability are difficult to guarantee, especially with DNN approximators.Existing safety approaches may require risky real-system data, extensive simulations, or uncertainty models, and often do not strictly prevent constraint violations.
  • Proposed approach: Robust MPC is used as an RL function approximator to provide safety and stability guarantees, while RL tunes MPC parameters to improve closed-loop performance.The framework also addresses constraint satisfaction during exploration, not only exploitation.
  • Efficiency: The method combines robust MPC predictions with a low-complexity uncertainty representation to reduce stored-data requirements and computational complexity.A nominal linear model, convexity, and a low-dimensional uncertainty-set approximation are used for efficiency.
  • Proposed approach: The proposed safe RL formulation enforces safety constraints during parameter updates through a constrained optimization problem.The approach can be applied directly to Q-learning, while actor-critic methods require adaptation for continuous safety-restricted input spaces.
  • Safety formulation: Safety is defined through constraints that ideally hold with unit probability during both exploitation and exploration, but finite samples cannot generally identify a set containing all future outcomes.The paper therefore treats strict universal safety as unattainable without additional assumptions on the process.

III. ROBUST MPC BASED ON INVARIANT SETS

The robust MPC formulation uses an affine nominal model and tube-based constraint tightening to represent uncertainty while retaining a parameterized MPC function approximator for RL.

  • Model and uncertainty: The paper focuses on an affine model because robust constraint satisfaction is especially difficult for general nonlinear systems.Nonlinear effects are represented as perturbations in the uncertainty model.
  • MPC approximator: The Q-function approximator is formulated using classic robust linear MPC with predicted states, inputs, costs, constraints, and a terminal condition.The MPC predictions use a nominal model, while uncertainty is handled through tightened path constraints and terminal constraints.
  • Parameterization: RL may adapt MPC parameters including cost, model, feedback, constraint, and uncertainty-set parameters collected in θ.Positive-definite cost parameters are assumed to support solvability, while feedback and uncertainty parameters affect tightening and terminal-set construction.

A. Recursive Robust Constraint Satisfaction

Tube-based robust MPC converts uncertainty into tightened constraints and an invariant terminal set, yielding recursive feasibility and safety under the assumed uncertainty-set model.

  • Constraint tightening: Robust MPC represents stochasticity and model uncertainty as additive disturbances contained in Wθ around nominal predictions.The uncertainty set is parameterized as a polytope, while affine modeling introduces conservatism for nonlinear systems.
  • Constraint tightening: Path constraints are tightened by adding worst-case disturbance effects, yielding ck = ¯c + dk for all disturbances in Wθ.When Wθ is polyhedral, the tightening problems reduce to linear programs.
  • Terminal set: The terminal set is designed to be robustly invariant and output admissible under a terminal feedback law, preserving feasibility under future disturbances.The construction uses recursively defined sets and an MRPI set, with finite determination under stability and interior-feasibility conditions.
  • Terminal set: Terminal constraints must also be tightened because the actual terminal state differs from the nominal prediction through the terminal error.The paper defines terminal tightening analogously to path-constraint tightening.
  • Recursive robust constraint satisfaction: If the terminal set is RPI and the initial MPC problem is feasible, recursive feasibility holds for all disturbances in Wθ and the real system satisfies safety constraints at all times.The result assumes the uncertainty inclusion condition ˆS+(s, a, θ) ⊇ S+(s, a).

B. Differentiability

The paper differentiates the robust MPC approximator and its tightening procedures so RL can update parameters with sensitivity-based methods. Under standard regularity conditions, the resulting value, action-value, and policy mappings are differentiable.

  • Sensitivity computation: RL parameter adaptation requires derivatives of the robust MPC problem and its constraint definitions with respect to θ.The derivatives include sensitivities of path, terminal, and constraint-tightening quantities.
  • Differentiability: Under LICQ, SOSC, and strict complementarity, the MPC value function, action-value function, and optimal solution are differentiable with respect to θ.The optimal solution determines the policy, so policy sensitivities are available under the same conditions.
  • Computational efficiency: Sensitivity evaluation is generally cheaper than solving the optimization problem after the KKT matrix has been factorized.The factorization can be reused for linear-system solves and value-function derivatives.
  • Computational efficiency: Constraint-tightening sensitivity calculations are separable across prediction times and constraint components, enabling parallel computation.Low-dimensional subproblems and reusable factorizations further improve efficiency.

C. Guaranteeing MPC Feasibility and LICQ

The paper addresses MPC infeasibility and sensitivity failures in RL-based MPC by relaxing selected constraints with an exact penalty, preserving feasible solutions while guaranteeing recursive feasibility and LICQ.

  • MPC Feasibility: MPC feasibility can fail when a newly observed disturbance lies outside the estimated uncertainty set, causing the controller to stop computing actions.The uncertainty set must then be updated, potentially losing recursive feasibility.
  • Sensitivity Computation: Sensitivity computation is valid only under LICQ, but the MPC problem is not guaranteed to satisfy LICQ.
  • Constraint Relaxation: A common exact-penalty relaxation introduces variables σk for path and terminal constraints and adds ρ⊤σk to the objective.Control-only constraints need not be relaxed, and state constraints should not be imposed at k = 0.
  • Guarantees: For sufficiently large finite ρ, the relaxed problem preserves the original solution whenever feasible while guaranteeing recursive feasibility and LICQ.
  • Safety Boundary: The relaxation addresses MPC infeasibility but does not remove the requirement that the uncertainty model correctly capture all possible state transitions for safety.

IV. SAFE DESIGN CONSTRAINT AND DATA MANAGEMENT

Safety is enforced through a sample-based design constraint requiring the learned uncertainty model to represent observed transitions, while data volume creates a real-time computational challenge.

  • Safe Design Constraint: Safety requires correctly identifying Wθ and gθ so the MPC formulation captures all possible state transitions.The sample-based SDC is introduced for use within RL formulations.
  • Data Management: High-rate operation produces large transition datasets, motivating retention of relevant data and compression into efficient data structures.

A. Set Membership and SDC

The paper defines set-membership constraints that make observed transitions belong to the learned dispersion set, then selects uncertainty-set parameters for closed-loop task performance rather than minimum volume alone.

  • Set Membership: The observed transition dataset D consists of tuples (s1, a1, s2), …, (sn, an, sn+1).
  • Set Membership: The SDC requires θ to satisfy gθ(sk+1, sk, uk) ≤0 for every transition in D.
  • Set Membership: SD contains parameters whose predicted dispersion sets include all transitions in D, whereas Ŝ+ is the outer approximation of possible next states.
  • Task-Oriented Identification: For a specific control task, θ may favor accurate approximation of task-relevant regions even if that increases the volume of Ŝ+.
  • Task-Oriented Identification: Set Membership Optimality defines θ as locally optimal when no nearby parameter yields lower closed-loop cost J(πθ).
  • Task-Oriented Identification: RL seeks the optimal set approximation by selecting θ to maximize the learned policy's closed-loop performance subject to the SDC.
  • Affine Formulation: Assuming affine gθ makes the SDC compatible with an affine model, with parameters A, B, b, M, and m potentially adapted by RL.

B. Model-Based Data Compression

The nominal model converts transition data into disturbance samples, reducing representation dimension and enabling efficient constraint checks; convex-hull storage is proved optimal, though model updates complicate maintenance.

  • Model-Based Compression: The nominal model restructures data into noise samples W, reducing the dataset representation dimension from 2ns+na to ns under state-action-independent uncertainty.
  • Model-Based Compression: The SDC becomes SD = SW := {θ | Mw ≤m, ∀w ∈W}, directly defining the uncertainty set used by MPC.
  • Model-Based Compression: The resulting dispersion approximation represents next states as As + Ba + b + w for disturbances satisfying Mw ≤m.
  • Efficient Computation: Constraint-tightening computations become linear programs, while membership checks for new states require one inequality evaluation and uncertainty validation checks each data point.
  • Convex-Hull Compression: The convex hull retains all necessary information because interior samples add no information to the uncertainty constraints.
  • Convex-Hull Compression: Theorem 1 states that the convex hull of W is an optimal data compression for the selected dispersion-set parametrization and MPC formulation.
  • Convex-Hull Compression: Each new sample is tested by an LP and added only when it lies outside the current convex hull, whose test has linear complexity in the number of vertices.
  • Limitations: Convex-hull size can still grow indefinitely, and updating A or B requires recomputing noise vectors for all recorded state-action pairs.Updates to b instead shift the noise set without state-action dependence.

C. Further Observations on the Sample-Based SDC

The sample-based uncertainty set trades exact data coverage against computational complexity. The paper therefore uses reinforcement learning to select a fixed, low-complexity representation while handling newly observed samples and feasibility effects.

  • Sample-based uncertainty representation: The convex hull of observed samples is volume- and cost-optimal but may contain too many facets for practical MPC constraint tightening.Its high-dimensional terminal constraint also increases computational cost.
  • Learning the uncertainty set: A fixed, low-complexity set W_θ is preferred in practice, motivating reinforcement learning to enforce set-membership optimality while reducing conservatism.This addresses the computational burden of the full convex hull.
  • Sample-based uncertainty representation: Uncertainty sets can use facet or vertex representations, parameterized respectively by inequalities or the polytope’s vertices.The facet form uses θ_W = (M, m), while the vertex form uses the listed vertices.
  • Sample-based uncertainty representation: The convex hull uses vertices because construction and incremental updating are simpler and less computationally demanding, despite costlier membership evaluation.The vertex form also simplifies enforcing the sample-dependent constraint.
  • Open representation question: The most convenient convex-hull representation remains unresolved, with future work considering combinations of facet and vertex forms.The paper identifies representation choice as an open research question.
  • Updating with new samples: New samples may fall outside the previous convex hull, requiring immediate adaptation of the sample-dependent constraint.Enlarging the uncertainty set further tightens constraints and can jeopardize recursive MPC feasibility.

V. SAFE RL MPC IMPLEMENTATION

The implementation makes reinforcement-learning updates subject to robust-MPC formulation, feasibility, and safety constraints. It also addresses nonsmooth sensitivities caused by active-set changes in constraint tightening.

  • Safe parameter updates: Safe RL updates the parameter θ through a constrained optimization problem rather than an unconstrained recursive gradient step.The constraint enforces the safety-related sample-dependent condition during parameter updates.
  • Safe parameter updates: The RL formulation includes positive-definite constraints for tractable MPC and imposes the sample-dependent constraint on the updated parameters.If model parameters A and B also change, the admissible parameter set must be modified accordingly.
  • Feasibility preservation: Feasibility can be preserved by updating only when feasible, reducing the step size, or imposing feasibility as an additional optimization constraint.Each strategy is presented as guaranteeing feasibility of the updated robust MPC scheme.
  • Nonsmooth sensitivities: Constraint-tightening sensitivities can be discontinuous at weakly active constraints, making the RL objective’s first-order sensitivities nonsmooth.The paper states that the problematic set has zero measure, so the RL solution is unaffected in principle.
  • Safe exploration: The safety concern extends to exploration, where perturbed actions must preserve constraint satisfaction rather than merely following the optimal policy.The exploration mechanism modifies the MPC cost while retaining the robust-MPC structure.

A. Safe Exploration

Safe exploration modifies the robust-MPC objective to select exploratory actions while retaining feasibility. Under the stated data and convergence assumptions, the resulting RL-MPC scheme is safe and optimal in the paper’s formal senses.

  • Safe exploration: Exploration perturbs the optimal policy, so actions must be selected from the feasible-action set without jeopardizing constraint satisfaction.The feasible-action set is implicitly approximated within robust MPC.
  • Safe exploration: The exploratory MPC objective adds either a distance penalty ρ∥u_0−q∥ or a linear term q^⊤u_0, with q chosen randomly.These modifications encourage exploration while preserving the robust-MPC problem structure.
  • Algorithmic flow: The implementation obtains exploratory actions from modified MPC and otherwise obtains the policy action from the standard MPC problem.The algorithm then observes the transition, updates the uncertainty set, and may perform an RL step.
  • Algorithmic flow: Parameter updates trigger recomputation of constraint tightening only when the updated parameters remain feasible.This links safe learning updates to the robust-MPC feasibility requirements.
  • Guarantees and limitations: Constrained exploration in policy-gradient methods can bias gradient estimates, requiring additional strategies to address the issue.This is identified as a limitation of the exploration procedure for those methods.
  • Guarantees and limitations: Theorem 2 bounds the probability of any constraint violation by 1−η(D) and asserts optimality under convergent RL, robust MPC, data handling, and exploration assumptions.The safety argument relies on new samples remaining inside the convex hull of prior samples.
  • Guarantees and limitations: Actor-critic methods may support set-membership optimality under convergence, whereas Q-learning may reasonably remain suboptimal.The distinction concerns convergence to a local minimum of J for the given parametrization.
  • Computational considerations: Batch RL can reduce how often the RL problem and constraint tightening are recomputed, leaving real-time requirements primarily to MPC.The reported Matlab implementation was only partially optimized, and constraint tightening dominated computation.

B. Discussion on the Proposed Approach

The discussion identifies limitations and open questions concerning safety assumptions, approximation quality, model adaptation, stability, and simulation design. It also reports simulations showing that RL can adapt uncertainty and feedback parameters to improve the robust MPC formulation.

  • 1) Safety: Safety relies on all future disturbances belonging to the assumed set, an assumption that can fail for unknown stochastic processes.When samples leave the convex hull, feasibility may be retained but safety can be temporarily lost.
  • 2) Approximation Quality: Exact recovery of value, action-value, and policy functions holds only on the feasible subset representable by the chosen noise-set parametrization.This is narrower than the corresponding result for a sufficiently rich MPC parametrization without approximate noise-set representation.
  • 2) Approximation Quality: Policy-gradient methods are attractive when the parametrization is imperfect, but exploration can bias gradients and hinder convergence to a local optimum.Q-learning and SARSA fit the action-value function rather than directly optimizing performance for the selected parametrization.
  • 3) Model Adaptation: Adapting the feedback matrix K is computationally manageable and may reduce constraint tightening for constraint components active during task execution.Adapting model parameters A, B, and b instead causes a large increase in computations, while choosing task-optimal K remains open.
  • 3) Model Adaptation: Adapting the model may improve approximation quality, but whether and how to do so advantageously remains unresolved because practical parametrizations are typically low-dimensional.Alternative fixed-safety and predictive-model formulations are proposed as future research directions.
  • A. Linear System: Simulations on a linear system and a nonlinear evaporation process illustrate the framework, while the linear example shows RL adapting uncertainty and feedback representations.The uncertainty hull has 28 facets, whereas the learned approximation uses 4; adapting K enlarges the RPI and terminal sets while reducing tightening.

B. Evaporation Process

The evaporation-process example applies robust linear MPC with RL-adjusted uncertainty and feedback parameters, illustrating how learning shifts safety sets toward the reference while retaining the paper’s safety scope.

  • Problem setup: The nonlinear evaporation process uses states X2 and P2, controls P100 and F200, and stochastic disturbances in X1, F1, T1, and T200.State and control bounds are imposed, with the process represented through a linearization around a nominal steady state for robust MPC.
  • Method: RL adjusts the cost gradient, uncertainty-set approximation Wθ, and feedback matrix K within the linearized robust MPC formulation.The LQR feedback K stabilizes model error when computing constraint tightening and the terminal set.
  • Learning effects: A better Wθ approximation moves the RPI and terminal sets closer to the setpoint for the specific control task.Figure 3 also reports stronger constraint tightening near the reference at k = 36 than afterwards.
  • Learning effects: 104 Q-learning samples lead to parameter stabilization toward the end of learning, while Wθ expands to better approximate the uncertainty set.Together with feedback-matrix adjustment, this shifts the terminal set toward the reference.
  • Conclusion and limitation: The paper reports strict constraint satisfaction given available data, while warning that strong nonlinearities may cause linear MPC to provide unsatisfactory performance or safety.The evaporation example demonstrates linear MPC applied to a nonlinear system, but the conclusion limits this success under strong nonlinearities.
  • Future work: Scenario-tree robust MPC, policy-gradient methods, and computational-geometry formulations are identified as future extensions.These directions are presented as ongoing research rather than results established by the evaporation simulation.
Loading 1906.04005v2…