PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-driven Economic NMPC using Reinforcement Learning

Sébastien Gros, Mario Zanon

arXiv:1904.04152v1eess.SY

TL;DR

RL provides data-driven control without a system model but lacks hard behavioral guarantees, whereas (E)NMPC has a mature closed-loop theory but depends on model quality. The paper adapts ENMPC costs and constraints to use it as an RL function approximator, showing that an inexact-model scheme can recover the real system’s optimal policy and supporting RL-based tuning of its parameters.

  • Problem

    RL lacks hard guarantees for approximate policies, while (E)NMPC performance depends on having a good model of the controlled system.

  • Method

    The paper parameterizes value functions and policies with (E)NMPC and adapts stage costs, terminal costs, constraints, and parameters using classic RL techniques.

  • Results

    Under stated conditions, an (E)NMPC scheme using an incorrect model can deliver the real system’s optimal policy, including for stochastic systems.

  • Takeaways & Limitations

    ENMPC can serve as an RL function approximator while retaining access to the established theory underlying (E)NMPC schemes and behavioral certificates.

Abstract

from arXiv · show

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature.

I. INTRODUCTION

RL enables model-free data-driven control but offers limited formal guarantees, while NMPC provides established closed-loop analysis yet depends on model accuracy. The paper proposes adapting costs and constraints so (E)NMPC can recover the real system’s optimal policy despite model mismatch, including stochastic dynamics.

  • RL motivation: RL commonly uses generic function approximators such as DNNs to learn policies or action-value functions, but their closed-loop behavior can be difficult to analyze formally.This complicates certificates, especially for safety-critical systems.
  • NMPC motivation: NMPC repeatedly solves a constrained finite-horizon optimal control problem using a system model, applies the first input, and replans from the next observed state.With a perfect model and adequate terminal cost, classic NMPC delivers the optimal policy; ENMPC permits a generic stage cost.
  • Problem: (E)NMPC performance relies on model quality, and fitting its model to real-system data does not necessarily recover the optimal policy and can even be counterproductive.The wrong-model optimization problem is also known in Real-Time Optimization, where the Modifier Approach adapts the cost rather than the model.
  • Contribution: The paper uses (E)NMPC instead of DNNs to parameterize value functions and policies, adapting stage cost, terminal cost, and constraints while leaving the model incorrect.The proposal applies to classic, economic, robust, and stochastic NMPC schemes and connects cost adaptation with strict dissipativity.
  • Practical implications: The framework enables classic RL techniques to tune NMPC parameters on the real system and can exploit ENMPC theory to provide behavioral certificates.The paper also states that a positive stage cost can be used when the optimal policy is stabilizing, while finite-horizon terminal effects motivate an additional condition.
  • Main result: Under stated conditions, a modified stage and terminal cost makes an inexact-model finite-horizon NMPC problem generate the true optimal policy and value functions.The reported identities include ˆV_N(s)=V⋆(s), ˆπ(s)=π⋆(s), and ˆQ_N(s,a)=Q⋆(s,a) for inputs with finite expected successor value.

III. ENMPC AS A FUNCTION APPROXIMATOR

The paper uses parametrized ENMPC schemes to approximate optimal policies and value functions despite inaccurate models, with RL adjusting their parameters toward better performance.

  • ENMPC schemes are parametrized to approximate the optimal policy and value functions even when their underlying model is inaccurate.The parametrization includes model, constraints, stage cost, terminal cost, and an additional cost term.
  • The scheme can represent a classic ENMPC formulation when γ = 1 and λθ = 0.
  • The proposed formulation uses relaxed mixed constraints and can become identical to the unrelaxed problem when the relaxation is exact.This occurs for sufficiently large relaxation weights when a feasible trajectory exists.
  • The parametrization satisfies the fundamental equalities underlying the Bellman equations.
  • With a rich enough parametrization and exact relaxation, the ENMPC scheme delivers π⋆, V⋆, and Q⋆ under the stated assumption.The result applies for states satisfying assumption (9).
  • Exact representation is generally impractical, so the paper uses limited parametrizations and RL to adjust θ without computing the modified stage cost directly.The modified cost may require knowledge of the real stochastic transition and can be extremely demanding to evaluate.

A. Robust NMPC Using Scenario Trees

Robust NMPC implemented with scenario trees fits the proposed framework by representing scenario dynamics as a stochastic process with discrete probabilities.

  • A scenario tree is represented as a stochastic process with a discrete probability distribution over its dynamics.
  • The scenario-tree dynamics use multiple models f1,...,fNs with associated probabilities W1,...,WNs satisfying Wi ≥ 0 and ΣWi = 1.
  • The framework treats the scenario probabilities as parameters that can be adjusted in the (E)NMPC scheme.

B. Model Parametrization

The ENMPC scheme can in principle capture the optimal policy without adapting its model, although model adaptation may provide additional approximation freedom.

  • The ENMPC scheme can in principle capture π⋆ without adjusting the model parametrization fθ.
  • Allowing model adaptation gives the NMPC scheme additional freedom to approximate π⋆ more closely.The paper identifies the interplay between model, cost, and constraint adaptation as ongoing research.

IV. REINFORCEMENT-LEARNING FOR ENMPC

The paper deploys classical RL to tune ENMPC parameters toward the optimal policy, using sensitivities obtained from the ENMPC optimization problem and its KKT conditions.

  • RL adjusts ENMPC parameters θ so that the resulting policy approximates the optimal policy as closely as possible.
  • The paper computes sensitivities of Qθ, Vθ, and πθ to support classical RL parameter updates.
  • The Lagrange formulation uses primal-dual variables and multipliers associated with the ENMPC constraints.
  • For ζ = 0, the augmented Lagrange function reduces to the Lagrange function associated with the NMPC problem.
  • The value-function gradients can be built as byproducts of solving the ENMPC problem.
  • The sensitivity formulas require regularity conditions, including no weakly active inequality constraints, linear independence, and second-order sufficiency.Interior-point methods are suggested to address weakly active constraints in practice.
  • TD-learning is applied to the ENMPC scheme in both on-policy and off-policy forms.

B. Q-learning for (E)NMPC

The paper adapts Q-learning and policy-gradient ideas to parametrized (E)NMPC schemes, while addressing finite-value requirements and practical safety limitations.

  • Q-learning updates NMPC parameters using temporal-difference errors, with actions selected by the current NMPC policy and optional random exploration.
  • Batch updates can learn alternative NMPC parameters off-policy while deploying the original parameterized scheme on the real system.
  • RL parameter updates have no guarantee of reaching the global optimum, although they improve closed-loop performance in practice over initial parameters.
  • Constraint relaxation keeps value functions finite after violations, so temporal-difference and policy updates remain well-defined.
  • Safety-critical constraint handling remains a limitation: crucial violations may be unacceptable, motivating robust formulations and further research.
  • Deterministic policy-gradient methods use an approximate action-value function within actor-critic updates, but computationally efficient ENMPC parametrizations remain future work.

V. RL AND STABLE ECONOMIC NMPC

This section develops a generalized cost modification for stable economic NMPC: it preserves the optimal policy while enabling positive-definite stage costs and value-function learning.

  • Economic NMPC permits generic stage costs, so its value function may not be positive-definite and stability proofs become challenging.
  • A cost modification replaces an indefinite stage cost with a positive-definite one while recovering the economic value and action-value functions.
  • The proposed modification generalizes standard cost rotation, preserves the optimal policy, and supports learning optimal value and action-value functions.
  • Choosing a state-only modification Λ(s,a)=λ(s) is simpler, but constructing a modification satisfying the required condition generally needs knowledge of the optimal policy.
  • Theorem 2 states that the modification preserves the optimal policy for the wrong model and original stage cost while transforming the associated value functions.
  • Stability guarantees discussed here concern the model used for NMPC predictions; guarantees for the real process are left to future work.

B. Strict Dissipativity

The section connects generalized cost modification to strict dissipativity, showing how a positive-definite stage cost can support stability while retaining economic optimal policies.

  • Strict dissipativity is used in economic NMPC to construct a positive-definite cost and Lyapunov function for closed-loop stability.
  • The dissipativity discussion restricts to deterministic systems with γ=1, while a mature stochastic dissipativity theory is not yet available.
  • If Λ minimizes at the optimal policy and λ satisfies the strict dissipativity inequality, the modified stage cost is bounded below by a positive-definite function.
  • This result allows a stabilizing optimal policy from a generic economic stage cost to be learned using a parametrization with positive-definite stage cost.
  • The proposed modification generalizes ENMPC cost rotation by allowing Λ(s,a) instead of only a state function λ(s).
  • The modification can be represented through the initial-state term λθ and combined with a positive-definite stage cost to approximate economic value and action-value functions.

VI. ANALYTICAL CASE STUDY: THE LQR CASE

The LQR case illustrates the theory analytically, including imperfect-model effects, cost modification, stability enforcement, and the quadratic connection to strict dissipativity.

  • The LQR example uses a centered linear-quadratic-Gaussian system with stochastic dynamics and associated quadratic value functions.
  • For the exact construction, the matrices ˆS=S and ˆK=K⋆ satisfy the required condition.
  • Using the imperfect model ˆA=2 yields a non-stabilizing solution with ˆA−ˆBK⋆=1.5, while the stabilizing DARE solution has ˆS=7, ˆK=4/3, and closed-loop factor 2/3.
  • For economic LQR with indefinite stage cost, quadratic modifiers are introduced through matrices δT, δN, and δR.
  • Stability enforcement requires choosing δR and δS so that W+δW_L is positive definite, which reduces to solving a linear matrix inequality.
  • In the linear-quadratic case, the relevant condition is strict dissipativity, and setting δR=0 gives the state-only modifier Λ(s,a)=λ(s)=sᵀδSs.
  • A quadratic stage-cost modifier can produce the nominal optimal feedback, but its associated value and action-value functions may both be zero.

VII. NUMERICAL EXAMPLES

This section introduces two examples illustrating the paper’s theoretical developments.

  • Two examples are proposed to illustrate the theoretical developments.

A. Linear MPC

The examples apply RL-based parameter tuning to linear MPC and an evaporation-process ENMPC, showing improved closed-loop behavior despite model mismatch and stochastic disturbances.

  • Linear MPC: The linear MPC example uses a prediction model with dynamics x_k+1 = A x_k + B u_k + b and RL-adjustable NMPC parameters.The setup uses horizon N = 10, discount factor γ = 0.9, and a Riccati-based terminal cost.
  • Linear MPC: The real linear process includes uniformly distributed noise e_k in [−10^-1, 0], which drives the first state below its lower bound.
  • Linear MPC: RL reduces the TD error toward zero and adjusts the stage gradient and model bias to avoid expensive state-bound violations.Early trajectories often violate x1 ≥ 0 and produce large control actions; learning modifies the stage gradient f and bias b.
  • Linear MPC: Restricting learning to constraint tightening improves MPC performance but does not reach the performance obtained when model parameters are also adapted.
  • Evaporation Process: The evaporation process has stochastic disturbances, state and input bounds, and an economic objective involving flow-related operating costs.The controlled states are concentration X2 and pressure P2, while the inputs are pressure P100 and flow F200.
  • Evaporation Process: The ENMPC parameters are initialized from nominal economic tuning, and learning converges to constant parameter values while reducing the average TD error.The parameters remain within the positive-definiteness constraint during learning.
  • Evaporation Process: RL-tuned NMPC improves the economic cost by about 7% in the considered evaporation-process scenario relative to nominal economic tuning.RL maintains higher X2 values to reduce quality-constraint violations.
  • Evaporation Process: The action-value function and policy are learned only approximately because the adapted cost and model do not share the real cost and model’s structure.The storage function λ_θ is used to keep the stage cost positive-definite.

VIII. CONCLUSIONS

The paper proposes ENMPC as an RL function approximator and shows that cost and terminal-cost adaptation can recover the real system’s optimal policy despite model mismatch. It also details classic RL tuning methods and illustrates them in simulations.

  • ENMPC can parametrize value functions or policies and deliver the real system’s optimal policy with an incorrect underlying model by adapting stage and terminal costs.The paper also connects positive costs with stability by construction when the optimal policy is stabilizing.
  • The paper illustrates classic RL methods for adjusting ENMPC parameters through simulations.
  • Future work will improve RL algorithms for ENMPC and combine existing model-tuning techniques with RL-based tuning.
Loading 1904.04152v1…