PaperlayerView exact revision on arXiv

Source-linked AI summary

Economic Model Predictive Control with Policy-Guided Terminal Ingredients

Salim Msaad, Robert D. McAllister

arXiv:2609.02628v1eess.SY

TL;DR

Economic MPC often relies on steady-state-based terminal ingredients, which are unsuitable when desirable operation is nonstationary or differs from the optimal steady state. This paper proposes PG-MPC, using a sub-optimal policy rollout to define terminal costs and constraints, and establishes feasibility and performance guarantees relative to that policy. The approach is demonstrated for energy management, where steady-state designs are inadequate because time-varying prices require load shifting.

  • Problem

    Steady-state-based terminal costs and constraints can be inadequate when a fixed operating point is unavailable, undesirable, or performs worse than another feasible operating trajectory.

  • Method

    PG-MPC constructs terminal costs and constraints around a center obtained by rolling out an available sub-optimal control policy.

  • Results

    The paper establishes closed-loop performance guarantees relative to the guiding policy and proves existence of solutions and positive invariance under its assumptions.

  • Takeaways & Limitations

    PG-MPC provides a policy-relative alternative to steady-state terminal design for economic MPC, demonstrated on energy management with time-varying prices.

Abstract

from arXiv · show

Conventional designs for model predictive control typically rely on terminal costs and constraints derived from a steady state to guarantee closed-loop stability and performance. However, this dependence on a steady-state assumption limits the applicability of this control method to systems in which such a fixed operating point is either not available or not desirable. This work introduces a novel framework, termed policy-guided MPC, to address this limitation. Our approach constructs terminal costs and constraints using a known sub-optimal control policy. Specifically, the terminal region is defined around a center determined by a rollout of the policy, and a penalty on deviation from this center is used to define the terminal cost. This method obviates the need for a steady state or reference trajectory. Closed-loop performance guarantees are established relative to the guiding policy, for both finite and infinite horizon problems. The effectiveness of the proposed framework is demonstrated through numerical simulations on an energy management example.

I. INTRODUCTION

Economic MPC is suited to optimizing operation when the desired behavior is not a fixed steady state or trajectory, but conventional guarantees often rely on steady-state-based terminal ingredients. The proposed PG-MPC framework instead uses a sub-optimal policy to construct terminal costs and constraints and provides guarantees relative to that policy.

  • Economic MPC directly optimizes metrics such as energy consumption, throughput, or profit rather than tracking a reference.
  • Existing economic MPC stability analyses commonly construct terminal ingredients around an optimal steady state, under assumptions such as linearity, convexity, duality, or dissipativity.
  • Steady-state-based terminal designs can perform poorly when another feasible operating trajectory is substantially better than the optimal steady state.
  • PG-MPC constructs terminal costs and constraints from an available sub-optimal control policy rather than a steady state.
  • The work establishes closed-loop performance guarantees relative to the guiding policy and demonstrates the approach in energy management with time-varying prices.

A. Problem Formulation

The paper formulates a constrained discrete-time system with a nonnegative stage cost and an infinite-horizon discounted objective. The setup includes standard continuity, closedness, compactness, and discount-factor assumptions.

  • The system has state x in X and input u in U, with X and U serving as state and input constraint sets.
  • The stage cost ℓ(x,u) quantifies the relative performance of state/input combinations.
  • The dynamics and stage cost are continuous, the state constraint set X is closed, and the input constraint set U is compact.
  • The standard optimal control problem minimizes an infinite-horizon discounted cost from the initial state.
  • The discount factor γ lies in (0, 1].

B. Available policy

The framework assumes an available sub-optimal policy whose rollout and value function support terminal design and analysis. The policy must satisfy constraints, have finite value, and be computationally efficient to evaluate.

  • The initial policy π may come from existing algorithms, domain expertise, or reinforcement learning, but must be computationally efficient to evaluate.
  • The rollout ϕπ(k; x) is the state at time k under the autonomous closed-loop dynamics x+ = f(x, π(x)).
  • The policy has a corresponding value function Jπ(x) satisfying the Bellman equation.
  • The policy is required to satisfy state and input constraints for every state in X.
  • The value function Jπ is assumed finite and Lipschitz continuous with constant L.

C. Policy-Guided MPC Framework

PG-MPC updates terminal ingredients around a policy-rollout center rather than a fixed steady state. Its design balances allowable deviation from the rollout against adherence to the policy, while the optimization remains receding-horizon.

  • C. Policy-Guided MPC Framework: PG-MPC defines the terminal cost Vf(x, xf) and terminal set Xf(xf) around a center xf updated using the available policy.
  • C. Policy-Guided MPC Framework: The policy approximation error d determines the tightness of performance bounds, but large ρ or τ = 0 reduces PG-MPC’s optimization freedom.
  • C. Policy-Guided MPC Framework: At each time step, PG-MPC solves a finite-horizon optimization over feasible trajectories reaching the terminal region while satisfying intermediate state constraints.
  • C. Policy-Guided MPC Framework: The center xf is initialized by rolling out π from the initial state for N steps and subsequently updated from the last predicted state.

III. CLOSED-LOOP GUARANTEES

The closed-loop analysis augments the system state with a policy-generated central point and establishes recursive feasibility under policy-guided MPC. The resulting extended trajectories remain well defined for all future times.

  • III. CLOSED-LOOP GUARANTEES: The extended state z := (x, xf) captures the physical state and the policy-guided central point used by the closed-loop dynamics.The extended dynamics are represented by a difference equation for z.
  • III. CLOSED-LOOP GUARANTEES: Each update applies the first input of the optimal trajectory, while the next central point is generated by rolling out policy π(·) from the final predicted state.This update is repeated at subsequent time steps.
  • III. CLOSED-LOOP GUARANTEES: Under Assumptions 1 and 2, an optimization solution exists for every feasible extended state, and the feasible set ZN is positively invariant.Positive invariance preserves feasibility after the closed-loop update.
  • III. CLOSED-LOOP GUARANTEES: At initialization, the central point is set to the state reached by rolling out policy π(·) from x for N steps.This policy rollout supplies a feasible way to reach the terminal region.
  • III. CLOSED-LOOP GUARANTEES: Consequently, the initial policy rollout makes the extended state feasible, and positive invariance ensures PG-MPC remains feasible with well-defined trajectories for all k ∈ I∞.The closed-loop extended state is initialized as ψ(0; x) = (x, ϕπ(N; x)).

A. Cost decrease inequality

The analysis introduces a shifted cost combining the optimal MPC cost with the guiding policy’s value function at the central point. This shifted cost satisfies a decrease inequality used for closed-loop guarantees.

  • A. Cost decrease inequality: The shifted cost is defined as the optimal MPC cost plus γ^N times the policy value function evaluated at the central point xf.It combines the MPC objective with the remaining discounted cost of policy π from xf.
  • A. Cost decrease inequality: The inequality follows by combining MPC optimality with the candidate solution obtained from the previous closed-loop optimization.The candidate is constructed for the updated pair of physical and central states.
  • A. Cost decrease inequality: Under Assumptions 1 and 2, the shifted cost satisfies a cost decrease inequality for every feasible pair (x, xf) ∈ ZN.The inequality is derived using the warm-start candidate and the Bellman equation for Jπ.

B. Nominal performance bounds

Policy-guided MPC provides finite- and infinite-horizon performance bounds relative to the guiding policy. The deviation is bounded by d, with dependence on the discount factor and prediction horizon, and d can be zero under specified choices.

  • B. Nominal performance bounds: The cumulative cost of policy-guided MPC remains within a bounded deviation d of the guiding policy’s cost, with the bound scaling with γ and N.These nominal bounds compare the closed-loop control law κ(·) directly with policy π(·).
  • B. Nominal performance bounds: For any finite horizon T ≥ 1, Theorem 1 establishes a transient performance bound for the policy-guided MPC closed loop.The result applies under Assumptions 1 and 2 and is obtained by repeatedly applying the cost decrease inequality.
  • B. Nominal performance bounds: The transient bound extends to asymptotic performance bounds for both discounted costs with γ ∈ (0, 1) and undiscounted costs with γ = 1.The infinite-horizon results are stated separately for the two discount-factor cases.
  • B. Nominal performance bounds: For γ = 1, the asymptotic result uses nonnegative stage costs and the boundedness of the policy value function along the policy trajectory.These properties imply Jπ(ϕπ(T; x))/T → 0 as T → ∞.
  • B. Nominal performance bounds: When γ = 1 and d = 0, the asymptotic policy-guided MPC cost bound holds without a deviation term.The special case d = 0 occurs when ρ is at least the Lipschitz constant L of Jπ or when τ = 0.

IV. NUMERICAL EXAMPLE

The energy-management example uses a two-battery system with time-varying prices, where steady-state terminal designs cannot support load shifting. PG-MPC uses a rule-based policy to center its terminal ingredients and achieves lower costs while retaining short-horizon optimization.

  • Energy management example: The system coordinates a high-efficiency small battery and a higher-capacity large battery to satisfy constant demand under time-varying energy prices.The operating horizon spans 24 days with 1152 half-hour time steps, and the objective is to minimize grid-purchased energy costs while satisfying constraints.
  • Limitations of steady-state designs: Any steady state fixes battery energy levels, preventing the load shifting needed to exploit time-varying prices.A precomputed periodic trajectory would require prior knowledge of the price profile and remain fragile to model mismatch and disturbances.
  • Standard MPC: Standard MPC with a 48-step horizon is myopic: it rarely charges the large battery and prefers the more efficient small battery.This limited-horizon behavior results in high operational costs and poor battery utilization.
  • Guiding policy: The rule-based policy controls only the large battery using the current energy price and a threshold-based charging and discharging strategy.The policy charges at a fixed rate below the threshold p̄ = 2 and discharges to meet demand when the price exceeds p̄.
  • PG-MPC design: PG-MPC uses the rule-based policy to determine the terminal-region center while retaining the standard MPC prediction horizon and constraints.The state and policy-generated center share the same time index, so the terminal ingredients are defined at the corresponding time step.
  • PG-MPC results: With τ = 1680 and ρ = 1.5, PG-MPC effectively uses both batteries and achieves substantially lower costs than standard MPC or the rule-based policy.The radius τ limits deviation from the policy rollout, while ρ controls how strongly the terminal state is pulled toward the policy-generated center.

V. CONCLUSION

The paper concludes that PG-MPC builds terminal costs and constraints from a control policy rather than a steady state. It provides policy-relative closed-loop guarantees and is demonstrated on an energy-management problem where steady-state designs are inadequate.

  • PG-MPC constructs terminal costs and constraints from a given control policy rather than a steady state.
  • The framework establishes closed-loop performance guarantees relative to the guiding policy.
  • The energy-management example demonstrates the approach where steady-state-based designs are inadequate.
  • Future work will examine additional terminal-cost information, stability for converging policies, and inherent robustness.
Loading 2609.02628v1…