PaperlayerView exact revision on arXiv

Source-linked AI summary

Infinite-Horizon Differentiable Model Predictive Control

Sebastian East, Marco Gallieri, Jonathan Masci, Jan Koutnik, Mark Cannon

arXiv:2001.02244v1math.OCcs.LGcs.SYeess.SY

TL;DR

Safety-critical imitation learning needs controllers that respect hard constraints, while MPC predictions can lose effectiveness when model dynamics are uncertain. This paper makes LTI quadratic MPC differentiable using DARE-based infinite-horizon design, pre-stabilization, and augmented-Lagrangian constraints. Numerical studies report learning infinite-horizon cost and guaranteeing hard constraints with short horizons; the method is limited to LTI systems and its DARE derivative scales poorly at larger dimensions.

  • Problem

    Imitation learning for safety-critical control needs effective learning and generalization, while MPC prediction inaccuracies can reduce effectiveness and robustness.

    Source: §1 · p. 1

  • Method

    The framework uses an LQR-derived DARE terminal cost and analytical DARE derivatives for gradient-based learning, with pre-stabilized MPC and augmented-Lagrangian state and input constraints.

    Source: §1 · pp. 1–2

  • Results

    The framework demonstrates learning infinite-horizon cost and guaranteeing hard constraints with a short finite prediction horizon on mass-spring-damper systems and a vehicle platooning model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The learned controller is reported to have local stability and intrinsic robustness guarantees under small model uncertainties.

    Source: §1 · p. 2

    The method is limited to LTI systems; extending it to time-varying or nonlinear models is non-trivial, and the DARE derivative's matrix operations do not scale well to large state and control dimensions.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and a pre-stabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in a set of numerical studies.

1 INTRODUCTION

The paper develops differentiable infinite-horizon LTI quadratic MPC for imitation learning, using a DARE-derived terminal cost and analytical DARE derivatives. Its formulation combines pre-stabilization with augmented-Lagrangian constraints, and numerical studies demonstrate learning the infinite-horizon cost and guaranteeing hard constraints with short prediction horizons.

Motivation

  • The framework targets imitation learning, where effective learning and generalization matter in safety-critical applications, and MPC is used for systems with hard safety constraints.MPC stability requires accounting for an infinite-horizon value function and candidate terminal controller; prediction inaccuracies can reduce effectiveness and robustness.Source: §1 · p. 1

Contributions

  • For LTI quadratic MPC, the framework uses differentiable QPs and optimal-control principles to learn an infinite-horizon controller.The contribution is framed as connecting MPC stability and infinite-horizon optimality to imitation learning.Source: §1 · pp. 1–2

Guarantees

  • The proposed controller is reported to have local stability and intrinsic robustness guarantees under small model uncertainties.The introduction states these guarantees for the learned MPC implemented in closed loop.Source: §1 · p. 2

Contributions

  • The DARE supplies infinite-horizon optimality and stability, while its analytically derived solution derivative enables gradient-based training.This connects MPC stability and infinite-horizon optimality results to imitation learning.Source: §1 · p. 2

MPC formulation

  • A pre-stabilizing state-feedback controller from the DARE conditions the QP, while an augmented-Lagrangian penalty keeps training feasible and enforces hard state and input constraints when sufficiently large.The total input is a perturbation of the feedback law; improved QP conditioning supports accurate solutions and backward-pass gradients.Source: §1 · p. 2

Demonstrations

  • The framework learns infinite-horizon cost and guarantees hard constraints with a short finite prediction horizon on mass-spring-damper systems and a vehicle platooning model.These are the numerical settings described in the introduction.Source: §1 · p. 2

2 DIFFERENTIABLE MPC

The section formulates constrained linear MPC and explains how differentiation, augmented-Lagrangian penalties, and pre-stabilization support learning. The method uses a finite-horizon QP while drawing on an unconstrained infinite-horizon LQR for its terminal design.

Linear quadratic MPC

  • MPC optimizes state and input over a finite prediction horizon, implements the first optimized input, and repeats the process at each time step.The section considers LTI systems and linear MPC with quadratic stage costs and box constraints.Source: §2 · p. 2
  • The augmented-Lagrangian slack variables preserve MPC feasibility at each learning iteration, while sufficiently high penalties exactly enforce state and input box constraints.The slacks represent state and control constraint violations; the constraints are exact when the slacks are zero.Source: §2 · p. 3

Pre-stabilised MPC

  • Pre-stabilization decomposes input into stabilizing feedback Kx_t and a perturbation, yielding predicted dynamics with A + BK and the same controlled system as the original formulation.The transformed problem is preferable computationally for open-loop unstable A when the prediction horizon is large, because the original matrices can be poorly conditioned.Source: §2 · p. 3

MPC derivative

  • If the system matrix A is learned, its eigenvalues may be unbounded at a given iteration, motivating attention to the conditioning of the MPC computation.The supplied passage raises this issue in the context of differentiable MPC.Source: §2 · p. 3
  • When the QP solution is unique, its KKT conditions define an implicit function that can be differentiated with respect to QP parameters and used for backpropagation through an MPC layer.The described parameter gradients include cost, dynamics, constraints, and state and input bounds.Source: §2 · pp. 3–4

Imitation Learning

  • The imitation-learning setup assumes precise state and control measurements, and optionally penalizes one-step prediction error alongside control imitation loss.The prediction-error penalty may not be needed for correct learning but might help stability.Source: §2 · p. 4

3 TERMINAL COST FOR INFINITE HORIZON

The MPC uses the DARE solution as its terminal cost, connecting finite-horizon control to infinite-horizon optimality and stability. The paper also derives the DARE solution’s analytic derivative for gradient-based learning and states the conditions under which the MPC guarantees hold.

3 TERMINAL COST FOR INFINITE HORIZON

  • For initial states in a specified closed, bounded set and zero disturbance, the MPC is infinite-horizon optimal for any horizon at least the state-dependent threshold.The paper notes that the threshold can be upper-bounded over a closed, bounded set of feasible states.Source: §3 · p. 4
  • With zero disturbance, the constraints remain feasible and the origin is asymptotically stable for initial states in Γ_N, for any horizon N ≥ 1.Source: §3 · p. 4
  • For any horizon N ≥ 1, bounded additive model error yields robust feasibility and input-to-state stability for initial states in Γ_N.The error must satisfy ∥ŵ_t∥ ≤ μ for all t ≥ 0.Source: §3 · p. 4
  • The state-dependent horizon threshold makes infinite-horizon optimality attainable with a finite prediction horizon, and can be upper-bounded over a closed, bounded feasible-state set.Source: §3 · p. 4
  • Using the LQR gain to pre-stabilize the dynamics keeps the QP matrices finite-norm and the MPC problem well-conditioned, supporting accurate solutions even over long horizons.Without pre-stabilization, matrices for an unstable open-loop system can become unbounded, making long-horizon solutions poorly conditioned and inaccurate.Source: §3 · pp. 4–5
  • The paper derives an analytic derivative of the DARE solution, enabling the DARE-based terminal cost and controller to be differentiated during gradient-based learning.The authors describe this as the first analytic DARE derivative using the differential calculus approach of Magnus & Neudecker (1999).Source: §3 · p. 5

4 NUMERICAL EXPERIMENTS

The experiments use second-order mass-spring-damper systems as an interpretable baseline and vehicle platooning as a higher-dimensional application.

4.1 MASS-SPRING-DAMPER

  • Seven mass-spring systems vary damping c across stable (c > 0) and unstable (c < 0) dynamics, with state and input constraints active in expert demonstrations.The damping coefficients are listed in Table 1.Source: §4.1 · p. 6, Table 1 · p. 6
  • Across generated systems, imitation loss converges to a low value, generally at a local minimum, while incorrect model convergence occurs only at N=2 for systems 2 and 4; longer horizons speed convergence, but evidence is inconclusive.Model loss was not used in training; it was shown only to indicate whether the model converges correctly.Source: §4.1 · pp. 6–7
  • With each system initialized at x0 = (0.5, 2), learned MPC closed-loop trajectories converge to expert trajectories at N = 6; state constraints are guaranteed for N ≥4, and controllers stabilize even at the shortest horizon.For systems 2 and 4, incorrect dynamics were learned, and state constraints are not guaranteed for N = 2.Source: §4.1 · p. 7

4.2 VEHICLE PLATOONING

  • The platoon study uses 10 vehicles, an 18-dimensional state, and 10 inputs, with 20 feasible initial conditions and a 30 m steady-state gap.The safe separation threshold is 10 m.Source: §4.2 · p. 7
  • All platoon controllers satisfy vehicle-separation constraints; at N=20, predictions match closed-loop behavior, while errors at shorter horizons are largest when state constraints are active.The authors suggest these shorter-horizon controllers deviate from predictions to satisfy constraints at later intervals.Source: §4.2 · pp. 8–9

4.3 LIMITATIONS

  • The approach is limited to LTI systems; extending it to time-varying or nonlinear models is non-trivial, and its DARE derivative scales poorly to large dimensions.The algorithm also requires a stabilizing DARE solution to exist, whose existence may be difficult to ensure during training.Source: §4.3 · p. 9

A MPC QUADRATIC PROGRAM

The passage fragments report equivalence reformulations for problems (2) and (4), and note that some quantities have conformal dimensions.

A MPC QUADRATIC PROGRAM

B OSQP DERIVATIVES

This passage describes how OSQP solution values identify active constraints, characterize the solution through KKT conditions, and support differentiation.

B OSQP DERIVATIVES

  • OSQP returns z, y, and s values that satisfy the solver conditions for the quadratic program.Source: B OSQP DERIVATIVES · p. 13
  • The sign of each y_i identifies whether the upper or lower constraint is strongly active, while y_i = 0 indicates inactivity.The solution is characterized by the KKT system.Source: B OSQP DERIVATIVES · p. 13
  • The active constraint sets define the matrix subsets used to differentiate the KKT system using the cited techniques.Source: B OSQP DERIVATIVES · p. 13

C PROOF OF PROPOSITION 1

The proof derives the proposition’s claims from earlier MPC stability results and properties of the linear model, strictly convex QP, and LQR terminal set.

  • The first claim follows from Scokaert & Rawlings (1998), while the next two rely on Limon et al. (2003; 2009).Source: C PROOF OF PROPOSITION 1 · p. 13
  • The closed-loop is Lipschitz because the model is linear and the controller solves a strictly convex QP.Source: C PROOF OF PROPOSITION 1 · p. 13
  • The LQR supplies a contractive terminal set, and the final claim uses eigenvalues of (A + BK)^N within the unit circle for all N ≥ 1.The proof also invokes a matrix-norm result: given a spectral-radius bound ρ, a matrix norm exists that is also less than ρ.Source: C PROOF OF PROPOSITION 1 · p. 13

D PROOF OF PROPOSITION 2

The appendix establishes differentiability of the DARE solution and describes a sampled verification procedure for reducing the MPC prediction horizon.

D PROOF OF PROPOSITION 2

  • A stabilizing solution to the DARE is unique under the stated spectral-radius condition, allowing the equation to be treated as an implicit function of A, B, Q, and R.The proof assumes (R + B⊤PB)−1 exists.Source: D PROOF OF PROPOSITION 2 · p. 13
  • Differentials of the DARE components are combined by the chain rule to relate dvecP to changes in A, B, Q, and R.The resulting relation is Z1dvecP = Z2dvecA + Z3dvecB + Z4dvecQ + Z5dvecR.Source: D PROOF OF PROPOSITION 2 · p. 13

E VERIFICATION AND REDUCTION OF THE PREDICTION HORIZON

  • The horizon-reduction procedure tests whether MPC actions stay within tolerance ϵ of the LQR action for all k ≥ N̄ using uniformly sampled states.It samples from the state constraints or an estimated region of attraction; if verification fails as the set is downscaled, it returns to training with an increased horizon.Source: E VERIFICATION AND REDUCTION OF THE… · p. 14

F PLATOON MODEL DERIVATION

The platoon-model derivation describes a mass-based vehicle model and its conversion to discrete time under a constant-input assumption.

G NONLINEAR MODELS

The authors limit the approach to LTI systems and describe a possible extension when an LQR value function is a local control Lyapunov function.

  • The described control input is bounded by b1 ≤ ut ≤ a1 for all t.Source: G NONLINEAR MODELS · p. 15
  • The approach is currently limited to Linear Time Invariant systems.Source: G NONLINEAR MODELS · p. 15
  • For systems where the LQR value function is a local control Lyapunov function, the authors state that stability and intrinsic robustness results are maintained.Source: G NONLINEAR MODELS · p. 15
  • The proposed check is to train a local linear model around the target equilibrium and test DARE solvability before imitation learning.The authors leave study of more general systems to future work and note the condition is a substantial assumption for very nonlinear systems.Source: G NONLINEAR MODELS · p. 15
Loading 2001.02244v1…