PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence

Kaiqing Zhang, Bin Hu, Tamer Başar

arXiv:1910.09496v3math.OCcs.LGeess.SY

TL;DR

The paper addresses global policy optimization for H2 linear control under an H∞ robustness constraint, where non-coercivity complicates feasibility preservation. It analyzes policy-gradient methods and shows that two search directions implicitly preserve robustness while converging globally despite nonconvexity. The resulting methods achieve globally sublinear convergence and locally linear, or locally Q-quadratic for Gauss–Newton, rates under additional conditions.

  • Problem

    H2 control with an H∞ robustness constraint is nonconvex and non-coercive, making it difficult to preserve feasibility and establish global convergence.

  • Method

    The paper analyzes policy-gradient-based methods for mixed H2/H∞ control, including Gauss–Newton and natural policy gradient updates, without explicit projection.

  • Results

    The Gauss–Newton and natural policy gradient methods implicitly preserve the H∞ constraint and converge globally to the optimal policy, with globally sublinear and locally linear rates; Gauss–Newton is locally Q-quadratic when η = 1/2.

  • Takeaways & Limitations

    The results show that suitable policy-search directions can maintain robustness and avoid suboptimal stationary points in this nonconvex control problem.

  • Takeaways & Limitations

    The analysis assumes an initial policy satisfying the specified H∞ norm constraint, and the paper does not resolve whether vanilla policy gradient has a constant stepsize guaranteeing global convergence and robustness preservation.

Abstract

from arXiv · show

Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $\mathcal{H}_\infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set, its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $\mathcal{H}_2$ linear control with $\mathcal{H}_\infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $\mathcal{H}_\infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, despite the nonconvexity of the problem, we show that these algorithms converge to the globally optimal policies with globally sublinear rates, avoiding all suboptimal stationary points/local minima, and with locally (super-)linear rates under certain conditions.

1 Introduction

The paper studies policy optimization for robust control, where stability, robustness, and safety constraints make the problem constrained and nonconvex. It focuses on mixed H2/H∞ design, whose non-coercive landscape complicates feasibility and convergence analysis.

  • Motivation: Policy optimization is constrained because the feasible set encodes stability, robustness, or safety requirements, while the cost may be undefined outside it.Projection can be computationally difficult because stability-feasible sets are nonconvex.
  • Motivation: H∞ constraints support robust stability and performance under model uncertainty, but produce a fundamentally different optimization landscape from LQR.The constraint limits the H∞ norm on an input-output channel and is motivated by robust and risk-sensitive control.
  • Challenges: Mixed H2/H∞ design lacks coercivity: the cost can remain finite near the feasibility boundary, unlike the LQR setting.This removes the straightforward argument that decreasing cost automatically keeps policy iterates feasible.
  • Contributions: The paper proposes three policy-gradient-based methods and proves that Gauss–Newton and natural policy gradient implicitly preserve the H∞ constraint.Their search directions bias iterates toward stabilizing controllers satisfying the robustness requirement without explicit projection.
  • Contributions: The two methods globally converge to the globally optimal policy despite nonconvexity, with globally sublinear and conditionally locally (super-)linear rates.The analysis shows that they avoid suboptimal stationary points and local optima.

2 Preliminaries

The preliminaries formulate LEQG and mixed H2/H∞ control through linear state-feedback policies, Riccati-based costs, and implicit H∞ feasibility constraints. The H∞ bound provides a robustness interpretation, while the feasible set may be unbounded and difficult to enforce directly.

  • 2.1 Motivating Example: LEQG: LEQG admits an optimal memoryless, stationary LTI state-feedback policy, so optimization can be reduced to the control gain K.The policy has the form u_t = −Kx_t.
  • 2.1 Motivating Example: LEQG: For LEQG, the objective is represented using a solution to an algebraic Riccati equation and reduces to the LQG cost as β approaches zero.The paper also rigorously establishes the LTI optimal-policy characterization and objective form.
  • 2.1.2 Implicit Constraint on H∞-Norm: Although LEQG appears unconstrained, its feasible gains are implicitly restricted by a lower-level set of the closed-loop H∞ norm.This constraint is hard to impose directly because it is a frequency-domain characterization, whereas the Riccati conditions provide a useful time-domain characterization.
  • 2.2 Bigger Picture: Mixed H2/H∞ Control Synthesis: Mixed H2/H∞ design minimizes an H2-based performance bound subject to an H∞ norm bound on the closed-loop transfer function.Under the stated state-feedback information structure, stationary current-state LTI feedback suffices for optimal performance.
  • 2.2 Bigger Picture: Mixed H2/H∞ Control Synthesis: The H∞ constraint ∥T(K)∥∞ < γ guarantees robustness against stable feedback perturbations whose induced ℓ2 gain is below 1/γ.This interpretation follows from the small gain theorem.
  • 2.1.2 Implicit Constraint on H∞-Norm: For LEQG, the Riccati conditions used to express the objective are equivalent to the H∞ feasibility constraint, making them necessary as well as sufficient.Thus every feasible controller has the stated Riccati-based objective form.

3 Landscape and Algorithms

The paper analyzes mixed H2/H∞ control as a nonconvex, noncoercive optimization problem and develops policy-gradient methods with convergence guarantees. Its landscape results characterize global optimizers, while the proposed updates use policy-gradient and Gauss–Newton directions motivated by LQR methods.

  • Optimization Landscape: The study covers discrete- and continuous-time mixed H2/H∞ settings and develops policy optimization algorithms with convergence guarantees.The discrete-time example also corresponds to linear exponential quadratic Gaussian control.
  • Optimization Landscape: The mixed H2/H∞ design problem is nonconvex, and its cost need not diverge as the controller approaches the feasible-set boundary.The nonconvex constraint set and lack of coercivity complicate standard policy-gradient convergence analysis.
  • Optimization Landscape: A global optimizer has the form K* = (R + B^T eP_K* B)^-1 B^T eP_K* A, and controllability at a stationary point makes it the unique global optimizer.When the relevant matrix is not full-rank, multiple stationary points may exist, but the global optimum retains this form.
  • Policy Optimization Algorithms: Three policy-gradient-based methods are introduced, with updates motivated by LQR policy optimization and using eP_K in place of P_K.The natural policy-gradient update relates to a Riemannian-manifold gradient, while the Gauss–Newton update is a quasi-Newton method.
  • Policy Optimization Algorithms: With η = 1/2, the Gauss–Newton update can be viewed as policy iteration for infinite-horizon mixed H2/H∞ design.The policy-gradient update can also be implemented model-free by estimating the gradient from sampled data.

4 Theoretical Results

The proposed natural policy-gradient and Gauss–Newton methods preserve the H∞ robustness constraint without projection and converge globally despite the noncoercive, nonconvex landscape. Their convergence is globally sublinear and locally linear, with Gauss–Newton achieving a local quadratic rate under an additional condition.

  • 4.1 Implicit Regularization: Finite-cost approaches to the robustness boundary invalidate LQR’s coercivity-based feasibility argument and can let descent directions leave the feasible region.The cost level sets need not be strictly separated from the infeasible complement.
  • 4.1 Implicit Regularization: Natural policy gradient and Gauss–Newton preserve feasibility under suitable constant stepsizes, yielding implicit regularization without projection.The updates remain stabilizing and satisfy the H∞ constraint throughout the iterations.
  • 4.2 Global Convergence: Both methods converge to the global optimum with an O(1/N) rate for the iteration-average gradient norm square, avoiding suboptimal stationary points and local minima.The specific search directions bias the iterates toward a globally optimal solution even when multiple stationary points exist.
  • 4.2 Global Convergence: Globally, the convergence is sublinear rather than LQR’s linear rate, but both methods become locally linear when DD^T > 0.The local linear result concerns convergence of the objective to its optimal value.
  • 4.2 Global Convergence: With η = 1/2, the Gauss–Newton update achieves local Q-quadratic convergence under the same additional condition.This is the stronger local rate established for the Gauss–Newton method.
  • 4.2 Global Convergence: Compared with prior projected methods, these guarantees require no projection or non-standard robust-control assumptions, and additionally cover continuous-time systems.The paper also reports the local superlinear rate for Gauss–Newton.

5 Proofs of Main Results

The proofs establish feasibility by constructing Lyapunov certificates from the current Riccati solution and perturbing them to obtain strict LMIs. Cost-difference and monotonicity arguments then yield global convergence, while local contraction arguments give linear or quadratic rates near the optimum.

  • 5 Proofs of Main Results: Feasibility follows by using P_K as a certificate, perturbing it by αP̄, and applying LMIs to show the updated controller remains stabilizing and H∞-robust.The perturbation makes the relevant inequality strict, while interpolation extends feasibility to smaller stepsizes.
  • 5 Proofs of Main Results: The cost-difference analysis shows that P_K decreases monotonically along the updates and converges to its value at the global optimum.The argument uses matrix inequalities and monotonicity of the transformed Riccati solution.
  • 5 Proofs of Main Results: The iteration-average stationarity measure converges to zero at O(1/N), and the limiting stationary point is the global optimal control gain.This conclusion is obtained by summing the per-iteration inequalities and using the global-optimality characterization.
  • 5 Proofs of Main Results: After the iterates enter a neighborhood of K*, a lower-level-set argument keeps them there and yields a uniform local linear contraction.Bounded solutions of the associated Lyapunov equation provide the uniform contraction factor.
  • 5 Proofs of Main Results: For Gauss–Newton with η = 1/2, the local contraction strengthens to Q-quadratic convergence of the objective near K*.The proof bounds the relevant Lyapunov solution uniformly inside the neighborhood.

6 Discussions

The mixed H2/H∞ problem is connected to zero-sum linear-quadratic games and risk-sensitive control, providing an alternative interpretation and a route to model-free policy optimization. The game formulation can reproduce the mixed-design optimum, while natural policy gradient can be implemented through an outer-loop game update.

  • 6 Discussions: The game formulation links risk-sensitive LEQG control to mixed design and identifies the game minimizer with the global mixed-design optimum.The correspondence replaces the disturbance weighting with the H∞ bound and preserves the relevant Riccati structure.
  • 6 Discussions: Mixed H2/H∞ control is equivalent to a zero-sum LQ game in which the adversarial player represents the worst-case disturbance.Maximizing over the disturbance policy produces a Riccati equation matching the mixed-design formulation.
  • 6 Discussions: Model-free policy-gradient methods can solve the associated game and thereby recover the mixed-design optimum under the stated conditions.The game’s stationary Nash equilibrium corresponds to the optimal mixed-design controller.
  • 6 Discussions: The natural policy-gradient update for mixed design is identical to the game’s exact outer-loop update, enabling a model-free implementation through sampled data.The inner update searches for the worst-case disturbance, while the outer update improves the controller.

7 Simulations

Simulations support the proposed methods' convergence, implicit robustness preservation, escape from suboptimal stationary points, and competitive solver performance across mixed H2/H∞ control cases.

  • Implicit Regularization & Global/Local Convergence: All four update rules converge successfully, with NPG and Gauss-Newton showing sublinear-to-linear behavior and Gauss-Newton at η = 1/2 showing super-linear convergence.These observations corroborate the stated global and local convergence theory.
  • Implicit Regularization & Global/Local Convergence: The H∞-norm remains below γ = 15.45 throughout the convergent runs, while γ^2I − D^⊤P_KD remains positive definite.This validates the implicit regularization result in the simulations.
  • Implicit Regularization & Global/Local Convergence: Near the feasibility boundary, vanilla PG violates the H∞ constraint and fails to converge even with stepsize 10^-7, whereas NPG and Gauss-Newton preserve feasibility.The initialization uses γ = 1.00001·∥T(K_0)∥∞, placing it very close to the boundary.
  • Escaping Suboptimal Stationary Points: With infinitely many stationary points, NPG and Gauss-Newton converge to K* in all 50 trials, while vanilla PG can become trapped at suboptimal stationary points.The result illustrates implicit regularization through search directions that avoid bad local minima.
  • Comparison with Existing H2/H∞ Control Solvers: Against HIFOO and h2hinfsyn, the PO methods preserve constraints and achieve competitive solutions, including approximately 5.93× faster runtime than HIFOO in Case 3 and 8×, 47×, and 295× speedups in Cases 4–6.For the simple 3 × 3 comparison, the PO methods are also around 4× faster than h2hinfsyn.
  • Comparison with Existing H2/H∞ Control Solvers: For γ = 1, the PO methods preserve the H∞ constraint and converge to solutions with small H2 and H∞ norms, while HIFOO's performance degrades more.More than 90% of the PO runtime is spent finding a robustly stable initialization, especially for smaller γ.

8 Concluding Remarks

The paper establishes policy optimization methods for mixed H2/H∞ linear control and shows that their search directions preserve robustness constraints without explicit projection. It also identifies global-convergence results and outlines limitations and extensions, including the need for a feasible initialization.

  • 8 Concluding Remarks: The paper identifies open directions involving other policy-optimization methods, model-free sample complexity, broader dynamical systems, and output-feedback controllers.Output-feedback analysis is harder because the policy landscape depends on controller parameterization order.
  • 8 Concluding Remarks: The methods require an initial policy satisfying the specified H∞-norm constraint, and finding such an initialization can dominate runtime in the reported comparison.The paper states that random search for a feasible initial controller can take more than 90% of total runtime, while HIFOO uses an in-house initialization method.
  • 8 Concluding Remarks: The analysis covers continuous-time mixed H2/H∞ design, including its Riccati formulation, bounded-real characterization, nonconvexity, and absence of coercivity.The H∞ bound is related to stabilizing Riccati solutions and an equivalent matrix inequality.
  • 8 Concluding Remarks: Both proposed updates converge globally to an optimal controller under the stated assumptions, while avoiding suboptimal stationary points even when multiple stationary points exist.The continuous-time results characterize the optimizer as K∗ = R−1B⊤PK∗ and establish convergence under detectability and stepsize conditions.
  • 8 Concluding Remarks: The proposed policy optimization methods preserve the H∞ robustness constraint through implicit regularization despite the mixed design problem’s nonconvexity and lack of coercivity.The natural policy-gradient and Gauss–Newton updates remain within the stabilizing, H∞-feasible set under suitable stepsizes.

A.3.2 Global Convergence

For continuous-time mixed H2/H∞ design, the natural policy-gradient and Gauss–Newton updates converge globally to an optimal controller under suitable initialization, detectability, and stepsize conditions. The global rate is sublinear, while stronger local rates hold under an additional disturbance condition.

  • A.3.2 Global Convergence: Both updates converge to the global optimum K∗ with an O(1/N) rate for the average stationarity measure.The convergence result does not require the controllability assumption used in the stationary-point characterization.
  • A.3.2 Global Convergence: The updates avoid spurious stationary points and converge to the globally optimal controller even when multiple stationary points may exist.This behavior is attributed to the implicit regularization of the natural policy-gradient and Gauss–Newton search directions.
  • A.3.2 Global Convergence: Only a globally sublinear rate is established for mixed design, unlike the linear rate available in the cited LQR setting.The paper connects the difference to gradient dominance holding only locally for mixed design problems.
  • A.3.2 Global Convergence: With DD⊤ > 0, both methods achieve locally linear objective convergence, and the Gauss–Newton update with η = 1/2 achieves locally Q-quadratic convergence.These stronger rates apply around the optimum under the assumptions of the global-convergence theorem plus DD⊤ > 0.

B.2 Proof of Lemmas 3.1 and A.2

The proof constructs explicit discrete- and continuous-time examples showing that the H∞-feasible controller set is nonconvex. Averaging two feasible stabilizing controllers can produce a controller violating the robustness bound.

  • B.2 Proof of Lemmas 3.1 and A.2: For the discrete-time example, ∥T(K1)∥∞ = 0.4350 and ∥T(K2)∥∞ = 0.7011, but their average has ∥T(K3)∥∞ = 1.6575.Thus, the H∞ lower-level set is nonconvex for γ ∈ (0.7011, 1.6575).
  • B.2 Proof of Lemmas 3.1 and A.2: In both examples, K1, K2, and K3 stabilize the system, so nonconvexity arises specifically from the H∞ constraint rather than loss of stability.The averaged controller remains stabilizing while violating the relevant robustness ranges.
  • B.2 Proof of Lemmas 3.1 and A.2: For the continuous-time example, ∥T(K1)∥∞ = 0.3860 and ∥T(K2)∥∞ = 0.5306, whereas their average has ∥T(K3)∥∞ = 1.1729.Therefore, the feasible controller set is nonconvex for γ ∈ (0.5306, 1.1729).

B.3 Proof of Lemmas 3.2 and A.3

The proof establishes that mixed H2/H∞ costs can remain finite as policies approach the robustness-feasible-set boundary, demonstrating noncoercivity. It also verifies differentiability of the Riccati solution and objective throughout the feasible set.

  • B.3 Proof of Lemmas 3.2 and A.3: Finite cost persists near the feasible-set boundary, so mixed H2/H∞ objectives are not coercive.For discrete-time settings, the Riccati solution and the relevant matrix condition remain finite and nonsingular near the boundary.
  • B.3 Proof of Lemmas 3.2 and A.3: A one-dimensional construction reaches the boundary with K → 1.2573 while P_K → 3.3752 and both costs remain finite.The example uses A = 2.75, B = 2, C^2 = 1, R = 1, D^2 = 0.01, and γ = 0.2101.
  • B.3 Proof of Lemmas 3.2 and A.3: Consequently, the objective J(K) is differentiable at every feasible policy, enabling the policy-gradient expressions used later.The argument combines differentiability of J with respect to P_K and differentiability of P_K with respect to K.
  • B.3 Proof of Lemmas 3.2 and A.3: The feasible set is open because each feasible policy admits strict linear-matrix inequalities that persist in a sufficiently small neighborhood.This openness supports applying the implicit function theorem to the Riccati fixed-point equation.
  • B.3 Proof of Lemmas 3.2 and A.3: The Riccati solution P_K is continuously differentiable with respect to K throughout the feasible set.The proof establishes invertibility of the relevant Jacobian and applies the implicit function theorem.

B.6 Proof of Propositions 3.4 and A.5

The proof shows that the mixed H2/H∞ objective has a unique stationary point, and that this point is the unique global optimizer under the stated assumptions.

  • B.6 Proof of Propositions 3.4 and A.5: The proof compares Riccati values through an auxiliary game and shows P_K ≥ P_K* for every feasible K.Monotonicity of the objective in P_K then gives J(K) ≥ J(K*) for all feasible policies.
  • B.6 Proof of Propositions 3.4 and A.5: The gradient factor Δ_K is positive definite, making the stationary-point characterization unique.The argument uses positivity of I − γ^−2D^T P_K D and an observability condition.
  • B.6 Proof of Propositions 3.4 and A.5: The necessary condition ∇J(K) = 0 yields a unique stationary policy, which is therefore the unique global optimizer.For one formulation, the policy is K* = (R + B^T P̃_K* B)^−1 B^T P̃_K* A.
  • B.6 Proof of Propositions 3.4 and A.5: In continuous time, detectability of (A,C) is additionally required for the game-theoretic proof.The objective remains monotone in the eigenvalues of P_K under this setting.

B.7 Proof of Lemma 5.1

The proof derives upper and lower bounds for the objective difference between two feasible policies using Riccati identities and iterative inequalities.

  • B.7 Proof of Lemma 5.1: For any two feasible policies K and K′, the proof first establishes invertibility of I − γ^−2P_K′DD^T.This follows from positivity of I − γ^−2D^T P_K′ D and Sylvester’s determinant identity.
  • B.7 Proof of Lemma 5.1: Riccati relations are unrolled iteratively to obtain the upper bound in (5.25).The bound follows after substituting the policy-difference expression into the recursive inequality.
  • B.7 Proof of Lemma 5.1: Interchanging K and K′ yields the corresponding lower bound in (5.26).The proof applies the same lemma with the roles of the two policies reversed and continues unrolling the resulting recursion.

B.8 Proof of Theorem A.6

The proof establishes that the Gauss–Newton and natural policy-gradient updates preserve the H∞ robustness-feasible set by constructing a strict Lyapunov inequality for each update.

  • B.8 Proof of Theorem A.6: A policy update K′ is feasible whenever there exists P > 0 satisfying the strict matrix inequality (B.36).The inequality implies that A − BK′ is Hurwitz, allowing the bounded-real lemma to certify ||T(K′)||∞ < γ.
  • B.8 Proof of Theorem A.6: The Gauss–Newton update with stepsize η = 1/2 satisfies the strict inequality and therefore preserves feasibility.The proof chooses P = P_K + αP̄, where P̄ solves a Lyapunov equation associated with the current closed-loop system.
  • B.8 Proof of Theorem A.6: For the natural policy-gradient update, stepsize η ≤ 1/(2||R||) suffices to preserve the same feasibility certificate.A sufficiently small α makes the combined inequality strict, after which the bounded-real lemma applies.
  • B.8 Proof of Theorem A.6: The certificate also guarantees existence of the stabilizing Riccati solution P_K′ ≥ 0 for the updated policy.This follows directly after establishing ||T(K′)||∞ < γ.

B.9 Proof of Theorem A.7

The continuous-time cost-difference lemma bounds changes in P_K through Lyapunov equations, enabling analysis of Gauss–Newton and natural policy-gradient updates. Both methods preserve feasibility, decrease P_K monotonically, and converge globally at a sublinear rate.

  • Cost-difference bounds: The cost-difference lemma compares P_K′−P_K with K′−K through Lyapunov equations and establishes matching upper and lower bounds.Stability of the relevant closed-loop matrices gives unique Lyapunov solutions whose ordering yields the bounds.
  • Gauss–Newton update: For step sizes η≤1/2, the Gauss–Newton update remains feasible and decreases P_K matrix-wise along the iterations.The resulting monotone sequence converges to a feasible limit.
  • Global convergence: The Gauss–Newton iterates converge to the global optimum K* with a globally sublinear O(1/N) rate, avoiding suboptimal stationary points.The limiting stationary condition is RK−B^⊤P_K=0, which Proposition A.5 identifies with the global optimum.
  • Natural policy gradient: For step sizes η≤1/(2∥R∥), the natural policy-gradient update likewise remains feasible and decreases P_K matrix-wise.The same cost-difference bounds support the descent argument for natural policy gradient.

B.10 Proof of Theorem A.8

The proof establishes local convergence near K* by controlling Lyapunov solutions on a neighborhood where the optimal closed-loop matrix is Hurwitz. It derives local linear convergence generally and Q-quadratic convergence for the specified step size under DD^⊤>0.

  • Local neighborhood: Under DD^⊤>0, the Gauss–Newton iterates eventually remain in a neighborhood of K* where the Lyapunov comparison matrix is uniformly bounded.A suitable lower-level set is contained in the neighborhood, preventing later iterates from leaving it.
  • Local linear convergence: The resulting trace sequence contracts locally at a uniform linear rate once an iterate enters that neighborhood.The contraction applies to Tr(P_KnDD^⊤).
  • Local quadratic convergence: For η=1/2, the Gauss–Newton update achieves Q-quadratic convergence of Tr(P_KnDD^⊤) around K*.The proof uses the special update K′=R^−1B^⊤P_K and bounds the local comparison constants.

C Auxiliary Results

The auxiliary results derive finite-horizon LEQG recursions and their infinite-horizon limits, then establish convergence and uniqueness properties for modified Riccati equations. These results identify the optimal stationary linear controller and objective value used elsewhere in the analysis.

  • LEQG recursion: Gaussian integration converts the finite-horizon LEQG dynamic-programming step into a quadratic control minimization with modified Riccati recursion updates.The transformed matrix eP_t and scalar λ_t produce the quadratic expression minimized over u_t.
  • Optimal control: Among controls with a well-defined objective, the optimal control is LTI state feedback μ_t=−K*x_t and its objective equals −β^−1logdet(I−βP_K*W).This conclusion follows from the stabilizing modified Riccati solution.
  • Infinite-horizon limit: The finite-horizon average converges to λ*=−β^−1 logdet(I−βP_K*W), with the terminal-value contribution vanishing as the Riccati iterates converge.The limiting value is attained by the stationary linear feedback K*.
  • Modified Riccati equation: The modified Riccati recursion is monotone and bounded, so it converges to a fixed point; the stabilizing fixed point is unique and is the minimal fixed point.Monotonicity follows from the order-preserving Riccati map and boundedness from the stabilizing solution.
  • Fixed-controller recursion: For a stabilizing controller K, the Riccati recursion is monotone and bounded and converges to the minimal stabilizing solution P_K.The recursion remains well defined because W^−1−βP_K is positive definite.
Loading 1910.09496v3…