Weak Convexity and Proximal Bundle Methods for Nonsmooth Policy Optimization in Robust Control

1 Introduction

• H∞ policy optimization is nonsmooth and nonconvex, with a generally nonconvex stabilizing-gain domain that may be disconnected under output feedback; prior work lacked deterministic non-asymptotic guarantees with feasibility preservation.Nonsmoothness comes from worst-case maximization in the H∞ norm.Source: §1 · pp. 1–2
• The paper develops a proximal bundle method as a tractable approximation of the proximal point method, replacing its exact subproblem with a convexified, piecewise-affine lower model.The structural analysis uses complementary frequency-domain and LMI representations of the cost.Source: §1 · p. 2
• The H∞ cost is weakly convex on convex subsets of sublevel sets, and the state-feedback case satisfies a weak PL inequality that makes every stationary point globally optimal.The weak-convexity guarantee is uniform over the prescribed sublevel set, unlike the local weak convexity from the lower-C2 property.Source: §1 · p. 2
• With η=Θ(ε), the proximal bundle method has O(ε^-4) complexity and preserves feasibility in both inner and outer loops.The method uses function-value and subgradient oracles, with an analytic inner-loop update under a suitable lower approximation.Source: §1 · p. 2

1.1 Related works

• Prior H∞ policy methods offered asymptotic or probabilistic guarantees, while earlier weakly convex bundle methods assumed convex feasible sets; stability makes this problem's feasible set nonconvex.The paper positions its guarantee as the first deterministic, non-asymptotic one for discrete-time H∞ policy optimization.Source: §1.1 · pp. 3–4

1.2 Paper organization

• The paper proceeds from problem formulation and landscape analysis to stationary-point properties, the proximal bundle method, numerical experiments, and conclusions.Source: §1.2 · p. 4

2.1 Policy optimization for robust H∞control

• The robust-control problem minimizes worst-case disturbance amplification over stabilizing static output-feedback policies for a discrete-time system.The disturbance is modeled as an adversarial input with bounded energy, and the initial state is fixed at zero.Source: §2.1 · pp. 4–5
• The formulation assumes stabilizability of (A, B), full row rank of Bw and C, positive-definite Q and R, and a nonempty stabilizing-gain set.The objective is the H∞ norm of the disturbance-to-output map; minimizing it or its square gives the same optimal solutions.Source: §2.1 · pp. 4–5

2.2 Preliminaries for nonsmooth nonconvex optimization

• The preliminaries define inexact subdifferentials and (η, ϵ)-stationarity to express approximate stationarity for nonsmooth, nonconvex objectives.When η and ϵ are zero, the definition recovers exact stationarity, 0 ∈ ∂f(x).Source: §2.2 · pp. 6–7

3.1 Basic landscape properties

• The stabilizing-gain set is open and generally nonconvex, while the H∞ cost is generally nonsmooth, nonconvex, coercive, and Lipschitz on each sublevel set.With full-state measurement the feasible set is path-connected; in general output feedback it may be disconnected.Source: §3.1 · p. 7
• Figure 1 illustrates nonsmooth costs, a disconnected feasible set, and a nonconvex cost for output feedback at α = 0.13.The examples show nonsmoothness in panels (a)–(b), disconnectedness in (c), and nonconvexity in (d).Source: Figure 1 · p. 8, §3.1 · p. 8
• For continuous-time H∞ control, coercivity need not hold, sublevel sets may not be compact, and a uniform Lipschitz constant on them is unclear.The stabilizing-gain set remains nonconvex and may be path-disconnected in general.Source: §3.1 · p. 8

3.2 Lower-C2 and weak convexity

• The H∞ cost is lower-C2, locally weakly convex, and subdifferentially regular, with its nonsmoothness represented through local maxima of smooth functions.The local weak-convexity constant may depend on the point and neighborhood, making it difficult to quantify explicitly.Source: §3.2 · pp. 9–10
• On any convex subset of a sublevel set, adding a uniform quadratic term makes the H∞ cost convex; compactness and a global smooth-component representation yield this uniform parameter.The resulting weak-convexity constant applies across the prescribed sublevel set, strengthening the local property implied by lower-C2 regularity.Source: §3.2 · p. 10

4.1 Subgradient characterization and Lipschitz continuity

• The Fréchet subdifferential is the convex hull of gradients from active smooth components, including cases with infinitely many maximizing frequencies; J is Lipschitz on convex subsets of sublevel sets.The subdifferential formula applies even when the set of maximizing frequencies is infinite.Source: §4.1 · pp. 11–13

4.2 Non-optimal stationary points and a weak PL condition

• Static output feedback may have a strict saddle; under full-state measurement (C = Inx), the H∞ problem satisfies a weak PL condition, and every stationary point is globally optimal.In Example 5 at α=0.14, Ks≈[−1.92, −0.26] has J(Ks)≈106.72 and Hessian eigenvalues λmin≈−174.41 and λmax≈2001.75.Source: §4.2 · pp. 13–14

4.3 Proof of the weak PL condition in Theorem 2

• For full-state feedback, a lifted hidden-convex formulation recovers J by partial minimization and supports the proof of the weak PL condition.Compact lifted sublevel sets and conditioning of the inverse map provide the bound used to establish the inequality.Source: §4.3 · pp. 15–17

5 A first-order proximal bundle method

• The analysis starts from a stabilizing gain and uses weak-convexity and Lipschitz constants on an enlarged sublevel set.The distance from the initial sublevel set to the enlarged set's boundary is strictly positive.Source: §5 · pp. 17–18

5.1 Proximal point method (PPM)

• With a sufficiently large proximal parameter, each proximal point subproblem has a unique minimizer, and all iterates remain in the initial sublevel set.The method finds an iterate with dist(0, ∂J(Kt)) ≤ ϵ within O(1/ϵ^2) iterations.Source: §5.1 · pp. 18–19

5.2 Proximal bundle method (PBM) as an inexact PPM

• The proximal bundle method approximates proximal point updates with an inner bundle procedure that enforces a relaxed descent condition at each outer iteration.This makes the method an inexact proximal point method, whereas exact solution of the proximal subproblem is not directly implementable.Source: §5.2.1 · p. 19, §5.1 · p. 19, §5.2 · p. 19

5.3 Non-asymptotic convergence guarantees

• For sufficiently large ρ, every inner-loop minimizer lies in the interior of the enlarged feasible sublevel set, despite the subproblem having no explicit stabilizing constraint.This establishes feasibility of inner iterates generated by unconstrained minimization.Source: §5.2.1 · p. 20, §5.3 · p. 23

5.2.1 A proximal bundle method for H∞optimization

• The bundle procedure builds piecewise-linear lower models and accepts a candidate when its test holds, ensuring descent in function value.The subproblem reduces to a quadratic program, and the two-cut model has a closed-form solution.Source: §5.2.1 · pp. 20–22
• A spectral bundle model can capture the cost's eigenvalue and frequency structure and reduce the subproblem to a simple quadratic program.For sufficiently large m, this model satisfies the lower-approximation property.Source: §5.2.1 · p. 22

5.3 Non-asymptotic convergence guarantees

• The convergence analysis treats total iteration complexity as the product of separate inner- and outer-loop bounds, using a small proximal gap to obtain an inexact subgradient guarantee.The section introduces this complexity analysis but does not yet state the resulting total bound in the supplied passages.Source: §5.3 · p. 23
• Under Assumptions 1 to 4, for H∞ policy optimization (8), Algorithm 3 with inner-loop Algorithm 2 yields an (η, ϵ)-stationary point, with {K_t} ⊂ K0 and {L_k} ⊂ int(eK0).Choose ρ > 0 as in Lemma 10.Source: §5.3 · p. 24
• The rate is the first deterministic non-asymptotic convergence guarantee with anytime feasibility for H∞ policy optimization, enabled by careful control of inner-iterate feasibility.The analysis relies on local weak convexity of J and the nonconvexity of K; setting η = Θ(ε) gives O(ε^-4) iterations.Source: §5.3 · p. 25
• Unlike standard subgradient methods, PBM accepts outer updates only after a descent test and keeps inner iterates feasible, providing an anytime feasibility guarantee.Standard subgradient analysis does not ensure feasibility on the nonconvex stabilizing set K.Source: §5.3 · p. 25
• In the state-feedback case, the weak PL condition improves the total complexity to O(ε^-3), with O(ε^-1) outer iterations and O(ε^-2) per inner loop.The weak PL condition also translates stationarity into global optimality.Source: §5.3 · p. 25

6.1 A simple academic example

• In the simple example, the plots show convergence toward stationarity and global optimality, with the best performance for PBM using the least conservative m = 0.005.The plots track objective gap and exact stationarity against total iterations, including inner-loop evaluations; flat regions represent null steps.Source: §6.1 · p. 26, Figure 4 · p. 26

6.2 Additional experiments

• The experiments compare PBM with subgradient and randomized smoothing methods on two partial-observation systems, including an open-loop unstable system and an aircraft model.The state-feedback cases are also compared, with the optimal value computable by the Riccati-based MATLAB solver hinfsyn.Source: §6.2 · p. 26, Figure 5 · p. 27
• For the open-loop unstable system, PBM has the most favorable performance and returns a near-optimal state-feedback solution faster than the other methods.The state-feedback comparison uses K0C as its initial point.Source: §6.2 · p. 27
• In the aircraft example, PBM achieves the best convergence speed and accuracy, obtains a near-optimal state-feedback solution, and does not increase the function value.The other methods, particularly RSM, do not always generate a descent direction.Source: §6.2 · p. 28

• The H∞ cost is weakly convex on convex subsets of sublevel sets, and in state feedback a weak PL inequality guarantees that stationary points are globally optimal.Source: §7 · p. 28
• The proximal bundle method preserves stability at every inner- and outer-loop iterate and finds an (η, ϵ)-stationary point within O(max{η^-4, ϵ^-2}) iterations.Source: §7 · p. 28
• Future work includes extending the analysis and algorithm to broader classes of robust control problems.Source: §7 · p. 28

• For continuous-time systems, the H∞ cost has an explicit subgradient because the cost has the lower-C2 property.The expression uses jω in place of e^jω in Proposition 1.Source: §A.1 · p. 32
• The continuous-time active set consists of frequency and unit-vector choices for which φc equals the cost Jc.Source: §A.1 · p. 33
• The set Θc specifies frequencies in [0, ∞] and unit-norm complex vectors u and v of the stated dimensions.Source: §A.1 · p. 33

• For a nonoptimal K, the proof chooses a positive β from the largest singular value of Γ and the distance between lifted points, then uses the arbitrariness of G in ∂Jlft.Source: §A.2 · p. 33
• The proof applies the convexity inequality to the lifted objective Jcvx and compares its value with that at an optimal triplet.Source: §A.2 · p. 33
• The chain rule maps subgradients through the transformation Υ, whose inverse Π has an invertible Jacobian Γ because Υ is a diffeomorphism.Source: §A.2 · p. 33

A.3 Proof of Lemma 5

• The classical result applies because the minimizing pair exists, the constraint is closed on its open domain, and J_lft is lower semicontinuous there.The proof also rules out feasible sequences approaching P not positive semidefinite and γ = 0.Source: §A.3 · p. 34
• The proof establishes bounded level sets by deriving uniform bounds on P and then K, while γ is bounded by the level value.It notes that an earlier proposition also gives this boundedness result.Source: §A.3 · p. 34
• The Fréchet and general subdifferentials coincide for J because the cost is subdifferentially regular, validating inclusion (58).In general, the Fréchet subdifferential is contained in the general subdifferential; the proof invokes regularity to obtain equality.Source: §A.3 · p. 34

B.1 Proof of Lemma 8

• The induction for Lemma 8 keeps each inner iterate within distance d of K_t, placing it in the interior of the neighborhood around K_0.The argument uses the optimality condition, Assumption 4, Lipschitz continuity of the bundle model, and the stated parameter inequality.Source: §B.1 · p. 35

B.2 Proof of Lemma 9

• The nearby point used in Lemma 9 lies in K_0 and is less than distance d from K, with its existence guaranteed by coercivity.The proof also bounds a quantity involving the objective difference and the distance between the points.Source: §B.2 · p. 35
• Lemma 9 constructs a local subgradient of the difference function whose norm is bounded by √2ρΔ_K.The construction uses a nearby point whose existence follows from coercivity and considers the largest ball on which weak convexity holds.Source: §B.2 · pp. 35–36

B.3 Proof of Lemma 11

• The accepted inner candidate is evaluated using its bundle model, which is bounded above by h_t because the true proximal point lies in the local ball.This is the stated setup for the acceptance argument.Source: §B.3 · p. 36
• The acceptance test supplies an inequality that the proof subsequently sums by telescoping.The supplied proof passage identifies the test and the telescoping step without stating the inequality's terms.Source: §B.3 · p. 36
• If min_t Δ_t ≤ σ, Lemma 9 implies that some K_t is an (η, ε)-stationary point.The proof separately considers the case min_t Δ_t > σ and derives a further bound.Source: §B.3 · p. 36