Source-linked AI summary
Algorithmic Optimality Guarantees for Nonsmooth $H_\infty$ Output-Feedback Policy Search
Ashkan Soleymani, Patrick Jaillet
TL;DR
Dynamic output-feedback H∞ policy search is nonconvex and nonsmooth, and prior optimality results leave a quantitative gap between approximate stationarity and value loss. The paper uses the extended convex lift to establish first-order and algorithmic guarantees, including recovery of an explicit ε-optimal stabilizing controller.
Problem
Prior dynamic output-feedback H∞ results certify exact Clarke-stationary controllers but do not bound value loss from approximate stationarity or provide convergence rates for finding an optimal policy.
Source: §I · p. 1Method
The paper uses the exact identity-gauge slice for local first-order analysis and the full nonstrict extended convex lift for global value computation and controller recovery.
Source: §I · p. 2Results
On compact exact slices, ε-stationarity yields O(ε) suboptimality, and nonstrict-feasibility bisection with one strict-feasibility recovery step returns an explicit ε-optimal stabilizing controller.
Source: §I · p. 2, §VI · p. 8Takeaways & Limitations
The lift’s local and global roles together provide a quantitative and algorithmic treatment of nonsmooth dynamic output-feedback H∞ policy search.
Source: §I · p. 2, §VI · p. 8
Abstract
from arXiv · showhide
We study continuous-time full-order dynamic output-feedback $H_\infty$ policy search, a nonconvex and nonsmooth problem. Direct policy search is a central paradigm in reinforcement learning and continuous control, but rigorous guarantees remain scarce in robust output-feedback settings. The $H_\infty$ problem is a canonical benchmark because it captures disturbance attenuation and robustness while exposing the hard nonsmooth geometry of policy-space optimization. We prove that on the exact identity-gauge slice of the extended convex lift, $\varepsilon$-stationarity yields $O(\varepsilon)$-suboptimality on compact exact slices, which in turn yields convergence-rate guarantees for nonsmooth policy-search methods. This result addresses the finite-time optimality-gap question raised by Guo and Hu [2022] in the more general dynamic output-feedback $H_\infty$ policy-search setting. We further use the established value equivalence supplied by extended convex lifting to formulate a nonstrict-feasibility bisection method with one final strict-feasibility recovery step, yielding an explicit $\varepsilon$-optimal stabilizing controller. These results provide a quantitative and algorithmic strengthening of prior qualitative optimality theory for nonsmooth $H_\infty$ policy search.
I. Introduction
Dynamic output-feedback policy search faces stationarity challenges because controller-state similarity creates redundant directions in partially observed settings. For H∞ search, the remaining gap is quantitative and algorithmic: the paper separates the lift’s global value role from the identity-gauge slice’s local certification role and develops associated guarantees.
I. Introduction
- Direct policy search optimizes a parameterized controller against closed-loop performance and is a core paradigm in reinforcement learning and continuous control.Its appeal includes end-to-end optimization with richly parameterized controllers and compatibility with model-based, model-free, and data-driven implementations.Source: §I · p. 1
- Controller-state similarity creates redundant directions in dynamic output feedback, complicating stationarity and potentially obstructing naive Euclidean descent.The paper notes that suitable reparameterizations or quotient-geometric views can restore meaningful first-order analysis.Source: §I · p. 1
- H∞ policy-space optimization is nonconvex and nonsmooth; prior work establishes qualitative globality under exact-stationarity or nondegeneracy conditions.Guo and Hu treat state feedback, while Tang and Zheng treat dynamic output feedback under a nondegenerate exact-certificate condition.Source: §I · p. 1
- The quantitative gap is converting approximate stationarity into a value-gap bound and algorithmic convergence rates for dynamic output-feedback H∞ search.Prior qualitative results certify exact Clarke-stationary points but do not provide this conversion.Source: §I · p. 1
- The extended convex lift has complementary roles: its full nonstrict set computes global value, while its exact identity-gauge slice supports local first-order certificates.Strict lifted points also support stabilizing-controller recovery.Source: §I · pp. 1–2
- The paper proves an exact reconditioning lemma, an O(ε) suboptimality bound on compact exact slices, convergence-rate results for nonsmooth search, and an ε-optimal stabilizing-controller recovery procedure.The reconditioning makes P12 = I; recovery combines nonstrict-feasibility bisection with one final strict-feasibility step.Source: §I · p. 2
A. Dynamic output-feedback H∞policy search
The section sets up continuous-time dynamic output-feedback policy search and its nonsmooth objective. It also states regularity properties of the objective and contrasts Clarke and Goldstein stationarity.
A. Dynamic output-feedback H∞policy search
- The policy-search setup uses a controller with its own state and views controller space as a finite-dimensional Euclidean space with the Frobenius inner product.The plant and controller notation identifies the exogenous input, control input, performance output, and measured output.Source: A. Dynamic output-feedback H∞policy… · p. 2
- The policy-search objective J is locally Lipschitz and subdifferentially regular on the stabilizing controller set.Its directional derivative equals the Clarke directional derivative and the maximum inner product over the Clarke subdifferential.Source: A. Dynamic output-feedback H∞policy… · p. 2
- The section distinguishes Clarke stationarity from Goldstein stationarity, noting the latter’s algorithmic tractability in contrast to Clarke stationarity’s finite-time intractability.The cited discussion states there is no algorithm that can find Clarke stationary points in finite time [22, Theorem 5].Source: A. Dynamic output-feedback H∞policy… · p. 3
B. Nondegenerate bounded-real certificates and Extended Convex Lifting
This section introduces the extended convex lift and the nondegenerate certificate setting for dynamic output-feedback H∞ policy search. It distinguishes algebraic nonstrict certificates from stabilizing certificates and motivates the paper’s quantitative focus.
B. Nondegenerate bounded-real certificates and Extended Convex Lifting
- A nonstrict certificate in Snd with a stabilizing controller implies J(K) ≤ γ, but the nonstrict inequality alone does not guarantee internal stability.The section distinguishes the algebraic certificate set from its stabilizing part.Source: B. Nondegenerate bounded-real… · p. 3
- The extended convex lift uses variables X, Y, M, H, F, L, and γ to define a convex lifted feasible set.The lifted matrix map is denoted M(z).Source: B. Nondegenerate bounded-real… · p. 3
- The ECL coordinate map is a real-analytic diffeomorphism whose first coordinate is the certificate block P12.Its inverse is Ψ, and the paper uses the map to work with lifted coordinates.Source: B. Nondegenerate bounded-real… · p. 3
- The prior globality result for exact Clarke-stationary controllers is qualitative, motivating a quantitative link between subdifferentials and suboptimality.The section frames its goal as an algorithmic framework for finding optimal dynamic output-feedback H∞ policies.Source: B. Nondegenerate bounded-real… · p. 3
III. Exact reconditioning and the identity gauge slice
Exact controller certificates can be reconditioned by controller-state similarity so that P12 = I, defining an identity-gauge slice. The slice provides an analytic chart for exact certificates and preserves realization-equivalent controller performance.
III. Exact reconditioning and the identity gauge slice
- Every exact nondegenerate controller has a realization-equivalent representative whose certificate satisfies P12 = I.The transformation preserves the H∞ norm and internal stability through controller-state similarity.Source: §III · p. 4
- The map ΨI is a real-analytic diffeomorphism from F onto the identity-gauge slice SI.Its construction fixes the first lifted coordinate at the identity matrix.Source: §III · p. 4
- The stabilizing part Fst is relatively open in F, and the slice chart restricts to an analytic bijection on that part.Every exact nondegenerate controller has a realization-equivalent representative in the stabilizing slice.Source: §III · p. 4
- The exact slice EI is relatively closed in Fst, so its intersection with a compact subset of Fst is compact.This compactness property supports analysis restricted to compact exact slices.Source: §III · p. 4
IV. Quantitative first-order error bounds on the exact slice
The exact-slice analysis connects first-order stationarity to value suboptimality, providing a basis for nonsmooth policy-search convergence guarantees. Its algorithmic implication depends on compact-slice and continuity conditions.
IV. Quantitative first-order error bounds on the exact slice
- Every Clarke-stationary point in the exact slice is globally optimal when the slice infimum equals the policy optimum.The lifted convex objective can also attain values arbitrarily close to the policy optimum.Source: §IV · p. 6
- On compact exact slices, Corollary 12 bridges approximate stationarity to value suboptimality when the slice contains a global optimizer.The corollary is deliberately abstract and does not establish that a particular algorithm generates exact-slice iterates.Source: §IV · p. 6
- Under one-sided Hausdorff continuity of the Clarke subdifferential, Goldstein-stationarity guarantees can be connected to convergence of the policy objective.This condition enables a reverse inequality up to an additive term.Source: §IV · p. 6
- INGD requires O~(1/(δ ε^3)) gradient and function evaluations to find a policy that is (ε, δ)-Goldstein stationary.The stated convergence-rate result assumes the subdifferential continuity condition.Source: §IV · p. 6
V. Global value computation and strict controller recovery in the convex lift
Established value equivalence for extended convex lifting separates global value computation from stabilizing-controller recovery. The resulting feasibility-bisection procedure uses nonstrict feasibility for value computation and strict lifted points for recovery.
V. Global value computation and strict controller recovery in the convex lift
- The ECL lifted problem and the policy objective have the same infimum, including for dynamic output-feedback H∞ control.This established value identity underpins the feasibility-bisection procedure.Source: §V · p. 6
- A nonstrict feasible lift need not decode to an internally stabilizing controller, whereas strict lifted points support stabilizing-controller recovery.The method distinguishes nonstrict value computation from strict recovery.Source: §V · pp. 6–7
A. Strict lifted points
Strict lifted points yield stabilizing controllers whose performance is strictly below the lifted objective value.
A. Strict lifted points
- For a strict lifted point, the recovered controller is stabilizing and has H∞ performance below γ(z).The argument uses positive definiteness of the recovered Lyapunov matrix and the strict bounded-real lemma.Source: A. Strict lifted points · p. 7
B. Strict nondegenerate approximation of stabilizing controllers
Every stabilizing controller admits a strict nondegenerate lifted approximation that preserves realization equivalence and can place the lifted objective arbitrarily close above the controller's value.
B. Strict nondegenerate approximation of stabilizing controllers
- For any stabilizing controller K and η > 0, a strict lifted point can recover a realization-equivalent controller with γ(z) = J(K) + η.The strict approximation is obtained by perturbing the certificate while preserving the required definiteness conditions.Source: B. Strict nondegenerate approximation… · p. 7
C. Value identity
The extended convex lift preserves the global optimal value without requiring attainment on a prescribed exact slice. This value identity concerns infima; a nonstrict-lift minimizer need not itself decode to a stabilizing controller.
C. Value identity
- The lifted-value identity equates the global policy-cost infimum with the infimum over the full nonstrict lift, without requiring attainment on a prescribed exact slice.The value theorem applies when the stabilizing-controller set C is nonempty.Source: C. Value identity · p. 7
- If a strict lifted point attains the lifted minimum, its decoded controller is globally optimal.The corollary assumes z⋆ lies in Fst and achieves the minimum over F.Source: C. Value identity · p. 7
- A minimizer of the nonstrict lift need not be a stable decoded controller, and degeneracy may prevent the lifted infimum from being attained.Strict lifted points can approach the optimal stabilizing-controller cost, so these issues do not change the value identity.Source: C. Value identity · pp. 7–8
D. Bisection over the convex lift
The convex-lift bisection procedure brackets the optimal value using nonstrict feasibility tests, then uses one strict-feasibility test to recover an ε-optimal stabilizing controller.
D. Bisection over the convex lift
- The feasibility problems are convex because their constraints are affine in the decision variables.The variables named in the passage are (X, Y, M, H, F, L).Source: D. Bisection over the convex lift · p. 8
- Feasibility is monotone in the upper-bound parameter, and a feasible nonstrict lift implies that the optimal value is no greater than that parameter.For any parameter above the optimal value, strict feasibility holds; strict feasibility also implies nonstrict feasibility.Source: D. Bisection over the convex lift · p. 8
- The procedure combines nonstrict-feasibility bisection with one final strict-feasibility recovery step to produce an ε-optimal stabilizing controller.The stated output satisfies J(Kε) < bγ ≤ J⋆ + ε.Source: D. Bisection over the convex lift · p. 8
- The search starts by doubling a feasible upper bound and then bisects an interval bracketing J⋆ until its length meets the stopping condition.The interval invariant is Lj ≤ J⋆ ≤ Uj, and its length halves at each bisection step.Source: D. Bisection over the convex lift · p. 8
VI. Conclusion
The paper gives quantitative and algorithmic guarantees for nonsmooth dynamic output-feedback H∞ policy search through the extended convex lift. It connects local stationarity bounds on an exact slice with global-value computation and stabilizing-controller recovery.
VI. Conclusion
- On the stable exact identity-gauge slice, approximate stationarity controls suboptimality on compact exact slices.The paper also reports convergence-rate analysis for nonsmooth policy-search methods.Source: §VI · p. 8
- The lifted-value identity supports nonstrict-feasibility bisection followed by strict-feasibility recovery of an explicit ε-optimal stabilizing controller.Source: §VI · p. 8
- Future work includes weakening the outer-continuity assumption, analyzing algorithms near the exact slice without exact iterates, and extending the framework to structured, reduced-order, or data-driven robust control.Source: §VI · p. 8