PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Gradient over History-Dependent Policy Classes for LQR with Domain Randomization

Tesshu Fujinami, Bruce D. Lee, Anastasios Tsiamis, Nikolai Matni, George J. Pappas

arXiv:2609.16300v1eess.SYcs.SY

TL;DR

The paper addresses limited understanding of policy-gradient optimization for domain-randomized LQR, including the limits of static controllers for simultaneous stabilization. It studies history-dependent FIR policies and a curriculum that expands controller memory, establishing global convergence under bounded heterogeneity. Its results motivate history-dependent policies for collections that static feedback cannot simultaneously stabilize.

  • Problem

    The effects of the objective, randomization distribution, and policy parameterization on domain-randomized LQR policy efficacy and convergence are not well understood, and static controllers may not simultaneously stabilize all systems.

    Source: §I · p. 1

  • Method

    The paper analyzes policy gradient for history-dependent FIR policies and proposes a curriculum that gradually expands controller memory during optimization.

    Source: §I · p. 1, B. Contribution · p. 2

  • Results

    Policy gradient over FIR policies converges globally to a sample-average objective minimizer under bounded heterogeneity and initial stabilization.

    Source: B. Contribution · p. 2, §III · p. 3

  • Takeaways & Limitations

    FIR policies motivate domain-randomized control for system collections that lack a simultaneously stabilizing static controller.

    Source: §VII · p. 8

    The optimality analysis introduces a heterogeneity assumption, so its convergence result is scoped to environments satisfying that condition.

    Source: B. Contribution · p. 2, §VI · p. 7

Abstract

from arXiv · show

Domain Randomization (DR) has been widely used to overcome the sim-to-real gap by training a controller on a distribution of simulated environments via reinforcement learning. While DR can achieve robust performance simply using controllers synthesized via policy gradient (PG) methods, the optimization landscape is not well understood, even in the case of linear quadratic regulator (LQR) objectives. To this end, we first study PG of domain randomized LQR over history-dependent policy classes, such as finite impulse response controllers, as they can extend the possibilities of simultaneous stabilization. Second, to find such a stabilizing controller, we propose a curriculum learning based algorithm which gradually expands the memory of the controller. Finally, we show that PG with the proposed algorithm converges globally to the minimizer of a sample average approximation of the DR objective under suitable bounds on the heterogeneity of environments. Empirical results support our findings and highlight promising directions for future work, including nonlinear domain-randomized control.

I. INTRODUCTION

The paper extends domain-randomized LQR policy-gradient analysis to history-dependent FIR policies, addressing limits of static controllers for simultaneous stabilization. It proposes gradually expanding controller memory and proves global convergence under bounded heterogeneity.

I. INTRODUCTION

  • DR averages control objectives over system parameters and is used in robot learning to introduce simulator-parameter robustness for sim-to-real transfer.The effects of objective, randomization distribution, and policy parameterization on policy efficacy and convergence remain poorly understood.Source: §I · p. 1

A. Related Work

  • Prior work analyzed DR-LQR policy gradients for static controllers; arbitrary history-dependent policies connect to dual control, whose solution is generally intractable.The paper instead studies linear, stationary FIR policies; DR is an average-case robustness problem, unlike worst-case H∞ control and µ-synthesis.Source: A. Related Work · p. 1

B. Contribution

  • Policy gradient over FIR policies converges globally to the sample-average objective’s minimizer under bounded heterogeneity.The stated result assumes an initially stabilizing controller.Source: B. Contribution · p. 2, §II · p. 3
  • A curriculum-learning approach gradually expands policy memory during optimization until satisfactory performance is achieved.Source: B. Contribution · p. 2
  • Illustrative examples show FIR policies can benefit simultaneous stabilization, and they are more amenable to optimization and outperform other dynamic parameterizations.The passage also describes a comparison with static policies for simultaneous stabilization.Source: B. Contribution · p. 2

II. PROBLEM FORMULATION

  • A one-step-memory policy simultaneously stabilizes two scalar systems that no static linear feedback controller can stabilize together.The example motivates FIR controllers because memory can also yield lower DR-LQR cost, while static controllers may have infinite cost.Source: §II · p. 2

III. GLOBAL CONVERGENCE OF POLICY GRADIENT

The paper establishes global convergence of policy gradient over FIR policy classes to a sample-average optimum under bounded heterogeneity and initial stabilization. Its guarantee depends on sampled systems being sufficiently close and on starting from a suitable stabilizing controller.

III. GLOBAL CONVERGENCE OF POLICY GRADIENT

  • Under bounded heterogeneity and initial stabilization, policy gradient with a suitable stepsize converges globally to a minimizer of the sample-average objective over FIR policies.The heterogeneity gap must be sufficiently small, and the convergence result concerns the sample-average objective.Source: §III · p. 3
  • The heterogeneity condition is checkable from problem data but may exclude simultaneously stabilizable systems that are not close under the stated measure.The authors leave relaxing this condition, or determining whether it is fundamental, for future work.Source: §III · pp. 3–4
  • The convergence guarantee requires an initial controller that simultaneously stabilizes the sampled systems and has cost within a factor of 8 of optimal.The paper later presents a scheme for finding such an initial controller.Source: §III · pp. 3–4
  • The theorem guarantees convergence for the sample-average objective; concentration arguments relate it to the population domain-randomization objective as the sample count grows.The cited argument extends from static to FIR policies.Source: §III · p. 4
  • The proof combines coercivity and smoothness of the sample-average cost with gradient domination, which holds under small heterogeneity.The lifted FIR-LQR state introduces singular noise covariance and state penalties, so existing analyses are insufficient directly.Source: §III · p. 4

A. Coercivity and Smoothness of the Sample Average Objective

The section establishes coercivity and smoothness of the FIR sample-average LQR objective on cost sublevel sets, supporting policy-gradient convergence to a fixed point.

A. Coercivity and Smoothness of the Sample Average Objective

  • The sample-average FIR-LQR objective is coercive and twice continuously differentiable, and is L-smooth on each finite cost sublevel set.Coercivity follows from the individual costs diverging toward the stability boundary and from the sample-average cost bounding each individual cost.Source: A. Coercivity and Smoothness of the… · p. 4
  • Establishing coercivity for FIR controllers requires handling degenerate noise in the lifted state, which prevents direct use of techniques from prior analyses.The proof uses the structure of state impulse responses to establish bounded controller gains.Source: A. Coercivity and Smoothness of the… · p. 4
  • Smoothness on the sublevel set ensures policy gradient remains in that set, decreases the objective, and converges to a fixed point.The section also establishes uniform bounds on each sampled system's cost within the sublevel set.Source: A. Coercivity and Smoothness of the… · pp. 4–5

B. Gradient Domination of the Sample Average Objective

The section establishes gradient domination for the sample-average objective when heterogeneity is small, addressing analytical difficulties caused by the lifted FIR-LQR formulation.

B. Gradient Domination of the Sample Average Objective

  • The lifted FIR-LQR state has degenerate process noise and state cost, so prior Lyapunov perturbation arguments cannot be applied directly.The authors derive new cost expressions to address this issue.Source: B. Gradient Domination of the Sample… · p. 5
  • Without a heterogeneity condition, the section establishes only approximate gradient domination for the sample-average objective.The bound is stated using a positive lower bound on the minimum eigenvalue of the optimal covariance matrices.Source: B. Gradient Domination of the Sample… · p. 5
  • Small heterogeneity yields gradient domination, the key property used with coercivity and smoothness to prove global convergence.The admissible heterogeneity threshold is explicit in the problem quantities and holds when the randomization support is sufficiently small.Source: B. Gradient Domination of the Sample… · p. 5
  • The heterogeneity bound scales with FIR order and does not capture cases where increasing order permits stabilization across larger heterogeneity gaps.The authors note that their convergence result still applies in the small-gap regime despite the nontrivial FIR optimization landscape.Source: B. Gradient Domination of the Sample… · p. 5

C. Proof of Theorem III.1

The proof combines the earlier landscape results to establish the convergence theorem, and the curriculum algorithm supplies an initial stabilizing controller while expanding FIR memory. Its components use discount annealing and gradual FIR-order selection.

C. Proof of Theorem III.1

  • Combining objective descent with gradient domination yields an iterate whose sample-average cost is within ε of the optimum after the stated iteration bound.The proof selects an iterate with a small gradient, applies gradient domination there, and uses monotonicity to transfer the guarantee to the final iterate.Source: C. Proof of Theorem III.1 · p. 6

IV. CURRICULUM LEARNING-BASED POLICY GRADIENT

  • The curriculum algorithm trades memory and online computation against stabilization capacity by gradually increasing FIR order when satisfactory performance is achievable.Its two components are discount annealing to find an initial stabilizing controller and order selection to expand memory.Source: §IV · p. 6

A. Discount Annealing

  • Discount annealing increases γ from 0 toward 1 while optimizing discounted sample-average objectives, maintaining the initial-iterate condition along the way.For γ below the stated stability threshold, scaling system matrices by √γ makes the discounted problem finite even for a non-stabilizing controller.Source: A. Discount Annealing · p. 6

B. FIR Order Selection

  • FIR-order selection gradually increases H because the required order for simultaneous stabilization is undecidable when there are at least three systems.The strategy seeks a small enough order that provides satisfactory performance.Source: B. FIR Order Selection · p. 6

C. Curriculum Learning-Based Policy Gradient

The curriculum increases FIR memory when discount-factor progress stalls, helping policy gradient find a simultaneously stabilizing controller. The experiments show convergence after sufficient memory and compare FIR optimization with a more expressive dynamic parameterization; the discussion notes limits to linear controllers and the heterogeneity condition.

C. Curriculum Learning-Based Policy Gradient

  • The curriculum raises FIR order H when the discount-factor increase falls below threshold τγ, then reinitializes γ.The algorithm first selects γ so K = 0 simultaneously stabilizes the systems and its discounted sample-average cost is bounded by Cinit.Source: C. Curriculum Learning-Based Policy… · pp. 6–7

A. Policy Gradient over FIR Controller classes

  • FIR reaches the sample-average cost minimizer after its order reaches H = 5, when the added memory enables simultaneous stabilization of the sampled systems.At lower orders, controller gains grow rapidly as γ approaches 1.Source: A. Policy Gradient over FIR Controller… · p. 7

B. Alternative Dynamic Parameterization

  • FIR is one realization of the paper's general dynamic policy parameterization.The excerpt introduces the alternative parameterization but does not provide its full form.Source: B. Alternative Dynamic Parameterization · p. 7
  • FIR attains lower cost in fewer iterations than the general dynamic controller (5.1 vs. 21.8), despite using longer history (H = 5 versus H = 2).The authors attribute easier optimization to FIR's smaller number of degrees of freedom, although it is a strict subclass of dynamic controllers.Source: B. Alternative Dynamic Parameterization · p. 7

VI. DISCUSSION

  • Some examples admit no linear controller that stabilizes the systems, so nonlinear parameterizations remain a proposed direction for overcoming this limitation.The discussion identifies Example VI.1 as an instance where no linear controller can stabilize the system.Source: §VI · p. 7
  • The sufficient heterogeneity condition for the optimality analysis scales inversely with FIR order H, a tension the authors leave for future study.The discussion contrasts this condition with the stabilization perspective, where more expressive controllers can improve the possibility of simultaneous stabilization.Source: §VI · pp. 7–8

VII. CONCLUSION

  • The paper proves policy-gradient convergence for FIR policies minimizing the sample-average domain-randomized LQR objective, motivated by cases where static feedback cannot simultaneously stabilize the systems.The conclusion presents this as a theoretical foundation for further study of policy classes in domain randomization.Source: §VII · p. 8

VIII. APPENDIX

The appendix sets up the FIR-LQR sample-average analysis and its assumptions, then addresses degeneracy in the lifted cost through exact reformulations. Its structural results establish an invertible noise-controllability factor and a matrix expression for the cost.

APPENDIX A NOTATION AND PRELIMINARIES

  • The preliminaries define the FIR policy class and sample-average objective, then represent the controller through a lifted state with an associated matrix-form cost.The appendix also fixes norm conventions and records optimal-policy notation.Source: Appendix A · p. 9
  • The appendix defines heterogeneity and gradient-size quantities, alongside model-size and optimal-policy notation used in the analysis.It also specifies the spectral and Frobenius norm conventions.Source: Appendix A · p. 9

APPENDIX B ASSUMPTIONS AND PROOF ROADMAP

  • The appendix records heterogeneity and initial-policy assumptions and outlines a five-step proof roadmap for the convergence analysis.The initial policy is required to have sample-average cost at most eight times the optimum; the heterogeneity bound is specified through a threshold in Lemma E.3.Source: Appendix B · p. 10

APPENDIX C THE LANDSCAPE OF A SINGLE FIR-LQR COST

  • The single-cost analysis uses exact reformulations with a non-degenerate H-step state-cost matrix ΣH(K) to address degeneracy in the lifted LQR cost.The appendix relates the lifted setting to coercivity and gradient-domination properties needed for the sample-average analysis.Source: Appendix C · p. 10, §C.1 · p. 10

C.1 Removing the degeneracy

  • The lifted-problem results show that CK is unit upper triangular and invertible, and express the per-system cost as J(K) = tr(PKW).The stated structures include the H-step noise-controllability factor and the Lyapunov-form cost.Source: §C.1 · p. 10

C.2 Coercivity, smoothness, and descent

For FIR-LQR, coercivity and smoothness support descent on the sample-average cost, while gradient domination and quadratic growth characterize the single-cost landscape.

C.2 Coercivity, smoothness, and descent

  • Coercivity and smoothness of the sample-average cost yield compact sublevel sets and an interior minimizer with zero gradient.The cost is twice continuously differentiable and L-smooth on each sublevel set.Source: §C.2 · p. 11
  • For step sizes 0 < α < 2/L, policy gradient stays in its initial sublevel set, does not increase cost, and its gradient converges to zero.The section also states convergence to a fixed point; the supplied passage does not give the gradient convergence rate.Source: §C.2 · p. 12

C.3 Gradient domination, quadratic growth

  • Gradient domination and quadratic growth are established for a single FIR-LQR cost, extending the corresponding LQR result.The paper identifies gradient domination as key to global convergence in non-convex optimization.Source: §C.3 · p. 12

C.4 Optimal FIR policy

For a single system, the optimal FIR policy uses only its first controller element, and its optimal cost matches the static LQR cost.

C.4 Optimal FIR policy

  • Because memory carries no benefit for a single system, the paper uses the static LQR gain Kstat as a reference controller for comparing systems in the next section.Source: §C.4 · p. 13
  • The optimal FIR cost for a single system equals the optimal static LQR cost.The section gives J(K⋆single) = tr(P⋆W) = tr(Pstat).Source: §C.4 · p. 13
  • The lifted DARE solution retains a diagonal structure when the controller has only a first element.The stated structure is P⋆ = diag(Pstat, p1Q, …, pH−1Q), with pj = H − j.Source: §C.4 · p. 13

APPENDIX D PERTURBATION ARGUMENT ON THE FIR-LQR COST

The appendix develops perturbation bounds for FIR-LQR costs and uses bounded heterogeneity to compare systems and control constants on an initial sublevel set.

APPENDIX D PERTURBATION ARGUMENT ON THE FIR-LQR COST

  • The perturbation analysis aims to establish sample-average gradient domination under a small heterogeneity gap between sampled systems.It develops bounds on closed-loop system-theoretic quantities; the named constants depend explicitly on H, τB, and J̄⋆.Source: Appendix D · p. 13, Table I · p. 14

D.1 Cost and Lyapunov bounds

  • Cost and Lyapunov bounds provide upper bounds on closed-loop system-theoretic quantities in terms of LQR cost and Lyapunov functions.These bounds are used throughout the remaining proof.Source: §D.1 · p. 14

D.2 Comparing systems and uniform constants on the sublevel set

  • Under the stated heterogeneity condition, the sample-average optimum lies in the initial sublevel set and individual sample costs there are bounded by 2JSA(K) ≤ J̃.On that set, the corollary also gives c ≥ c̄ and ν ≤ ν̄.Source: §D.2 · p. 15

D.3 Perturbation across sytems

The appendix bounds perturbations across sampled systems and gains, supplying ingredients for the FIR-LQR heterogeneity analysis.

D.3 Perturbation across sytems

  • Lemma D.3 states a perturbation bound for i,j ∈ [M] and K ∈ Kζ0, using constants CA, CΣ, and CP from Table I.Source: §D.3 · p. 16

H ˜J/¯c. For DH,

  • Separate results bound perturbations in EK and in the policy gradient across systems.The supplied passage names the EK and gradient perturbation results without giving their full inequalities.Source: H ˜J/¯c. For DH · pp. 16–17

D.4 Perturbation across gains

  • The appendix also provides a gain-perturbation bound for two gains in the initial sublevel set.The result is stated for Δ = K′ − K and a system indexed by i.Source: §D.4 · p. 17

APPENDIX E GRADIENT DOMINATION AND GLOBAL CONVERGENCE

Under Assumptions B.1–B.2, policy gradient over FIR policies converges globally to a minimizer of the sample-average objective. A canonical-form controller also identifies a simultaneously stabilizing controller, though it reaches a suboptimal solution compared with FIR.

APPENDIX E GRADIENT DOMINATION AND GLOBAL CONVERGENCE

  • The excerpt identifies fgrad and fhet as explicit, finite, positive functions of (H, τB, J̄⋆), and specifies K∈Kζ0 with εgrad(K)≤ ε̄grad under Assumptions B.1–B.2.Source: Appendix E · p. 18
  • With stepsize α = 1/L, policy gradient over FIR policies reaches ε-optimality after N ≥ 2Lζ0/(c̄ε) iterations under Assumptions B.1–B.2.The theorem concerns the sample-average FIR-LQR objective.Source: Appendix E · pp. 18–19

APPENDIX F FURTHER EXPERIMENTS IN SECTION V-B

  • The canonical-form controller identifies a simultaneously stabilizing controller using H = 2 history, while optimization remains stable despite its larger parameter count.The passage presents this as further support for a tradeoff: limiting controller degrees of freedom can simplify optimization while retaining strong performance.Source: Appendix F · p. 19, Figure 3 · p. 19
  • Compared with FIR in Figure 2a, the canonical class attains a suboptimal solution, potentially because its DK = 0 constraint cannot represent the best controller.The stated explanation is explicitly presented as a possibility.Source: Appendix F · p. 19
Loading 2609.16300v1…