Policy Iteration for Domain Randomized Linear Quadratic Systems

Motivation

• Domain randomization trains policies across sampled dynamics but generally lacks theoretical guarantees.Robust control provides worst-case guarantees, whereas domain randomization optimizes average performance across systems.Source: §I · p. 1

Background

• LQR provides an analytically tractable benchmark for studying generalization and robustness under uncertain dynamics.Prior domain-randomized LQR work learns one linear state-feedback controller minimizing average quadratic cost across systems.Source: §I · p. 1

Related Work

• Policy iteration can avoid long rollouts, exploit Bellman operators, and often converge faster than policy-gradient methods.The paper positions policy iteration as useful for data-driven control and as a second-order optimization interpretation.Source: §I · p. 1

Research Gap

• Existing policy-iteration analyses largely address single-system settings rather than domain-randomized control.This gap motivates extending policy iteration to multiple systems with uncertain dynamics.Source: §I · p. 1

Contributions

• The proposed domain-randomized LQR policy iteration method preserves joint stability, monotonically decreases the sample-average objective, and supports convergence analysis.Under a gradient-dominance condition, the analysis establishes a linear convergence rate.Source: §I · pp. 1–2

Optimization View

• LQR can be formulated as a nonconvex optimization problem over the stabilizing linear feedback gain K.For stabilizing K, the infinite-horizon cost and its gradient admit analytical expressions.Source: C. Optimization-Based LQR · p. 2
• Gradient-based algorithms apply because the LQR objective has a closed-form gradient on the stabilizing gain set.The gradient-descent update is presented as an optimization formulation of LQR.Source: C. Optimization-Based LQR · p. 2
• Policy-gradient steps remain within a bounded stabilizing sublevel set and converge linearly to the global LQR optimum.The stated result applies when initialization lies in Kγ for finite positive γ.Source: C. Optimization-Based LQR · p. 2

Domain Randomization

• Domain-randomized LQR models uncertain dynamics with a distribution over system matrices and seeks one controller minimizing expected cost.The jointly stabilizing gain set requires ρ(A+BK)<1 for every system in the distribution support.Source: D. Domain Randomization · p. 2
• Simultaneous stabilizability makes the domain-randomized feasible set nonempty and can hold when system heterogeneity is sufficiently small.The assumption is stated as nonemptiness of the jointly stabilizing gain set.Source: D. Domain Randomization · p. 2

Classical LQR

• Classical policy iteration alternates policy evaluation and improvement, and its first-order condition coincides with the classical policy-improvement equation.The optimization view treats the method as alternating minimization over value and policy variables.Source: §III · p. 3

Sample-Average Formulation

• The domain-randomized objective is approximated by averaging costs and gradients over finitely many systems sampled from the dynamics distribution.For each sampled system, policy evaluation supplies the associated value and covariance matrices.Source: §IV · p. 3, A. Policy Iteration for DR-LQR · p. 3

Policy Improvement

• The multi-system policy-improvement equation generalizes the single-system rule and couples controller updates across all sampled systems.Unlike the single-system case, the domain-randomized update does not reduce to an independent closed-form Riccati update.Source: A. Policy Iteration for DR-LQR · p. 3
• Vectorization converts the policy-improvement equation into a Sylvester-type linear matrix equation with a closed-form solution.Positive definiteness of Rj,i and Σj,i guarantees existence and uniqueness of the solution.Source: A. Policy Iteration for DR-LQR · p. 3

Algorithm

• Each iteration evaluates value and covariance matrices for every sampled system, then updates the policy using the coupled rule and an appropriate step size.The algorithm starts from a stabilizing controller.Source: A. Policy Iteration for DR-LQR · p. 3

KKT Derivation

• The policy-improvement step follows by applying first-order optimality conditions to the constrained sample-average objective.This derivation connects the constrained optimization formulation to the policy-iteration update.Source: B. KKT Derivation of the… · p. 4

1) Sample-average DR-LQR formulation:

• The sample-average DR-LQR formulation uses one shared linear state-feedback controller across sampled systems.Source: 1) Sample-average DR-LQR formulation · p. 4

2) Lagrangian:

• The constrained formulation introduces Lyapunov variables and a Lagrangian for the sample-average objective.Source: 2) Lagrangian · p. 4

3) KKT conditions:

• The first-order optimality conditions include stationarity with respect to each covariance matrix Σ_j.Source: 3) KKT conditions · p. 4
• The KKT system also includes stationarity with respect to K and primal feasibility of the constraints.Source: 3) KKT conditions · p. 4
• Policy evaluation computes P_j,i and Σ_j,i for the current stabilizing controller K_i before policy improvement solves for the next controller.Source: 4) From KKT conditions to PI · p. 4

5) Interpretation as a Newton-like policy update:

• The method alternates policy evaluation at K_i with policy improvement using a linearized KKT condition.The evaluation matrices remain fixed while computing the improvement controller.Source: 5) Interpretation as a Newton-like… · p. 4
• Damping selects α so every sampled closed-loop matrix remains Schur stable while the sample-average cost decreases sufficiently.Source: 5) Interpretation as a Newton-like… · p. 4
• The policy-improvement equation follows from KKT stationarity after freezing the current policy-evaluation quantities.Source: 5) Interpretation as a Newton-like… · p. 4
• The analysis covers stability preservation, objective descent, and subsequential convergence, with a linear rate under gradient dominance.Source: §V · p. 4
• Smoothness and gradient dominance are imposed on the compact stabilizing set to support the convergence analysis.Source: §V · p. 4

A. Closed-Loop Stability Preservation

• Remaining in the stabilizing region is necessary because leaving it can make the sample-average objective unbounded or undefined.Source: A. Closed-Loop Stability Preservation · p. 5
• A positive stabilizing step size exists at every iteration, and compactness yields a uniform lower bound across iterations.Source: A. Closed-Loop Stability Preservation · p. 5

B. Descent Property

• The policy-improvement direction has a uniformly negative inner product with the sample-average gradient whenever it is nonzero.Source: B. Descent Property · pp. 5–6
• For 0 < α ≤ min{ᾱ, c_d/L}, the updated controller has strictly lower sample-average cost.Source: B. Descent Property · p. 6
• The step-size restriction balances the stability bound ᾱ with the smoothness-based decrease condition c_d/L.Source: B. Descent Property · p. 6
• Combining descent with stability preservation produces stabilizing controllers with strictly decreasing objective values.Source: B. Descent Property · p. 6

• A sufficiently small step size α = min{¯α, cd/L} provides sufficient decrease under the theorem’s assumptions.Source: C. Subsequential Convergence · p. 6
• Theorem 4 keeps every iterate in the jointly stabilizing compact set Kjs,γ, ensuring boundedness and accumulation points.Source: C. Subsequential Convergence · p. 6
• The sample-average objective JSA(Ki) is monotone nonincreasing and convergent.Stability confines the sequence to a compact region where JSA is bounded below.Source: C. Subsequential Convergence · p. 6
• The update norms satisfy ∥∆i∥F → 0, while the gradients satisfy ∥∇JSA(Ki)∥F → 0.Source: C. Subsequential Convergence · pp. 6–7
• Every accumulation point of the controller sequence is a stationary point of JSA.This follows from vanishing gradients and continuity of the gradient on the jointly stabilizing set.Source: C. Subsequential Convergence · pp. 6–7

• Under gradient dominance, the algorithm converges globally to the minimizer K⋆ at a linear rate.Theorem 5 states that a contraction factor ρ ∈ (0,1) exists for any initial controller in Kjs,γ.Source: D. Linear Convergence · p. 7
• The sufficient-decrease bound links objective reduction to the squared gradient norm, enabling the contraction argument.Source: D. Linear Convergence · p. 7

• The experiment uses a two-dimensional discretized inverted pendulum with mass and pole length independently perturbed by ±25%.Source: §VI · p. 7
• The sample-average objective is approximated with M = 50 sampled systems, using α = 0.1 for PI and α = 0.001 for PG.Source: §VI · p. 7
• From K0 = [−16, −40], PI initially moves toward the interior of the stabilizing set before its direction becomes better aligned with K⋆.The later alignment produces accelerated convergence.Source: §VI · p. 7, Figure 2 · p. 8

• The algorithm preserves closed-loop stability and monotonically decreases the sample-average objective at every iteration.Source: §VII · p. 8
• The iterates converge subsequentially to stationary points, with a global linear convergence rate under gradient dominance.Source: §VII · p. 8
• Future directions include stochastic or time-varying domains, finite-sample data-driven guarantees, and nonlinear systems.Source: §VII · p. 8