PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Iteration for Domain Randomized Linear Quadratic Systems

Abbas Pasdar, Farnaz Adib Yaghmaie

arXiv:2609.04794v1math.OCeess.SY

TL;DR

The paper asks how to learn one state-feedback controller that performs well across systems with uncertain dynamics while retaining stability and convergence guarantees. It extends policy iteration to domain-randomized LQR with a stabilizing step-size rule. The method preserves stability, monotonically decreases the sample-average objective, converges subsequentially to stationary points, and achieves global linear convergence under gradient dominance.

  • Problem

    Learning a single controller that minimizes average cost across uncertain systems is important, but domain randomization generally lacks theoretical stability and convergence guarantees.

  • Method

    The paper derives policy evaluation and improvement steps for a sample-average domain-randomized LQR objective and uses a step-size rule that preserves joint stability.

  • Results

    The algorithm preserves closed-loop stability and monotonically decreases the sample-average objective, with subsequential convergence to stationary points and global linear convergence under gradient dominance.

  • Takeaways & Limitations

    Policy iteration provides a theoretically analyzed approach to optimizing average performance across uncertain linear systems while maintaining stability across the sampled systems.

Abstract

from arXiv · show

In this work, we study policy optimization under domain randomization for linear quadratic control, focusing on learning a single state-feedback controller that minimizes the average cost across systems with uncertain dynamics. We propose a policy iteration algorithm with a step-size rule that preserves stability across all sampled systems at each iteration. We show that the method yields monotonic improvement of the sample-average objective and that a stabilizing step size always exists. Under standard smoothness assumptions, the iterates converge subsequentially to stationary points, and under a gradient-dominance condition, we obtain a global linear convergence rate.

I. INTRODUCTION

The paper addresses learning controllers that generalize across uncertain dynamics, contrasting domain randomization’s average-performance objective with robust control’s worst-case guarantees. It extends policy iteration to domain-randomized LQR and establishes stability, descent, and convergence results.

  • Motivation: Domain randomization trains policies across sampled dynamics but generally lacks theoretical guarantees.Robust control provides worst-case guarantees, whereas domain randomization optimizes average performance across systems.
  • Background: LQR provides an analytically tractable benchmark for studying generalization and robustness under uncertain dynamics.Prior domain-randomized LQR work learns one linear state-feedback controller minimizing average quadratic cost across systems.
  • Related Work: Policy iteration can avoid long rollouts, exploit Bellman operators, and often converge faster than policy-gradient methods.The paper positions policy iteration as useful for data-driven control and as a second-order optimization interpretation.
  • Research Gap: Existing policy-iteration analyses largely address single-system settings rather than domain-randomized control.This gap motivates extending policy iteration to multiple systems with uncertain dynamics.
  • Contributions: The proposed domain-randomized LQR policy iteration method preserves joint stability, monotonically decreases the sample-average objective, and supports convergence analysis.Under a gradient-dominance condition, the analysis establishes a linear convergence rate.

C. Optimization-Based LQR

The paper formulates LQR as smooth optimization over stabilizing feedback gains and connects policy-gradient and policy-iteration perspectives to domain-randomized control. Domain-randomized LQR seeks one jointly stabilizing controller minimizing cost across a distribution of systems.

  • Optimization View: LQR can be formulated as a nonconvex optimization problem over the stabilizing linear feedback gain K.For stabilizing K, the infinite-horizon cost and its gradient admit analytical expressions.
  • Optimization View: Gradient-based algorithms apply because the LQR objective has a closed-form gradient on the stabilizing gain set.The gradient-descent update is presented as an optimization formulation of LQR.
  • Optimization View: Policy-gradient steps remain within a bounded stabilizing sublevel set and converge linearly to the global LQR optimum.The stated result applies when initialization lies in Kγ for finite positive γ.
  • Domain Randomization: Domain-randomized LQR models uncertain dynamics with a distribution over system matrices and seeks one controller minimizing expected cost.The jointly stabilizing gain set requires ρ(A+BK)<1 for every system in the distribution support.
  • Domain Randomization: Simultaneous stabilizability makes the domain-randomized feasible set nonempty and can hold when system heterogeneity is sufficiently small.The assumption is stated as nonemptiness of the jointly stabilizing gain set.

III. OPTIMIZATION-DERIVED DYNAMIC PROGRAMMING FOR LQR

The paper interprets classical policy iteration as alternating optimization over value and policy variables, then extends this construction to sampled domain-randomized LQR. The resulting update couples all sampled systems through their policy-evaluation quantities.

  • Classical LQR: Classical policy iteration alternates policy evaluation and improvement, and its first-order condition coincides with the classical policy-improvement equation.The optimization view treats the method as alternating minimization over value and policy variables.
  • Sample-Average Formulation: The domain-randomized objective is approximated by averaging costs and gradients over finitely many systems sampled from the dynamics distribution.For each sampled system, policy evaluation supplies the associated value and covariance matrices.
  • Policy Improvement: The multi-system policy-improvement equation generalizes the single-system rule and couples controller updates across all sampled systems.Unlike the single-system case, the domain-randomized update does not reduce to an independent closed-form Riccati update.
  • Policy Improvement: Vectorization converts the policy-improvement equation into a Sylvester-type linear matrix equation with a closed-form solution.Positive definiteness of Rj,i and Σj,i guarantees existence and uniqueness of the solution.
  • Algorithm: Each iteration evaluates value and covariance matrices for every sampled system, then updates the policy using the coupled rule and an appropriate step size.The algorithm starts from a stabilizing controller.

B. KKT Derivation of the Policy-Iteration

The policy-improvement step is derived from first-order optimality conditions for a constrained sample-average problem.

  • KKT Derivation: The policy-improvement step follows by applying first-order optimality conditions to the constrained sample-average objective.This derivation connects the constrained optimization formulation to the policy-iteration update.

1) Sample-average DR-LQR formulation:

The sample-average DR-LQR problem is formulated with Lyapunov constraints, and its policy-improvement step is obtained from the KKT conditions of the constrained optimization problem.

  • 1) Sample-average DR-LQR formulation:: The sample-average DR-LQR formulation uses one shared linear state-feedback controller across sampled systems.
  • 2) Lagrangian:: The constrained formulation introduces Lyapunov variables and a Lagrangian for the sample-average objective.
  • 3) KKT conditions:: The first-order optimality conditions include stationarity with respect to each covariance matrix Σ_j.
  • 3) KKT conditions:: The KKT system also includes stationarity with respect to K and primal feasibility of the constraints.
  • 3) KKT conditions:: Policy evaluation computes P_j,i and Σ_j,i for the current stabilizing controller K_i before policy improvement solves for the next controller.

5) Interpretation as a Newton-like policy update:

The method alternates policy evaluation with a KKT-derived policy-improvement update and uses damping to preserve joint stability. Under smoothness and gradient-dominance assumptions, stability is maintained and linear convergence is obtained.

  • 5) Interpretation as a Newton-like policy update:: The method alternates policy evaluation at K_i with policy improvement using a linearized KKT condition.The evaluation matrices remain fixed while computing the improvement controller.
  • 5) Interpretation as a Newton-like policy update:: Damping selects α so every sampled closed-loop matrix remains Schur stable while the sample-average cost decreases sufficiently.
  • 5) Interpretation as a Newton-like policy update:: The policy-improvement equation follows from KKT stationarity after freezing the current policy-evaluation quantities.
  • 5) Interpretation as a Newton-like policy update:: The analysis covers stability preservation, objective descent, and subsequential convergence, with a linear rate under gradient dominance.
  • 5) Interpretation as a Newton-like policy update:: Smoothness and gradient dominance are imposed on the compact stabilizing set to support the convergence analysis.
  • A. Closed-Loop Stability Preservation: Remaining in the stabilizing region is necessary because leaving it can make the sample-average objective unbounded or undefined.
  • A. Closed-Loop Stability Preservation: A positive stabilizing step size exists at every iteration, and compactness yields a uniform lower bound across iterations.

B. Descent Property

The policy-improvement direction is a strict descent direction, and an admissible step size yields strict monotonic decrease of the sample-average objective while preserving stability.

  • B. Descent Property: The policy-improvement direction has a uniformly negative inner product with the sample-average gradient whenever it is nonzero.
  • B. Descent Property: For 0 < α ≤ min{ᾱ, c_d/L}, the updated controller has strictly lower sample-average cost.
  • B. Descent Property: The step-size restriction balances the stability bound ᾱ with the smoothness-based decrease condition c_d/L.
  • B. Descent Property: Combining descent with stability preservation produces stabilizing controllers with strictly decreasing objective values.

C. Subsequential Convergence

The proposed iterates remain stable, decrease the sample-average objective monotonically, and converge subsequentially to stationary points under the stated assumptions.

  • A sufficiently small step size α = min{¯α, cd/L} provides sufficient decrease under the theorem’s assumptions.
  • Theorem 4 keeps every iterate in the jointly stabilizing compact set Kjs,γ, ensuring boundedness and accumulation points.
  • The sample-average objective JSA(Ki) is monotone nonincreasing and convergent.Stability confines the sequence to a compact region where JSA is bounded below.
  • The update norms satisfy ∥∆i∥F → 0, while the gradients satisfy ∥∇JSA(Ki)∥F → 0.
  • Every accumulation point of the controller sequence is a stationary point of JSA.This follows from vanishing gradients and continuity of the gradient on the jointly stabilizing set.

D. Linear Convergence

With gradient dominance, the previously established stationary-point convergence strengthens to global linear convergence toward the optimal controller.

  • Under gradient dominance, the algorithm converges globally to the minimizer K⋆ at a linear rate.Theorem 5 states that a contraction factor ρ ∈ (0,1) exists for any initial controller in Kjs,γ.
  • The sufficient-decrease bound links objective reduction to the squared gradient norm, enabling the contraction argument.

VI. NUMERICAL ANALYSIS

Numerical experiments on a domain-randomized inverted pendulum compare policy iteration with policy gradient and SDP-based updates, illustrating initialization-dependent trajectories.

  • The experiment uses a two-dimensional discretized inverted pendulum with mass and pole length independently perturbed by ±25%.
  • The sample-average objective is approximated with M = 50 sampled systems, using α = 0.1 for PI and α = 0.001 for PG.
  • From K0 = [−16, −40], PI initially moves toward the interior of the stabilizing set before its direction becomes better aligned with K⋆.The later alignment produces accelerated convergence.

VII. CONCLUSION

The paper develops policy iteration for learning one controller across uncertain systems, establishing stability, monotonic improvement, stationary-point convergence, and conditional global linear convergence.

  • The algorithm preserves closed-loop stability and monotonically decreases the sample-average objective at every iteration.
  • The iterates converge subsequentially to stationary points, with a global linear convergence rate under gradient dominance.
  • Future directions include stochastic or time-varying domains, finite-sample data-driven guarantees, and nonlinear systems.
Loading 2609.04794v1…