PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Driven LQR with Uncertainty Refinement and Domain Randomization---Part~II: Fisher-Majorized Certified Policy Iteration

Hamidreza Modares

arXiv:2610.33795v1eess.SYcs.SY

TL;DR

Domain-randomized LQR can reduce the conservatism of worst-case design, but sampled stability alone does not certify unsampled models. The paper combines partial Bellman evaluation, Fisher-based covariance control, and support-wide stability correction; its iterates robustly stabilize the full uncertainty support while monotonically decreasing the fixed-batch objective.

  • Problem

    Worst-case robust LQR can be conservative or infeasible for large uncertainty sets, while domain-randomized methods do not certify stability of unsampled models from sample-wise stability alone.

    Source: §I · p. 1

  • Method

    The method uses partial Bellman evaluation and Fisher-based covariance bounds to construct certified policy steps, then checks and corrects them to enforce stability over the full uncertainty support.

    Source: §I · p. 1, 3) Certified upper model · p. 7

  • Results

    Every iterate robustly stabilizes the full support and monotonically decreases the fixed-batch objective, with the gap contracting linearly to a floor set by target dispersion and measurement noise.

    Source: §I · p. 1

  • Takeaways & Limitations

    FM-DR-GPI reduces conservatism relative to common worst-case certification while establishing constrained stationarity without gradient dominance or small-heterogeneity assumptions, even with measurement noise.

    Source: §I · p. 1

Abstract

from arXiv · show

Building on the uncertainty description developed in Part~I, this paper introduces a Fisher-majorized domain-randomized generalized policy iteration framework for data-driven linear quadratic regulator (LQR) synthesis. At each iteration, partial Bellman evaluation over a fixed batch of uncertainty realizations produces a structured Sylvester policy-improvement step. A residual-weighted Fisher curvature controls the policy-induced variation in the stationary covariance and converts the Sylvester model into a certified upper bound on the cost change, thereby guaranteeing descent for every accepted step. A check-and-correct mechanism then certifies stability over the entire uncertainty support, rather than only over the sampled realizations. Consequently, every iterate is robustly stabilizing and monotonically decreases the fixed-batch objective. The Fisher truncation and stability correction become inactive after finitely many times. The objective gap then contracts linearly to a floor determined by the dispersion of the realization-wise Riccati targets and by measurement noise. In the absence of measurement noise, this floor vanishes when the realization-wise targets coincide.

I. INTRODUCTION

The paper addresses the tension between conservative worst-case robust LQR and domain-randomized optimization that does not certify stability beyond sampled models. It combines Part I’s uncertainty description with a partitioned, support-wide stability certificate for domain-randomized LQR.

I. INTRODUCTION

  • Worst-case robust LQR can be conservative or infeasible for large uncertainty sets, while sample-wise stability in domain randomization does not certify unsampled models.The paper therefore seeks expected-performance optimization with stability guaranteed over the full uncertainty support.Source: §I · p. 1

A. Part I Model and Certified Uncertainty

  • Part I provides a refined model-set cover containing the exact refined set on its confidence event, which underpins the uncertainty description used here.The model uses state and input data to parametrize control gains and includes certified noise bounds.Source: A. Part I Model and Certified… · pp. 1–2

B. Uncertainty-Partitioned Domain-Randomized LQR

  • The domain-randomized LQR problem partitions the uncertainty coefficient domain into polytopic cells and optimizes a sample-average objective whose guarantees are transferred to the full problem.The regional contraction constraints are specified for every uncertainty realization in each cell.Source: B. Uncertainty-Partitioned… · p. 2

III. ROBUST-STABILITY CERTIFICATION

  • A single common contraction certificate can fail even when every fixed model is stable, whereas cell-specific polyhedral certificates certify stability across the partition.In the example, the common-certificate test fails because the endpoint product has spectral radius about 1.30, while cell certificates achieve contraction factor 0.995.Source: §III · p. 2

A. Bounded-Complexity Inner Controller Sets

The section constructs bounded-complexity inner polytopes within certified controller sets, preserving feasibility while enlarging the approximation where it restricts a proposed step.

A. Bounded-Complexity Inner Controller Sets

  • The initial inner controller polytope is built from boundary points extended along simplex directions from the center of the largest inscribed Euclidean ball.The ball center and radius are computed for the certified controller set, and the initial polytope is computed offline.Source: A. Bounded-Complexity Inner Controller… · p. 3
  • When the inner polytope restricts a proposed direction, enrichment adds a certified endpoint to expand its reach while keeping it inside the parent certified set.The largest step within the inner polytope is computed using convex-combination weights on its stored vertices.Source: A. Bounded-Complexity Inner Controller… · p. 3

IV. CERTIFIED DR OBJECTIVE AND FISHER GEOMETRY

  • The following section constructs the objective and realization-wise Fisher quantities used by FM-DR-GPI.Source: §IV · p. 3

A. Realization-Dependent LQR Cost

The paper defines a realization-dependent surrogate LQR cost and a Fisher measure of policy-induced transition changes. These quantities support a frozen-covariance Sylvester improvement model with a unique unconstrained minimizer.

A. Realization-Dependent LQR Cost

  • The realization-dependent surrogate LQR cost is averaged over the desired uncertainty distribution and approximated for synthesis by a fixed-batch sample-average objective.The surrogate replaces the unknown realization-wise cost, and the batch uses fixed uniform weights.Source: A. Realization-Dependent LQR Cost · p. 4

B. Fisher Geometry of Closed-Loop Policy Perturbations

  • The realization-wise Fisher sensitivity measures, to second order in step size, how a gain perturbation changes the transition law.A certified quadratic form is introduced because the support-wide Fisher sensitivity is not directly computable.Source: B. Fisher Geometry of Closed-Loop… · p. 4

V. FISHER-MAJORIZED CERTIFIED GPI

  • The method connects the sample-average objective, support-wide Fisher form, and certified controller sets by deriving a frozen-covariance Sylvester model, bounding its omitted covariance change, and checking steps for stability and descent.Source: §V · p. 4

A. Exact one-step policy-improvement model

  • The exact one-step policy-improvement model is a convex quadratic in the candidate gain, and its unconstrained minimizer is unique.The model is first-order in the cost difference, rather than the exact finite difference.Source: A. Exact one-step policy-improvement… · p. 5

B. Partial Evaluation and Certified Upper Model

Partial Bellman evaluation makes the policy-improvement model computable without exact Lyapunov solutions. Regional contraction certificates and Fisher bounds control the covariance mismatch and yield a support-wide safe-step region.

B. Partial Evaluation and Certified Upper Model

  • Finite Bellman sweeps replace exact Lyapunov solutions, and the resulting computable upper model supplies a policy direction and certified step.A scalar search may extend the step along the same stability-certified ray.Source: B. Partial Evaluation and Certified… · p. 5

1) Partial evaluation:

  • The partial-evaluation recursions preserve positive-semidefinite value and covariance bounds, giving the quadratic model a positive lower curvature bound.The stated bound is f_Mt ⪰ 2σ0 I_dK, with σ0 = λmin(R)λmin(W̄) > 0.Source: 1) Partial evaluation · p. 5

2) Certified covariance bounds and Fisher region:

  • A support-wide Fisher bound controls the covariance change caused by a policy displacement, bounding the perturbed stationary covariance between scaled versions of the current covariance.For gains in the certified set, the covariance perturbation bound applies for every admissible uncertainty realization.Source: 2) Certified covariance bounds and… · p. 6
  • The Fisher region enforces the prescribed covariance distortion over the support, and the current-gain region contains the certificate-wide region, becoming less conservative when the current policy contracts more strongly.Source: 2) Certified covariance bounds and… · p. 6

3) Certified upper model:

A residual-weighted Fisher correction bounds the covariance effect omitted by the frozen-covariance Sylvester model. This upper curvature makes a negative model step a certificate of actual fixed-batch descent.

3) Certified upper model:

  • The exact cost difference uses the updated stationary covariance, whereas the Sylvester model freezes the current covariance.The correction targets this discrepancy.Source: 3) Certified upper model · pp. 6–7
  • The Fisher form controls policy-induced covariance variation, while the covariance-weighted LQR curvature and residual interaction quantify its cost effect.Together these corrections make the augmented curvature an upper bound on the omitted covariance change.Source: 3) Certified upper model · p. 7
  • The e-curvature correction bounds the frozen-covariance error, so a Fisher-feasible step with a negative model guarantees descent of the actual objective.The bound applies when both gains are certified and the Fisher budget is satisfied.Source: 3) Certified upper model · p. 7
  • Finite Bellman sweeps provide tracked matrices for a computable batch-averaged curvature; under exact evaluation, both model buffers vanish.The smooth model generates the direction, while its buffered version certifies the step.Source: 3) Certified upper model · p. 7

4) Check-and-correct Sylvester step:

The check-and-correct update combines a Sylvester direction with a controller-polytope correction and a Fisher-certified step length. It preserves full-support stability and decreases the fixed-batch objective, with scalar line search needed only finitely often.

4) Check-and-correct Sylvester step:

  • The update first obtains a Sylvester direction, then corrects it for full-support stability and uses the Fisher model to certify a descent length.A scalar search may extend the certified step along the same stability-certified ray.Source: 4) Check-and-correct Sylvester step · p. 8
  • Among eligible certified inner sets, the method selects the bundle allowing the longest move and freezes it after the switching budget is exhausted.If the unconstrained endpoint is infeasible, it solves the e-model over the certified polytope.Source: 4) Check-and-correct Sylvester step · p. 8
  • The Fisher check can truncate a step before the stability boundary, and homogeneity gives the valid range along the stable ray.The certified correction step remains included in the scalar search interval.Source: 4) Check-and-correct Sylvester step · p. 8
  • The controller-polytope correction keeps the gain contractive across uncertainty cells, while the certified step preserves descent of the fixed-batch objective.The line search is used only when the Fisher check fails and only finitely many times.Source: 4) Check-and-correct Sylvester step · p. 8

VI. CONVERGENCE AND GUARANTEES

The convergence analysis separates population improvement from realization-wise dispersion, then establishes fixed-batch stationarity and whole-sequence convergence under exact evaluation. It also identifies conditions under which the Fisher truncation and line search become inactive.

VI. CONVERGENCE AND GUARANTEES

  • The analysis transfers the convergence guarantees from the fixed-batch objective to the population surrogate, Problem 1, and the true plant.The proof framework distinguishes the batch, population, and true-plant objectives.Source: §VI · pp. 8–9

A. Population improvement and dispersion

  • The population Sylvester model separates the removable common-step decrement from dispersion among realization-wise Riccati targets.The population improvement potential and dispersion characterize this split.Source: A. Population improvement and dispersion · p. 9

B. SAA stationarity and whole-sequence convergence

  • Every accumulation point is constrained stationary for the fixed-batch problem, without requiring low dispersion among realization-wise targets.After finite certificate switching and line-search use, exact evaluation eventually yields full Sylvester steps and whole-sequence convergence; the Fisher truncation also becomes inactive.Source: B. SAA stationarity and whole-sequence… · p. 9

C. Linear rate with a dispersion floor

The paper gives an eventual linear rate for the fixed-batch gap under exact evaluation and an interior-limit condition, then transfers guarantees through uniform SAA bounds to population and true-plant objectives.

C. Linear rate with a dispersion floor

  • Under exact fixed-batch evaluation and an interior limit, the SAA gap contracts linearly to a floor set by target dispersion and measurement noise.With zero measurement noise, the floor vanishes when the realization-wise targets agree; no rate is claimed at a boundary limit.Source: C. Linear rate with a dispersion floor · p. 10

D. Uniform SAA transfer

  • A uniform SAA event bounds cost and gradient errors over the certified domain, so the bounds remain valid for an inner set selected using the same batch.The event supports population-surrogate descent guarantees.Source: D. Uniform SAA transfer · p. 10

E. Transfer to the Problem-1 objective and the true plant

  • With probability at least 1−δI−δ, every iterate stabilizes the true plant and the stated population, Problem-1, and true-plant bounds hold.The Problem-1 gap retains a fixed-batch and certificate floor.Source: E. Transfer to the Problem-1 objective… · p. 11

F. Interpretation and special cases

In the noise-free special case, the objective-gap floor vanishes when realization-wise directions coincide. The simulation illustrates the trade-off between expected cost and stability over the full uncertainty set, while the conclusion states how the guarantees extend beyond the fixed batch.

F. Interpretation and special cases

  • With zero measurement noise, the objective-gap floor vanishes when realization-wise Riccati improvement directions coincide.The special case assumes exact evaluation and inactive certificate constraints for the classical Newton-like update.Source: F. Interpretation and special cases · p. 11

VII. SIMULATION RESULTS

  • FM-DR-GPI is stable over the full set (0.98), while policy gradient is stable on 20,000 samples (0.96) but unstable over the set (1.11).The destabilizing models occupy a corner with large a12 and a21 that Pdes practically never visits.Source: Figure 1 · p. 12, §VII · p. 12
  • Stability correction acts at all 97 iterations, while Fisher correction acts for the first 46 iterations and one line search is performed.At convergence, two of the 21 inequalities are active.Source: §VII · p. 12
  • The domain-randomized cost of FM-DR-GPI is 8.2% above policy gradient, the stated price of stability over the certified set.The unconstrained minimizer has spectral radius 1.11 on the set.Source: §VII · p. 12

VIII. CONCLUSION

  • The analysis establishes SAA stationarity and transfers guarantees to population and true-plant costs, with heterogeneity, certificate compression, sampling, and evaluation errors entering separately.Adaptive certificate design and richer uncertainty decompositions are identified as future work.Source: §VIII · pp. 12–13

APPENDIX A PROOF OF LEMMA 1

The appendices establish how retained uncertainty images yield stability constraints, characterize and construct certified gain sets, and relate certificate bounds to true-plant costs.

APPENDIX A PROOF OF LEMMA 1

  • Retaining only extreme uncertainty images preserves the relevant convex hull; the resulting linear certificate conditions imply strict contractive invariance and Schur stability.The conditions are converted to linear inequalities in vec(K).Source: Appendix A · p. 13

APPENDIX B PROOF OF PROPOSITION 1

  • For fixed certificate facets and gain-region vertices, the maximizing certificate vertices are among the retained ones, establishing the stated constraint characterization.The proof uses the normal cones of the gain region.Source: Appendix B · p. 13

APPENDIX C PROOF OF LEMMA 2

  • The gain set Cj is compact with nonempty interior, and the associated linear program computes its largest contained Euclidean ball.The ball's center lies in the interior of Cj.Source: Appendix C · p. 13

APPENDIX D PROOF OF PROPOSITION 2

  • The certificate-set enrichment procedure adds vertices without removing stored ones, so it terminates after at most pmax − (dK + 1) enrichments.Each successful enrichment adds one vertex.Source: Appendix D · p. 13

APPENDIX E PROOF OF LEMMA 3

  • The certificate-based cost upper-bounds true cost; as the data-supported model set contracts to the true model, the bounds converge uniformly.The convergence follows from the Part I consistency results for the model and noise-covariance estimates.Source: Appendix E · p. 14

APPENDIX F PROOF OF LEMMA 4

The appendices bound Fisher information and regional covariance quantities, then use covariance mismatch control to bound the policy-induced cost change. The resulting correction terms reflect both Fisher curvature and the Riccati residual.

APPENDIX F PROOF OF LEMMA 4

  • Conditioning on measurement noise yields a simpler fixed-covariance transition model; Jensen’s inequality bounds marginal Fisher information by the joint Fisher information.Averaging over the stationary state uses Xt(η) + V̄.Source: Appendix F · p. 14

APPENDIX G PROOF OF LEMMA 6

  • Regional contraction certificates provide uniform bounds on the value matrix and stationary covariance across each uncertainty cell.The bounds follow from contraction in the certificate gauge and summing geometric series.Source: Appendix G · p. 14

APPENDIX H PROOF OF LEMMA 7

  • The covariance mismatch between policies is bounded multiplicatively using the Fisher quantity, provided both gains lie in the certified stable set.The proof derives an order bound on X′ − X from the perturbation of the covariance operator.Source: Appendix H · pp. 14–15

APPENDIX I PROOF OF THEOREM 1

K DX1/2∥F = √ψ a, because E⊤

  • The correction combines a Fisher term for covariance error with a residual-weighted term for the cross interaction between Riccati residual and covariance error.The residual weight arises specifically from that cross term.Source: K DX1/2∥F = √ψ a, because E⊤ · p. 15

APPENDIX J PROOF OF PROPOSITION 3

The appendices bound the batch cost change, derive a realization-wise cost identity and its quadratic surrogate, then establish eventual full-step convergence to a critical point.

APPENDIX J PROOF OF PROPOSITION 3

  • The proof bounds the batch cost change using the quadratic model and controls the evaluation-residual terms by 2ϵ_g^t∥d∥².This supports the stated inequalities for feasible steps satisfying the stability and Fisher constraints.Source: Appendix J · p. 15

APPENDIX K PROOF OF LEMMA 9

  • Expanding the new-policy dynamics and covariance equations yields a realization-wise cost-change identity, whose terms are collected into Φ_K(Δ) and Ξ_K(Δ).The proof then takes expectation over the uncertainty distribution.Source: Appendix K · p. 15

K RKDH

  • The expected surrogate h(K;D) equals n(K) plus a quadratic model with gradient ∇f(K) and curvature M_K; positive definiteness gives a unique minimizer.Its minimum is h⋆(K)=n(K)−δ⋆(K).Source: K RKDH · p. 15

APPENDIX M PROOF OF THEOREM 2

  • Once the feasible polytope is fixed, the QP optimality condition yields constrained stationarity at limit points.The proof uses the closedness of the normal-cone graph to pass this condition to a convergent subsequence.Source: Appendix M · p. 16
  • After finitely many iterations, the Fisher check and line search are inactive; under exact evaluation, updates use full steps and the iterates converge to a critical point.The proof establishes sufficient decrease, relative error, and finite-length convergence on the fixed feasible set.Source: Appendix M · p. 16
Loading 2610.33795v1…