PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Certainty-Equivalence Approach to Direct Data-Driven LQR Design

Florian Dörfler, Pietro Tesi, Claudio De Persis

arXiv:2109.06643v1math.OC

TL;DR

Data-driven LQR methods face a trade-off between model-based certainty-equivalence designs and direct methods that account for data uncertainty. This paper reformulates indirect least-squares certainty-equivalence LQR as a direct regularized problem, and reports stability and performance guarantees at sufficiently high signal-to-noise ratio. A numerical case study also reports strong empirical performance when certainty-equivalence and robustness-promoting regularizers are blended.

  • Problem

    Data-driven LQR spans indirect designs based on identified models and direct designs, as well as certainty-equivalence and robust approaches that differ in whether they account for uncertainty.

    Source: §I · p. 1

  • Method

    The paper transforms indirect least-squares identification followed by nominal LQR design into an equivalent single-level direct formulation with a regularizer accounting for least-squares fitting.

    Source: §I · p. 1

  • Results

    At sufficiently large SNR, certainty-equivalence LQR yields stabilizing controllers irrespective of noise statistics, and its sub-optimality gap scales linearly with SNR.

    Source: §I · p. 1

  • Takeaways & Limitations

    The formulation can be augmented with a robustness-promoting regularizer, and the case study reports remarkable empirical performance with mixed regularization.

    Source: §VII · p. 7

Abstract

from arXiv · show

The linear quadratic regulator (LQR) problem is a cornerstone of automatic control, and it has been widely studied in the data-driven setting. The various data-driven approaches can be classified as indirect (i.e., based on an identified model) versus direct or as robust (i.e., taking uncertainty into account) versus certainty-equivalence. Here we show how to bridge these different formulations and propose a novel, direct, and regularized formulation. We start from indirect certainty-equivalence LQR, i.e., least-square identification of state-space matrices followed by a nominal model-based design, formalized as a bi-level program. We show how to transform this problem into a single-level, regularized, and direct data-driven control formulation, where the regularizer accounts for the least-square data fitting criterion. For this novel formulation we carry out a robustness and performance analysis in presence of noisy data. Our proposed direct and regularized formulation is also amenable to be further blended with a robust-stability-promoting regularizer. In a numerical case study we compare regularizers promoting either robustness or certainty-equivalence, and we demonstrate the remarkable performance when blending both of them.

I. INTRODUCTION

The paper bridges direct and indirect, and robust and certainty-equivalence, data-driven LQR by reformulating indirect least-squares identification followed by nominal design as a direct regularized problem. It motivates this approach through trade-offs in handling noisy data and sets up LQR as an H2-optimal state-feedback design.

I. INTRODUCTION

  • Indirect methods make it hard to propagate data uncertainty through identification, while direct methods can be more sensitive to inexact data and require robustification that may diminish performance.Source: §I · p. 1
  • The proposed direct formulation is equivalent to indirect certainty-equivalence LQR and adds a regularizer accounting for least-squares data fitting.The starting approach identifies state-space matrices from noisy data, then designs a nominal model-based controller.Source: §I · p. 1

A. Model-based linear quadratic optimal control

  • LQR seeks a state-feedback gain that makes A + BK Schur and minimizes the H2 norm of the disturbance-to-performance transfer function.The setup assumes (A, B) is stabilizable and uses positive-definite weighting matrices Q and R.Source: A. Model-based linear quadratic… · p. 2

B. Subspace relations in state-space data, ordinary least-square identification, and certainty-equivalence control

The conventional certainty-equivalence approach identifies a state-space model from data by ordinary least squares and then designs a controller using that model. Under the identifiability condition, the procedure recovers the optimal controller with noise-free data.

  • The rank condition on the data matrix ensures that the system matrices can be recovered in the noiseless case and is generically necessary for data-driven LQR design.With noiseless data, the condition is satisfied when the input is persistently exciting and (B, A) is controllable.Source: B. Subspace relations in state-space… · p. 2
  • Ordinary least squares estimates the system matrices from the input, state, and successor-state data, and certainty-equivalence design substitutes those estimates into the LQR problem.The approach is formalized as a bi-level program.Source: B. Subspace relations in state-space… · p. 2
  • With identifiability and noise-free data, the certainty-equivalence program is feasible and returns the optimal controller because the estimated matrices equal the true matrices.Source: B. Subspace relations in state-space… · p. 2

III. CERTAINTY EQUIVALENCE AS REGULARIZED & DIRECT DATA-DRIVEN LQR

The section develops a direct, data-matrix formulation of certainty-equivalence LQR and shows how to relate it to the indirect identification-and-design program. The formulation uses a regularizer that accounts for implicit identification.

III. CERTAINTY EQUIVALENCE AS REGULARIZED & DIRECT DATA-DRIVEN LQR

  • The bi-level certainty-equivalence program can be recast as a single-level convex program with a regularizer accounting for implicit identification.Source: §III · p. 2

A. Direct design & LQR parameterization by data matrices

  • Under the rank condition, a data-matrix parameterization replaces the closed-loop matrix A + BK with (X1 − D0)G and yields the control gain K = U0G.Source: A. Direct design & LQR… · p. 3
  • With noisy data, disregarding the unknown disturbance term gives a direct certainty-equivalence formulation; a particular regularized version is equivalent to the indirect program.The regularized and unregularized direct formulations coincide for noise-free data for every λ ≥ 0.Source: A. Direct design & LQR… · p. 3

B. A direct version of the certainty-equivalence LQR

The indirect certainty-equivalence LQR problem can be reformulated as a direct program without explicit system identification. Penalizing an orthogonality constraint yields a regularized formulation equivalent to the indirect one for sufficiently large λ.

B. A direct version of the certainty-equivalence LQR

  • The direct formulation adds an orthogonality constraint that makes G unique, and eliminating variables from either formulation yields the same compact problem.The indirect formulation eliminates the identified matrices; the direct formulation eliminates G.Source: B. A direct version of the… · p. 3
  • The direct and indirect formulations have identical cost functions and feasible sets.The direct formulation bypasses explicit system identification while retaining the certainty-equivalence design.Source: B. A direct version of the… · p. 3
  • Lifting the orthogonality constraint into the objective produces a regularized direct formulation.The constraint measures distance from G to the range space of W0.Source: B. A direct version of the… · p. 3
  • For sufficiently large λ, the regularized direct problem coincides with the original direct problem; for general λ, it lower-bounds that problem.The standard indirect certainty-equivalence problem is equivalent to the regularized formulation when λ is sufficiently large.Source: B. A direct version of the… · pp. 3–4
  • With noise-free data, the direct and regularized problems coincide for every λ ≥ 0.The theorem relating the feasible sets remains valid even if those sets are empty, for example when the identified pair is not stabilizable.Source: B. A direct version of the… · pp. 3–4

C. Tractable convex problem formulation

The direct and regularized formulations are converted into convex programs through variable substitutions, elimination, and a Schur complement. An epigraph formulation then supports exact penalization of the orthogonality constraint.

C. Tractable convex problem formulation

A. Preliminary considerations

A regularizer in the direct certainty-equivalence formulation penalizes large-norm solutions and relates the design to stability. Under identifiability, sufficiently high SNR ensures feasibility, closed-loop stability, and a bounded sub-optimality gap.

A. Preliminary considerations

  • The single-level direct program searches for a solution satisfying a Lyapunov inequality, while its regularizer penalizes G with large norm.The norm penalty helps make the Lyapunov inequality used in the program imply the desired stability constraint.Source: A. Preliminary considerations · p. 4
  • For sufficiently large λ, the direct program returns the least-Frobenius-norm solution, suggesting certainty-equivalence LQR has some noise robustness.The paper follows this observation with a rigorous analysis.Source: A. Preliminary considerations · p. 4
  • A different stability-promoting regularizer proposed in penalizes trace(GPG⊤), accounting for the full term multiplying the stability constraint.Source: A. Preliminary considerations · p. 4

B. Main robustness and performance result

  • Under identifiability, SNR is positive and compares useful information W0 with noise D0; without identifiability, it is zero.Source: B. Main robustness and performance… · p. 4
  • With identifiability and SNR above a threshold depending on ε, the certainty-equivalence problem is feasible and its solution stabilizes the system with a sub-optimality gap.The theorem states that for every ε > 0, some threshold ν suffices; the supplied passage does not give the gap expression.Source: B. Main robustness and performance… · p. 4

C. Proof of Theorem 4.2

The proof establishes the theorem’s high-SNR guarantees by bounding feasibility and performance quantities using auxiliary results. These bounds yield closed-loop stability and the stated sub-optimality guarantee.

C. Proof of Theorem 4.2

  • The feasibility argument compares the candidate optimal controller’s data-dependent terms and shows the required inequality holds at sufficiently large SNR.The candidate uses the ground-truth gain K⋆ and its controllability Gramian P⋆.Source: C. Proof of Theorem 4.2 · p. 5
  • The proof uses auxiliary results from, with full proofs supplied because the design program differs.Source: C. Proof of Theorem 4.2 · p. 5
  • Under identifiability, Lemma 4.3 says that if (8) is feasible and its η1 condition holds, K ensures stability; Lemma 4.4 says its η2 condition ensures feasibility of (8).Source: C. Proof of Theorem 4.2 · p. 5
  • For the stability and sub-optimality bound, the proof bounds P and K using the objective, then controls the remaining noise-dependent term through SNR.Source: C. Proof of Theorem 4.2 · p. 5
  • Combining the feasibility and stability inequalities yields the theorem’s closed-loop performance bound.Source: C. Proof of Theorem 4.2 · p. 5

A. Scaling of performance as SNR−1

The analysis relates certainty-equivalence performance to SNR and contrasts it with robust design across uncertainty regimes. It also describes sample-size conditions and the benefits of the equivalent single-level formulation.

A. Scaling of performance as SNR−1

  • For sufficiently large SNR, the sub-optimality gap decreases as SNR^-1, independently of noise statistics, matching the scaling of least-squares estimation error.The analysis is non-asymptotic and gives performance and stability bounds as functions of SNR.Source: A. Scaling of performance as SNR−1 · p. 5

B. Certainty-equivalence versus robust control

  • Robust approaches typically better ensure closed-loop stability, while certainty-equivalence can perform better when uncertainty is small.The cited comparison describes robust methods as less performing in low-uncertainty regimes.Source: B. Certainty-equivalence versus robust… · pp. 5–6

C. Sample complexity and Gaussian noise statistics

  • The non-asymptotic analysis permits nearly optimal controllers at small sample sizes, requiring only T ≥ n + m for identifiability.With Gaussian noise, averaging multiple experiments yields an explicit high-confidence SNR bound that decays inversely with the square root of the experiment count [16, Section 6.2].Source: C. Sample complexity and Gaussian… · p. 6

D. Comparison of direct and indirect problem formulations

  • The bi-level certainty-equivalence problem is equivalent to single-level formulations that support direct uncertainty mapping and further robustification.The regularized formulation permits smooth trade-offs between performance and robustness; the multi-criteria design jointly accounts for identification and control goals.Source: D. Comparison of direct and indirect… · p. 6

E. Data-dependent stability test

The section gives a data-dependent stability test, reports simulation comparisons across regularizers, and concludes that the direct formulation can interpolate between certainty-equivalence and robust design.

E. Data-dependent stability test

  • A known upper bound on noise magnitude enables a data-based closed-loop stability certificate; with K = 0, the condition tests open-loop stability from noisy data.The condition implies the stability guarantee described in the analysis.Source: E. Data-dependent stability test · p. 6

VI. NUMERICAL SIMULATIONS

  • Certainty-equivalence controllers are less robust but significantly outperform robust controllers when stabilizing; blending their regularizer with the robust-stability regularizer also gives remarkable empirical performance.Table I compares approaches across noise variances.Source: §VI · p. 6, Table I · p. 7
  • For the direct regularized approach, when λ ≥ 0.0028, empirical errors for this data set ranged from 5.922e-4 (minimum) to 0.0094 (maximum), with median error M = 0.0026.The figure summarizes 100 trials, each collecting T = 20 state and input samples.Source: §VI · p. 6, Figure 1 · p. 7

VII. CONCLUSIONS

  • The proposed direct regularized formulation is equivalent to indirect certainty-equivalence LQR, admits robustness-promoting regularization, and can interpolate between robust and certainty-equivalence design.The conclusion also points to extensions across system classes, identification criteria, regularizers, and control objectives.Source: §VII · p. 7

APPENDIX

The appendix proves feasibility relations used to connect the certainty-equivalence and direct formulations.

A. Proof of Lemma 4.3

  • Under condition (26), the proof rescales P to show feasibility for (11), using G = W† and the preceding feasibility relations.The proof then invokes [16, Lemma 2].Source: A. Proof of Lemma 4.3 · p. 7

B. Proof of Lemma 4.4

  • The second proof uses condition (27) to rescale the optimal solution of (11) into a feasible solution of (13), then relates it to feasibility for (8).The cost comparison uses optimality of (P, K) for (8).Source: B. Proof of Lemma 4.4 · pp. 7–8
Loading 2109.06643v1…