Source-linked AI summary
Data-Driven Policy Iteration Without an Initial Stabilizing Policy: A Finite-Horizon Bootstrap Method
Jiacheng Wu, Yang Zhu
TL;DR
Infinite-horizon data-driven PI lacks a self-starting policy-evaluation step because it requires a stabilizing gain that is difficult to certify when system matrices are unknown. This paper bootstraps PI with finite-horizon evaluation for a shifted system and an independent data-driven Lyapunov certificate. Numerical studies demonstrate the proposed method’s effectiveness.
Problem
Method
The method applies finite-horizon PI to a shifted system using original-plant trajectory data, then independently certifies the candidate gain before initializing infinite-horizon PI.
Source: §1 · p. 2, §2.1 · p. 3Results
In the batch-reactor study, the data-derived gain passed the Lyapunov certificate and subsequent off-policy RL converged to the optimal solution within a few iterations.
Source: §5.1 · p. 10Takeaways & Limitations
The finite-horizon bootstrap provides a route to an admissible initializer without requiring an initially stabilizing policy.
Source: §6 · p. 12The stated assumptions require the shifted pair (Aρ, B) to be stabilizable and (Aρ, Q^1/2) to be detectable.
Source: §3 · p. 3
Abstract
from arXiv · showhide
This article investigates data-driven policy iteration (PI) for continuous-time linear systems without requiring an initially stabilizing policy. Standard infinite-horizon PI is not self-starting because its policy-evaluation step is well posed only when the feedback gain is stabilizing. However, verifying this property is difficult when the system matrices are unknown. To remove this requirement, we develop a finite-horizon bootstrap method. The key idea is to perform policy evaluation over a compact interval for a shifted system, where the evaluation equation is well defined for arbitrary bounded time-varying policies. We show that, for a sufficiently long horizon, the initial-time optimal gain of the shifted finite-horizon problem, when applied as a constant feedback gain, achieves a prescribed stability margin for the original system. We then derive a data-driven implementation from an off-policy identity evaluated along trajectories of the original plant. We use basis-function approximations to reconstruct the finite-horizon value matrix and policy, and we characterize the resulting error through a perturbed policy-improvement recursion. A data-driven Lyapunov certificate is further introduced to verify admissibility of the candidate gain before it is used to initialize infinite-horizon PI. Numerical studies of a batch reactor and a two-mass-spring system demonstrate the effectiveness of the proposed bootstrap method.
1 Introduction
Infinite-horizon PI is attractive for data-driven LQR but requires an admissible initial gain, which is difficult to construct when system matrices are unknown. The paper proposes a finite-horizon bootstrap and contrasts it with VI, bias-PI, and homotopy approaches.
1 Introduction
- Infinite-horizon PI requires a stabilizing initial gain, making data-driven initialization difficult when unknown system matrices are needed to construct and certify admissibility.Without stabilization, the infinite-horizon cost may be nonfinite and its Lyapunov equation does not define a valid value matrix.Source: §1 · p. 1
- VI avoids an initial stabilizing gain but may require step-size selection, boundedness verification, and an explicit switching criterion before PI begins.Its first-order updates may also converge more slowly than PI’s local Newton iteration.Source: §1 · p. 1
- Bias PI shifts the stability requirement to an auxiliary matrix, while larger bias can expand initialization at the cost of making the iteration more VI-like.Its implementation requires choosing a sufficiently large bias for well-posed auxiliary evaluation.Source: §1 · p. 1
- Homotopy methods construct a stability-preserving path through auxiliary infinite-horizon problems, so each intermediate policy must still stabilize its problem.The paper’s Table 1 compares initialization requirements across continuous-time data-driven RL methods.Source: §1 · p. 2, Table 1 · p. 2
- The bootstrap uses finite-horizon evaluation, which is well defined for arbitrary bounded intermediate policies, to remove the need for an initially stabilizing policy.The method performs this evaluation on a compact interval and uses the resulting gain to initialize infinite-horizon PI.Source: §1 · p. 2
- The paper establishes that the finite-horizon initial-time gain approaches the shifted infinite-horizon optimum as the horizon grows, and develops an off-policy implementation with error analysis and an independent admissibility certificate.The implementation uses original-plant trajectories; basis-approximation and regression errors are treated through a perturbed policy-improvement recursion.Source: §1 · p. 2
2 Preliminaries and problem statement
The problem is to construct from measured data a constant admissible gain that can initialize infinite-horizon PI when the system matrices are unavailable. The proposed bootstrap uses finite-horizon evaluation for a shifted system, where arbitrary bounded policies are evaluable, and sufficiently long horizons yield an ε-admissible gain for the original system.
2 Preliminaries and problem statement
- The system matrices are unavailable, and the target is a constant admissible gain from measured data to initialize infinite-horizon PI.The assumed setting includes a stabilizable pair (A, B) and a detectable pair (A, Q^1/2).Source: §2 · pp. 2–3
- Infinite-horizon policy evaluation is restricted to stabilizing gains because a nonstabilizing closed loop can make the unbounded cost nonfinite and invalidate its value matrix.The standard PI iteration therefore requires an admissible initial gain.Source: §2.1 · p. 3, §2 · p. 3
2.1 Why a finite-horizon bootstrap method
- On a compact interval, the terminal-value evaluation equation has a unique solution for any bounded time-varying gain, without requiring closed-loop exponential stability.This makes finite-horizon evaluation available before an admissible gain has been found.Source: §2.1 · p. 3
- The method evaluates the shifted dynamics with Aρ = A + ρI, where ρ > ε, and uses the finite-horizon optimal gain at the initial time as a candidate for the original system.As the horizon increases, that gain approaches the stabilizing infinite-horizon gain for the shifted problem.Source: §2.1 · p. 3
- For sufficiently large horizons, continuity of the spectral abscissa ensures that the initial-time shifted finite-horizon gain is ε-admissible for the original system.The schematic depicts increasing horizons and the first certified horizon Tε.Source: §2.1 · p. 3, Figure 1 · p. 3
3 Model-based finite-horizon bootstrap algorithm
The model-based bootstrap uses finite-horizon policy iteration on a shifted system, where evaluation is well posed without an initially stabilizing policy. For a sufficiently long horizon, its initial-time gain provides a stability-margin-qualified initializer for infinite-horizon policy iteration.
3 Model-based finite-horizon bootstrap algorithm
- Finite-horizon policy evaluation is well posed for any bounded time-varying policy, so the bootstrap does not require a stabilizing initial gain.After convergence, the policy at time zero is used as the constant-gain initializer for infinite-horizon PI.Source: §3 · p. 4
- The algorithm repeats finite-horizon PI at an expanding horizon until the candidate passes the admissibility check, then returns it to initialize infinite-horizon PI.The horizon is multiplied by γ > 1 after a failed check.Source: §3 · pp. 4–5
- Under stabilizability and detectability of the shifted pair, fixed-horizon policy iteration is well defined and converges to the finite-horizon optimal policy.The theorem allows bounded piecewise-continuous initial policies; the value matrices are monotone and converge uniformly.Source: §3 · pp. 3–5
- For sufficiently large horizons, the initial-time finite-horizon optimal gain approaches the shifted infinite-horizon optimal gain and becomes ε-admissible for the original system.The procedure verifies admissibility directly and enlarges the horizon when the candidate fails the spectral test.Source: §3 · pp. 4–5
4 Data-driven finite-horizon bootstrap algorithm
The data-driven method reconstructs the finite-horizon policy iteration from behavior trajectories of the original plant, without requiring trajectories of the shifted system. It treats approximation and regression errors as perturbations, then certifies the candidate constant gain before returning it as an initializer.
4 Data-driven finite-horizon bootstrap algorithm
- The off-policy implementation uses behavior data from the original plant, with the effect of shifting the system represented explicitly in the identity.Only original-system behavior data over the enlarged horizon are needed when the horizon grows.Source: §4 · pp. 5–6
- Basis-function expansions convert the unknown time-varying value matrix and policy into coefficient matrices that can be estimated from the off-policy identity.The value-matrix basis enforces the terminal constraint through its endpoint condition and terminal matrix.Source: §4 · p. 6
- The data-driven recursion is an inexact finite-horizon policy-improvement iteration whose perturbation captures basis truncation and data-conditioning effects.Uniformly small perturbations yield practical convergence, while vanishing perturbations recover convergence to the finite-horizon optimal policy.Source: §4 · p. 8
- For a sufficiently long horizon and under the stated perturbation conditions, the candidate gain at time zero is ε-admissible after sufficiently many iterations.The theoretical neighborhood radius is model dependent and cannot be checked directly from data.Source: §4 · p. 8
- A data-driven Lyapunov test certifies the candidate when the reconstructed Lyapunov matrix is positive definite and the data satisfy the richness condition.The resulting certificate establishes ε-admissibility for the original system.Source: §4 · pp. 8–9
- The bootstrap uses an inner fixed-horizon PI loop and expands the horizon after a failed certificate, returning the candidate once the test succeeds.The returned candidate initializes subsequent infinite-horizon PI.Source: §4 · p. 9
5 Illustrative examples
In the batch-reactor study, the model-based and data-driven bootstrap methods reach admissibility at a horizon of 0.3052. On the two-mass-spring system, Algorithm 2 certifies an initializer at the first tested horizon and uses fewer iterations than the compared methods.
5.1 Case study of the batch reactor system
- At T_s = 0.3052, the model-based bootstrap accepts an ε-admissible initializer with α(A − BK_c) < −0.2, as the closed-loop spectral abscissa decreases monotonically with horizon.The reported closed-loop poles are −0.5822 ± 0.5852i, −8.3664, and −7.6089.Source: Table 2 · p. 10, §5.1 · p. 10
- At T_s = 0.3052, data-driven finite-horizon PI converges in five iterations, and its Lyapunov certificate succeeds with relative residual 7.432 × 10^−4 and λ_min(P_c) = 5.938 × 10^−2.Candidates at the first five horizons are rejected; the certified gain then initializes off-policy RL, which converges to the optimal solution within a few iterations.Source: §5.1 · p. 10
- After bootstrap initialization, both ∥P[j] − P*∥ and ∥K[j] − K*∥ decrease rapidly in the batch-reactor off-policy RL stage.The paper reports that the subsequent off-policy RL converges to the optimal solution within a few iterations.Source: §5.1 · p. 10, Figure 5 · p. 11
5.2 Comparative study of the two-mass-spring system
- Algorithm 2 succeeds at the first tested horizon, T_s = 0.1, and converges in three finite-horizon PI iterations, versus 101 stabilizing iterations for homotopic PI and 193 iterations for hybrid PI.The certified candidate has α(A − BK_c) = −1.0445 × 10^−2, λ_min(P_c) = 10.84354, and relative residual 4.732 × 10^−4.Source: Figure 8 · p. 11, §5.2 · p. 12
6 Conclusion
The paper presents a finite-horizon bootstrap for data-driven infinite-horizon policy iteration without an initially stabilizing policy. It reconstructs a candidate gain from plant trajectories and verifies its admissibility with a data-driven Lyapunov certificate before using it to initialize infinite-horizon PI.
6 Conclusion
- The bootstrap generates an admissible candidate gain by exploiting finite-horizon policy evaluation for a shifted system.The finite-horizon evaluation is well posed for arbitrary bounded time-varying policies; increasing the horizon enlarges the candidate-gain margin.Source: §6 · p. 12
- An off-policy implementation reconstructs the time-varying value matrix and policy from measured trajectories.The trajectories are collected from the original plant.Source: §6 · p. 12
- A data-driven Lyapunov certificate verifies the candidate gain before it initializes infinite-horizon PI.Numerical studies demonstrate the effectiveness of the method.Source: §6 · p. 12