PaperlayerView exact revision on arXiv

Source-linked AI summary

Critic-Free Policy Iteration for Continuous-Time Zero-Sum Games: A Policy-Space Riccati Approach

Jiacheng Wu, Yang Zhu, Hongye Su

arXiv:2609.11564v1eess.SYcs.SY

TL;DR

Data-driven methods for continuous-time zero-sum games typically retain a value-matrix critic alongside policy gains. This paper formulates policy iteration directly in joint policy space and removes the value-matrix term from its data-driven update; its results separate unique policy recovery from critic identifiability.

  • Problem

    Data-driven zero-sum-game methods typically estimate policy gains together with value-matrix parameters, although the critic is not part of the final feedback law.

    Source: §I · p. 1

  • Method

    The paper constructs a joint-policy PGRE and uses endpoint projection of an integral identity to obtain a fixed-batch actor-only regression for unknown dynamics.

    Source: §I · pp. 1–2

  • Results

    The policy-space formulation corresponds to symmetric GARE solutions, and its data-driven update reproduces model-based policy iteration with a necessary and sufficient actor rank condition invariant across stabilizing iterations.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Unique policy recovery does not require value-matrix identifiability, and separating critic parameters yields computational and memory benefits as state dimension grows.

    Source: §I · p. 2, §VIII · p. 11

Abstract

from arXiv · show

This paper develops a critic-free policy iteration (PI) method for continuous-time linear zero-sum games. The central idea is to characterize the saddle-point policies directly in the joint policy space, rather than treating the quadratic value matrix as an iterative variable. A policy game Riccati equation (PGRE) is introduced whose unknowns are policy gains only. Its solutions are shown to be in one-to-one correspondence with the symmetric solutions of the game algebraic Riccati equation. Based on a stabilizing anchor, PI is performed directly in the actor space. The actor-space Jacobian is nonsingular at every stabilizing policy, and the resulting policy sequence coincides with that of simultaneous PI. For unknown dynamics, a data-driven algorithm uses a single batch of data and nullspace projection of endpoint increments to eliminate the value matrix, yielding an actor-only regression. A necessary and sufficient rank condition for unique policy recovery is established and shown to be iteration-invariant, demonstrating that critic identifiability is unnecessary. A power systems frequency-regulation example verifies convergence and policy recovery, while scalability tests demonstrate substantial reductions in computational and memory requirements.

I. INTRODUCTION

Continuous-time zero-sum games link saddle-point feedback to stabilizing GARE solutions, but data-driven methods typically estimate both policy gains and a value-matrix critic. This paper develops policy-space and data-driven critic-free methods, with actor recovery separated from critic identifiability.

I. INTRODUCTION

  • For continuous-time linear systems with quadratic performance, a stabilizing GARE solution characterizes saddle-point control and disturbance policies.Unknown system matrices motivate learning these policies from measured trajectories.Source: §I · p. 1
  • Existing data-driven zero-sum-game PI methods typically estimate policy gains jointly with quadratic value-matrix parameters.The paper identifies the critic as an intermediate variable rather than part of the final feedback law.Source: §I · p. 1
  • The paper introduces a joint-policy PGRE whose solutions correspond directly to symmetric GARE solutions, without the Hamiltonian-observability construction used in.The formulation uses a Sylvester lifting and supports actor-space Newton updates from a stabilizing anchor.Source: §I · pp. 1–2
  • For unknown dynamics, endpoint projection removes the value-matrix term from a fixed-batch integral identity, yielding an actor-only regression.The policy-recovery rank condition is separated from critic identification.Source: §I · p. 2
  • The framework derives a necessary and sufficient actor rank condition that remains invariant across stabilizing iterations, so unique policy recovery does not require an identifiable value matrix.The paper also reports convergence and policy recovery in a frequency-regulation example, with computational and memory benefits in scalability tests.Source: §I · p. 2

II. SYSTEM MODEL AND ZERO-SUM GAME

The paper formulates an infinite-horizon continuous-time zero-sum game with unknown linear-system dynamics and quadratic running payoff. Under stated stabilizability, detectability, and attenuation-level conditions, a stabilizing GARE solution characterizes the saddle-point policies.

II. SYSTEM MODEL AND ZERO-SUM GAME

  • The model uses a continuous-time linear system with state, control, and exogenous disturbance, while the system matrices are unknown.The disturbance is specified as square-integrable over [0, ∞).Source: §II · p. 2
  • The running payoff is quadratic in state and control, with a negative disturbance term weighted by the prescribed attenuation level.The infinite-horizon payoff is defined from this running payoff.Source: §II · p. 2
  • A saddle-point policy pair minimizes the upper value against admissible control and disturbance policies.The formulation assumes (A, B1) stabilizable and (Q^1/2, A) detectable, with feasible attenuation γ > γ* and a stabilizing GARE solution.Source: §II · p. 2
  • The corresponding saddle-point policies are characterized through the value matrix, which the paper later avoids treating as an iterative variable.The system and payoff setup thus supplies the model-based target for the policy-space reformulation.Source: §II · p. 2

III. POLICY GAME RICCATI EQUATION

The policy game Riccati equation (PGRE) recasts the GARE and saddle-point stationarity conditions as equations in joint policy gains. Under its spectral assumption, PGRE solutions correspond one-to-one with symmetric GARE solutions, while stability and feasibility select game-relevant policies.

III. POLICY GAME RICCATI EQUATION

  • The joint policy variable combines minimizing control and maximizing disturbance gains, allowing the GARE and stationarity conditions to be expressed using policy gains alone.The reformulation rewrites the quadratic Riccati term as a quadratic function of the joint policy matrix.Source: §III · pp. 2–3
  • The PGRE consists of (m + q)n scalar polynomial equations of degree at most two in the same number of policy variables.Its construction uses a Sylvester lifting under an open-loop spectral condition.Source: §III · p. 3
  • The PGRE and symmetric GARE solution sets are in one-to-one correspondence.This correspondence does not require a matching condition between the control and disturbance channels or injectivity of B_g^T.Source: §III · p. 3
  • A PGRE zero need not be stabilizing or admissible for the infinite-horizon game; closed-loop stability and γ > γ* select the game-relevant solution.The correspondence covers all symmetric GARE solutions, not only stabilizing ones.Source: §III · p. 3

IV. MODEL-BASED CRITIC-FREE RL ALGORITHM

A fixed stable anchor yields an anchored PGRE whose zeros correspond to symmetric GARE solutions and support Newton iteration directly in policy space. At every stabilizing policy the actor-space Jacobian is nonsingular, including with the indefinite game weighting.

IV. MODEL-BASED CRITIC-FREE RL ALGORITHM

  • A fixed anchor with Hurwitz closed-loop matrix removes the open-loop spectral requirement from the Newton formulation.The anchored lifting is defined through a fixed Lyapunov operator, which is invertible under the anchor assumption.Source: §IV · p. 3
  • The anchored PGRE compensates for replacing A with the fixed stable matrix A_a through two cross terms.Its residual is differentiable, with the quadratic perturbation term of order O(∥E∥^2).Source: §IV · pp. 3–4
  • The anchored PGRE solutions correspond one-to-one with symmetric GARE solutions, and coincide with the unanchored PGRE solutions when its additional spectral assumption holds.At an anchored PGRE zero, the lifting satisfies the closed-loop policy-evaluation equation and the gains satisfy stationarity.Source: §IV · p. 4
  • The actor-space Jacobian is nonsingular at every stabilizing policy, so the Newton direction is unique there.The proof requires R_g to be symmetric and nonsingular, not positive definite; this permits the indefinite game weighting.Source: §IV · p. 4

A. Critic-Free Policy Iteration Design

The method computes policy updates directly in actor space, with a unique Newton direction at stabilizing policies. Its updates match simultaneous policy iteration, including the offline SPUA policy sequence and convergence.

A. Critic-Free Policy Iteration Design

  • The actor-space Newton direction is unique at each stabilizing policy, with only the sn policy-increment entries unknown.The fixed operator supplies the required value sensitivity without constructing a policy-dependent critic matrix.Source: A. Critic-Free Policy Iteration Design · pp. 4–5
  • The critic-free Newton update is exactly equivalent to a simultaneous PI step, and its policy sequence matches offline SPUA.The paper establishes equality of the generated policy sequences by induction.Source: A. Critic-Free Policy Iteration Design · pp. 4–5
  • The sequence converges to Θ* and (Li, Ki) converges to (L*, K*).The convergence conclusion follows from the cited offline SPUA result.Source: A. Critic-Free Policy Iteration Design · p. 5

V. DATA-DRIVEN CRITIC-FREE RL ALGORITHM

The data-driven method uses one batch of trajectory data and endpoint nullspace projection to eliminate the value matrix, leaving a policy-only residual. Under the stated assumptions, its regression recovers the same updates as the model-based Newton method.

V. DATA-DRIVEN CRITIC-FREE RL ALGORITHM

  • One batch of state, control, and disturbance data supports an actor-only policy residual after endpoint-increment projection eliminates the value matrix.The algorithm uses the data-driven formulation to avoid requiring system matrices for the model-based anchor-response operator.Source: §V · p. 5
  • Assumption 3 requires ν ≥ sn; N ≥ dn + sn ensures this dimension requirement but does not ensure full column rank, which also requires excitation in both components of va(t).The sample-count bound follows from rank(Dx) ≤ dn.Source: §V · p. 5
  • If Assumption 3 holds, the projected data residual and policy residual have the same zero set, corresponding to symmetric GARE solutions.The zero-set equivalence follows from full column rank of Za.Source: §V · p. 6
  • Actor informativity is independent of the chosen orthonormal nullspace basis, as are the rank condition, residual zero set, and uniquely recovered actor.The projected quantities change only by a left orthogonal transformation when the basis changes.Source: §V · p. 6
  • Under Assumptions 2 and 3, Γi and Λi have full column rank and yield unique, mutually equivalent solutions producing the model-based Newton policy.The actor-only regression and its algebraic rearrangement are equivalent to the model-based Newton update.Source: §V · p. 6

VI. DATA INFORMATIVITY ANALYSIS

The section characterizes when collected data uniquely determine a policy update without identifying the value matrix. Endpoint projection removes the nuisance critic without losing actor solutions, and actor informativity remains valid across stabilizing iterations.

VI. DATA INFORMATIVITY ANALYSIS

  • The analysis characterizes when collected data uniquely determine the policy update without identifying the value matrix.The policy update is uniquely determined if and only if the stated rank condition holds.Source: §VI · p. 7
  • Left-nullspace projection eliminates the nuisance critic parameter losslessly with respect to the actor variable.The resulting actor solution set is the same as that of the actor–critic regression.Source: §VI · p. 7
  • Actor informativity can be checked once from the fixed data batch and remains valid through stabilizing iterations without critic identifiability.Table I summarizes the rank conditions and the quantities they uniquely determine.Source: Table I · p. 7
  • Endpoint projection eliminates the nuisance critic by passing to the quotient space modulo im(C), preserving the feasible actor set.Any left annihilator with kernel im(C) produces the same feasible actor set.Source: §VI · p. 7

A. Identifiability and Data Requirements

Theorem 3 separates the data needed to identify the critic from the data needed to identify a policy update. Unique policy recovery requires the actor rank condition, not critic identifiability.

A. Identifiability and Data Requirements

  • Theorem 3 distinguishes the information required for critic identification from that required for policy identification.Thus data can identify a policy update without identifying the critic.Source: A. Identifiability and Data Requirements · p. 7
  • Joint actor–critic identifiability requires rank(Dx) = dn and rank(Λi) = sn; the first condition gives critic uniqueness and the second gives policy uniqueness.The actor condition is necessary and sufficient for unique policy identification.Source: A. Identifiability and Data Requirements · p. 7
  • The sample-count bound N ≥ dn+sn is conservative when rank(Dx) < dn and does not guarantee actor identifiability.Condition (70) is necessary only; full column rank also requires adequate excitation.Source: A. Identifiability and Data Requirements · p. 7

B. Iteration-Invariant Actor Informativity

Actor informativity is determined by the fixed data batch and remains valid across stabilizing policy iterations. It can be checked once without requiring critic identifiability.

B. Iteration-Invariant Actor Informativity

  • Actor informativity depends only on the fixed data batch, not on the stabilizing policy iterate.Source: B. Iteration-Invariant Actor… · p. 7
  • At every stabilizing iterate, the actor-informativity condition is necessary and sufficient because the actor-space Jacobian is nonsingular.The proof relates the ranks of Γi, Λi, and Za.Source: B. Iteration-Invariant Actor… · p. 8
  • Actor informativity can be verified once from the fixed batch and remains valid throughout stabilizing iterations without critic identifiability.Table I summarizes the rank conditions and the quantities they uniquely determine.Source: B. Iteration-Invariant Actor… · p. 8

VII. ILLUSTRATIVE EXAMPLE

The frequency-regulation example tests model-based equivalence, data-driven policy recovery, data informativity, and scalability. The reported results include policy recovery without critic identifiability and reduced computational and memory requirements after critic elimination.

VII. ILLUSTRATIVE EXAMPLE

  • Algorithm 1 and SPUA produce identical gain trajectories at every iteration and converge to the same optimal gains.Source: §VII · p. 8, Figure 1 · p. 8
  • Endpoint projection combines weighted endpoint increments along a nullspace direction so their complete sum vanishes, eliminating the value-matrix term.The figure illustrates interval states used to form Dj and coefficients αj1 for the nullspace direction.Source: §VII · p. 8, Figure 2 · p. 9
  • Algorithm 2 recovers the optimal saddle-point gains after four iterations without explicitly estimating the value matrix, and all state components converge toward equilibrium.The trajectory shown uses the learned gains K4 and L4.Source: §VII · p. 8, Figure 3 · p. 9, Figure 4 · p. 9
  • Cases I and II recover the optimal gains to numerical precision, although Case II has a nonidentifiable critic; Case III has a nonunique policy update.The comparison shows policy recovery depends on actor informativity rather than critic identifiability.Source: §VII · pp. 9–10, Table II · p. 10
  • At n = 50, the critic-free and joint actor–critic formulations use 100 and 1375 unknowns, respectively; their parameter dimensions grow linearly and quadratically with n.The actor-only regression has sn = 2n unknowns, compared with sn + dn for the joint regression.Source: §VII · p. 10, Table III · p. 10
  • At n = 50, regression working memory is approximately 0.15 MB for critic-free updates and 15.5 MB for joint actor–critic updates.The per-update speedup grows from a few-fold in low-dimensional systems to several tens at n = 50.Source: §VII · p. 10, Figure 6 · p. 10
  • At n = 50, retained sufficient-statistics memory is approximately 4 MB after projection versus 44 MB for raw joint actor–critic statistics, an approximately 11-fold reduction.The projected and raw representations scale approximately as O(n3) and O(n4), respectively.Source: §VII · p. 10, Figure 7 · p. 11
  • At n = 50, the one-time projection overhead is amortized after approximately three policy updates.The critic-free method is initially more expensive, but its lower recurring update cost yields an increasing cumulative advantage beyond break-even.Source: §VII · pp. 10–11, Figure 8 · p. 11

VIII. CONCLUSION

The conclusion argues that continuous-time zero-sum game policies can be improved without using the value matrix as an intermediate variable. Endpoint projection retains the information needed for unique policy updates, even when the critic is not identifiable.

VIII. CONCLUSION

  • The PGRE determines the same joint-policy update directly that conventional policy iteration constructs using the value matrix.This equivalence holds under the stated conditions.Source: §VIII · p. 11
  • Endpoint projection removes the value-matrix term while retaining the information needed for policy improvement, so unique policy updates do not require critic identifiability.The usual full-rank condition for a joint actor–critic regression is stronger than policy improvement requires.Source: §VIII · p. 11
  • Because critic parameters grow quadratically with state dimension, separating policy recovery from critic identification yields increasing computational and memory benefits.Numerical studies corroborate the identifiability result and demonstrate these benefits.Source: §VIII · p. 11
Loading 2609.11564v1…