Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence

Problem formulation

• Mixed H2/H∞ control minimizes an H2-norm upper bound subject to an H∞ constraint that guarantees robustness against model uncertainty.The framework also includes linear risk-sensitive control under suitable H2 bounds.Source: §1 · pp. 3–4

Challenges

• Unlike LQR, the mixed-design cost remains finite near the H∞ feasibility boundary, so cost descent cannot ensure constraint preservation or global convergence with a constant stepsize.The H∞ constraint is frequency-domain and difficult to enforce through direct projection, especially when the system model is unknown.Source: §1 · p. 4, Figure 1 · p. 5

Main contributions

• The Gauss–Newton and natural policy-gradient methods implicitly regularize iterates toward stabilizing controllers satisfying the H∞ constraint, without explicit projection or regularization.This property depends on the algorithms’ search directions, unlike LQR’s algorithm-agnostic stability preservation.Source: §1 · pp. 4–5
• Both methods converge to the globally optimal policy despite nonconvexity, with globally sublinear and locally (super-)linear rates under certain conditions, avoiding spurious stationary points and local optima.The paper also studies discrete- and continuous-time settings and derives results for linear risk-sensitive control.Source: §1 · p. 4

Related work

• Existing mixed H2/H∞ and safe-RL approaches generally lack guarantees for both global convergence and on-the-fly H∞ constraint satisfaction, while convex lifted methods may be inefficient or not model-free.The paper positions direct policy search as a route toward model-free robust control design.Source: §1 · pp. 5–6

2.1 Motivating Example: LEQG

• LEQG admits an optimal memoryless, stationary LTI state-feedback policy, so optimization can be reduced without loss of optimality to the control gain K.This motivates gain-based policy optimization for the discrete-time setting.Source: §2.1 · p. 9

2.1.1 Cost Closed-Form

• For stabilizing K satisfying the Riccati conditions, the LEQG objective has a closed form through P_K, reducing to the LQG cost as β approaches zero.The paper notes that both the LTI optimal-controller result and this objective form were not rigorously established previously for LEQG; the former was hypothesized in Glover and Doyle (1988).Source: §2.1.1 · pp. 9–10

2.1.2 Implicit Constraint on H∞-Norm

• The apparently unconstrained LEQG problem has an implicit feasible set equal to stabilizing controllers satisfying an H∞-norm bound, which is frequency-domain, potentially unbounded, and difficult to enforce directly (Glover and Doyle (1988)).The Riccati conditions underlying the cost representation are later shown equivalent to this H∞ constraint.Source: §2.1.2 · p. 10

2.2 Bigger Picture: Mixed H2/H∞Control Synthesis

• Mixed H2/H∞ synthesis uses stationary LTI state-feedback under the stated information structure, constrains ||T(K)||∞ below γ for robust stability, and includes LEQG as a special case.The robustness interpretation follows the small-gain theorem, while LEQG is obtained by substituting β, W, and Q with γ^-2, DD^T, and C^T C, respectively.Source: §2.2 · pp. 11–12

2.3 Bounded Real Lemma

• The discrete-time Bounded Real Lemma converts the H∞ constraint into equivalent stabilizing Riccati-equation and Riccati-inequality conditions, enabling time-domain analysis of policy optimization.For LEQG, these conditions are necessary as well as sufficient for the Riccati-based objective form, so every feasible K has that form.Source: §2.3 · pp. 12–13

3 Landscape and Algorithms

• The section studies representative discrete- and continuous-time costs and develops policy-optimization algorithms for mixed H2/H∞ control.The analyzed settings include a discrete-time combination that also solves discrete-time LEQG problems, while techniques can apply to other setting combinations.Source: §3 · p. 13, §3.1 · p. 13

3.1 Optimization Landscape

• Mixed H2/H∞ design is nonconvex, and its discrete-time cost need not diverge as gains approach the feasible-set boundary.The noncoercivity arises because the Riccati-based evaluation equation can remain well-defined with finite cost near the boundary, unlike the coercive LQR landscape.Source: §3.1 · p. 14
• When the relevant controllability condition holds, the only stationary point is the unique global optimizer, whose gain has the form K* = (R + B^T eP_K* B)^-1 B^T eP_K* A.Without controllability, multiple stationary points may exist, but the global optimizer retains this form; analogous landscape results extend to the other objectives.Source: §3.1 · p. 15

3.2 Policy Optimization Algorithms

• The algorithms include policy-gradient, natural policy-gradient, and Gauss–Newton updates, with the Gauss–Newton method reducing to policy iteration when η = 1/2.Model-free policy-gradient updates can use sampled-data gradient estimates, whereas natural policy-gradient and Gauss–Newton implementations require additional quantities that are not directly available through zeroth-order methods.Source: §3.2 · pp. 15–16

4.1 Implicit Regularization

• Natural PG and Gauss–Newton are implicitly regularized: from any feasible gain, suitable constant-step updates remain stabilizing and satisfy the H∞ constraint without projection.The proof constructs a Lyapunov function from the current Riccati solution, then perturbs it to obtain strict feasibility; direct projection is difficult, and vanilla PG is not covered theoretically.Source: §4.1 · pp. 16–18

4.2 Global Convergence

• From any feasible initialization, both updates converge to the global optimum with an O(1/N) rate for the iteration-average squared gradient norm, without requiring controllability.Their search directions avoid suboptimal stationary points and converge to the optimal stationary point satisfying E_K = 0, while preserving robustness throughout the iterations.Source: §4.2 · p. 19
• The global results require an initially stabilizing controller that already satisfies the H∞ robustness constraint, reflecting the feasibility requirement of iterative local search.Unlike projection-based guarantees for related zero-sum LQ games, these results use implicit regularization without projections or non-standard robust-control assumptions.Source: §4.2 · pp. 19–20
• Under DD^T > 0, both methods converge locally linearly in objective value, while Gauss–Newton with η = 1/2 achieves a locally Q-quadratic rate.The faster local rates rely on local gradient dominance around the optimum.Source: §4.2 · p. 19

5.1 Proof of Theorem 4.3

• For any stepsize η ∈ [0,1/2], the interpolated gain Kη between K and K′ satisfies theorem conditions i)–iii), with P > 0 certifying K′ ∈ K.Source: §5.1 · pp. 20, 22

5.2 Proof of Theorem 4.4

• For Gauss–Newton updates with η ∈ [0,1/2], P_K decreases matrix-wise, and the iterates converge to the global optimum K* at O(1/N).The argument uses the discrete-time cost difference bounds to obtain monotonicity and convergence to the stationary condition E_K = 0, which is globally optimal.Source: §5.2 · p. 25
• Natural policy-gradient updates converge to the global optimum at O(1/N) when η ≤ 1/(2∥R + B^⊤eP_KB∥), while P_K decreases matrix-wise.The proof applies the same cost-difference bounds and monotonicity argument to the natural policy-gradient update.Source: §5.2 · pp. 25–26

5.3 Proof of Theorem 4.6

• Once iterates enter a neighborhood of K*, the surrogate Tr(P_KDD^⊤) contracts linearly, yielding a local linear convergence rate for J(K).Monotonicity keeps a sufficiently low-level set inside the neighborhood, and bounded Lyapunov solutions provide uniform contraction.Source: §5.3 · p. 28
• For Gauss–Newton updates with η = 1/2, the objective J(K) converges Q-quadratically around K* after the iterates enter the invariant neighborhood.The proof combines local linear contraction of the surrogate with a quadratic bound on the update error, with the relevant constants uniformly bounded thereafter.Source: §5.3 · p. 28

6.1 Connection to Zero-Sum LQ Games

• The mixed-design problem is equivalent to minimizing the zero-sum LQ game's value over stabilizing gain pairs, with the disturbance maximization producing the corresponding Riccati equation and global optimum.This equivalence identifies the mixed-design objective with the game formulation and maps the game's equilibrium solution to the mixed-design optimum.Source: §6.1 · p. 30

6.2 Model-Free Algorithms

• Model-free policy-gradient methods for the zero-sum LQ game can therefore solve the mixed-design problem, because the game solution is equivalent to the mixed-design global optimum.The game can be sampled through trajectory data, unlike the direct natural policy gradient for mixed design.Source: §6.2 · p. 30
• Under certain conditions, jointly stationary policy gradients coincide with the Nash equilibrium, while some prior policy-gradient methods require projection when updating the disturbance policy.Double-loop methods remove that projection requirement, although the cited passage presents this as a development direction rather than a quantified result.Source: §6.2 · pp. 30–31
• The disturbance update searches for the worst-case disturbance underlying the H∞ norm, and for fixed K it converges to the best response L(K), supporting algorithm feasibility.The game simulator uses the mixed-design simulator with disturbance modeled as w_t = −Lx_t.Source: §6.2 · p. 31
• The outer-loop natural policy-gradient update for K is identical to the mixed-design update and converges to the game's Nash equilibrium without requiring direct trajectory sampling of the original gradient.The model-free algorithms use the game formulation to implement the mixed-design natural policy gradient.Source: §6.2 · pp. 30–31

7.1 Implicit Regularization & Global/Local Convergence

• All four convergent update rules preserved the H∞ bound γ = 15.45, maintained γ^2I − D^T P_K D > 0, and exhibited sublinear-to-linear or super-linear convergence.NPG and Gauss–Newton initially showed sublinear gradient-norm convergence followed by linear convergence, while Gauss–Newton with η = 1/2 displayed super-linear convergence.Source: Figure 2 · p. 32, Figure 3 · p. 32, §7.1 · p. 33
• Vanilla policy gradient violated the H∞ constraint and failed to converge from an initialization with γ = 1.00001·||T(K0)||∞, even using stepsize 10^-7.The existence of a constant stepsize guaranteeing both global convergence and robustness preservation for vanilla policy gradient remains unresolved.Source: §7.1 · p. 33

7.2 Escaping Suboptimal Stationary Points

• Across 50 trials with infinitely many stationary points, NPG and Gauss–Newton always converged to K*, whereas vanilla policy gradient could become trapped at suboptimal stationary points.The result supports implicit regularization through search directions that avoid bad local minima and favor the global optimum.Source: Figure 4 · p. 33, §7.2 · p. 34

7.3 Comparison with Existing H2/H∞Control Solvers

• 5.93× faster runtime than HIFOO was obtained for γ = 5, while γ = 3 produced a HIFOO violation of ||T(K)||∞ = 3.4353 despite policy-optimization feasibility.The policy-optimization methods also achieved lower H2 and H∞ norms in both comparisons; against h2hinfsyn, they were around 4× faster on the simple case.Source: §7.3 · pp. 35–36, Table 1 · p. 36
• 8×, 47×, and 295× faster runtimes than HIFOO were achieved in Cases 4–6, while h2hinfsyn could hardly solve Cases 5–6 and returned no solution.The methods reached almost identical control gains without constraint violations, with advantages increasing as problem dimension grew.Source: §7.3 · pp. 36–37, Table 3 · p. 37

8 Concluding Remarks

• The paper concludes that its proposed policy-optimization methods converge globally for H2 control with H∞ robustness guarantees and exhibit implicit regularization despite noncoercivity.The authors present the work as an initial step toward reinforcement-learning algorithms for robust and risk-sensitive control.Source: §8 · pp. 37–38
• Future directions include analyzing other policy-optimization methods, extending the theory to LQ games and broader dynamical systems, comparing model-based and model-free efficiency, and handling nonsmooth H∞ synthesis.The current algorithms require an initially feasible policy; output-feedback extensions are more difficult because the optimization landscape depends on controller parameterization.Source: §8 · pp. 38–39

A.1 Formulation

• The continuous-time mixed H2/H∞ problem is a constrained nonconvex optimization with a robustness condition ∥T(K)∥∞ < γ and a Riccati-based H2 objective.Under state feedback, stationary policies u = −Kx suffice, and the continuous-time cost uses the solution of a continuous-time Riccati equation.Source: §A.1 · p. 48

A.1.1 Bounded Real Lemma

• The continuous-time Bounded Real Lemma characterizes the H∞ constraint through equivalent Riccati and positive-definite LMI conditions.For stabilizing K, the norm bound is equivalent to existence of a stabilizing Riccati solution and P > 0 satisfying the strict matrix inequality.Source: §A.1.1 · pp. 48–49

A.2.1 Optimization Landscape

• Unlike coercive LQR, continuous-time mixed H2/H∞ design is nonconvex and its objective is not coercive.This lack of coercivity motivates the implicit-regularization analysis for policy-based algorithms.Source: §A.2.1 · p. 49
• Under controllability conditions, any stationary point is unique and equals the global optimizer, whose gain has the form K* = R^-1B^TP_K*.The required controllability condition holds when DD^T > 0.Source: §A.2.1 · pp. 49–50

A.2.2 Policy Optimization Algorithms

• The continuous-time algorithms include policy-gradient, natural policy-gradient, and Gauss–Newton updates derived from the policy gradient.The vanilla policy-gradient update is given explicitly, while the natural PG and Gauss–Newton updates receive the implicit-regularization guarantee.Source: §A.2.2 · p. 50, §A.3.1 · p. 50

A.3.1 Implicit Regularization

• Natural policy-gradient and Gauss–Newton updates preserve stability and the specified H∞-norm robustness constraint under suitable stepsizes.The updated gain remains stabilizing and admits a stabilizing Riccati solution, as established using the Bounded Real Lemma.Source: §A.3.1 · p. 50

A.3.2 Global Convergence

• Both updates (A.6) and (A.7) converge globally to K* with averaged stationarity residual decreasing at O(1/N).Theorem A.7 assumes K0 ∈ K, finite ∥K0∥, detectability, and the stepsizes of Theorem A.6.Source: §A.3.2 · p. 51
• Implicit regularization avoids spurious stationary points and bad local minima without requiring controllability, yielding the globally optimal K*.The result applies even when multiple stationary points may exist.Source: §A.3.2 · p. 51
• Under DD^T > 0, both updates converge locally linearly, while Gauss–Newton with η = 1/2 converges locally Q-quadratically.Global convergence remains sublinear, whereas the faster rates hold near the optimum under the additional condition.Source: §A.3.2 · p. 51

B Supplementary Proofs

• The supplementary section supplies proofs for the paper’s previously stated results.These proofs support the convergence and robustness claims developed in the main text.Source: B Supplementary Proofs · p. 52

B.1 Proof of Lemmas 2.7 and A.1

• For discrete time, the Bounded Real Lemma and stabilizing-solution uniqueness show equivalence among the Riccati, H∞-norm, and LMI robustness conditions.The surrogate transfer function satisfies ∥eT(K)∥∞ < 1 exactly when ∥T(K)∥∞ < γ, and the corresponding LMI reduces to Lemma 2.7.Source: §B.1 · p. 52
• For continuous time, the Bounded Real Lemma, stabilizing-solution uniqueness, and KYP Lemma establish the Riccati and LMI characterizations of ∥T(K)∥∞ < γ.The continuous-time LMI follows from the KYP condition via a Schur complement, completing Lemma A.1.Source: §B.1 · pp. 52–53

Discrete-time

• Discrete-time: stable gains K1, K2, and K3 have H∞ norms 0.4350, 0.7011, and 1.6575, proving nonconvexity for γ ∈ (0.7011, 1.6575).K3 = (K1 + K2)/2 remains stabilizing, but its H∞ norm exceeds those of K1 and K2.Source: §B.2 · p. 53

Conclusion

• Thus, the H∞-norm constraint set is nonconvex despite stability of the endpoint gains and their midpoint in both time settings.The discrete-time and continuous-time constructions complete the nonconvexity argument.Source: §B.2 · p. 53

Continuous-time

• Continuous-time: ∥T(K1)∥∞ = 0.3860, ∥T(K2)∥∞ = 0.5306, and ∥T(K3)∥∞ = 1.1729; therefore, K is not convex when γ ∈ (0.5306, 1.1729).Source: §B.2 · p. 53

B.3 Proof of Lemmas 3.2 and A.3

• The cost need not diverge at the robustness-set boundary: under DD^T > 0, J(K) remains finite as K approaches ∂K.A one-dimensional construction gives K → 1.2573, P_K → 3.3752, and finite costs for (2.11) and (2.12).Source: §B.3 · pp. 53–54
• For costs (2.11) and (2.12), finiteness additionally requires I − γ^-2D^TP_KD to remain nonsingular, which the one-dimensional counterexample verifies at the boundary.In the example, 1 − γ^-2D^2P_K → 0.2354 > 0 while P_K remains finite and positive.Source: §B.3 · pp. 53–54

B.4 Proof of Lemmas 3.3

• The proof of Lemma 3.3 shows that P_K is continuously differentiable in K throughout the open feasible set K.The implicit-function argument uses strict LMIs, continuity of the Riccati fixed-point map, and invertibility implied by the stability condition.Source: §B.4 · pp. 55, 57
• This differentiability makes J(K) differentiable everywhere on K and supports the stated policy-gradient recursion through matrix differential and Riccati identities.The derivation uses the invertible Jacobian of the vectorized fixed-point equation and verifies the gradient elementwise before assembling the matrix form.Source: §B.4 · pp. 57–58

B.5 Proof of Lemma A.4

• For the objective in (A.2), the feasible policy set K is open because strict LMI feasibility persists in a sufficiently small neighborhood of every K.The proof then applies the implicit function theorem to the vectorized Riccati equation.Source: §B.5 · p. 58
• The Jacobian with respect to vec(P_K) is invertible because A − BK + γ^-2DD^TP_K is Hurwitz, yielding continuous differentiability of P_K and J(K).The Jacobian is expressed with Kronecker products, and the Hurwitz property supplies the required invertibility.Source: §B.5 · pp. 58–59
• The resulting policy gradient is ∇J(K) = 2[RK − B^TP_K]Λ_K.The formula follows by combining the Riccati differential, Lyapunov equations, and trace identities.Source: §B.5 · p. 59

Proposition 3.4

• For Proposition 3.4, comparing game values yields P_K ≥ P_K* and therefore J(K) ≥ J(K*) for every feasible stabilizing K.The argument applies Bas¸ar and Bernhard (1995) to identify the maximizing disturbance and uses determinant monotonicity to transfer the Riccati ordering to the objective.Source: §B.6 · p. 60
• For Proposition 3.4, positive-definiteness of Δ_K makes K* = (R + B^T P_K* B)^-1 B^T P_K* A the unique stationary point and global optimizer.The proof establishes Δ_K > 0 using observability and Zhou et al. (1996, Lemma 21.2), then applies the necessary condition ∇J(K) = 0.Source: §B.6 · p. 60

Proposition A.5

• For Proposition A.5, continuous-time game theory and eigenvalue monotonicity similarly show J(K) ≥ J(K*) whenever P_K ≥ P_K*.The proof requires detectability of (A,C) to apply Bas¸ar and Bernhard (1995, Theorem 4.8).Source: §B.6 · p. 60
• For Proposition A.5, Λ_K > 0 implies that K* = R^-1 B^T P_K* is the unique stationary point and hence the unique global optimizer.Positive definiteness follows from controllability, equivalently observability of the transposed pair, via Zhou et al. (1996, Lemma 3.18 (iii)).Source: §B.6 · pp. 60–61

B.7 Proof of Lemma 5.1

• Lemma B.1 shows that I − γ^−2P_K′DD^⊤ is invertible for K,K′ ∈ K, using positivity of I − γ^−2D^⊤P_K′D and the matrix inversion lemma.The determinant identity establishes invertibility, while the matrix inversion lemma yields the required equivalent relation.Source: §B.7 · p. 61
• Applying the Riccati equation and Lemma B.1, then unrolling the resulting inequality by induction, gives the upper bound in (5.25).The proof rewrites the expression using P_K and substitutes the resulting expansion into (B.33).Source: §B.7 · p. 62
• Interchanging K and K′ in Lemma B.1 supplies a lower bound, and continued unrolling yields (5.26), completing the proof.The lower-bound expression is substituted into (B.34) before the final unrolling step.Source: §B.7 · p. 62

Certificate and feasibility

• A certificate P > 0 satisfying (B.36) implies that K′ is stabilizing and that ∥T(K′)∥∞ < γ, so K′ remains feasible.The LMI yields a Hurwitz closed loop and is equivalent to the H∞ robustness constraint via Lemma A.1.Source: §B.8 · p. 63

Gauss–Newton update

• For the Gauss–Newton update with η = 1/2, choosing P = P_K + α P̄ and sufficiently small α makes (B.36) hold.The construction uses the Lyapunov equation for the stable closed-loop system and bounds the resulting terms so their sum is negative.Source: §B.8 · pp. 63–64
• The same certificate argument preserves feasibility along the interpolated Gauss–Newton updates K_η = (1 − 2η)K + 2ηK′ for η ∈ [0, 1/2].Convexity of quadratic functions combines the LMIs for K and K′ to establish the constraint for the entire interpolation range.Source: §B.8 · p. 64

Natural policy gradient

• For natural policy gradient, sufficiently small α establishes (B.36) when η ≤ 1/(2∥R∥), proving ∥T(K′)∥∞ < γ and guaranteeing a stabilizing Riccati solution.The proof again sets P = P_K + α P̄, bounds the terms in the LMI, and applies Lemma A.1.Source: §B.8 · p. 64

Continuous-Time Cost Difference Lemma

• Lemma B.2 bounds the Riccati-cost difference PK′ − PK from above, and also from below when A − BK′ + γ^-2DD⊤PK is Hurwitz.The bounds follow from Lyapunov equations and the positive-semidefinite term γ^-2∆P⊤DD⊤∆P.Source: §B.9 · pp. 64–65

Lyapunov Lower Bound

• Lemma B.3 lower-bounds the positive Lyapunov-equation solution MK under K⊤RK ≤ CK·I using the stability quantity ωK.The proof combines Riccati-equation eigenvalue bounds with the relationship between operator norm and numerical radius.Source: §B.9 · pp. 66–67

Gauss–Newton Update

• For Gauss–Newton updates with η ≤ 1/2, PK decreases matrix-wise, and the iterates converge to the global optimum K* with sublinear O(1/N) rate.The update remains in the feasible set by Theorem A.6; trace bounds force convergence to RK − B⊤PK = 0, which Proposition A.5 identifies as K*.Source: §B.9 · pp. 67–68

Natural Policy Gradient Update

• For natural policy-gradient updates with η ≤ 1/(2∥R∥), PK likewise decreases matrix-wise along feasible iterates.The proof applies the same continuous-time cost-difference and trace arguments used for Gauss–Newton updates.Source: §B.9 · p. 68

• The iterates eventually remain in a neighborhood of K*, where the Lyapunov solution M_K,K* is uniformly bounded, yielding uniform local linear contraction.A suitable lower-level set is contained in the neighborhood, so once entered, the iterates cannot leave it.Source: §B.10 · p. 69
• For η = 1/2, the policy update yields Q-quadratic convergence of Tr(P_Kn DD^T) around K*.The relevant constant becomes uniformly bounded after the iterates remain in the local neighborhood.Source: §B.10 · pp. 69–70
• Using analogous arguments, the proof also establishes a local linear rate with a different contraction factor.Source: §B.10 · p. 70

C Auxiliary Results

• The Gaussian-integral lemma converts exponential quadratic expectations into modified quadratic forms when I −βPZ > 0.The resulting matrix is e_P = P + βP(Z−1 −βP)−1P.Source: C Auxiliary Results · p. 71
• The LEQG optimum is J* = −β−1 logdet(I −βP_K*W), achieved among well-defined controls by stationary LTI state feedback μ_t = −K*x_t.The gain K* is obtained from the stabilizing solution of the modified Riccati equation under W−1 −βP_K* > 0.Source: C Auxiliary Results · pp. 71, 73
• The finite-horizon dynamic-programming update is quadratic in the control, and minimizing it yields the modified Riccati recursion and associated scalar log-determinant terms.The key update is [F_u(V_t)](x) = c(x,u) + (Ax + Bu)^T e_Pt(Ax + Bu) + λ_t.Source: C Auxiliary Results · p. 72
• The modified Riccati recursion converges from admissible P_0 to the unique stabilizing fixed point P_K*, using monotonicity and boundedness.The assumptions require P_0 ≤ Q, P_0 ≥ 0, and W−1 −βP_0 > 0, with W−1 −βP_K* > 0.Source: C Auxiliary Results · pp. 74–75
• For any stabilizing fixed controller K, the Riccati recursion converges to the unique minimal stabilizing solution P_K under W−1 −βP_K > 0.The recursion is monotone and bounded when initialized with 0 ≤ P_0 ≤ Q, so its limit is P_K.Source: C Auxiliary Results · p. 75

D Pseudocode

• Algorithm 1 estimates ∇_L C(K,L) and Σ_K,L for stabilizing (K,L) using uniformly perturbed policies, m trajectories, and rollout length R.It returns empirical gradient and correlation estimates from sampled perturbations of radius r.Source: D Pseudocode · p. 76
• Algorithm 2 estimates L(K) through iterative policy-gradient or natural-policy-gradient updates initialized at a stabilizing L_0.The natural update multiplies the estimated gradient by the inverse estimated correlation matrix.Source: D Pseudocode · p. 76
• Algorithm 3 estimates K* by perturbing each outer iterate, calling Inner-NG for each perturbed policy, and applying policy-gradient or natural-gradient updates.It returns the final outer iterate K_T after T iterations.Source: D Pseudocode · p. 77