PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy-Gradient Transfer under a Shared State Estimator

Dongming Wang, Wei Ren

arXiv:2610.21693v1eess.SYcs.SY

TL;DR

Policy-gradient transfer can fail when a simulator and physical plant differ, even when both use the same state estimator. This paper bounds their gradient discrepancy using noise-weighted model errors without an estimator-accuracy guarantee, and derives occupancy-weighted, ascent-direction, and finite-training guarantees. Simulations illustrate the bounds and their conservatism.

  • Problem

    When simulator and plant models differ, their policy gradients may misalign, so a simulator-improving update can reduce plant return.

    Source: §I · p. 1

  • Method

    The paper bounds gradient discrepancy by noise-weighted dynamics and observation-model errors, using the shared fixed estimator without requiring an accuracy guarantee.

    Source: §I · p. 1

  • Results

    Simulations on a scalar system and a pendulum illustrate the bounds; on the pendulum, an identified residual model changes estimated mean gradient alignment from negative to positive.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The results provide an occupancy-weighted refinement, a sufficient ascent condition, and a finite-budget bound on expected squared plant gradient for simulator-only training.

    Source: §I · pp. 1–2, §IV · p. 5

Abstract

from arXiv · show

This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.

I. INTRODUCTION

The paper studies when simulator policy gradients can transfer to a plant controlled through the same estimator, since model mismatch can make a simulator-improving update reduce plant return. It develops a gradient-discrepancy analysis for differing dynamics and observation models without requiring an estimator-accuracy guarantee.

I. INTRODUCTION

  • A simulator-improving policy update may decrease plant return when model mismatch makes the plant and simulator gradients poorly aligned.A sufficiently small positive step improves the plant objective when the gradients have a positive inner product.Source: §I · p. 1
  • Existing results do not bound exact closed-loop gradient differences for partially observed plant–simulator pairs sharing a fixed estimator without model derivatives or estimator-accuracy assumptions.The paper identifies this as the gap it addresses.Source: §I · p. 1

A. Two environments, one observer

  • The analysis compares systems whose dynamics and observation maps may differ but whose policy, estimator, reward, stopping rule, initial law, and noise covariances are common.The estimator processes measurements in both environments, but model mismatch can still change its memory and subsequent actions.Source: §I · p. 1, A. Two environments, one observer · p. 2
  • A shared measurable estimator is treated as common post-processing, allowing trajectory-law comparison without an estimator-accuracy bound.The estimator may be built from a model other than the training simulator.Source: §I · p. 1, A. Two environments, one observer · p. 2

B. Stopping, reward, objective

The paper formulates a stopped, discounted reward objective for the plant and simulator, with trajectories absorbed when they leave the live set. Its squashed Gaussian actor has bounded latent means and first- and second-order parameter derivatives, supporting score-based analysis.

B. Stopping, reward, objective

  • Trajectories are absorbed at a cemetery state when the physical gate or live-set condition is violated, and rewards are zero after absorption.The objective also has an H-step truncation, with discount γ in (0, 1).Source: B. Stopping, reward, objective · p. 2
  • The environment, estimator, gates, initial law, reward, discount, and noise covariances are fixed independently of the policy parameter.If identification establishes the model-error bounds on a high-probability event, the conclusions apply on that event.Source: B. Stopping, reward, objective · p. 2

C. Actor

  • The actor squashes a Gaussian latent action into the open action box, with uniformly bounded latent means and first two parameter derivatives.The latent mean depends on the estimator readout, and the covariance is fixed and positive definite.Source: C. Actor · p. 2
  • The policy score is defined at live steps, while scores vanish after absorption; the squashing map preserves the score of the action density.The score is built from the latent Gaussian and the parameter derivative of its mean.Source: C. Actor · p. 2

III. POLICY-GRADIENT TRANSFER

The paper establishes score identities and smoothness for the stopped objective, then uses them to support policy-gradient transfer analysis. Bounded score moments and a Lipschitz gradient provide the regularity needed for that analysis.

A. Differentiating the stopped objective

  • For each environment, the stopped expected reward is twice continuously differentiable, and its gradient is the discounted sum of reward-weighted score expectations.The gradient is Lipschitz with constant LJ.Source: A. Differentiating the stopped objective · p. 3
  • The cumulative score has second moment at most (t+1)G2, which controls score variability over a finite stopped trajectory.The one-step score has conditional mean zero and bounded second moment.Source: A. Differentiating the stopped objective · p. 3
  • A likelihood-ratio representation expresses stopped expected rewards under a reference policy using a finite history likelihood ratio.The history-law changes arise from the latent Gaussian action distributions.Source: A. Differentiating the stopped objective · p. 3
  • Polynomial envelopes and Gaussian score moments justify differentiating under the integral, while uniform convergence justifies termwise differentiation of the discounted objective.The resulting derivatives are continuous and yield the stated smoothness bound.Source: A. Differentiating the stopped objective · p. 3

B. From model error to gradient error

The section compares plant and simulator trajectory laws through their shared estimator to bound policy-gradient discrepancy. The resulting bound needs no estimator-accuracy or contraction guarantee, but is worst-case and not uniform as noise vanishes.

B. From model error to gradient error

  • The shared measurable estimator acts as a common pushforward, allowing the local plant and simulator laws to be compared without an estimator-accuracy term.The local comparison uses the noise-weighted discrepancies between dynamics and observation maps.Source: B. From model error to gradient error · pp. 3–4
  • Theorem 1 uses trajectory-law comparison and score-moment bounds to control the difference between exact plant and simulator policy gradients.The gradient integrand is the same measurable function of the trajectory under both laws, and its second moment is bounded in each environment.Source: B. From model error to gradient error · pp. 3–4
  • The bound contains no estimator-accuracy or contraction constant, because the observer is common post-processing of measurements, though it still shapes both closed-loop objectives.The bound is worst-case and is not uniform as noise vanishes because it uses relative entropy scaled by inverse noise covariances.Source: B. From model error to gradient error · p. 4
  • The passage states that no matching lower bound is given, so the discrepancy bound’s tightness is not established.The supplied text does not provide the lower-bound expression.Source: B. From model error to gradient error · p. 4

C. Occupancy and ascent consequences

The section refines worst-case transfer error using the plant’s occupancy and derives a sufficient condition for simulator-gradient steps to improve the plant objective. The occupancy refinement is useful only when its losses are sufficiently small.

C. Occupancy and ascent consequences

  • The occupancy-weighted transfer bound replaces worst-case model error with dynamics and observation losses evaluated under the plant’s occupancy.Their discounted sum is related to the combined expected losses by the tower property.Source: C. Occupancy and ascent consequences · p. 4
  • The occupancy-weighted first term improves on the comparison only if Ef + Eh < 2a/(γ(1 − a^2)).The passage compares this first term with the model-independent term.Source: C. Occupancy and ascent consequences · p. 4
  • If the simulator gradient norm exceeds a valid bound on gradient discrepancy, then sufficiently small positive steps along it increase the plant objective.The discrepancy bound may be chosen from either the worst-case transfer bound or the square root of the occupancy-weighted bound.Source: C. Occupancy and ascent consequences · p. 4

IV. FINITE-BUDGET TRAINING GUARANTEE

The finite-budget result bounds expected squared plant-gradient magnitude at a uniformly random simulator-training iterate using optimization, truncation, and model-error terms. The training procedure uses stopped simulator rollouts and requires no plant interaction after identification.

IV. FINITE-BUDGET TRAINING GUARANTEE

  • Algorithm 1 averages gradients from B independent simulator rollouts stopped at length K, updates parameters with step size α, and returns a uniformly selected iterate.The procedure uses fresh randomness and resets, with no critic, advantage normalization, gradient clipping, or data-dependent stopping of training.Source: §IV · p. 4
  • The rollout gradient estimator has truncation bias bounded by bK and a second-moment bound, which are used in the randomized-output analysis.The proof combines these bounds with smoothness and a bounded objective range to control expected gradient magnitude.Source: §IV · pp. 4–5
  • The proposition bounds expected squared plant-gradient magnitude at the random output iterate using optimization, truncation, and model terms.The reported guarantee requires at most BTK simulator transitions and no plant interaction after identification.Source: §IV · p. 5
  • Proposition 2 does not establish monotone improvement, last-iterate convergence, optimality, or closed-loop stability.The stated guarantee concerns expected squared gradient at a random iterate.Source: §IV · p. 5

A. Shared protocol

The experimental protocol uses a shared fixed EKF recursion for each plant–simulator pair, while estimates and covariances evolve separately. It measures gradients with a frozen predictable baseline and uses independent data streams for calibration, measurement, training, and evaluation.

A. Shared protocol

  • Table I is identified as the benchmark table, and its caption states that both systems use γ = 0.95 and rewards in (0, 1] with Rmax = 1.The supplied caption gives no benchmark values or comparison.Source: Table I · p. 5
  • The scalar system and pendulum use a shared fixed EKF recursion within each plant–simulator pair, while the estimator’s estimates and covariances evolve separately.True states are available for calibration and evaluation but not to the controller.Source: A. Shared protocol · p. 5
  • Gradient estimates use a predictable baseline fitted on an independent pilot batch and frozen, with the same baseline on both sides of each pair.The resulting finite-horizon estimator is unbiased for the truncated objective’s gradient.Source: A. Shared protocol · p. 5
  • Figure 1 compares S1 squared gradient discrepancy with κ2 and, for P1, plant-gradient inner products with nominal and learned simulator gradients.The caption specifies H = 512 and the interval procedures, but does not state a winner or outcome.Source: Figure 1 · p. 6
  • Measurement, training, pilot, and evaluation streams are distinct, and reported intervals are approximate and pointwise.The protocol uses block intervals for S1 and a percentile cluster bootstrap over twelve datasets for P1.Source: A. Shared protocol · p. 6

B. Gradient discrepancy at fixed policies

At fixed policies, the estimated squared gradient discrepancy stays below the theoretical bound and scales with model error, but observer and noise changes affect realized discrepancies and the P1 bounds are often too conservative to certify them.

T1 and T2 (zero error and scaling)

  • Across all 30 tested cells with positive λ, the estimated squared discrepancy lies below the bound and grows with κ2, with log–log slopes of 1.05 to 1.08.At λ = 0, paired gradients agree to floating-point precision despite estimation error of 0.11 to 0.17; this checks pairing, not the theorem.Source: B. Gradient discrepancy at fixed… · p. 6

T3 (observer and noise level)

  • At λ = 1, squared discrepancies vary substantially across frozen observers, while the learned EKF has the smallest estimation error in both tested policies.For policies (0, −1, 0) and (0.4, −1, 0), discrepancies are 0.18 and 1.3 under the learned EKF, versus 0.47 and 1.2 under the nominal EKF.Source: B. Gradient discrepancy at fixed… · p. 6
  • As noise scaling changes κ2 from 0.54 to 34.7, the measured squared discrepancy remains between 0.41 and 0.55, while the squared uniform bound changes by a factor of 64.These settings vary the noise and EKF jointly and do not isolate a zero-noise limit.Source: B. Gradient discrepancy at fixed… · p. 6

T4 (bound magnitudes)

  • For the S1 policies and horizons tested, every measured squared discrepancy is below the squared finite-horizon bound by a factor of at least 200.For P1, the plug-in minimum is the model-independent cap M2, and the transfer and deployment expressions provide no useful certificate.Source: B. Gradient discrepancy at fixed… · p. 6

C. Training outcomes on the pendulum

On the pendulum, the learned residual simulator improves plant returns relative to the nominal simulator, especially from weak initialization, while measured gradient alignment favors the learned simulator but the theoretical bounds do not certify the training outcomes.

Training setup

  • Training compares nominal, residual-augmented, and privileged exact-model simulators under frozen learned EKFs across twelve datasets, with random-output checkpoints evaluated on held-out plant episodes.Runs use T = 256, B = 16, K = 256, and three optimizer seeds per dataset; the weak-initialization arm uses two.Source: C. Training outcomes on the pendulum · pp. 6–7

T6 (alignment at common parameters)

  • At the nominal run’s output in the weak-initialization arm, learned-simulator gradient alignment is positive in 12 of 12 datasets, while nominal-simulator alignment is negative in 10 of 12.In the original arm, the corresponding inner products are negative for N and positive for L in 12 of 12 datasets each.Source: C. Training outcomes on the pendulum · p. 7

T7 (plant gradient at the training output)

  • Estimated squared plant gradients at random outputs are lower for L than N in both initialization arms, but an ordering of upper bounds does not imply the same ordering of exact gradient norms.The paired N−L differences are 7.4 [0.3, 17] in the original arm and 80 [65, 96] in the weak-initialization arm.Source: C. Training outcomes on the pendulum · p. 7

Plant return

Observer dependence

  • The return-change pattern depends on the shared estimator, which the uniform bound does not resolve.With the nominal EKF, every initial-policy and LQR episode exits through the physical gate; the initial policy’s RMS angle-estimation error is 0.22 rad, versus 0.052 under Fℓ.Source: C. Training outcomes on the pendulum · p. 8

VI. CONCLUSION AND FUTURE WORK

The paper bounds plant–simulator policy-gradient discrepancy under a shared fixed observer and derives related ascent and finite-training guarantees. Pendulum results show improved estimated alignment with an identified residual model, but the bounds are conservative and realized discrepancy depends on observer and noise.

Main results

  • The plant–simulator policy-gradient difference is bounded linearly by noise-weighted dynamics and observation-model errors without an estimator-accuracy guarantee.The paper also derives an occupancy-weighted bound, a sufficient ascent condition, and a finite-budget bound on expected squared plant gradient at a random training iterate.Source: §VI · p. 8

Simulation results

  • On the pendulum, an identified residual model changes estimated mean gradient alignment from negative to positive, but the bounds exceed measured discrepancies by several orders of magnitude and do not certify the outcomes.Realized discrepancy depends on the observer and noise level, which the uniform bound does not resolve.Source: §VI · p. 8

Future work

  • Future work will consider sharper constants, finite-data certification of model errors, unequal noise covariances, and quantitative links to control performance.Source: §VI · p. 8

Related papers

Loading 2610.21693v1…