PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal-Difference Calibration of Black-Box Simulators

Min Zeng, Xin Wen, Xi Chen, Yichen Zhang, Wen-Xin Zhou

arXiv:2610.35311v1stat.MLcs.LG

TL;DR

The paper studies how to calibrate black-box simulators when target-system observations are limited to one-step transitions and simulator values are noisy, while gradients and policy values are unavailable. It develops Bellman-orthogonal derivative-free TD calibration and inference; its guarantees depend on simulator specification, and a second TD stage corrects remaining value discrepancy without extra simulator queries.

  • Problem

    Calibrating simulators is difficult when the target supplies one-step transitions, simulator evaluations are noisy, and simulator gradients and the policy value are unavailable.

    Source: §1 · p. 2

  • Method

    The paper defines calibration through Bellman orthogonality and develops derivative-free TD algorithms using random-direction finite differences, plus a few-replicate online bootstrap.

    Source: §1 · pp. 2–4

  • Results

    Under correct specification, the calibration target recovers the physical parameter and DF-Ind and DF-Fwd can attain rollout-budget rate exponents approaching 1/2; misspecification generally reduces their exponents to 1/3 and 1/4, respectively.

    Source: §1 · pp. 2, 4

  • Takeaways & Limitations

    A second-stage TD recursion estimates and corrects the remaining value discrepancy without additional simulator queries.

    Source: §1 · p. 2

    Along Markov trajectories, conventional multiplier inference can be inconsistent when the TD score has temporal dependence; correct specification restores valid inference.

    Source: §5 · pp. 22–23

Abstract

from arXiv · show

We develop a derivative-free temporal-difference (TD) framework for calibrating black-box simulators using one-step transitions from a target system and noisy simulator-generated continuation values. The calibration parameter is defined through Bellman orthogonality, recovering the physical parameter under correct simulator specification and yielding a near-optimal $L_2$ approximation of the policy value under misspecification. We propose three TD algorithms based on random-direction finite differences, distinguished by their query costs and reuse of noisy simulator evaluations. We establish almost-sure convergence, last-iterate mean-square error bounds, and central limit theorems for Polyak-Ruppert averages across four query-noise regimes. Our analysis reveals a specification-dependent bias phenomenon: for two algorithms, the finite-difference bias of the TD estimating equation vanishes at the calibration target under correct specification, even when the gradient surrogate remains biased. This permits a fixed perturbation radius and rate exponents approaching $1/2$ in the simulator rollout budget, whereas misspecification generally recovers the classical Kiefer-Wolfowitz rates. We also develop a few-replicate online bootstrap that yields Student-$t$ calibrated confidence intervals under negligible query noise, and identify a covariance mismatch that can invalidate conventional multiplier inference along Markov trajectories. Finally, a second-stage TD recursion corrects the remaining value discrepancy without additional simulator queries.

1 Introduction

The paper addresses simulator calibration when target observations are limited to one-step transitions and simulator values are noisy and gradients unavailable. It introduces Bellman-orthogonal derivative-free TD methods, with guarantees that depend on simulator specification, and develops online inference and value correction.

1 Introduction

  • The framework calibrates black-box simulators from target-system one-step transitions and noisy simulator continuation values, without requiring direct policy-value observations or simulator gradients.Simulation can support policy evaluation when direct experimentation is costly or impractical, while retaining physically interpretable structure.Source: §1 · pp. 2–3
  • Bellman orthogonality defines the calibration target, recovering the physical parameter under correct specification and yielding an L2 value approximation within a constant factor of the best simulator-family approximation under misspecification.The target avoids estimating the additional instrument needed to recover the conventional L2 calibration target from Bellman residuals.Source: §1 · p. 2
  • Three random-direction finite-difference TD algorithms trade simulator query cost against reuse of noisy evaluations, with convergence, last-iterate mean-square error, and averaged central limit guarantees across four query-noise regimes.DF-Sym saves one query per update while retaining DF-Ind's second-order accuracy; persistent query noise can add non-negligible variability through products of query errors.Source: §1 · p. 3
  • Under correct specification, DF-Ind and DF-Fwd's estimating-equation bias cancels at the target despite gradient-surrogate bias, allowing fixed perturbation radii and rollout-budget rate exponents approaching 1/2.Under misspecification, the cancellation generally fails, with attainable exponents reduced to 1/3 for DF-Ind and 1/4 for DF-Fwd.Source: §1 · pp. 3–4
  • The few-replicate online multiplier bootstrap provides confidence intervals without trajectory storage, but conventional multiplier inference can mismatch long-run covariance along Markov trajectories.The mismatch distorts coverage under misspecification in an example, while correct specification restores valid inference.Source: §1 · p. 4
  • A second-stage TD recursion estimates the remaining value discrepancy from calibration TD residuals and corrects it without additional simulator queries.The paper establishes joint asymptotic normality for calibration and correction estimators, enabling inference on corrected value.Source: §1 · p. 4

2 Temporal-Difference Calibration

This section defines a policy-specific TD calibration target using one-step Bellman discrepancies and value gradients. It relates that target to L2 calibration, establishes approximation and stability properties, and sets out the sampling and inference framework.

Temporal-Difference Calibration

  • The TD calibration target is a root of the population semigradient equation, formed by weighting the target system's one-step Bellman discrepancy by the simulator value gradient.Because the equation uses transitions under the policy weighted by μ, the target generally depends on the policy, discount factor, and state weighting, and calibrates value prediction for that policy.Source: §2 · pp. 6–7
  • The policy value is not directly observed, so TD calibration replaces the value error in the L2 orthogonality condition with the one-step Bellman discrepancy.The TD instrument is the value gradient, whereas the L2 first-order condition uses a discounted accumulation of gradients over current and past states that is not directly available from one-step observations.Source: §2 · pp. 6–7
  • Under misspecification, calibration targets generally depend on the instrument, while correct specification makes the value-matching parameter a common root for admissible instruments.Thus the TD target is policy-oriented, and its relationship to L2 calibration is not generally equality under misspecification.Source: §2 · p. 7
  • With correct specification, a unique TD root and unique L2 projection make both targets equal the value-matching parameter θ†; under misspecification, the TD-calibrated value’s L2 error is within a constant factor of the best simulator-family approximation.For any θ with V(·; θ) in L2(μ), its policy-value error is at most (1 − γ)^−1 times its Bellman-discrepancy error.Source: §2 · pp. 7–8
  • At a correctly specified root, local stability is equivalent to positive definiteness of the value-gradient second-moment matrix; under misspecification, discrepancy and simulator curvature can compromise stability even when that matrix is positive definite.The stated Hurwitz condition under misspecification requires the discrepancy-curvature product to be smaller than (1 − γ) times the matrix's minimum eigenvalue.Source: §2 · p. 8
  • The benchmark recursion uses exact gradients and stationary i.i.d. one-step samples, while its averaged estimator has a score covariance with transition and Bellman-discrepancy contributions.The discrepancy contribution vanishes when the simulator is correctly specified and the TD root is unique.Source: §2 · pp. 8–9

3 Methodology

The methodology combines random-direction finite differences of noisy simulator values with one-step TD residuals to calibrate without simulator gradients. Its three algorithms trade query reuse and cost against dependence in the noisy update factors.

3 Methodology

  • Random-direction finite differences of noisy simulator evaluations approximate unavailable gradients and are combined with noisy TD residuals to form derivative-free updates.The complete TD increment’s finite-difference bias depends on how its residual and gradient-surrogate factors are combined.Source: §3 · p. 9, §3.3 · p. 11

3.1 Noisy value queries

  • A noisy value query averages Mt independent simulator rollouts, with higher precision as the number of rollouts increases.Each rollout returns a random discounted cumulative reward.Source: §3.1 · p. 9

3.2 Random-direction surrogates and finite-difference bias

  • Two-sided finite differences allow a more slowly shrinking perturbation radius for the same approximation accuracy, reducing query-noise amplification when queries are noisy.DF-Sym retains second-order finite-difference accuracy while removing the separate central query; DF-Ind also has second-order accuracy, whereas DF-Fwd has generic first-order bias.Source: §3.2 · p. 10, §3.3 · p. 11

3.3 Derivative-free temporal-difference learning

  • Three derivative-free TD variants combine noisy TD residuals with finite-difference gradient surrogates, differing in their finite-difference construction and simulator-query reuse.DF-Ind uses separate queries; DF-Fwd and DF-Sym reuse evaluations, with query noise coupling the residual and gradient surrogate.Source: §3.3 · pp. 10–11

4 Theoretical Analysis

The section analyzes how stochastic-approximation error, finite-difference bias, and amplified query noise shape convergence and averaged inference across query-noise regimes. It characterizes when fixed perturbation radii preserve centered root-T inference under correct specification and how rollout replication and simulator accuracy inform design choices.

4.1 Last-iterate convergence

  • Under the stated rate conditions, the iterates converge almost surely, with error controlled by stochastic-approximation, squared finite-difference-bias, and amplified query-noise terms.For persistent query noise, balancing bias and noise gives rates O(t^-α/2) for DF-Fwd and O(t^-2α/3) for DF-Ind and DF-Sym; query reuse does not change the last-iterate rate order.Source: §4.1 · pp. 13–14

4.2 Averaged inference across query-noise regimes

  • Polyak–Ruppert averages satisfy a unified central limit theorem across query-noise regimes when the corresponding rate conditions hold and finite-difference bias is negligible at the relevant rate.In the fast-decaying regime, all three designs share score covariance Q; when query noise is not negligible, query design determines its contribution to the limiting covariance. The averaged limit also supports inference for the calibrated value.Source: §4.2 · pp. 14–16
  • With persistent query noise, DF-Sym has smaller asymptotic covariance than DF-Ind when the standardized fourth moment µ4 is below 5.For ζ > 0, their noise-product contributions are lower order and the designs have the same leading query-noise contribution; the µ4 < 5 comparison applies at ζ = 0.Source: §4.2 · p. 16

4.3 Fixed perturbation radius under correct specification

  • Under correct specification, DF-Ind and DF-Fwd preserve θ∗ as the mean-field root at any fixed perturbation radius, whereas DF-Sym may shift it through its midpoint residual.For sufficiently small fixed radii, Theorem 3 establishes convergence for DF-Ind and DF-Fwd without the usual finite-difference bias conditions; fixed radius also prevents query-noise amplification from growing over time.Source: §4.3 · p. 17

4.4 Design trade-offs under budgets of transitions and rollouts

  • In the fast-decaying regime, DF-Ind and DF-Sym have two-sided accuracy and the same gradient-query noise level, but DF-Sym uses one fewer query; DF-Fwd uses three queries and has four times the gradient-noise factor.Under persistent noise, the covariance comparison between DF-Ind and DF-Sym supplies an additional design criterion.Source: §4.4 · p. 18
  • For centered root-T inference, first- and second-order designs require rollout budgets of order T^(2+ε) and T^(3/2+ε), respectively; rollout-indexed rate exponents approach 1/4 and 1/3.Second-order accuracy also lowers the fast-decaying finite-difference-bias threshold from β > 1/2 to β > 1/4, and the persistent-noise thresholds from 1/4 to 1/6, allowing slower radius shrinking and replication growth.Source: §4.4 · pp. 18–19
  • DF-Sym can shift the target through its midpoint residual unless the value function is affine, and query reuse can add bias under fixed replication when query variance depends on θ.DF-Ind and DF-Fwd preserve θ∗ at sufficiently small fixed radius under correct specification.Source: §4.4 · pp. 18–19
  • Under correct specification, DF-Ind and DF-Fwd retain centered root-T inference with a fixed radius when ζ > 0, while the exponent 1/{2(1 + ζ)} approaches 1/2 as ζ ↓ 0.In the persistent regime, fixed rollout replication preserves the same exponents, changing the rate in T but not the rollout-budget frontier; the preferred design depends on believed simulator accuracy.Source: §4.4 · p. 20

5 Few-replicate Online Bootstrap

The paper develops a fixed-small-replicate online multiplier bootstrap for DF-TD, with bootstrap paths sharing observations and perturbation directions while using independent multipliers and fresh simulator queries. Its Student-t intervals have stated validity conditions, including a covariance-matching boundary under Markov trajectory sampling.

Bootstrap construction

  • Each bootstrap path costs as many simulator rollouts as the main path, so B directly controls simulation cost and motivates a fixed, small replicate count.Source: §5 · p. 20
  • A fixed number B ≥2 of bootstrap paths supports online DF-TD inference in the fast-decaying regime, with each path sharing the transition and perturbation direction but using an independent multiplier and fresh local queries.The paths run in one pass without storing the sample trajectory.Source: §5 · pp. 20–21

Inference

  • The joint Gaussian limit supports Student-t calibrated intervals centered at the main estimator, and the results also hold at deterministic checkpoints under fixed-radius conditions.For fixed-radius conditions, the covariance in the limits is replaced.Source: §5 · p. 21

Trajectory sampling

  • Along Markov trajectories, conventional multiplier inference can be inconsistent when the score is temporally dependent, because bootstrap paths capture lag-zero rather than long-run covariance.Calibration holds coordinatewise if the corresponding Markov and standard covariance diagonals match; a martingale-difference score is sufficient.Source: §5 · pp. 22–23

6 Correcting the Calibrated Value

A second-stage TD recursion uses calibration residuals to estimate the remaining value discrepancy without additional simulator queries. Its correction is a projected approximation whose error depends on how well the chosen feature span represents that discrepancy.

Correction method

  • A second-stage TD recursion estimates the remaining value discrepancy from residuals already computed during calibration, without additional simulator queries.The corrected value adds the feature-based correction to the calibrated simulator value.Source: §6 · pp. 23–24
  • The correction uses features χ chosen to capture discrepancy left by calibration, and gradient features alone yield no correction because their Bellman-orthogonality moment is zero.Source: §6 · p. 23

Approximation properties

  • The projected correction error is bounded by (1 −γ)^-1 times the L2 projection error of the discrepancy, and exact representation in the feature span recovers the policy value almost everywhere.Source: §6 · p. 24

Inference

  • The calibration and correction estimators have a joint asymptotic limit, and the online bootstrap extends to corrected-value intervals; the Jacobian structure requires no additional stability condition beyond the stated assumptions.For DF-Ind and DF-Fwd, the second stage has no finite-difference bias.Source: §6 · p. 24

7 Discussion

Simulator specification affects both derivative-free rates and multiplier-bootstrap validity: correct specification preserves cancellations for DF-Ind and DF-Fwd, while misspecification generally removes them. The paper identifies several extensions, including inference for misspecified Markov trajectories and richer or adaptive procedures.

Specification

  • Correct specification preserves the estimating-equation cancellation for DF-Ind and DF-Fwd, while misspecification generally changes attainable rates and multiplier-inference validity.Source: §7 · p. 25

Open directions

  • Fixed-B inference for misspecified simulators along Markov trajectories remains an open extension, with block multipliers or alternative random-scaling inference identified as possibilities.Source: §7 · p. 25
  • Richer estimating equations and adaptive allocation of simulator rollouts are proposed as future directions for moving calibration closer to the L2 projection and allocating queries by observed variability.Source: §7 · p. 25
Loading 2610.35311v1…