PaperlayerView exact revision on arXiv

Source-linked AI summary

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu

arXiv:2610.28915v1cs.LGmath.OC

TL;DR

Continuous-time control often must be learned from a fixed batch of discrete-time transitions, while discrete-time Q-learning loses its action-ranking signal as the sampling interval shrinks. This paper develops model-free PhiBE policy iteration with a single critic and reuses the same batch through policy updates. Its analysis gives a local end-to-end guarantee and quantifies a trade-off between discretization and finite-sample error.

  • Problem

    Continuous-time control objectives must often be learned from fixed discrete-time off-policy data, and recovering the continuous-time dynamics from such observations can be ill-posed.

    Source: §1 · pp. 1–2

  • Method

    PhiBE-PI learns a single state-action critic directly from sampled increments and reuses one batch for successive policy updates without explicitly estimating drift or diffusion.

    Source: §1 · p. 3, §6 · p. 21

  • Results

    The linear-function-approximation analysis establishes a local end-to-end guarantee and shows that smaller Δt improves the population surrogate but worsens the finite-sample radius at fixed sample size.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Finer time discretization does not necessarily improve policy performance because reducing discretization error can increase statistical error at a fixed sample size.

    Source: §1 · p. 3, §6 · p. 21

    The guarantee is local and covers linear function approximation with hard-greedy improvement under stated regularity and coverage assumptions; the CartPole softmax implementation is outside its scope.

    Source: §6 · p. 21

Abstract

from arXiv · show

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.

1 Introduction

Continuous-time control is defined by a continuous-time objective, while learning may have only fixed off-policy discrete transitions; standard discrete-time Q-learning loses its action-ranking signal as the sampling interval shrinks. The paper proposes a model-free, one-batch PhiBE policy-iteration method and analyzes its local end-to-end guarantee.

1 Introduction

  • The continuous-time objective must be learned from finite-Δt transitions, while recovering a continuous-time model from discrete observations can be ill-posed and propagate errors into policy improvement.The sampled-data setting uses a fixed off-policy batch.Source: §1 · pp. 1–2
  • Known-model policy iteration preserves continuous-time action ranking and has strong convergence results, but depends on an accurate generator that discrete observations may not identify [14].Model-recovery errors can propagate through policy evaluation and improvement.Source: §1 · p. 2
  • Discrete-time Q-learning fits one-step data but its advantage vanishes as Δt →0, while DAU and martingale q-learning preserve the action signal but jointly learn value and action-dependent critics.These limitations motivate learning the state-action equation directly from sampled increments.Source: §1 · p. 2
  • PhiBE-PI learns a single state-action critic from sampled increments and uses greedy improvement, reusing one off-policy batch without explicitly fitting drift or diffusion.The implementation uses linear function approximation.Source: §1 · p. 3
  • PhiBE-Q supplies a continuous-time-scale state-action ranking, while PhiBE and Optimal-PhiBE contribute finite-step moment surrogates for policy evaluation and value-level control [14] [25] [1].Source: §1 · p. 3
  • The analysis gives a uniform finite-sample evaluation bound for data-dependent greedy policies and a local end-to-end guarantee that separates four error sources and exposes a Δt bias–variance trade-off.Smaller Δt improves the population surrogate but worsens the finite-sample radius at fixed sample size.Source: §1 · p. 3

2 Problem setup and the finite-observation gap

The paper studies continuous-time control from fixed-step off-policy transitions, where policy improvement needs an action-ranking signal that remains meaningful as the sampling interval shrinks. It motivates a single-critic PhiBE-Q construction as a way to connect that sampled-data setting to continuous-time policy iteration.

2.1 Continuous-time control with discrete-time off-policy data

  • The target is discounted continuous-time control of an unknown diffusion, but the learner observes only fixed-step, off-policy one-step transitions under piecewise-constant actions.The transition kernel is induced by holding each action constant over an interval of length ∆t.Source: §2.1 · pp. 3–4

2.2 Why a value function alone is not enough: policy iteration needs action ranking

  • A value function alone cannot improve a policy: greedy updates require an action-ranking score, which cannot be evaluated directly when the dynamics are unknown.Known-dynamics policy iteration uses an instantaneous advantage, or HJB residual, to rank actions at each state.Source: §2.2 · p. 4

2.3 Two routes: finite-step equations and continuous-time scores

  • Discrete-time Q-learning's action advantage is O(∆t) and becomes ill-conditioned as ∆t →0, while continuous-time alternatives lack a standalone one-step critic equation for policy iteration.DAU and martingale q-learning jointly learn a value function and action-dependent critic; the direct optimal-q construction also does not proceed through iterative policy improvement, and generic statewise normalization is difficult.Source: §2.3 · pp. 5–6

2.4 Continuous-time Q-functions and their PhiBE approximation

  • PhiBE-Q uses one state-action critic whose policy average recovers the value function and whose action ranking is independent of ∆t.The PhiBE construction replaces unknown generator coefficients with conditional moments of observed one-step increments.Source: §2.4 · p. 6

3 PhiBE-PI

PhiBE-PI repeatedly evaluates a policy with a single state-action critic and improves it using a fixed off-policy batch. The framework separates the policy-iteration template from the evaluation approximation and includes a linear Galerkin, hard-greedy instantiation.

3.1 An Abstract Policy-Iteration Framework

  • The abstract framework separates policy-iteration structure from the choice of function approximator and evaluation procedure.It supports repeated policy evaluation and improvement using one off-policy batch.Source: §3 · p. 6, §3.1 · p. 7
  • Evaluation uses sampled residuals whose conditional expectation, for fixed π and Q∆t, equals the residual of the PhiBE-Q equation.This allows logged increments to be used without separately estimating the moment coefficients.Source: §3.1 · p. 7
  • The framework permits greedy, softmax, or conservative policy updates while reusing the same batch throughout iteration.The conservative update mixes the current policy with a candidate policy using step size η.Source: §3.1 · p. 7

3.2 Linear Features and Hard-Greedy Improvement

  • PhiBE-PI combines linear residual projection for policy evaluation with hard-greedy policy improvement.This is the concrete instantiation of the broader framework.Source: §3.2 · p. 8
  • The linear Galerkin evaluator represents the critic with features Φ and its policy average with the policy-averaged features Φπ.The fitted residual is empirically orthogonal to the features, with the sample PhiBE-Q residual replacing the standard Bellman residual.Source: §3.2 · p. 8
  • Hard-greedy policies are indexed by normalized coefficient directions because positive rescaling preserves the maximizing actions.For each direction, a measurable selector chooses an action maximizing the feature score.Source: §3.2 · p. 8

4 Local End-to-End Error Decomposition and Main Guarantee

The section decomposes the returned policy’s error into continuous-time-to-surrogate transfer, policy-evaluation, and policy-iteration components, then combines them into a local end-to-end guarantee under A1–A3 and the one-batch sampling model. The analysis accounts for policies generated using the same batch and characterizes how approximation, evaluation, policy iteration, sampling interval, data, feature quality, action coverage, and iteration count shape the bound.

4.1 Master decomposition and stochastic transfer

  • The master decomposition separates optimal-value and returned-policy transfer errors from the returned policy’s suboptimality in the surrogate problem.The evaluation density may differ from the batch’s state distribution, and returns remain full-space discounted returns rather than stopping at the evaluation-region boundary.Source: §4.1 · p. 9

4.2.1 Population Galerkin approximation

  • The uniform Galerkin error is controlled by the best approximation of each policy’s surrogate evaluator in the feature space, uniformly over the feature-induced policy class.The error vanishes when all such evaluators lie in the feature space; enriching features can reduce it when uniform best-approximation error tends to zero and amplification constants stay bounded.Source: §4.2.1 · p. 13

4.2.2 Uniform one-batch finite-sample error

  • Uniform one-batch finite-sample control evaluates every policy selected from the batch on one shared high-probability event, without a union bound over iterations.For fixed discount, reducing the sampling interval requires a larger batch to maintain the same finite-sample accuracy.Source: §4.2.2 · pp. 13–14

4.3 Policy-iteration error

  • The policy-iteration bound propagates Q-evaluation error through greedy updates, with coverage and action-wise error variation governing how easily errors can mislead action selection.More accurate evaluation reduces the bound; its described remainder also includes a decreasing initial-policy term, an O(∆t) surrogate-covariance perturbation, and a localization term that need not vanish as the evaluation region grows.Source: §4.3 · pp. 14–15

4.4 Local end-to-end main theorem

  • The end-to-end theorem combines transfer, policy-evaluation, and policy-iteration bounds to control the returned policy in the original problem.The preceding decomposition identifies the two transfer terms as replacing the continuous-time problem with its surrogate and the remaining term as surrogate suboptimality.Source: §4 · p. 8, §4.1 · p. 9, §4.4 · p. 15
  • Under A1–A3 and the one-batch sampling model, Theorem 4.14 gives a local end-to-end control guarantee for Algorithm 2’s output policy when β and Δt meet stated conditions and n meets Theorem 4.8’s sample-size condition.The uniform evaluation event has probability at least 1−δ; the bound’s coverage and envelope quantities may depend on R and (n, δ; β, Δt), but not on the realized batch or iteration index.Source: §4.4 · p. 15
  • Under polynomial action coverage and Δt ≤1, Remark 4.15 specializes the end-to-end guarantee to a leading-order bound.The supplied passage does not include the bound’s expression; its constants depend on fixed coverage, envelope, and feature parameters, and its explicit factor R comes from policy-iteration propagation.Source: §4.4 · p. 15
  • More expressive features can reduce ϵgal but may require more data and better Gram conditioning; decreasing Δt lowers deterministic error while increasing sampling error at fixed n.A larger batch can compensate for the sampling effect, while increasing β reduces the displayed sampling contribution but changes value functions and thus ϵgal.Source: §4.4 · pp. 15–16
  • Larger state dimension d or weaker action coverage (larger qac) slows bound decay; for the action cube with conditional behavior density bounded below, qac = da.More policy-improvement steps reduce the initial-policy contribution, while remaining errors depend on features, data, and sampling interval.Source: §4.4 · p. 16

5 Experiments

Across three control settings, the experiments examine policy quality, error sources, and deployment at frequencies different from training. PhiBE-PI shows bias–variance trade-offs in LQR and smooth control, and benefits from finer CartPole deployment updates while BE-PI performs better when training and deployment intervals match.

5.1 Smooth control with an analytic optimal value

  • The smooth-control experiment tests hard-greedy PhiBE-PI against an analytic optimum while varying batch size, sampling interval, and policy-iteration updates.The setup satisfies A1–A3, but does not certify the theorem’s remaining quantitative sufficient conditions.Source: §5.1 · p. 17
  • At ∆t = 0.02, smooth-control value gaps decrease with more data, while the smallest tested gap for n = 10000 occurs at ∆t = 0.3.At n = 2000 and ∆t = 10^-4, the mean gap is 0.817 and 34 trials retain a direction change above 10^-7 after 20 updates.Source: §5.1 · p. 17, Figure 2 · p. 18

5.2 End-to-end LQR scaling in β and ∆t

  • The LQR comparison uses the same off-policy batch and features for PhiBE-PI and BE-PI, and distinguishes discretization error from statistical error.This scalar LQR setting uses a drift-only evaluator.Source: §5.2 · pp. 17–18
  • As β varies at ∆t = 1, PhiBE-PI has lower total gain error than BE-PI throughout the tested range, with discretization dominating for both methods.The experiment uses 200 trials with 2 × 10^6 samples per trial.Source: §5.2 · p. 18, Figure 3 · p. 19
  • At β = 0.5, PhiBE-PI’s finite-sample error grows at small ∆t and discretization error at large ∆t, with minimum total error near ∆t = 0.065.BE-PI has a larger discretization floor and higher total error at every tested step size.Source: §5.2 · p. 18, Figure 3 · p. 19

5.3 CartPole with different training and deployment frequencies

  • At ρ = 1, BE-PI has lower mean absolute angle and longer mean survival than PhiBE-PI, despite PhiBE-PI’s slightly higher pre-failure reward.The respective angles are 0.0486 and 0.0846 radians, and survival times are 5.37 and 1.23 s.Source: §5.3 · p. 20, Figure 4 · p. 20
  • At every finer tested CartPole deployment frequency, PhiBE-PI outperforms BE-PI on reward, angular error, survival time, and failure rate.At ρ = 100, reward, absolute angle, and survival are 0.749, 0.0171 radians, and 8.87 s for PhiBE-PI, versus 0.543, 0.0686 radians, and 3.19 s for BE-PI.Source: §5.3 · p. 20, Figure 4 · p. 20
  • CartPole uses fixed coarse training data and varies deployment action-update frequency; the PhiBE-PI rollouts’ longest mean survival occurs at ρ = 10.Mean survival is 9.44 s at ρ = 10, with a 52.1% failure rate, versus 61.9% at ρ = 100.Source: §5.3 · pp. 19–21

6 Conclusion

The conclusion presents PhiBE-PI as a model-free, single-critic method that reuses a fixed off-policy batch and has a local guarantee for its linear implementation. The appendix supports the analysis with policy-average derivative identities, regularity and stability results, and stated limits on the guarantee.

6 Conclusion

  • PhiBE-PI learns a single state-action critic from sampled increments, reuses a fixed off-policy batch, and has a local end-to-end guarantee for its linear-function-approximation implementation.The experiments show sampling-interval bias–variance trade-offs; CartPole benefits from finer deployment updates, while BE-PI does better when training and deployment intervals coincide.Source: §6 · p. 21
  • The guarantee is local and limited to linear function approximation with hard-greedy improvement under the stated regularity and coverage assumptions.The softmax CartPole implementation lies beyond this guarantee; soft updates and nonlinear approximation are identified as future analysis directions.Source: §6 · p. 21

A Likelihood-Ratio Derivatives of Policy Averages

  • For policy averages, the appendix gives gradient identities and notes that target-policy action samples yield unbiased estimators without extra simulator transitions.For a deterministic policy, the derivatives are those of Q(s, π(s)) wherever this composition is differentiable.Source: A Likelihood-Ratio Derivatives of… · p. 24

B.1 Regularity of the moment coefficients

  • Under A1, the moment coefficients have uniform regularity and consistency bounds for sufficiently small ∆t.The appendix states that the covariance surrogate may depend on action, with an O(∆t) deviation from Σ0.Source: §B.1 · pp. 24–26

B.2 Regularity of composed features and closed-loop coefficients

  • Under A1 and A2, composed features and closed-loop coefficients have uniform regularity bounds over greedy directions.The appendix also gives an action-Lipschitz bound for features on the specified region.Source: §B.2 · p. 26

B.3 Population Galerkin stability

  • Under A1–A3, the appendix establishes uniform population Galerkin stability, with a stated lower bound for sufficiently large β.It does not require symmetry of Au or stationarity of the sampling law.Source: §B.3 · p. 27

B.4 Well-posedness of policy evaluation

The appendix establishes well-posed policy evaluation and fixed-policy transfer, then develops a finite-sample evaluation bound that extends to a greedy policy class through a finite net.

B.4 Well-posedness of policy evaluation

  • Under A1 and A2(a)–(b), the policy-evaluation equations have unique bounded classical solutions identified with discounted returns for sufficiently small ∆t.The proposition applies to every greedy-direction policy and β > 0.Source: §B.4 · p. 27

B.5 Uniform gradients of the policy values

  • For sufficiently large β, the appendix establishes uniform gradient bounds for policy values over the policy family and sampling interval.The constants are independent of the greedy direction, ∆t, and β.Source: §B.5 · p. 28

B.6 Fixed-policy transfer

  • The fixed-policy transfer result bounds the difference between true and surrogate policy values under the stated small-step and discount conditions.Its constants are independent of β, the policy direction, ∆t, n, and N.Source: §B.6 · p. 28

C Sample Complexity for Local One-Batch Policy Evaluation

  • The sample-complexity analysis proves a fixed-policy finite-sample bound and extends it to the greedy class using a finite net.Its constants depend on fixed-region and local-feature bounds, while dependence on sample size, confidence, ∆t, β, and dimensions is displayed explicitly.Source: C Sample Complexity for Local… · pp. 28–29

C.1 One-sample summands for the Galerkin objects

The appendix expresses the empirical Galerkin objects through one-sample summands evaluated at sampled starting states.

C.1 One-sample summands for the Galerkin objects

  • The empirical Galerkin system’s one-sample summands define the sample quantities used to form its matrix and vector estimates.The passage identifies the empirical objects as averages of summands and their population counterparts as expectations.Source: §C.1 · p. 29
  • Features and feature derivatives in the summands are evaluated at sampled starting state S, not next state S′.The stated feature bounds apply at starting states in K and actions in A.Source: §C.1 · p. 29
  • Direction-Lipschitz bounds make the summands continuous in u almost surely, ensuring the relevant suprema are measurable.The analysis abbreviates logarithmic factors using ℓ_n,δ and Λ_n,δ.Source: §C.1 · p. 29

C.2 Fixed-policy error and sample complexity

The appendix bounds fixed-policy evaluation error through reward-vector and Galerkin-matrix errors, then extends control uniformly over greedy policies. It also outlines the population approximation argument using residual projection and stability.

C.2 Fixed-policy error and sample complexity

  • Under A1–A3 and the stated stability and sampling-step conditions, a fixed-policy sample-radius condition ensures the empirical Galerkin matrix is invertible with high probability.The prediction error is controlled by reward-vector and Galerkin-matrix errors.Source: §C.2 · p. 29

C.3 Uniform extension over the greedy class

  • One high-probability event controls the stated bounds simultaneously for every policy in the greedy class.The proof extends from a finite sphere net to the whole sphere using Lipschitz bounds.Source: §C.3 · p. 31
  • Theorem 4.8 gives a refined finite-sample error budget for the uniform extension over greedy policies.The supplied passage identifies the budget but does not provide its terms.Source: §C.3 · p. 31

D Population Galerkin Approximation

  • The population approximation argument bounds the projected residual by integration by parts, then applies the population stability estimate.The argument uses the state–action equation and sampling measure μK.Source: D Population Galerkin Approximation · p. 32

D.1 Fixed-policy Galerkin approximation

The paper bounds fixed-policy Galerkin approximation error and extends it uniformly over the greedy class. It then sets up policy-iteration propagation through value regularity and contraction.

D.1 Fixed-policy Galerkin approximation

  • Above the stability threshold, the fixed-policy approximation amplification factor is bounded uniformly in the time step by C0 + C1/β, although the best-approximation error may still depend on β and ∆t.The fixed-policy bound is obtained by applying the estimate to the policy’s Galerkin approximation.Source: §D.1 · p. 33

D.2 Uniform Galerkin approximation

  • The uniform Galerkin approximation bound holds over the greedy policy class with a constant independent of n, N, β, and ∆t.The Gram matrix converts coefficient error into prediction error, while αΦ controls amplification from solving the Galerkin system.Source: §D.2 · p. 34

E Policy-iteration propagation

  • The policy-iteration analysis first establishes value regularity and contraction under a local residual budget, then derives that budget from Q-evaluation error.This separates propagation of a residual into value error from the step that obtains the residual.Source: E Policy-iteration propagation · p. 34

E.1 PI setup and regularity

  • The algorithmic policy path remains in ΠΦ through its initialization and the rule that retains the previous policy when the fitted coefficient is zero.Source: §E.1 · p. 34

E.1.1 Control equations and coefficient conditions

  • The surrogate HJB equation’s bounded classical solution is identified with the optimal value, supplying derivative bounds used in the analysis.Source: §E.1.1 · p. 34

E.1.2 Value derivative bounds

The paper establishes discount-uniform derivative bounds for fixed-policy values and regularity of the surrogate optimal value. Its greedy-improvement analysis compares action scores without computing a common Hessian term.

E.1.2 Value derivative bounds

  • For β ≥ βgrad, the local gradient bound sharpens to Cgrad/β, while the local Hessian bound is C and the gradient bound is otherwise Cβ^-1/2.The fixed-policy derivative estimates are independent of β, u, and ∆t for β ≥ 1 and the stated small-step range.Source: §E.1.2 · p. 35
  • The surrogate optimal value is a bounded classical solution of its HJB equation for β ≥ 1 and the stated small-step range.Source: §E.1.2 · p. 35

E.1.3 Greedy improvement and score residual

  • The algorithm does not compute the common Hessian term because it is independent of action and is used only to compare scores in the proof.Source: §E.1.3 · p. 36

E.2 PI contraction from a local residual budget

The policy-iteration analysis relates value error to a local score-residual budget and obtains that budget from Q-evaluation error. The argument is localized to a ball and uses action coverage and interpolation.

E.2 PI contraction from a local residual budget

  • The value error on BR is bounded in terms of the score residual on B2R.The residual bound is the condition used for the local contraction argument.Source: §E.2 · p. 37
  • The abstract policy-iteration theorem establishes error contraction under a local residual budget.The proof derives a weighted error estimate and applies an ABP estimate.Source: §E.2 · p. 37

E.3 From Q-evaluation error to a local residual budget

  • The local score residual is bounded from Q-evaluation error under μK using uniform score-error bounds, action coverage, and interpolation on B2R.Source: §E.3 · p. 39

E.3.1 Score-error envelopes

The appendix bounds score errors along the policy-iteration path and transfers evaluation error to action-max error using action coverage. It then converts evaluation error into a residual budget that holds across iterations.

E.3.1 Score-error envelopes

  • Deterministic envelopes bound score error for every policy on the algorithmic path, with constants independent of the realized batch and iteration index.The constants may depend on β, n, δ, and ∆t.Source: §E.3.1 · p. 40

E.3.2 Action-max norm transfer

  • Action coverage and action-Lipschitz continuity transfer L2 error to action-max error, with the transfer function tending to zero as the L2 error vanishes.For finite action sets, a positive lower bound on action probabilities permits the conversion without an action-Lipschitz bound.Source: §E.3.2 · pp. 40–41

E.3.3 Residual budget along the PI path

  • The local residual budget along the policy-iteration path is controlled by the evaluation error and the action-coverage and score-error envelope quantities.The map Γ is nondecreasing and tends to zero at zero when its parameters are fixed.Source: It satisfies · p. 42, §E.3.3 · p. 42

It satisfies

  • The same residual budget applies at every iteration in the policy-iteration path.The proof applies the bound for each 1 ≤ k ≤ N.Source: It satisfies · p. 42

E.4 Assembly of the main-text PI propagation result

The proof combines the local residual budget with the policy-iteration contraction estimate to obtain a propagation bound. Its propagation constant is independent of the radius R, while the residual-budget term may depend on R.

E.4 Assembly of the main-text PI propagation result

  • An L2 evaluation error at most ϵQ implies a local score residual bounded by Γ(ϵQ) at every iteration.The proof applies the residual bound from Lemma E.10 across the policy-iteration path.Source: §E.4 · p. 43
  • The propagation constant Cprop is independent of R, while Γ(ϵQ) retains the sample-size, coverage, and envelope dependence and may depend on R.The proof uses a probability density supported in BR.Source: §E.4 · p. 43
  • When the working diffusion is action-independent, the diffusion-perturbation term vanishes in the contraction proof.The same argument then gives the propagation result.Source: §E.4 · p. 43
Loading 2610.28915v1…