Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach

Introduction

• The framework addresses a gap in policy evaluation research, where most methods target discrete finite-state MDPs and continuous-setting studies largely concern deterministic systems.The paper focuses on general Markov diffusion processes and carries out its theoretical analysis in continuous time and space.Source: §1 · p. 2
• The analysis remains continuous until the algorithmic stage, and the limit of a convergent time-discretized algorithm approaches the continuous-problem solution as the mesh shrinks, with a convergence rate provided.Time is discretized only for implementation; state and action are not discretized at the outset.Source: §1 · pp. 2–3
• The continuous analogue of mean-square TD-error minimization targets the martingale’s quadratic variation, making it inconsistent with the policy-evaluation objective in stochastic environments.The paper argues that this explains why the method is unsuitable for policy evaluation when the environment is stochastic.Source: §1 · p. 4
• Minimizing the proposed martingale loss is equivalent to minimizing mean-square error between the approximate and true value functions.The loss is based on the total mean-square error between the martingale process and its terminal value, and is implementable on samples.Source: §1 · p. 4
• Martingale orthogonality conditions provide a unified interpretation of TD errors and algorithms including TD(λ), LSTD, and GTD.The framework builds on the value function along the state process combined with accumulated running payoff being a martingale.Source: §1 · pp. 3–4

1 Introduction

• TD(λ) applies stochastic approximation to the moment conditions, LSTD solves them explicitly when linear, and GTD solves quadratic forms of them.The conditions arise by introducing a finite number of suitable test functions.Source: §1 · p. 4
• Test-function choices determine how the true value function is approximated; different TD(λ) choices may therefore converge to different limits.Source: §1 · p. 4
• The framework points to data-driven PDE solving through the connection among PDEs, BSDEs, and policy evaluation for Markov diffusion processes.Source: §1 · pp. 4–5

2 Problem Formulation and Preliminaries

• Policy evaluation estimates J(t, x) from multiple trajectories of a fixed-policy diffusion without knowing its drift, diffusion, reward, or terminal-reward functions.The setup covers on-policy episodic learning with offline or online updates; the authors caution that one trajectory without resets is generally insufficient for finite-horizon learning.Source: §2 · pp. 5–6
• Regularity assumptions require continuous coefficients and rewards, Lipschitz and linear-growth drift and diffusion, and polynomial-growth rewards to ensure well-posedness and finite J.The first three conditions ensure a unique strong solution with uniformly bounded moments.Source: §2 · pp. 6–7
• The value function is characterized by a Feynman–Kac PDE, but unknown model parameters and high state dimension motivate the trajectory-based approach.The analysis assumes the PDE has a classical solution with polynomial growth.Source: §2 · pp. 7–8
• A forward–backward stochastic differential equation represents the solution through Y_s = J(s, X_s) and Z_s as the spatial derivative of J.This relationship follows by applying Itô’s formula to J.Source: §2 · p. 8
• The value function is characterized by a martingale: the process formed from J and accumulated rewards is a square-integrable martingale, and the converse identifies J.The converse holds for a continuous candidate function satisfying the terminal condition.Source: §2 · p. 8
• The admissible κ processes are specified on [0,T] as F_t-progressively measurable, with an expectation condition whose expression is incomplete in the excerpt.A separate fragment ends with F([0,T];Y) =, but does not provide enough context to state its meaning.Source: §2 · p. 9

3.1 Doya’s TD algorithm for deterministic dynamics

• Doya (2000) extends TD to continuous time and space for deterministic dynamics, defining an instantaneous TD error rate and minimizing its mean-square error with gradient descent.The resulting residual-gradient update can use sampled payoffs and states without knowing model parameters.Source: §3.1 · pp. 9–10

3.2 Mean-square TD error for stochastic dynamics

• In stochastic dynamics, Bellman consistency still holds and yields a Bellman error via Itô’s formula, but the Brownian-motion example shows discretized MSTDE can fail to learn the true value function.For the true value function the Bellman error is zero; nevertheless, the conditional expectation needed to compute it requires model knowledge.Source: §3.2 · pp. 11–12
• For diffusion dynamics, the pathwise TD error rate is undefined because sample paths are intrinsically nondifferentiable, while Bellman error requires dynamics knowledge unavailable from samples alone.The conditional expectation in Bellman error requires the dynamics or generator; the authors note that discretized MSTDE can still be computed, but it does not necessarily yield the correct solution.Source: §3.2 · pp. 11–12

3.3 Theoretical characterization of mean-square TD error

• In Example 1, SGD minimizing MSTDE∆t leads to θ*_MSTDE = -3/2, rather than the desired θtrue = 0; minimizing the martingale loss generates the correct solution.Source: Figure 1 · p. 13, §3.3 · p. 13
• Minimizing MSTDE is inappropriate for stochastic policy evaluation because it is equivalent to minimizing the martingale’s expected quadratic variation.That quadratic variation should not be minimized as the objective for estimating the value function.Source: §3.3 · p. 13
• In Example 2, MSTDE selects θ*=(-2, 0, 0), yielding J(t, x)=(2t−1)x² instead of the true J(t, x)=x².The example shows that the MSTDE minimizer can produce a learned value function that deviates from the true one.Source: §3.3 · p. 15
• As the time discretization shrinks, minimizing MSTDE converges to minimizing the martingale’s quadratic variation, whose leading term favors the smoothest value-function process.This differs from policy evaluation: the true value function is tied to the mean-square Bellman error, while MSTDE’s minimizer is primarily determined by quadratic variation.Source: §3.3 · pp. 15, 17

3.4 Online mean-square TD error algorithms

• With the uniform distribution on r0,T, the objective is the same optimization problem as offline learning on the whole trajectory; theoretically, it leads to the same undesired solution determined by Theorem 1.Source: §3.4 · p. 17

4 Martingale Perspective and Approach

• The section rejects MSTDE as a policy-evaluation objective and develops criteria based on the martingality of process M, connecting them to classical MDP TD algorithms.In Examples 1 and 2, CTD(0) and CTD(1) lead to the desired solution.Source: §4 · pp. 17–19

4.1 Offline learning: Martingale loss function

• The martingale-loss algorithm uses full trajectories to compare predicted values with realized reward-to-go and is the continuous-time counterpart of gradient Monte Carlo.It is intended for offline learning, where updates follow observation of a full trajectory.Source: §4.1 · pp. 19–20
• Minimizing martingale loss is equivalent to minimizing mean-square value error; convergent discrete-time minimizers approach the continuous-time counterpart as the mesh shrinks, with a convergence rate.The true value function is not observable from data, whereas the martingale loss is implementable.Source: §4.1 · pp. 20–21

4.2 Online and offline learning: TD based on martingale orthogonality conditions

• ML converged to ≈ -2.12568, CTD(0) to ≈ -1.83923, and CTD(1) to ≈ -2.12568 in Example 4; the two CTD methods differ because they use different test functions.ML and CTD(1) coinciding is reported as a coincidence.Source: Figure 6 · p. 23, §4.2 · p. 23
• Because martingale loss requires whole trajectories, the section proposes a different approach based on martingale orthogonality conditions for online learning.The passages also identify the mean-square TD error as an incorrect value-function learning objective.Source: §4.2 · pp. 22–23
• Solving finite martingale orthogonality conditions with suitable test functions yields continuous-time counterparts of well-known TD algorithms for online and offline learning.The conditions arise from the martingale property, and the choice of test functions and solution method determines the resulting algorithm.Source: §4.2 · pp. 23–24
• Applying stochastic approximation to the conditions recovers TD(0) and TD(λ), with parameter updates possible after an episode or at each time step.TD(λ) uses a parameter λ to weight historical predictions; λ = 0 gives TD(0).Source: §4.2 · p. 24
• With linear function approximation, the conditions form a linear system whose sample-based solution recovers LSTD(0), assuming the required matrix inverse exists.Expectations can be estimated from multiple trajectories or, under an ergodicity condition, by long-time averages from one trajectory.Source: §4.2 · p. 25
• Different choices in the GMM solution yield gradient TD methods, including GTD(0), GTD2, and TDC, which minimize a quadratic form of the conditions.GTD2 and TDC use two time scales to update an auxiliary quantity and the value-function parameters.Source: §4.2 · p. 26
• As the mesh shrinks, any convergent subsequence of discretized TD-type parameters converges to a solution of the continuous-time moment conditions; an additional assumption yields a convergence rate.The result is stated under Assumptions 1, 2, 3, and 5, with the rate requiring Assumption 4 as well.Source: §4.2 · p. 27
• As the mesh shrinks, any convergent subsequence of discretized GMM minimizers converges to the continuous GMM minimizer; an additional assumption yields a convergence rate.The result assumes Assumptions 1, 2, 3, and 5, and additionally Assumption 4 for the rate.Source: §4.2 · p. 28
• Convergent CTD(0) and CTD(λ) target solutions to their chosen moment conditions; Example 1–2 algorithms converge, but absent solutions can prevent convergence and test functions affect the result.The authors report that solutions may differ with test-function choice even when a unique solution exists and the algorithm converges.Source: §4.2 · p. 28
• In Example 3, the nonzero test function gives the sole solution θ* = 0, whose value function is 0 rather than the true value function x.Source: §4.2 · p. 29
• In Example 4, CTD(0) targets θ ≈ -1.83923, whereas CTD(1) targets θ ≈ -2.12568, the latter matching the martingale-loss solution.Source: §4.2 · pp. 29–30
• CGTD methods minimize quadratic forms of the moment conditions even when a solution does not exist or is not unique, and are described as more robust.In Example 5, CGTD yields a solution when the CTD(0) moment condition has none.Source: §4.2 · pp. 30–31
• The continuous-time CGTD result identifies its minimizer through projection of Bellman error onto the span of the test functions, generalizing the discrete-time MSPBE result.Theorem 5 presents the continuous-time version for a more general choice of test functions.Source: §4.2 · p. 30
• For Example 5, ML converges to θ ≈ -0.875301, CTD(0) diverges, and CGTD reaches the MSPBE minimizer θ = -1.The moment condition has no solution for CTD(0), while CGTD minimizes the mean-square projected Bellman error.Source: Figure 7 · p. 31, §4.2 · p. 31

5.1 Extension to discounted case

• For discounted evaluation, full-trajectory martingale loss is impractical at an infinite horizon; truncation permits offline learning, while single-trajectory online learning favors TD.The section also adapts the martingale loss and orthogonality conditions to discounting.Source: §5.1 · pp. 31, 33

5.2 Extension to non-Markovian setting

• For fractional Brownian motion with H=0.75, both ML and CTD(0) converge, though less sharply than in Markov examples, and CTD(0) performs better.The comparison uses mean-square value errors evaluated on 1,000 independent trajectories, with standard deviations shown as shaded areas.Source: §5.2 · p. 34, Figure 8 · p. 34

5.3 Option-like payoff

• For the first two option-pricing error criteria, offline ML and online CTD(0) converge to zero quickly, while deep BSDE takes longer; PE-method errors remain for the final criterion.The policy-evaluation methods are presented as an alternative way to solve the Black–Scholes PDE; the deep BSDE benchmark requires perfect knowledge of model parameters, including the driving-noise samples.Source: §5.3 · p. 36
• The option-like payoff comparison shows the martingale-loss method stays nearly flat from the start, whereas the BSDE method oscillates dramatically before converging to zero.The passage describes the ML curve as close to zero and almost flat from the start.Source: §5.3 · p. 36
• The martingale-loss objective approximates the value function itself in mean square without targeting its derivatives, making policy evaluation flexible for tasks such as solving a PDE.The passages caution that policy-evaluation methods may not accurately approximate derivative values.Source: §5.3 · p. 36
• ML achieves the smallest value-function error, with online CTD(0) only slightly behind, while deep BSDE can ultimately learn the derivative.The methods illustrate a distinction between estimating the value function itself and learning its derivative.Source: §5.3 · p. 36
• The method cannot directly evaluate an option price because its relation holds in the risk-neutral world, where data cannot be observed; pricing from physical-probability data remains open.The stated limitation concerns direct use for option pricing, not the comparison of methods in the example.Source: §5.3 · p. 36

5.4 Infinite time horizon linear-quadratic problem

• CLSTD(0) converges fastest because its linear parameterization lets it explicitly solve the moment conditions; CGTD2 converges faster than CTD(0) as a gradient-based algorithm.Figure 10 reports that all algorithms converge to the correct value function, with CTD(0) the slowest.Source: §5.4 · p. 37, Figure 10 · p. 38

6.1 Choice of test functions

• Test-function choice affects both convergence behavior and the sense in which the learned value function approximates the truth.The conventional choice can be unbiased yet inconsistent when parameter variance grows, whereas a tailored choice bounds volatility at the cost of bias.Source: §6.1 · pp. 38–39
• For Brownian motion, the conventional test function produces an asymptotically unbiased estimate but a variance that grows without bound, making the estimator inconsistent.The example attributes the variance growth to the non-stationarity of Brownian motion.Source: §6.1 · p. 39
• Choosing ξt = 1 bounds the volatility term and reduces variance, but introduces bias in the mean through correlation in the drift.The simulation reports more volatile parameter estimates over time with the conventional choice and lower variance but a slightly displaced average with the tailored choice.Source: §6.1 · pp. 39–40

6.2 Function approximation: global vs sectional

• Sectional approximation requires parameters for each time point and creates an online update problem because future states are unavailable.The global parameterization updates the bivariate value function as a whole, including future predictions and revised past evaluations.Source: §6.2 · p. 41
• Global and sectional approximation perform similarly with offline ML, while global approximation performs significantly better with online CTD(0).In this example, the computational-cost difference is insignificant because the sectional approach has few unknown parameters.Source: §6.2 · p. 41, Figure 12 · p. 42

7 Conclusions

• Policy evaluation is equivalent to enforcing the martingality of a stochastic process, providing a common perspective on popular PE algorithms.The framework supports modifying these algorithms to solve PE in continuous settings.Source: §7 · p. 42
• The martingale perspective may help study off-policy evaluation, state-action value estimation, and convergence analysis in continuous settings, where these problems remain open.The authors also suggest it may inspire questions not posed by traditional RL research.Source: §7 · p. 42
• The paper formulates PE for Itô processes, while the martingale view is generalizable to jump diffusions, non-Markov processes, and semimartingales.The authors choose Itô processes for their convenient, well-studied properties and ability to model many real-life dynamics.Source: §7 · p. 42

Appendix A: A Summary of Popular PE Methods

• Table 1 organizes popular PE methods by method and representative algorithm, online or offline applicability, objective, and convergent point.The table summarizes the interpretations developed in the paper.Source: Table 1 · p. 43, Appendix A · p. 43

Appendix B: Stochastic Control Formulation of RL

• In the RL setting, model functions are typically unknown, and PE seeks to estimate the value function from sample trajectories without knowing those functions.The general RL task is to optimize the value function over admissible policies; PE fixes a policy and estimates its value.Source: Appendix B · p. 44

Appendix C: Martingale in Discrete-time Markov Reward Pro-

• The paper shows that a martingale property also holds in the classical discrete-time reinforcement-learning MDP formulation.Source: cesses · p. 44
• The appendix focuses on finite-horizon episodic tasks and notes that infinite-horizon continuing tasks can be studied similarly.Source: cesses · p. 44
• The martingale condition for the discrete-time process is equivalent to the value function satisfying the Bellman equation.The Bellman equation can therefore characterize policy evaluation in this setting.Source: cesses · p. 45
• The martingale loss function and martingale orthogonality condition recover conventional policy-evaluation algorithms for discrete-time MRPs.Source: cesses · p. 45

Proof of Proposition 1

• Using the Markov property, the proof establishes that the constructed process is a martingale.Source: Proof of Proposition 1 · p. 45
• Conversely, if the constructed process is a martingale, its conditional expectation yields the corresponding value-function relation.Source: Proof of Proposition 1 · p. 46
• Setting the relation’s time argument to the initial time identifies the candidate function with the value function.Source: Proof of Proposition 1 · p. 46

Proof of Theorem 1

• An error bound for the Bellman-error expression separates the discretization error from the reward-approximation error.The bound uses the triangle inequality, with the reward-approximation term tending to zero as the mesh shrinks.Source: Proof of Theorem 1 · p. 49
• The stepwise reward approximation converges in L2 to the reward process as the mesh size tends to zero, independently of θ.Source: Proof of Theorem 1 · p. 49
• Under Assumption 4, the proof states that the leading term in the discretization bound is O(∆t).Source: Proof of Theorem 1 · p. 49
• For the martingale-loss comparison, the proof notes that one term in the objective difference does not depend on θ.Source: Proof of Theorem 2 · p. 50

Proof of Proposition 2

• The increment is decomposed into a drift term A_t and a diffusion term B_t dW_t, with A_t identified as the generator applied to J^θ plus the reward.Source: Proof of Proposition 2 · p. 51
• The proof tests the drift over an interval using its sign; the stochastic-integral term has zero expectation under the stated integrability condition.Source: Proof of Proposition 2 · p. 51
• The proof concludes that A_t = 0 almost surely, which makes M^θ a martingale.Source: Proof of Proposition 2 · p. 51
• The discretization error bound tends to zero as ∆t tends to zero, and the proof uses its leading term to obtain a convergence rate.Source: Proof of Theorem 3 · p. 52

• The proof uses an inner product in an L2 space over [0,T].Source: Proof of Theorem 5 · p. 54
• Projection orthogonality makes the residual κ−Πθκ orthogonal to ξθ,pj for any κ in L2.The stated condition applies for j=1,…,L1.Source: Proof of Theorem 5 · p. 54
• The ij-th entry of the L1×L1 matrix Aθ is specified using an expression involving ξθ,pi.Source: Proof of Theorem 5 · p. 54