Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms

1 Introduction

• Continuous-time reinforcement learning remains underdeveloped: prior work is largely discrete-time, while continuous-time studies often assume deterministic systems or specific models.Discrete-time solutions can be highly sensitive to time discretization (Tallec et al. (2019); Yildiz et al. (2021)).Source: §1 · pp. 2, 4
• Policy gradient optimizes value over a parameterized policy family, avoiding action maximization and enabling efficient learning when useful policy classes are known.The paper contrasts this approach with Hamiltonian maximization, which requires environmental knowledge and can be computationally demanding in high dimensions.Source: §1 · p. 3
• The paper develops model-free, data-driven actor-critic algorithms for general continuous-time and space reinforcement learning, covering episodic and continuing as well as online and offline tasks.Its analysis is conducted in continuous time, with time discretized only at the final algorithmic implementation stage.Source: §1 · p. 4
• The analysis extends policy-evaluation theory from deterministic to stochastic policies, including martingale characterization, martingale loss, and orthogonality conditions.This extension is technically non-trivial and supports the paper’s policy-gradient analysis and actor-critic algorithms.Source: §1 · p. 4

Policy-gradient representation

• The policy-gradient representation becomes an auxiliary policy-evaluation problem, with its reward transformed into functions evaluable from samples and the current value function.The transformation uses integration by parts and Itô’s formula to avoid requiring the Hamiltonian’s functional form.Source: §1 · p. 4

Offline and online learning

• The forward-looking representation uses future states and is suited only to offline learning, while online policy updates use martingale orthogonality conditions and stochastic approximation.The online conditions come from the first-order condition of the policy gradient.Source: §1 · p. 4

Actor-critic algorithms

• The paper combines its policy-gradient methods with Jia and Zhou (2022a) policy-evaluation methods to propose actor-critic algorithms for episodic and continuing/ergodic tasks.The paper develops policy-gradient theory, general actor-critic algorithms, and an extension to ergodic tasks.Source: §1 · pp. 4–5

Model-free scope

• Unlike related model-based policy-updating methods, the paper offers general model-free actor-critic algorithms, subject to diffusion-process dynamics, applicable to the cited problem settings.Wang and Zhou (2020)’s Hamiltonian-maximization method requires market knowledge; Dai et al. (2020) and Guo et al. (2022) also rely on policy differentiation and are model-based.Source: §1 · pp. 4–5

Simulations

• The simulations apply the algorithms to Wang and Zhou (2020)’s mean–variance portfolio-selection problem and report that they outperform significantly.The paper demonstrates the algorithms with two concrete examples.Source: §1 · p. 5

Problem formulation

• The controlled state evolves over a finite horizon according to an SDE with drift b, diffusion σ, action a_s, and Brownian environmental noise.The state dynamics and reward functions define the classical stochastic control problem.Source: §2 · pp. 5–6
• The control objective is to maximize expected total reward, combining a running reward with an end-of-period lump-sum reward and discount factor β.The optimal strategy is chosen for each initial time–state pair.Source: §2 · p. 6
• In reinforcement learning, the agent lacks the environment’s functional forms and learns by trying actions and observing states and payoffs.Offline learning repeats action sequences over the same period, whereas online learning updates actions using historical observations.Source: §2 · p. 7
• A stochastic policy samples actions from time–state-dependent distributions independently of Brownian noise, and an entropy regularizer weighted by γ encourages exploration.Wang et al. (2020) use this continuous-time, continuous-space formulation to balance exploration and exploitation.Source: §2 · pp. 7–8

2 Problem Formulation and Preliminaries

• Under Assumptions 1, an admissible policy yields a unique strong solution with the stated growth condition and a finite expected payoff.The growth condition is stated for any μ ≥ 2.Source: §2 · p. 9
• The relaxed control problem is mathematically equivalent to the sample-based problem, serving value-function analysis while the latter uses observable samples.Taking expectation over action randomization reduces the sample-based formulation’s expectation to that of the relaxed formulation.Source: §2 · p. 9
• The paper mainly studies finite-horizon episodic tasks; as T tends to infinity, the formulation covers discounted continuing tasks under suitable conditions, and Section 4 considers an ergodic alternative.The stated example condition for the discounted formulation is that β is large enough.Source: §2 · p. 9

3.1 Policy Evaluation

• For stochastic policies, the value function is characterized by a PDE and equivalently by a martingale condition and martingale orthogonality.This extends the martingale characterization of policy evaluation from deterministic to stochastic policies.Source: §3.1 · p. 10

3.2 Policy Gradient

• Policy-gradient computation becomes a policy-evaluation problem with a different reward, using an expected representation of the gradient.The representation can be evaluated from trajectory samples and an estimated value function from the preceding policy-evaluation step.Source: §3.2 · pp. 13–14
• The gradient representation’s auxiliary reward involves an environment-dependent operator that is unavailable directly, but Itô’s lemma and martingality enable incremental estimation from samples.The unknown stochastic-integral term does not contribute to the expectation under suitable regularity conditions.Source: §3.2 · p. 13
• The policy gradient separates into a score-function term multiplied by a continuous-time TD error and a regularizer-derived term that vanishes without regularization.The score-function term’s TD error also appears in the martingale orthogonality condition.Source: §3.2 · p. 14
• The policy-gradient representation in (14) is exactly the continuous-time version of advantage actor–critic, so the authors do not add other baseline functions.Theorem 3 assumes an interior optimal policy satisfying the first-order condition; its test-function equations can use past observations only, enabling online learning. The condition is necessary, and different test functions yield infinitely many equations.Source: §3.2 · pp. 15–16

3.3 Actor–Critic Algorithms

• Actor–critic combines policy evaluation and policy gradients, jointly seeking optimal value-function and policy parameters through alternating updates, with time discretized for implementation.The paper presents TD-type algorithms as illustrations rather than comparisons of which algorithm performs better.Source: §3.3 · pp. 17, 20
• The policy gradient of the value function is given by (14) and can be estimated; online, (14) is not implementable, so stochastic approximation solves optimal condition (18) using past observations through time t.Source: §3.3 · p. 17
• The illustrative algorithms admit variants tailored to the learning problem through test-function choices and different combinations of policy-gradient and policy-evaluation methods.The authors note that their examples use TD-type algorithms and that test functions determine the sense in which value and policy are approximated.Source: §3.3 · pp. 17, 20

4 Extension to Ergodic Tasks

• The section extends the results and algorithms to ergodic, long-term-average tasks.Ergodic tasks are presented as one formulation of continuing tasks, where learning can use a single trajectory.Source: §4 · p. 20
• Ergodic policy evaluation represents the solution as a state function J and a scalar value V, with V independent of the initial state and time.J is not unique: adding a constant to it also gives a solution.Source: §4 · p. 21
• For a parameterized policy, the ergodic Feynman–Kac representation expresses the value gradient using a different running reward.Ergodicity makes the integrand asymptotically unbiased for the gradient as time tends to infinity.Source: §4 · p. 22
• Because ergodic tasks are continuing, one online algorithm estimates the policy gradient from its long-run representation as the state process approaches stationarity.The representation’s long-time average converges to an expectation under the stationary measure.Source: §4 · p. 22
• A second algorithm uses a test function and stochastic approximation to solve the policy optimality condition through martingale orthogonality.The two algorithms differ by the presence of the test function.Source: §4 · p. 23

5 Applications

• The section reports simulations in two applications: finite-horizon mean–variance portfolio selection using multiple simulated stock-price episodes, and ergodic linear–quadratic control using a single sample trajectory.Source: §5 · p. 23

5.1 Mean–Variance Portfolio Selection

• Mean–variance portfolio selection is time-inconsistent because of its variance term, so it cannot be solved directly by dynamic programming.The application adds an entropy regularizer to incentivize exploration.Source: §5.1 · p. 25
• The portfolio application presents offline and online algorithms that apply this paper’s methods to the exploratory mean–variance problem.The authors state that they replicate Wang and Zhou (2020)’s simulation study and present Algorithms 4 and 5.Source: §5.1 · pp. 26–27
• The offline Algorithm 4 achieves significantly higher out-of-sample average Sharpe ratios in most market scenarios than Wang and Zhou (2020), with underperformance occurring mainly in some scenarios.The comparison covers different geometric-Brownian-motion market specifications; the supplied passages do not give specific Sharpe-ratio values.Source: §5.1 · p. 27
• In a few low-return scenarios, the learned policy has larger volatility and less stable out-of-sample performance, but its lower average Sharpe ratios than Wang and Zhou (2020) are statistically inconclusive.These cases occur when the actual stock return is low (µ = 0, ±0.1), and the learned policy's standard deviations are large.Source: §5.1 · p. 30
• Offline learning outperforms online learning on Sharpe ratio in most cases and is always preferred for stably reaching the 40% annual target return.The comparison is reported across the experiments summarized in Tables 2 and 3.Source: §5.1 · p. 30, Table 2 · p. 32, Table 3 · p. 33
• Offline learning bootstraps multiple one-year episodes from stored data, whereas the online 20-year setting uses 20 sequential episodes; nonstationarity, storage, and computation can motivate online learning.The paper notes that offline learning may overfit nonstationary data, while incremental online learning can reduce storage costs and computation time.Source: §5.1 · p. 30

5.2 Ergodic Linear–Quadratic Control

• With sufficient learning, policy parameters and average reward converge toward theoretical optima, but average reward converges more slowly and can approach only the optimal level less exploration cost.Figure 1 depicts the learned policy and average reward; the reward benchmark is limited by entropy-regularized exploration.Source: §5.2 · pp. 34–35, Figure 1 · p. 35
• Average reward initially decreases on the illustrated trajectory, possibly because the state process has not reached stationarity and early policies generate poor trials.After sufficient time, the policies and average reward begin converging toward their theoretical optimal values.Source: §5.2 · pp. 34–35

6 Conclusion

• The paper finds that policy gradient is intimately related to policy evaluation, allowing Jia and Zhou’s (2022a) martingale characterization to be applied to policy gradient.Combining results across the trilogy, it proposes online and offline actor–critic algorithms that learn value functions and stochastic policies simultaneously and alternatingly.Source: §6 · pp. 35–36
• The trilogy conducts theoretical analysis in continuous time and discretizes time only for implementation, with calculus, stochastic calculus, stochastic control, and differential equations among its available tools.The authors point to Doya (2000) when discussing this approach versus discretizing time at the outset.Source: §6 · p. 36
• The policy-gradient algorithms inherit mesh-size convergence from their derivation through policy-evaluation martingality, following Jia and Zhou (2022a).The conclusion notes that discrete-time versions of the trilogy’s algorithms are well known and their convergence is established.Source: §6 · p. 36
• Unlike the MDP derivation involving a Q-function, the paper represents policy gradient through the expected integration of a term involving the value function.The conclusion identifies this distinction as a difference between the derivations and representations of policy gradient and MDP results.Source: §6 · p. 36
• Open questions include episodic-RL regret bounds, interpreting Q-functions and Q-learning in continuous settings, and how exploration temperature affects actor–critic performance.The conclusion notes that continuous-time RL remains in its infancy.Source: §6 · p. 36

Appendix A. Connections with Policy Gradient in Discrete Time

• The appendix frames extension to continuous time around what the Q-function should be and notes that Jia and Zhou (2022b) studied this question.Without a policy regularizer, the appendix says its discrete-time expression coincides with policy gradient (14) at γ = 0.Source: Appendix A · p. 38

ments

• The appendix’s proofs are similar to analogous results in Wang and Zhou (2020) and Wang et al. (2020), respectively.Source: ments · p. 39

Appendix B1. Mean-Variance Portfolio Selection

• For an omniscient agent solving the stated problem, the optimal policy is deterministic, and the resulting discounted wealth process is a geometric Brownian motion.The appendix then computes its expected terminal criterion.Source: Appendix B1. Mean-Variance Portfolio… · p. 39
• The appendix also expresses the terminal criterion in terms of z and relates it to the initial value x0 and w*.The displayed expressions give the terminal criterion for the omniscient and entropy-regularized cases, respectively.Source: Appendix B1. Mean-Variance Portfolio… · p. 39
• When the agent knows the true model but must use a stochastic policy with entropy regularization, the appendix gives an optimal stochastic policy.Its terminal-criterion calculation uses the same value of w* as the omniscient case.Source: Appendix B1. Mean-Variance Portfolio… · p. 39

Appendix B2. Ergodic Linear-Quadratic Control

• The two benchmarks used for comparison in Section 5.2 are the omniscient optimal level and that level less exploration cost, defined with model coefficients known to the agent.The appendix computes both values using the Hamilton-Jacobi-Bellman approach.Source: Appendix B2. Ergodic Linear-Quadratic… · pp. 39–40
• Matching coefficients in the HJB equation gives algebraic equations whose solution yields the omniscient optimal reward and corresponding policy.The HJB calculations assume the true model and maximize long-term average reward.Source: Appendix B2. Ergodic Linear-Quadratic… · p. 40
• The entropy-regularized optimal policy has the same k1 and k2 as the unregularized optimum, and its value is the omniscient optimal level less exploration cost.The difference between the omniscient optimal level and this value is the exploration cost due to randomization.Source: Appendix B2. Ergodic Linear-Quadratic… · pp. 40–41

• In the proofs, C1, C2, and similar symbols denote constants independent of variables such as t, x, and a, and the same symbol need not have the same value in different places.Source: Appendix C · p. 42

Proof of Lemma 1

• The averaged drift coefficient is locally Lipschitz continuous and has linear growth in x.These conditions support the SDE existence argument.Source: Proof of Lemma 1 · p. 42
• Standard SDE theory gives a unique strong solution to (5), and Cauchy–Schwarz, Burkholder–Davis–Gundy, and Gronwall inequalities are used in the subsequent estimate.The estimate uses growth conditions on ˜b and ˜σ.Source: Proof of Lemma 1 · pp. 42–43

• The transformation ˜v(t, x) = e^−βt v(t, x) gives the terminal condition ˜v(T, x) = e^−βT h(x), and (8) implies the transformed equation.Source: Proof of Lemma 2 · p. 43
• Applying the analogous transformation ˜J(t, x; π) = e^−βt J(t, x; π), equation (6) yields the corresponding equation for the policy value.Source: Proof of Lemma 2 · p. 43
• It suffices to prove that the viscosity solution ˜v of (35) coincides with ˜J(·, ·; π).Source: Proof of Lemma 2 · p. 43
• Beck et al. (2021, Corollary 3.3) applied to SDE (5) establishes the result, since Lemma 1 verifies its sufficient conditions under Assumption 1 and Definition 1.Source: Proof of Lemma 2 · p. 43

Proof of Theorem 1

• Theorem 1’s first statement follows from the discounting transformation, Jia and Zhou (2022a, Proposition 1), and the SDE solution’s Markov property.The proof uses the same discounting transformation as the proof of Lemma 2.Source: Proof of Theorem 1 · p. 43
• Theorem 1’s second statement uses Jia and Zhou (2022a, Proposition 4) to obtain martingale orthogonality for ˜Xπ, then combines equations to derive (9).The orthogonality condition uses a measurable functional ξ of time and ˜Xπ.Source: Proof of Theorem 1 · pp. 43–44

Proof of Theorem 2

• The proof reduces to showing (12) equals (14), applies Itô’s lemma up to localized stopping times, and concludes by dominated convergence as n tends to infinity.Source: Proof of Theorem 2 · pp. 44, 46

Proof of Theorem 3

• Theorem 3’s proof follows the approach of Theorem 1 and Jia and Zhou (2022a, Proposition 4), using that π˚ satisfies (17).It suffices to consider ζ = 0 by Theorem 1; the final equality follows from (17).Source: Proof of Theorem 3 · p. 46

Proof of Lemma 3

• A localization argument similar to that in Theorem 2’s proof can show a limit superior as T tends to infinity involving EPW.Source: Proof of Lemma 3 · p. 47
• The analysis also establishes an (F ˜Xπ, P^W)-martingale, from which the lemma’s second conclusion follows as in Theorem 1’s proof.Source: Proof of Lemma 3 · p. 47