Source-linked AI summary
Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
Yanwei Jia, Xun Yu Zhou
TL;DR
The paper addresses the limited theory and algorithms for reinforcement learning in continuous time and space, particularly policy improvement under stochastic policies. It represents policy gradients as an auxiliary policy-evaluation problem and develops actor–critic algorithms for online and offline learning. The resulting policy-gradient methods inherit convergence as the discretization mesh goes to zero, while the broader continuous-time setting remains an active area for future work.
Problem
Continuous-time reinforcement learning lacks general policy-gradient and actor–critic algorithms, while existing approaches are predominantly limited to discrete-time systems.
Method
The paper converts policy gradients into an auxiliary policy-evaluation problem and derives model-free actor–critic algorithms using martingale-based representations and first-order conditions.
Results
The policy-gradient algorithms inherit convergence as the discretization mesh goes to zero from the underlying martingale-based policy-evaluation analysis.
Takeaways & Limitations
The analysis supports actor–critic reinforcement learning algorithms for continuous-time, continuous-space tasks under stochastic policies, including episodic and continuing settings.
Takeaways & Limitations
Continuous-time reinforcement learning remains in its infancy, with open questions still surrounding the broader setting.
Abstract
from arXiv · showhide
We study policy gradient (PG) for reinforcement learning in continuous time and space under the regularized exploratory formulation developed by Wang et al. (2020). We represent the gradient of the value function with respect to a given parameterized stochastic policy as the expected integration of an auxiliary running reward function that can be evaluated using samples and the current value function. This effectively turns PG into a policy evaluation (PE) problem, enabling us to apply the martingale approach recently developed by Jia and Zhou (2021) for PE to solve our PG problem. Based on this analysis, we propose two types of the actor-critic algorithms for RL, where we learn and update value functions and policies simultaneously and alternatingly. The first type is based directly on the aforementioned representation which involves future trajectories and hence is offline. The second type, designed for online learning, employs the first-order condition of the policy gradient and turns it into martingale orthogonality conditions. These conditions are then incorporated using stochastic approximation when updating policies. Finally, we demonstrate the algorithms by simulations in two concrete examples.
1 Introduction
The paper addresses the open problem of reinforcement learning in continuous time and space by developing policy-gradient and actor–critic methods for stochastic policies. Its approach connects policy gradients to policy evaluation, yielding model-free algorithms for episodic, continuing, online, and offline tasks.
- Motivation: Discrete-time solutions can be highly sensitive to time discretization in continuously evolving applications such as trading, autonomous driving, and robot navigation.The paper motivates continuous-time methods for environments requiring ultra-high-frequency or continuous interaction.
- Motivation: Continuous-time reinforcement learning remains largely uncharted, especially for stochastic systems with continuous state and action spaces.Existing work is mostly restricted to discrete-time settings or deterministic continuous systems.
- Contribution: The paper studies policy improvement under stochastic policies, completing the exploration and policy-evaluation components developed in earlier work.This places the contribution within the three-part reinforcement-learning procedure of exploration, evaluation, and improvement.
- Algorithms: Policy gradients avoid action maximization by sampling from a known parametric distribution and can exploit prior knowledge about potentially optimal policy classes.The authors contrast this with Hamiltonian-based improvement, which requires environmental knowledge and can be computationally demanding in high dimensions.
- Algorithms: The resulting actor–critic algorithms are model-free and data-driven, covering episodic and continuing tasks as well as online and offline learning.The paper develops multiple algorithms by learning value functions and stochastic policies simultaneously or alternatingly.
- Contribution: The gradient of the value function is represented in the same form as a policy-evaluation value function, effectively turning policy gradients into an auxiliary policy-evaluation problem.The auxiliary running reward can be evaluated from samples together with the current value function, although its initial form depends on the Hamiltonian.
2 Problem Formulation and Preliminaries
The paper formulates continuous-time stochastic control and reinforcement learning over finite horizons, with policies randomizing actions based on time and state. The objective combines running and terminal rewards, discounting, and entropy-regularized exploration.
- The state evolves according to a stochastic differential equation driven by Brownian motion, with actions serving as controls.
- The control objective maximizes expected discounted running rewards plus a terminal lump-sum reward over a finite planning horizon.
- In reinforcement learning, the environment dynamics and reward functions are unknown, so the agent learns through trial-and-error observations of states and payoffs.
- A stochastic policy samples actions from a probability distribution conditioned on the current time–state pair, independently of environmental Brownian noise.
- The regularized formulation adds an exploration term to the reward, with entropy as the specified regularizer and γ controlling its weight.
3 Theoretical Foundation of Actor–Critic Algorithms
The paper develops a martingale-based theoretical foundation for actor–critic learning with stochastic policies. It characterizes policy evaluation and rewrites policy-gradient computation as policy evaluation, supporting offline and online algorithms.
- Policy Evaluation: The value function satisfies a terminal condition and growth bound, and under the stated characterization equals the policy value.
- Policy Evaluation: Policy evaluation characterizes the value function through a martingale condition and extends prior results from deterministic to stochastic policies.
- Policy Evaluation: Policy evaluation algorithms use martingale losses, orthogonality conditions, or quadratic forms, yielding offline or online procedures depending on the objective.
- Policy Gradient: The policy gradient is represented as an expected integration of an auxiliary running reward, whose terms can be estimated from sampled trajectories and an estimated value function.
- Policy Gradient: This representation turns policy-gradient computation into policy evaluation with a different reward function, while Itô’s lemma and martingality remove the unobservable stochastic-integral contribution.
- Actor–Critic Algorithms: The paper develops two algorithmic routes: future-trajectory gradient estimation for offline learning and first-order-condition martingale methods for online learning.
4 Extension to Ergodic Tasks
The paper extends its policy-evaluation and policy-gradient framework to ergodic tasks with long-term average objectives. It derives online algorithms based on long-run gradient representations and martingale orthogonality.
- Ergodic tasks use long-term average rewards, with a scalar value independent of the initial state and time under ergodicity.
- Because ergodic tasks are continuing, the proposed algorithms are online and can estimate gradients from long-run averages as the state process approaches stationarity.
- The ergodic policy-evaluation problem seeks a state function and scalar value satisfying a martingale characterization.
- The ergodic policy gradient is represented through a different running reward using an ergodic Feynman–Kac formula.
- A second algorithm incorporates a test function and stochastic approximation to solve the optimality condition, differing from the first by the presence of η.
5 Applications
The applications evaluate offline and online actor–critic algorithms in mean–variance portfolio selection and ergodic linear–quadratic control. Simulations examine out-of-sample portfolio performance, convergence, and the trade-off between policy learning and reward accumulation.
- Applications: The experiments cover mean–variance portfolio selection with simulated stock-price data and ergodic linear–quadratic control with a single sample trajectory.The portfolio experiments use geometric Brownian motion under different market-parameter specifications.
- Algorithms: The online actor–critic procedure samples actions from a parameterized stochastic policy, updates test functions and learning variables, and iteratively advances the environment state.Its inputs include initial states, learning rates, value and policy forms, regularization, test functions, and a temperature parameter.
- Mean–Variance Portfolio Selection: Mean–variance portfolio selection minimizes terminal-wealth variance subject to a target expected return, but its variance term makes the problem time-inconsistent and unsuitable for direct dynamic programming.An embedding transformation converts the constrained objective into an expectation-only problem that is time-consistent and compatible with dynamic programming.
- Mean–Variance Portfolio Selection: The proposed offline algorithm achieves significantly higher out-of-sample average Sharpe ratios than the Wang and Zhou (2020) strategy in most scenarios.The comparison uses terminal-wealth mean, variance, and Sharpe ratio over independent experiments.
- Results: Offline learning outperforms online learning in Sharpe ratio in most ergodic-task cases and is more stable for reaching the 40% annual target return.In low-return scenarios, the learned policy has larger volatility and less stable out-of-sample performance, while the methods remain statistically indistinguishable in some cases.
- Ergodic Linear–Quadratic Control: In the online control experiment, learned policy parameters and average reward eventually converge toward theoretically optimal values, although reward convergence takes longer than policy convergence.The average reward initially decreases because early poor performance continues to affect the running average.
6 Conclusion
The paper develops a continuous-time policy-gradient framework by relating policy gradient to policy evaluation through martingale methods, yielding online and offline actor–critic algorithms. It positions these results within a broader continuous-time RL program while identifying several unresolved questions.
- Contributions: PG is intimately related to PE, allowing the martingale characterization of PE to be applied to continuous-time policy-gradient problems.The paper derives policy-gradient representations involving the value function and uses them to formulate PE-style methods.
- Contributions: The paper proposes online and offline actor–critic algorithms for general model-free RL tasks, updating value functions and stochastic policies simultaneously and alternatingly.These algorithms combine the theoretical results developed across the paper’s continuous-time RL framework.
- Continuous-time framework: The analysis is conducted in continuous time and space, with time discretized only during algorithm implementation rather than at the outset.This preserves access to calculus, stochastic calculus, stochastic control, and differential-equation tools.
- Theoretical implications: Any convergent time-discretized PE algorithm converges as the mesh size vanishes, and the paper states that the same convergence holds for its derived PG algorithms.The latter conclusion follows because the PG algorithms are derived from martingality for PE.
- Open questions: The continuous-time RL agenda remains open, including regret bounds, the interpretation of Q-functions and Q-learning, and temperature-parameter dependence in exploratory actor–critic methods.These questions define important scope boundaries for the current theory rather than reported failures of the proposed algorithms.
Appendix B. Theoretical Results Employed in Simulation Experi-
The appendix summarizes theoretical results used in the two simulation studies and points to proofs analogous to results from the preceding papers in the research program.
- Appendix scope: The appendix summarizes the theoretical results employed in the paper’s two simulation studies.It is provided for the reader’s convenience before the simulation-specific applications.
- Proof relationship: The proofs are similar to analogous results in Wang and Zhou (2020) and Wang et al. (2020), respectively.The appendix therefore connects the simulation theory to results established in the related papers.
Appendix B1. Mean-Variance Portfolio Selection
This appendix derives benchmark solutions for a mean-variance portfolio problem, including deterministic optimal control and stochastic policies under entropy regularization.
- Optimal policy: With the true model known and no entropy constraint, the omniscient optimal policy is deterministic.The appendix derives the associated optimal policy and value through the portfolio dynamics and reward equations.
- Portfolio dynamics: Under the optimal deterministic policy, the resulting wealth process is a geometric Brownian motion with an explicitly computable expected terminal value.The appendix gives the expected terminal wealth in terms of the initial wealth and model parameters.
- Exploration benchmark: When entropy regularization forces stochastic policies, the appendix defines the omniscient optimal level less exploration cost as a benchmark.The benchmark compares the regularized stochastic solution with the omniscient optimal level, with their difference interpreted as exploration cost.
Appendix B2. Ergodic Linear-Quadratic Control
This appendix constructs theoretical benchmarks for ergodic linear-quadratic control using HJB equations and algebraic coefficient matching, including an entropy-regularized stochastic-policy benchmark.
- Benchmarks: The simulations compare learning against the omniscient optimal level and the omniscient optimal level less exploration cost.These quantities are introduced as formal benchmarks for the ergodic control experiment.
- HJB derivation: The HJB equation is solved by substituting a quadratic value-function form and matching coefficients of x^2, x, and the constant term.This produces a system of algebraic equations for the unknown coefficients.
- Unregularized solution: Solving the algebraic equations yields the omniscient optimal reward and its corresponding optimal policy.The policy is expressed using the coefficients determined by the HJB system.
- Regularized solution: With entropy regularization, the optimal policy becomes stochastic, and the appendix derives its associated value and state dynamics.The regularized policy is compared with the unregularized solution under the same model coefficients.
- Benchmark relationship: The solutions satisfy ˜k2 = k2 and ˜k1 = k1, while the difference between the two benchmark levels represents the exploration cost from randomization.The appendix identifies the regularized long-term value with the omniscient optimal level less exploration cost.
Appendix C. Proofs of Statements
The proofs use generic constants independent of variables such as t, x, and a, while allowing the same notation to denote different values in different places.
- Generic constants C1, C2, … are independent of variables including t, x, and a.
- The same generic constant notation may represent different values at different points in the proofs.
Proof of Lemma 1
The lemma establishes regularity and growth properties for the relevant coefficients, then uses standard SDE arguments and moment estimates to obtain the desired result.
- The proof concludes after examining the relevant expression and identifying the final result as evident.
- The coefficient is locally Lipschitz continuous and has linear growth in x.
- These coefficient properties imply unique existence of the strong solution to SDE (5) by standard SDE theory.
- Cauchy–Schwarz and Burkholder–Davis–Gundy inequalities are used to derive estimates from SDE (5).
- Gronwall’s inequality is then applied to the resulting expectation bound as a function of the time horizon.
Proof of Lemma 2
The proof applies a discounting transformation to the value and objective functions, then invokes viscosity-solution results and prior policy-evaluation propositions.
- The transformation ˜v(t,x) = e^-βt v(t,x) converts the terminal value to e^-βT h(x) and yields equation (8).
- Applying the same transformation to J(t,x;π) makes equation (6) available in discounted form.
- The proof identifies the viscosity solution of (35) with the discounted objective using Beck et al. (2021, Corollary 3.3).
- The required conditions for that result are verified using Assumption 1, Definition 1, and Lemma 1.
Proof of Theorem 1
The theorem proof combines a discounting argument, policy-evaluation results, martingale orthogonality, localization, and convergence arguments to establish its statements.
- The first theorem statement follows from Jia and Zhou (2022a, Proposition 1) together with the Markov property of SDE (5).
- The second statement uses a martingale orthogonality condition for ˜Xπ derived from Jia and Zhou (2022a, Proposition 4).
- A measurable functional ξ is introduced, and combining two equations yields equation (9).
- Dominated convergence and related bounds establish the limiting result, while the proof also uses that π˚ satisfies (17).
- For ζ = 0, the argument derives a finite, x-independent limit and identifies a martingale before concluding the lemma.