PaperlayerView exact revision on arXiv

Source-linked AI summary

q-Learning in Continuous Time

Yanwei Jia, Xun Yu Zhou

arXiv:2207.00713v4cs.LGcs.AIq-fin.CP

TL;DR

Continuous-time Q-learning lacks an action-sensitive Q-function because the conventional Q-function collapses to the value function. This paper develops a discretization-independent q-learning theory and martingale-based actor-critic algorithms, establishing the q-function’s connection to Hamiltonians and SARSA-style learning.

  • Problem

    In continuous time, the conventional Q-function collapses to an action-independent value function, preventing it from ranking current actions.

    Source: §1 · p. 2

  • Method

    The paper defines a first-order q-function and characterizes it with martingale conditions to develop on-policy and off-policy actor-critic algorithms.

    Source: §1 · p. 3, §3.2 · p. 12

  • Results

    The q-function is linked to the Hamiltonian, while the resulting temporal-difference algorithm recovers a version of SARSA.

    Source: §1 · p. 4, §8 · p. 42

  • Takeaways & Limitations

    The q-function supplies an action-sensitive building block for continuous-time reinforcement learning and supports a theoretical foundation covering general policy improvement in on-policy and off-policy settings.

    Source: §8 · pp. 41–42

    The paper does not analyze convergence rates for its q-learning algorithms, leaving this as an outstanding research question.

    Source: §8 · p. 42

Abstract

from arXiv · show

We study the continuous-time counterpart of Q-learning for reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation introduced by Wang et al. (2020). As the conventional (big) Q-function collapses in continuous time, we consider its first-order approximation and coin the term ``(little) q-function". This function is related to the instantaneous advantage rate function as well as the Hamiltonian. We develop a ``q-learning" theory around the q-function that is independent of time discretization. Given a stochastic policy, we jointly characterize the associated q-function and value function by martingale conditions of certain stochastic processes, in both on-policy and off-policy settings. We then apply the theory to devise different actor-critic algorithms for solving underlying RL problems, depending on whether or not the density function of the Gibbs measure generated from the q-function can be computed explicitly. One of our algorithms interprets the well-known Q-learning algorithm SARSA, and another recovers a policy gradient (PG) based continuous-time algorithm proposed in Jia and Zhou (2022b). Finally, we conduct simulation experiments to compare the performance of our algorithms with those of PG-based algorithms in Jia and Zhou (2022b) and time-discretized conventional Q-learning algorithms.

1 Introduction

The paper develops a continuous-time q-learning theory for entropy-regularized diffusion-based RL, replacing the collapsing conventional Q-function with a time-discretization-independent q-function. It characterizes q-functions through martingale conditions, establishes policy improvement and on-/off-policy learning, and derives actor-critic algorithms connected to SARSA and policy-gradient methods.

On- and off-policy learning

  • The martingale characterizations support both on-policy and off-policy learning, allowing a target policy’s value and q-functions to be learned from data generated by either the target or an arbitrary behavior policy.This preserves a central advantage of conventional Q-learning in the continuous-time setting.Source: §1 · pp. 2–4

Motivation

  • The conventional Q-function collapses to the action-independent value function in continuous time, while the q-function provides a time-discretization-independent analogue of the advantage rate for ranking actions.The q-function avoids sensitivity to observation and intervention frequency without imposing restrictive action-process regularity.Source: §1 · pp. 2–3

Policy improvement

  • The q-function equals the entropy-regularized Hamiltonian plus temporal dispersion, and the theory establishes policy improvement through Gibbs measures based on the scaled q-function.When the Gibbs normalizing constant is unavailable, policy improvement is instead established through a stronger Kullback–Leibler divergence result.Source: §1 · pp. 3–4

q-learning theory

  • The paper characterizes a policy’s q-function and value function through martingale conditions under an enlarged filtration containing environmental noise and action randomization.The characterization applies jointly to the value and q-functions and extends the martingale perspective previously used for policy evaluation and policy gradients.Source: §1 · pp. 3–4

Algorithms

  • The resulting actor-critic algorithms recover a version of SARSA and, when Gibbs normalization is unavailable, recover a policy-gradient actor-critic algorithm from Jia and Zhou (2022b).The algorithms are designed in sample-driven, model-free fashion according to the martingale characterizations.Source: §1 · p. 4

2 Problem Formulation and Preliminaries

This section formulates the classical continuous-time stochastic control problem, then introduces entropy-regularized exploratory reinforcement learning and establishes preliminary PDE, HJB, Gibbs-policy, and policy-improvement results. It concludes that model-dependent Hamiltonians are not directly implementable in reinforcement learning, motivating q-learning.

2.1 Classical model-based formulation

  • The classical problem controls an SDE-driven state process to maximize expected discounted instantaneous and terminal rewards, with discount factor β measuring impatience.The Hamiltonian combines instantaneous reward with the control’s impact on system dynamics, and its maximization yields the optimal feedback policy.Source: §2.1 · pp. 5–6
  • Under continuity, Lipschitz, linear-growth, and polynomial-growth assumptions, the stochastic control problem is well posed and characterized by an HJB equation whose solution is the optimal value function.The HJB solution may be understood as a viscosity solution under appropriate conditions.Source: §2.1 · pp. 5–6

2.2 Exploratory formulation in reinforcement learning

  • In reinforcement learning, the agent lacks the environment functions and instead learns through trial-and-error actions, observed states, rewards, and continual policy updates.The formulation applies to offline repeated trials and online updating from historical observations.Source: §2.2 · pp. 6–7
  • Exploration uses a stochastic feedback policy sampled independently of Brownian environmental noise, with action randomization represented by an expanded filtration and an associated random-coefficient SDE.Under the stated assumptions, the policy-generated SDE has a unique strong solution, and entropy regularization encourages exploration with temperature parameter γ.Source: §2.2 · pp. 7–8
  • The random-coefficient formulation is mathematically equivalent to a projected exploratory formulation, while the former supports value-function analysis and the latter supplies observable samples for algorithm design.The entropy and moment conditions ensure the exploratory payoff is finite and policy admissibility can be verified from policy conditions.Source: §2.2 · pp. 8–9

2.3 Some useful preliminary results

  • For an admissible policy, the value function satisfies a Feynman–Kac PDE, while the optimal exploratory value satisfies an exploratory HJB equation with a Gibbs or Boltzmann optimal policy.With entropy regularization, Gibbs sampling generates trial-and-error strategies; in linear-action, quadratic-payoff systems, the Gibbs policy becomes Gaussian.Source: §2.3 · pp. 9–10
  • The Gibbs measure also yields a general policy-improvement theorem, but its Hamiltonian depends on unknown model parameters, preventing direct application to reinforcement-learning algorithms and motivating q-learning.The classical and exploratory control problems’ theoretical properties and well-posedness are established under the stated conditions.Source: §2.3 · p. 10

3 q-Function in Continuous Time: The Theory

The section develops a time-discretization-independent continuous-time q-function by extracting the action-dependent first-order term from a collapsing Q-function. It establishes martingale characterizations for policy-specific and optimal value–q pairs, supporting both on-policy and off-policy learning.

3.1 Q-function

  • The continuous-time Q-function collapses to the value function as ∆t→0, while its first-order expansion motivates a distinct q-function that avoids upfront time discretization.The expansion contains the value function as its leading action-independent term, the Hamiltonian plus temporal value change at first order, and an o(∆t) residual; the approach differs from Tallec et al. (2019) but reaches the analogous limiting advantage object.Source: §3.1 · pp. 11–12

3.2 q-function

  • The q-function is the ∆t-derivative of the Q-function, represents an instantaneous advantage rate, and enables policy improvement through a Gibbs measure over q-values without time-discretization dependence.Only the Hamiltonian depends on the action, and the resulting Gibbs policy provides a continuous-time interpretation and refinement of the Hamiltonian conjecture of Gao et al. (2020) and Zhou (2021).Source: §3.1 · p. 11, §3.2 · pp. 12–13
  • Martingale conditions characterize the q-function under a target policy in both on-policy and off-policy settings, with enlarged-filtration conditions requiring tests that depend on past and current actions.Theorem 7 jointly characterizes value and q functions through a single martingale condition, while terminal and normalization conditions identify the correct soft q-function; a Gibbs policy generated by the q-function yields the optimal pair.Source: §3.2 · pp. 13–15

3.3 Optimal q-function

  • The optimal value and q-function can be identified by maintaining martingality under any admissible policy, enabling off-policy learning from data generated by a single behavior policy without iterative policy improvement.The direct equation for the optimal q-function is necessary but insufficient without the underlying primitives; Theorem 9 supplies the additional martingale characterization and makes the constraints policy-independent.Source: §3.3 · pp. 16–17

4 q-Learning Algorithms When Normalizing Constant Is Available

This section develops actor–critic q-learning algorithms when the Gibbs normalizing constant is available, using martingale conditions to jointly learn value and q-functions and update policies. It also shows that the approach is continuous-time and can recover a modified SARSA algorithm.

4.1 q-Learning algorithms

  • The algorithms jointly learn value and q-functions while updating the policy, with q serving as both critic and actor through the Gibbs policy.When the normalizing constant is computable, the policy can be obtained directly from the q-function, avoiding separate policy learning; martingale losses, stochastic approximation, and GMM yield offline and online variants.Source: §4 · p. 17, §4.1 · pp. 17–18
  • The martingale-based methods use full-horizon losses or test-function orthogonality, but finite test functions may only approximate the condition and stochastic approximation need not converge.Test-function selection depends on (J, q), while full-trajectory gradient updates are analogous to gradient Monte Carlo or TD(1).Source: §4.1 · pp. 18–19

4.2 Connections with SARSA

  • The q-learning formulation learns the zeroth- and first-order terms of the conventional Q-function independently of ∆t and recovers a modification of SARSA.Its update differs from the modified SARSA rule by a mean-zero term caused by action randomization, which may make the q-learning algorithm less noisy and faster to converge.Source: §4.2 · pp. 20, 24, 25

5 q-Learning Algorithms When Normalizing Constant Is Unavailable

This section addresses soft q-learning when Gibbs normalizing constants are unavailable, using tractable policy families and deriving policy-improvement and policy-gradient connections. It also shows that trajectory-level q-values and value-function approximation enable actor–critic updates without learning the full q-function.

5 q-Learning Algorithms When Normalizing Constant Is Unavailable

  • Computing the Gibbs normalizing constant is often daunting or impossible in high-dimensional settings, motivating algorithms that avoid enforcing its availability.The constraints in Theorems 7 and 9 require the normalizing constant, creating the central computational obstacle.Source: §5 · p. 25

5.1 A stronger policy improvement theorem

  • Tractable policy families with computable densities provide updates that provably improve the current policy, even when the target policy is not reached.The stronger policy-improvement result generalizes Theorem 2 and supports optimizing updates within a tractable family.Source: §5.1 · pp. 25–26
  • The update requires only trajectory q-values rather than the full q-function, making temporal-difference learning sufficient for policy improvement.For tractable distributions, the relevant normalization constraints can be computed and enforced, while trajectory values are easier to estimate.Source: §5.1 · p. 26

5.2 Connections with policy gradient

  • The continuous-time q-learning update recovers the entropy-regularized policy-gradient rule of Jia and Zhou (2022b), yielding an actor–critic algorithm based on value-function approximation.Policy learning does not require martingale conditions, although value-function learning may use them as in Jia and Zhou (2022a).Source: §5.2 · p. 27

6 Extension to Ergodic Tasks

This section extends the continuous-time q-learning framework to ergodic tasks, optimizing a regularized long-run average with time-homogeneous dynamics and stationary policies. It characterizes ergodic value and q-functions through martingale conditions and derives corresponding learning algorithms, including an online example.

Ergodic formulation

  • Ergodic tasks maximize a regularized long-run average over [0, ∞), with time-independent b, σ, and r, h = 0, and stationary admissible policies.Ergodicity makes the long-run value independent of the initial state and time; the value function J is defined only up to an additive constant.Source: §6 · pp. 27–28

Ergodic characterization

  • Theorem 12 characterizes the ergodic value V, value function J, and q-function through martingale conditions for on-policy and off-policy learning.The theorem also provides a policy-improvement condition: if a policy follows the Gibbs distribution induced by the q-function, it is optimal and the associated V is the optimal value.Source: §6 · pp. 29–30

Ergodic q-learning algorithms

  • The resulting ergodic q-learning algorithms jointly learn V, Jθ, and qψ, connect naturally to SARSA and Jia and Zhou (2022b), and include an online Algorithm 4.The section states that the algorithmic discussions parallel the earlier constructions while presenting Algorithm 4 as an ergodic-task example.Source: §6 · p. 30

7 Applications

The applications evaluate q-learning against policy-gradient and conventional Q-learning methods in mean–variance portfolio selection and on- and off-policy ergodic LQ control. Across these settings, q-learning is generally competitive or more stable, while conventional Q-learning is sensitive to time discretization and policy gradient fails off-policy.

7.1 Mean–variance portfolio selection

  • In mean–variance portfolio selection, q-learning and policy gradient have close overall performance, with q-learning stronger in volatile markets while conventional Q-learning is usually worst or divergent.Results change little across time discretizations, but q-learning’s terminal variance can exceed policy gradient’s when |µ| is small or σ is large.Source: §7.1 · p. 34, Table 1 · p. 35, Table 2 · p. 36

7.2 Ergodic linear–quadratic control

  • In on-policy ergodic LQ control, q-learning and policy gradient converge faster and achieve higher running rewards than conventional Q-learning, with q-learning slightly outperforming policy gradient.Q-learning has the slowest convergence; policy gradient initially learns some parameters faster but overshoots, whereas q-learning converges more smoothly.Source: Figure 1 · p. 38, §7.2 · pp. 39–40, Figure 2 · p. 39
  • Conventional Q-learning worsens sharply as ∆t decreases, whereas q-learning and policy gradient remain robust across discretizations in on-policy ergodic LQ control.At ∆t = 0.01, conventional Q-learning shows almost no improvement, while the other two methods retain robust performance.Source: Figure 3 · p. 39, §7.2 · p. 40

7.3 Off-policy ergodic linear–quadratic control

  • In off-policy ergodic LQ control, policy gradient diverges, conventional Q-learning remains discretization-sensitive, and q-learning is the most stable method, converging in every scenario.With coarse ∆t = 1, q-learning and Q-learning converge to the same limits, which differ from the continuous-time optimum because finite sums poorly approximate the theoretical integrals.Source: §7.3 · p. 40, Figure 4 · p. 41

8 Conclusion · Appendix A. Soft Q-learning in Discrete Time

The paper establishes a continuous-time q-learning theory for general policy improvement, covering on- and off-policy learning through a q-function tied to the Hamiltonian. It also identifies martingale-based learning links, computational implications for model-free optimization, unresolved convergence analysis, and discrete-time soft Q-learning analogies.

8 Conclusion

  • The conclusion situates this theory within prior continuous-time RL work on optimal exploration samplers, policy evaluation, and policy gradients, noting that Jia and Zhou (2022b) transformed policy gradients into policy evaluation.PG is described as a special case of policy improvement, whereas discrete-time policy improvement can proceed through Q-functions.Source: §8 · p. 41
  • The paper fills the continuous-time Q-learning gap by introducing a first-order q-function that supports general policy improvement and covers both on- and off-policy learning.The conventional Q-function lacks action information in continuous time, whereas the q-function retains it; its policy-relevant component is the Hamiltonian, explaining Boltzmann exploration in classical RL.Source: §8 · pp. 41–42
  • The q-function is characterized as the compensator preserving martingality of value-plus-cumulative-reward processes under enlarged policy-randomization filtrations.This enables simultaneous q-function and value-function learning through martingale-based PE techniques, while the resulting TD algorithm connects to SARSA.Source: §8 · p. 42
  • The paper leaves convergence-rate analysis as an open problem because discrete-time Q-learning results do not extend directly to continuous state spaces, nonlinear approximations, and differing Q- versus q-function behavior.Stochastic approximation is proposed as a possible framework for future convergence analysis.Source: §8 · p. 42
  • For policy optimization, learning the Hamiltonian or q-function is more efficient and robust than separately estimating model coefficients, while temporal differences can make learning data-driven.The conclusion contrasts this model-free route with model-based estimation and highlights reduced over-parameterization, sensitivity, and error accumulation.Source: §8 · p. 42

Appendix A. Soft Q-learning in Discrete Time

  • The appendix reviews soft Q-learning for finite, time-homogeneous discrete-time MDPs and presents analogies to the paper’s major continuous-time results.It emphasizes the martingale perspective as an underexplored discrete-time viewpoint that may reveal further insights for discrete-time RL.Source: Appendix A · p. 44
  • The appendix’s discrete-time setup uses finite state and action spaces, transition probabilities p(x′|x, a), expected rewards r(x, a), discount factor β ∈(0, 1), and stochastic policies π(·|x).The policy π(·|x) is a probability mass function over actions.Source: Appendix A · p. 44

Appendix A1. Q-function associated with an arbitrary policy … Appendix B5. Q-learning for ergodic LQ control

The appendices establish martingale characterizations linking Q-functions, value functions, and advantages, then specify time-discretized Q-learning variants for normalization regimes and ergodic control applications.

Appendix A1. Q-function associated with an arbitrary policy

  • For an arbitrary policy, the advantage-compensated process is a martingale under both target and behavior policies, explaining Q-learning’s on- and off-policy validity.The required filtration is crucial, whereas the policy-gradient compensator generally supports on-policy but not off-policy learning; suitable stochastic approximation yields SARSA.Source: Appendix A1. Q-function associated… · pp. 44–46

Appendix A2. Q-function associated with the optimal policy

  • For the optimal policy, the softmax Bellman relation yields the foundation for off-policy Q-learning and a martingale characterization valid under any policy.The optimal-policy construction is analogous to the continuous-time optimal q-function characterization.Source: Appendix A2. Q-function associated… · p. 47

Appendix B. ∆t-Parameterized Q-Learning

  • The conventional approach learns the Δt-parameterized Q-function, treating Δt as a parameter and using these algorithms for simulation comparisons.The appendix distinguishes this big-Q approach from the time-discretization-independent little q-learning framework.Source: Appendix B · p. 47

Appendix B1. When normalizing constant is available

  • When the normalizing constant is available, the associated Gibbs policy is explicit and conventional Q-learning produces a direct parameter-update rule.The appendix identifies SARSA as an example of this conventional update method.Source: Appendix B1. When normalizing constant… · pp. 47–48

Appendix B2. When normalizing constant is unavailable

  • When normalization is unavailable, tractable policy densities approximate the Gibbs policy, and the algorithm updates their parameters using evaluated Q-functions.This extends the normalization-available construction through a family of easily calculated policies.Source: Appendix B2. When normalizing constant… · p. 48

Appendix B3. Ergodic tasks

  • For ergodic tasks, the appendix derives parallel Q-function, value-function, and intertemporal-increment relations and corresponding algorithms.The displayed updating rules illustrate the case where the normalizing constant is explicitly available.Source: Appendix B3. Ergodic tasks · pp. 48–49

Appendix B4. Q-learning for mean–variance portfolio selection

  • For mean–variance portfolio selection, a concave parameterized Q-function induces an explicit Gaussian policy, with parameters updated using the derived Q-learning rules.Concavity is enforced through −e−ψ1 < 0, and the Lagrange multiplier is updated analogously to Algorithm 5.Source: Appendix B4. Q-learning for… · p. 49

Appendix B5. Q-learning for ergodic LQ control

  • For ergodic LQ control, a general quadratic Q-function exploits quadratic value and Hamiltonian structure, omits an arbitrary constant, and enforces concavity in actions.The optimal value is an additional parameter, and Q-learning updates follow the derived derivative and parameter rule.Source: Appendix B5. Q-learning for ergodic LQ… · pp. 49–50

Appendix C. Proofs of Statements … Proof of Corollary 5

The appendix proves the entropy-maximizing policy characterization underlying Theorem 2, establishes policy improvement and optimality, and derives Proposition 3 and Corollary 5 through short-time decomposition and direct implication.

Proof of Theorem 2

  • Lemma 13 identifies a unique entropy-maximizing density for the action-wise optimization problem.The proof reformulates the constrained density problem, relaxes the density constraint, and uses the unique inner maximizer to establish uniqueness.Source: Proof of Theorem 2 · p. 50
  • Theorem 2’s proof relies on the Hamiltonian formulation and on Theorem 6, whose proof is independent of Theorem 2 and its proof.The appendix explicitly introduces the q-function notation for the proof and invokes the Hamiltonian H defined in (3).Source: Proof of Theorem 2 · p. 51
  • Applying Lemma 13 within the equivalent formulation shows that the policy improvement operator π′ = Iπ satisfies J(t, x; π) ≤ J(t, x; π′).The argument applies Itô’s lemma to the value function under one policy and the state process under another, then uses localization and moment estimates before sending n →∞.Source: Proof of Theorem 2 · pp. 51–52
  • If the policy improvement operator leaves π unchanged, the resulting value function satisfies both the PDE and HJB equation, proving π is optimal.The fixed-point condition gives J∗(t, x) and, via Lemma 13, the HJB equation and optimal value characterization.Source: Proof of Theorem 2 · p. 52

Proof of Proposition 3

  • Proposition 3 follows by splitting [t, T] into [t, t + ∆t) and [t + ∆t, T], conditioning on the intermediate state, and applying Itô’s lemma.The final equality and its error order arise from approximating the integral in the short-time decomposition.Source: Proof of Proposition 3 · p. 53

Proof of Corollary 5

Proof of Theorem 6 … Erratum to “q-Learning in Continuous Time”

The proofs establish martingale characterizations linking value and q-functions to admissible policies, then derive optimality and policy-improvement consequences through Feynman–Kac, Itô calculus, and KL-divergence arguments. The final theorem proof is omitted as parallel to earlier results.

Proof of Theorem 6

  • Theorem 6 is proved by showing the constructed stochastic process is a martingale when q̂=q, and conversely that the martingale condition forces q=q̂.The converse uses finite-variation local-martingale arguments, continuity, Lebesgue differentiation, Fubini’s theorem, and full support of admissible policies.Source: Proof of Theorem 6 · pp. 54–55

Proof

  • The proof of the companion characterization derives Ĵ=J(·,·;π) from the martingale condition and Feynman–Kac uniqueness, then identifies q̂ with the associated q-function.Under the Gibbs-density condition, the policy is optimal and Ĵ is the optimal value function.Source: Proof · pp. 55–56

Proof of Proposition 8

  • The proof of Proposition 8 obtains its stated identities by exponentiating and integrating the defining q-equation, then comparing with the exploratory HJB equation.The remaining expression follows from the referenced value-function relation.Source: Proof of Proposition 8 · p. 57

Proof

  • The optimal-policy proof constructs the Gibbs density from the optimal q-function, verifies the martingale characterization under that policy, and concludes optimality via Theorem 2.It also uses bq∗(t, x, a)=γ log cπ∗(a|t, x) to satisfy the policy constraint.Source: Proof · p. 57

Proof of Theorem 10

  • Theorem 10’s proof evaluates the KL divergence and applies the same argument as Theorem 2 to show J(t, x; π′) ≥ J(t, x; π).This establishes the value-improvement inequality for the compared admissible policies.Source: Proof of Theorem 10 · p. 58

Proof of Lemma 11

  • Lemma 11’s second statement follows by applying Itô’s lemma to the value function under one policy along the process generated by another, using the Feynman–Kac formula and localization.The first statement is cited as previously proved in Jia and Zhou (2022b, Lemma 7).Source: Proof of Lemma 11 · p. 58

Proof of Theorem 12

  • Theorem 12’s proof is omitted because it proceeds in parallel with the proofs of Theorems 6 and 7.The supplied text characterizes the argument only by this parallelism.Source: Proof of Theorem 12 · p. 59

1 Introduction

The section identifies a measurability gap in continuum-independent action sampling for continuous-time RL and adopts discretely sampled action processes as a rigorous alternative.

1 Introduction

  • Jia and Zhou (2023) introduce the continuous-time q-function and martingale characterizations for data-driven learning of q- and value functions, but assume continuum independent policy sampling.At each time–state pair, an action is sampled from π and applied instantaneously, generating continuous-time state, action, and reward processes.Source: §1 · p. 60

1.1 Measure-Theoretical Issue with Continuum Sampling

  • Continuum-independent sampling may fail progressive measurability, so the resulting drift and stochastic integrals need not be well defined.Szpruch et al. (2024) and Bender and Thuan (2024) identify this gap, motivating an erratum to the important results of Jia and Zhou (2023).Source: §1.1 · p. 60

1.2 Discretely Sampled Processes

  • The erratum adopts Jia et al. (2025)’s framework, sampling policy actions only on a time grid to produce an adapted, well-posed discretely sampled state process.The action process is simple and adapted, while the state solves an SDE driven by the grid-held action aδ(s).Source: §1.2 · pp. 60–61

2 Martingale Characterizations for q-Learning with Discretely Sampled Processes

The section revises martingale characterizations for q-learning using discretely sampled actions while keeping the q-function and exploratory value function independent of the sampling grid. It establishes on-policy and off-policy characterizations that jointly identify value and q-functions and can certify optimality.

Setup

  • The q-function and exploratory value function remain defined independently of discrete sampling, while prior martingale theorems from Jia and Zhou (2023) extend to discretely sampled state–action processes on any time grid.The state remains continuous in time; only actions are sampled discretely, and discretely sampled-policy rewards converge to the exploratory value function as grid size vanishes [Jia et al. (2025)].Source: §2 · p. 61

Theorem 6

  • For a given policy, the q-function is characterized equivalently by martingale conditions along discretely sampled processes, including converse identification from an alternative policy.The result applies for every initial state and any time grid, and continuity plus zero quadratic variation support the converse identification.Source: §2 · pp. 61–63

Joint characterization

  • A subsequent martingale characterization jointly identifies the value function and q-function in both on-policy and off-policy settings.If the relevant martingale conditions hold under another policy, they still recover the value and q-functions associated with the target policy.Source: §2 · p. 64

Optimality

  • When the q-function induces the Gibbs policy density condition, the characterized policy is optimal and the associated value function is the optimal value function.This optimality conclusion is an additional condition in the joint value–q martingale characterization.Source: §2 · p. 64

Proof

The proofs establish martingale characterizations of value and q-functions, including optimal and ergodic settings, with optimality arising under the Gibbs policy condition. They also note that the revised theoretical results do not affect the algorithms or numerical experiments.

Proof

  • Martingale conditions characterize the policy’s value function and q-function through the Feynman–Kac PDE and terminal condition.Uniqueness of the PDE solution yields ˆJ ≡ J(·, ·; π) and ˆq ≡ q(·, ·, ·; π).Source: Proof · p. 65
  • For the finite-horizon optimal case, the Gibbs policy generated by the optimal q-function is optimal, while a corresponding martingale condition identifies the optimal value and q-functions.The converse states that if the martingale condition holds for some admissible policy, then the candidate functions are optimal.Source: Proof · pp. 65–66
  • For ergodic tasks, analogous martingale conditions are necessary and sufficient for identifying the policy’s value, value function, and q-function.The characterization applies across arbitrary time grids and exploratory dynamics, including the converse direction under an admissible policy.Source: Proof · pp. 66–67
  • In the ergodic setting, when the policy satisfies the Gibbs condition in any characterized case, it is optimal and the associated ˆV is the optimal value.This conclusion is stated after the three ergodic characterization cases.Source: Proof · p. 67
  • The revised theoretical results do not change the algorithms or numerical experiments because those procedures use discretely sampled state processes.The proof of the ergodic theorem is omitted as parallel to the proofs of Theorems 6 and 7.Source: Proof · p. 67

Related papers

Loading 2207.00713v4…