PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Deep Q-learning methods robust to time discretization

Corentin Tallec, Léonard Blier, Yann Ollivier

arXiv:1901.09732v2cs.LGstat.ML

TL;DR

DRL can be sensitive to time discretization in near-continuous environments, a concern for settings such as continuous control, robotics, and video games. The paper analyzes this sensitivity and develops an off-policy approach; it reports that standard Q-learning methods degrade as timesteps shrink while its algorithm is more robust.

  • Problem

    The paper investigates DRL sensitivity to time discretization in near-continuous environments, where changing action frequency or frame rate matters.

    Source: §1 · p. 1

  • Method

    The paper analyzes the continuous-time limit of Q-learning and uses the analysis to develop an off-policy approach with timestep-consistent exploration.

    Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 6

  • Results

    Standard Q-learning methods degrade as timesteps shrink, while the proposed algorithm demonstrates substantial robustness to changes in time discretization.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The paper concludes that theoretical analysis can guide a practical off-policy deep-RL algorithm with improved timestep robustness, confirmed empirically.

    Source: §7 · p. 9

    The algorithm requires the timestep δt, which is typically available or set by the practitioner.

    Source: §6 · p. 8

Abstract

from arXiv · show

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitivity to time discretization in near continuous-time environments as a critical factor; this covers, e.g., changing the number of frames per second, or the action frequency of the controller. Empirically, we find that Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small time steps. Formally, we prove that Q-learning does not exist in continuous time. We detail a principled way to build an off-policy RL algorithm that yields similar performances over a wide range of time discretizations, and confirm this robustness empirically.

1. Introduction

The paper identifies sensitivity to time discretization as a factor limiting DRL in near-continuous environments. It analyzes why standard Q-learning degrades as timesteps shrink and proposes an off-policy algorithm designed for greater robustness.

1. Introduction

  • DRL sensitivity to hyperparameters, implementation details, and environment changes, together with sample inefficiency, limits its application in real-world settings.The paper also notes that environment sensitivity can prevent transfer from imperfect simulators to real-world scenarios.Source: §1 · p. 1
  • The study focuses on time-discretization sensitivity in near-continuous environments, including continuous control, robotics, and many video games.The motivating example contrasts 50 observations and actions per second with 10.Source: §1 · p. 1
  • Standard Q-learning methods, including DQN and DDPG, degrade as timesteps shrink because individual actions contribute less to return, and the Q-function collapses to the value function in continuous time.The paper formally shows that Q-learning is ill-behaved in this setting.Source: §1 · pp. 1–2
  • The analysis leads to design choices for architecture, exploration, and learning-rate scaling so the off-policy algorithm can remain viable at small timesteps.The target is an algorithm that admits a continuous-time limit as the timestep approaches zero.Source: §1 · pp. 1–2
  • The proposed algorithm demonstrates substantial robustness to time-discretization changes, unlike standard Q-learning methods.The experiments compare DAU with DDPG for continuous actions and DQN for discrete actions.Source: §1 · pp. 1–2

2. Related Work

Building on, the paper extends Advantage Updating with deep-network experiments, formal results, and timestep-invariant exploration. It distinguishes this continuity problem from multiscale reinforcement learning and contrasts its generic deep-RL focus with earlier continuous-space-time approaches.

2. Related Work

  • Building on, the paper modifies Advantage Updating’s normalization, tests it across deep-RL environments, proves collapse and non-collapse results, and studies timestep-invariant exploration and robustness.Earlier testing was limited to a quadratic-linear problem.Source: §2 · p. 2
  • Unlike the value-advantage parameterization of (Wang et al., 2015), Advantage Updating changes learning scalings rather than simply reparameterizing Q.The passage says the Wang et al. approach does not yield an invariant algorithm for small δt.Source: §2 · p. 2
  • The studied continuity effect is distinct from multiscale RL and can arise even when the environment has only one timescale.The Q-to-V collapse is described as an intrinsic mathematical effect of time continuity.Source: §2 · p. 2
  • Earlier continuous-space-time work includes grid methods for solving the HJB equation and algorithms such as advantage updating, while this paper targets generic deep-RL algorithms.The grid approach assumes extensive state visitation and is described as impractical for larger-dimensional problems.Source: §2 · p. 2

3. Near Continuous-Time Reinforcement Learning

The section formalizes near-continuous environments as small-timestep discretizations of continuous-time MDPs and defines scalings that preserve meaningful returns and value functions. It then shows that the action-value function collapses to the value function as the timestep approaches zero, leaving no continuous-time Q-learning.

3. Near Continuous-Time Reinforcement Learning

  • The section focuses on discretized environments with small timesteps, termed near-continuous, and notes that stochastic-dynamics extensions are not formally treated.The authors believe the results extend to states following a stochastic differential equation.Source: §3 · p. 3, §3.1 · p. 3

3.1. Framework

  • Near-continuous MDPs discretize an underlying continuous-time process at timestep δt; suitable δt-dependent discount and reward scalings give non-trivial returns and value functions as δt → 0.The framework aims for quantities to converge to meaningful limits; formal statements and technical assumptions are in the supplementary material.Source: §3.1 · p. 3
  • Scaling the discount factor keeps the physical-time horizon stable as δt → 0, whereas leaving it constant makes the agent increasingly short-sighted.The scaled decay factor is used for both Deep Advantage Updating and classical deep Q-learning baselines.Source: §3.1 · p. 3

3.2. There is No Q-Function in Continuous Time

  • As δt → 0, Qπ collapses to Vπ, so it carries no action-ranking information and cannot support continuous-time Q-learning.The action-specific effect on Q is O(δt), making actions difficult to rank, especially with an approximate Q-function.Source: §3.2 · p. 4

4. Reinforcement Learning with a Continuous-Time Limit

The section builds a discrete off-policy algorithm with a continuous-time limit by preserving action-ranking information, using exploration with a meaningful limit, and scaling learning rates for well-behaved parameter trajectories. Its components address advantage learning, timestep-invariant exploration, algorithm updates, and learning-rate scaling.

4.1. Advantage Updating

  • The rescaled advantage has a non-degenerate action-dependent limit, unlike Q, and can retain information for ranking actions as δt approaches zero.If a policy strictly dominates another, its limiting advantage is positive for some state; approximate Q rankings can be unreliable when approximation error exceeds O(δt).Source: §4.1 · p. 4
  • Defining Aψ(s, a) by subtracting its value at the policy action enforces the consistency condition needed to identify the learned advantage.Without this constraint, shifting Vθ and Aψ by a function-dependent amount can leave Q unchanged while making Aψ unstable or uninterpretable as δt approaches zero.Source: §4.1 · p. 5

4.2. Timestep-Invariant Exploration

  • Perturbing advantage values with discretized continuous-time Ornstein–Uhlenbeck noise yields a nontrivial exploration scheme in the limit, whereas ε-greedy exploration tends to collapse to a deterministic policy.The OU noise trajectories retain Brownian behavior with a recall force toward zero.Source: §4.2 · p. 5

4.3. Algorithms for Deep Advantage Updating

  • DAU learns Vθ and Aψ using variants of Q-learning, approximately minimizing the squared Bellman residual while enforcing the advantage consistency equation.For continuous actions, a policy neural network is trained to maximize Aψ(s, πφ(s)); for discrete actions, maximization is exact.Source: §4.3 · p. 6

4.4. Scaling the Learning Rates

  • Learning rates of order δt are the only scaling that yields convergent continuous parameter trajectories: β = 1 converges, β > 1 leaves parameters at initialization, and β < 1 can send them to infinity in arbitrarily small physical time.The parameterization uses ηVδt = αVδt^β and ηA = αAδt^β.Source: §4.4 · p. 6

5. Experiments

The experiments compare DAU with DDPG for continuous actions and DQN for discrete actions across time discretizations. DAU is generally more robust in the reported benchmarks, though the results are less clear on Cartpole and Half-Cheetah.

Experiments

  • The experiments target robustness across time discretizations and compare DAU with DDPG for continuous actions and DQN for discrete actions, but do not evaluate on-policy methods.Reported quantitative results are averaged over five runs.Source: §5 · pp. 6–7

Qualitative experiments: Visualizing policies and values

  • On pendulum swing-up, DAU’s policy and value function converge quickly across tested discretizations without specific tuning, while DDPG’s learned functions vary substantially with discretization.Figure 1 plots value over angle and angular velocity, with lighter pixels indicating higher values.Source: §5 · p. 7, Figure 1 · p. 7

Quantitative experiments

  • DAU is robust to δt variations with reasonable performance on Pendulum, BipedalWalker, and Ant, while DDPG varies with δt; DAU is not clearly invariant on Half-Cheetah, and Cartpole results are noise-dominated.DDPG degrades as δt decreases on Ant and Half-Cheetah, and becomes more variable during Pendulum learning at small δt.Source: §5 · p. 7

6. Discussion

DAU shows improved timestep robustness in the reported environments, but practical invariance is qualified by environment behavior and implementation choices. The discussion identifies assumptions and algorithm components that can affect whether the theoretical limit applies.

6. Discussion

  • DAU appears more timestep-robust than DDPG across environments, though some practical results still fall short of the theoretical invariance.Figure 2 plots scaled return against physical training time for multiple discretizations; the discussion notes remaining issues on some environments.Source: §6 · pp. 7–8, Figure 2 · p. 8
  • The algorithm requires knowledge of δt, usually available from the frame rate, actuator or observation frequency, or simulator timestep.The analysis can then provide relevant hyperparameter scalings once the timestep’s order of magnitude is known.Source: §6 · p. 8
  • The continuous-time analysis assumes a continuously differentiable value function, so it holds only almost everywhere when different policy regimes create discontinuities.In pendulum swing-up, the optimal value function can be discontinuous at the boundary between states requiring oscillation and those that do not.Source: §6 · p. 8
  • A fixed-size replay buffer breaks strict timestep invariance because its sample freshness in physical time changes with δt.A buffer of size proportional to 1/δt would keep memory fixed in physical time, at greater memory cost.Source: §6 · p. 8
  • In stochastic near-continuous environments, gradient variance typically scales as O(1/δt), which may interact with the learning-rate scaling through RMSProp.The passage contrasts this with deterministic environments, where gradients typically scale as O(1) in δt.Source: §6 · p. 8
  • For recurrent models or partial observability, memory must persist for a non-infinitesimal physical time as δt approaches zero.The continuous-time analysis should in principle be repeated for every component of the system.Source: §6 · pp. 8–9

7. Conclusion

The paper reports that Q-learning methods fail to learn with small time steps, both theoretically and empirically, and that theoretical analysis helps construct an off-policy deep RL algorithm with improved timestep robustness confirmed in experiments.

7. Conclusion

  • Q-learning methods fail to learn with small time steps both theoretically and empirically; analysis supports an off-policy deep RL algorithm with improved timestep robustness, confirmed experimentally.Source: §7 · p. 9

A. Proofs

The proofs establish continuous-time limits for discretized trajectories and derive conditions for value convergence, policy improvement, and learning-rate behavior. They also characterize the limiting dynamics of ε-greedy exploration.

A. Proofs

  • The proof strategy is largely standard calculus and numerical-scheme analysis, except Theorem 8’s reinforcement-learning argument for policy improvement using the continuous-time advantage.Source: A. Proofs · pp. 10–11
  • Discretized trajectories converge to their continuous-time dynamics under the stated smoothness and boundedness assumptions.The appendix frames the first result as convergence of discretized trajectories and proves it using standard Euler-scheme reasoning.Source: A. Proofs · p. 10
  • For bounded Lipschitz rewards, the value function converges across discretizations under the theorem’s stated assumptions.Source: A. Proofs · p. 10
  • If both policies’ reward-rate functions are continuous and both value functions are continuously differentiable, then Aπ(s, π′(s)) ≥ 0 for every state implies Vπ(s) ≤ Vπ′(s) for every state.Source: A. Proofs · p. 11
  • Learning-rate scaling η = αδt^β yields continuous parameter trajectories for β = 1, stationary trajectories for β > 1, and possible unbounded growth for β < 1.Source: A. Proofs · p. 12
  • With ε-greedy exploration, trajectories converge as δt approaches zero to solutions of a deterministic equation.The result covers a policy mixing deterministic exploitation with noise-policy actions at probability ε per step.Source: A. Proofs · p. 13

B. Implementation details

The implementation section specifies the DAU, DDPG, and DQN variants and compares scaled and unscaled baseline settings. DAU uses value, advantage, and policy components with timestep-scaled learning rates.

B. Implementation details

  • Continuous DAU updates value, advantage, and policy parameters with learning rates scaled by δt.Its implementation also takes the timestep and discount factor as inputs.Source: B. Implementation details · p. 14
  • The DDPG and DQN baselines are tested both with timestep-dependent scalings and with parameters fixed to a reference timestep.In the scaled setting, rewards and learning rates scale with δt; in the fixed setting, these use the reference δt0 = 1e-2 while the discount factor remains scaled.Source: B. Implementation details · pp. 14, 16
  • The baseline algorithms use replay-buffer transitions containing state, action, reward, termination signal, and next state.Their pseudocode describes collecting transitions and sampling random batches for learning.Source: B. Implementation details · p. 15

B.1. Global hyperparameters

The appendix lists shared training hyperparameters, network architecture, replay and interaction settings, and discount scaling used across experiments.

B.1. Global hyperparameters

  • All networks use two 256-unit LayerNorm/ReLU hidden layers, with an additional tanh layer on policy networks and optional running mean-standard-deviation input normalization.The normalization statistics are computed separately for each input feature over previously encountered samples.Source: §B.1 · p. 16
  • The replay buffer is cyclic with capacity 1,000,000, and RMSprop is used without momentum with α = 1 − δt (or 1 − δt0 for unscaled DDPG and DQN).Source: §B.1 · p. 16
  • The physical γ is 0.8 and is scaled as γδt, including for unscaled DQN and DDPG.Source: §B.1 · p. 16
  • Training uses batch size 256, 10 environment steps between learning steps, 50 learning updates, and 256 parallel environments.Together, the parallel environments produce 2,560 environment interactions between learning steps.Source: §B.1 · p. 16
  • Exploration uses an OU process with κ = 7.5 and σ = 1.5; α1 is 0.1 and α2 is 0.03 unless otherwise stated.The listed α1 values are α̃Q, αV, and αA; α2 values are α̃π and απ.Source: §B.1 · p. 16

B.2. Environment dependent hyperparameters

Environment-specific settings specify state normalization, selected α2 and τ values, and tested discretization ranges for six environments.

B.2. Environment dependent hyperparameters

  • Ant and Cheetah use continuous actions, state normalization, and discretization values [0.05, 0.02, 0.01, 0.005, 0.002].Source: §B.2 · p. 16
  • Bipedal Walker uses continuous actions and state normalization, α2 = 0.02, and discretization values [0.01, 0.005, 0.002, 0.001].Source: §B.2 · p. 16
  • Cartpole uses discrete actions, α2 = 0.02, τ = 0, and discretization values [0.01, 0.005, 0.002, 0.001, 0.0005].Source: §B.2 · p. 16
  • Pendulum uses continuous actions, α2 = 0.02, τ = 0, and discretization values [0.01, 0.005, 0.002, 0.001, 0.0005].Source: §B.2 · p. 16

C. Additional results

The additional results section presents pendulum policy and value-function visualizations and learning curves on classic-control benchmarks.

C. Additional results

  • Figure 3 depicts pendulum swing-up policies from unscaled DDPG and AU at different physical training times, with angle and angular velocity axes and action indicated by pixel brightness.Lighter pixels indicate actions closer to 1 and darker pixels actions closer to −1.Source: Figure 3 · p. 17
  • Figure 4 depicts pendulum swing-up policies from scaled DDPG and AU at different physical training times, with angle and angular velocity axes and action indicated by pixel brightness.Lighter pixels indicate actions closer to 1 and darker pixels actions closer to −1.Source: Figure 4 · p. 17
  • Figure 5 depicts pendulum swing-up value functions from scaled DDPG and AU at different physical training times, with angle and angular velocity axes and value indicated by pixel brightness.Lighter pixels indicate higher values.Source: Figure 5 · p. 18
  • Figure 6 presents DAU and scaled DDPG learning curves on classic-control benchmarks across time discretizations, plotting scaled return against physical time spent in the environment.Source: Figure 6 · p. 18

Related papers

Loading 1901.09732v2…