Source-linked AI summary
How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
Vincent François-Lavet, Raphael Fonteneau, Damien Ernst
TL;DR
Neural-network function approximation can become unstable when combined with Q-learning recursion, making the discount factor's role in deep Q-learning stability and convergence worth examining. The paper adapts the discount factor and learning rate during convergence; across six tested Atari games, the approach improved performance.
Problem
Because neural-network function approximation can become unstable under a Q-learning recursion, the paper examines the discount factor's role in deep reinforcement learning stability and convergence.
Source: §1 · p. 1Method
The paper adapts the discount factor and learning rate along the way to convergence in deep Q-learning.
Source: §5 · p. 6Results
Takeaways & Limitations
The authors conclude that adapting the discount factor and learning rate can speed convergence and improve the quality of the learned Q-function in the tested Atari setting.
Source: §5 · p. 6A lower discount factor can decrease exploration and leave an agent at risk of a local optimum.
Source: §4.3 · p. 5
Abstract
from arXiv · showhide
Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma.
1 Introduction
Deep neural networks can approximate value functions for complex, high-dimensional tasks, but their combination with Q-learning can be unstable. This paper examines whether changing the discount factor can improve deep reinforcement learning, motivated by starting with short-term rewards and later weighting delayed rewards more.
1 Introduction
- Neural networks can become unstable when used as function approximators with a Q-learning recursion, despite their usefulness for high-dimensional inputs.Large or continuous state spaces can make comprehensive value-function representation difficult.Source: §1 · p. 1
- The paper investigates how the discount factor may affect deep reinforcement learning stability and convergence, and reports that increasing it can improve learning quality.It also discusses connections to neural-network instability and the exploration/exploitation dilemma.Source: §1 · p. 1
- The motivation is to begin with a short-term reward horizon and progressively give more weight to delayed rewards, by analogy with delayed gratification.The introductory analogy describes delaying rewards as potentially advantageous when pursuing longer-term goals.Source: §1 · pp. 1–2
2 Problem Formulation
The formulation defines expected discounted rewards for a policy, a stochastic state-transition model, and a Bellman solution. For complex problems, it describes approximating the value function with a neural network and updating its parameters toward a target.
2 Problem Formulation
- The task is to maximize expected future rewards, represented as the maximum sum of rewards discounted by γ under a stochastic behavior policy.The policy assigns probabilities to actions in each state.Source: §2 · p. 2
- The environment is a time-invariant stochastic discrete-time system; in the experiments, Atari pixels represent states and legal game actions form the action space.Disturbances are generated from a time-invariant conditional probability distribution given state and action.Source: §2 · p. 2
- The Q-value function's unique solution is described through the Bellman equation, with H mapping functions over state-action pairs to real values.The passages identify H's domain and range but do not provide its full definition.Source: §2 · p. 2
- For complex problems, a neural network parameterizes Q(s, a; θk), and stochastic gradient descent updates it toward a target that uses reward and a discounted next-state value.The target includes parameters from a previous Q-network; α is the scalar step size called the learning rate.Source: §2 · p. 2
3 Instabilities of the online neural fitted Q-learning
Online neural fitted Q-learning can propagate approximation errors unpredictably, so convergence may be slow or unreliable. The paper frames adaptive discounting as a way to balance policy complexity against learning stability and speed.
3 Instabilities of the online neural fitted Q-learning
- Neural-network generalization can create unpredictable value changes across state-action space, propagate errors, and undermine convergence guarantees.Experiments have reported that online updates may converge slowly or unreliably.Source: §3 · pp. 2–3
- The discount factor and network architecture control the number of possible policies, creating a trade-off between complex policy classes and overfitting risk versus limited representation.The paper relates this to the bias-variance trade-off in supervised learning.Source: §3 · p. 3
- With inaccurate prior dynamics information, an optimal policy may use a planning horizon shorter than the problem's evaluation horizon; longer horizons also raise computational cost.The passage gives a small number of available tuples as an example of limited dynamics information.Source: §3 · p. 3
- In neural fitted value learning, higher discount factors increase policy-class complexity but can also propagate errors and destabilize learning.Replay memory, error clipping, a separate target network, convolutional architectures, and double Q-learning are cited as practical safeguards.Source: §3 · p. 3
- The paper investigates adaptive discounting to soften errors during learning while targeting high policy-class complexity and reducing error propagation.The authors frame this as a trade-off between target-policy performance, stability, and learning speed.Source: §3 · p. 3
4 Experiments
An increasing discount factor produced faster learning on four of five tested games and results similar to the constant-factor benchmark on the fifth. The experiments also examine high-discount instability, a decreasing learning rate, exploration, and a proposed actor-critic update scheme.
4 Experiments
- Four of five games learned faster with an increasing discount factor, while the fifth was similar to the constant γ = 0.99 benchmark.Figure 2 summarizes relative improvements after 20M steps against the constant-factor benchmark.Source: §4 · p. 4, Figure 2 · p. 4
4.1 Convergence of the neural network
- Beyond γ ≈ 0.99, increasing the discount factor without additional care severely degraded scores, with overestimation linked to poor policies.The authors attribute high-discount instability to overestimation in the average V value.Source: §4.1 · p. 4
4.2 Further improvement with an adaptive learning rate
- Pairing a higher initial learning rate with a rate that falls as the discount factor rises improved scores on all tested games.The schedule starts at 0.005, twice the original benchmark rate; the authors also report that V decreases as γ is held fixed and the rate is lowered, indicating reduced Q-value overestimation.Source: §4.2 · p. 4, Figure 4 · p. 5
4.3 Exploration / Exploitation Dilemma
- A lower discount factor can reduce exploration and leave the agent in a local optimum; in Seaquest, an adaptive exploration rule allowed escape.The authors identify this as a risk of value-iteration learning and illustrate the rule in Figure 6.Source: §4.3 · p. 5, Figure 6 · p. 6
4.4 Towards an actor-critic algorithm
- The paper presents a general update scheme intended to further improve deep reinforcement learning performance.Source: §4.4 · p. 6
5 Conclusion
The approach adapts the discount factor and learning rate and reports improved performance on six tested Atari 2600 games. The conclusion proposes further online adaptation and combinations with other deep reinforcement learning advances.
5 Conclusion
- Improved performances were reported on the 6 tested Atari 2600 games.The conclusion does not specify a comparison baseline in this passage.Source: §5 · p. 6
- The approach adapts the discount factor and learning rate to speed convergence and improve learned Q-function quality.Source: §5 · p. 6
- Further experiments could develop online adaptation of the discount factor, learning rate, and possibly exploration level.Source: §5 · p. 6
- Figure 7 sketches an actor-critic-type algorithm that adaptively manages discount rate, learning rate, and exploration level.Source: Figure 7 · p. 7
- The conclusion proposes combining the approach with double Q-learning, prioritized experience replay, dueling networks, or recurrent architectures.Source: §5 · pp. 6–7
A Appendix
The appendix lists one results summary for an increasing discount factor and another for an increasing discount factor paired with a decreasing learning rate.
A Appendix
- Table 1 summarizes results for an increasing discount factor.Source: Table 1 · p. 9
- Table 2 summarizes results for an increasing discount factor associated with a decreasing learning rate.Source: Table 2 · p. 9
- The two table captions distinguish an increasing-discount schedule alone from one paired with a decreasing learning rate.Source: Table 2 · p. 9, Table 1 · p. 9