Source-linked AI summary
Dueling Network Architectures for Deep Reinforcement Learning
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, Nando de Freitas
TL;DR
Deep reinforcement learning commonly relies on standard neural-network architectures rather than designs tailored to model-free RL. This paper introduces a dueling architecture that separates state value and action advantage estimation, improving evaluation with similar-valued actions and achieving state-of-the-art Atari 2600 performance.
Problem
Most reinforcement-learning approaches use standard neural-network architectures, leaving room for architectures better suited to model-free reinforcement learning.
Method
The dueling architecture uses separate value and advantage streams that share a common feature-learning module.
Results
Duel Clip did better than Single Clip on 75.4% of Atari games (43 out of 57) and achieved human-level performance on 42 of 57 games.
Takeaways & Limitations
The architecture improves policy evaluation as redundant or similar actions are added and supports state-of-the-art performance on Atari 2600.
Takeaways & Limitations
The separate value and advantage estimates are not necessarily accurate estimators of their corresponding true functions.
Abstract
from arXiv · showhide
In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.
1. Introduction
The paper introduces the dueling architecture, a neural network for model-free reinforcement learning that separately represents state values and action advantages while sharing feature learning. It is designed to improve policy evaluation with redundant actions and achieves strong Atari 2600 performance, while remaining compatible with existing algorithms.
- Architecture: The dueling architecture separates state-value and state-dependent action-advantage representations into two streams sharing a convolutional feature-learning module.A special aggregating layer combines the streams to estimate the state-action value function Q.
- Motivation: The architecture can learn which states are valuable without learning every action’s effect, especially when actions do not meaningfully affect the environment.This is useful in states with redundant or similar actions.
- Results: The dueling architecture more quickly identifies the correct action during policy evaluation as redundant or similar actions are added.The experiments demonstrate this advantage directly in policy-evaluation settings.
- Results: On Atari 2600, a single agent using the same structure and hyper-parameters plays 57 games from image pixels and scores, with results improving over single-stream baselines.Combining prioritized replay with the dueling network produces the paper’s strongest reported Atari result.
- Architecture: Unlike prior advantage-updating methods, the dueling architecture decouples representation from the reinforcement-learning algorithm and can combine with many model-free algorithms.A single deep model represents V(s) and A(s, a), combining them to produce Q(s, a).
2. Background
The background defines reinforcement learning’s sequential decision-making setting, discounted value functions, and the advantage function. It then reviews DQN, experience replay, Double DQN, and prioritized replay as foundations for the dueling architecture.
- Reinforcement learning setup: An agent interacts with an environment over discrete time steps, observes states, selects actions from a discrete set, and receives rewards.In Atari, each state is represented by M consecutive image frames.
- Value functions: The agent maximizes expected discounted return, with γ controlling the trade-off between immediate and future rewards.The discounted return sums future rewards weighted by powers of γ.
- Value functions: The value function measures how good a state is, while Q measures an action’s value and the advantage function measures its importance relative to the state value.The advantage function is defined by subtracting the state value from the Q function, and its policy-weighted expectation is zero.
- Deep Q-networks: DQN approximates high-dimensional value functions with a neural network, stabilizing learning by freezing a separate target network and training with experience replay.Experience replay reuses samples and reduces update variance by uniformly sampling less-correlated mini-batches.
- Algorithmic foundations: The paper uses Double DQN to mitigate overoptimistic estimates and evaluates the dueling architecture with both uniform and prioritized replay baselines.Prioritized replay increases sampling probability for experiences with high expected learning progress, measured by absolute TD-error.
3. The Dueling Network Architecture
The dueling network is a single Q-network that separates state-value and action-advantage estimation into two fully connected streams after shared convolutional layers. An identifiability-aware aggregation combines these streams while preserving compatibility with existing Q-learning algorithms.
- Architecture: The architecture uses shared convolutional layers followed by separate fully connected streams for a scalar state value and an |A|-dimensional action-advantage vector.The streams are combined to produce one Q value for each action.
- Identifiability: Naively summing value and advantage is unidentifiable because shifting a constant between the two estimators leaves Q unchanged and performs poorly in practice.The parameterized Q estimate does not by itself guarantee that either stream accurately estimates its intended function.
- Identifiability: The proposed aggregation forces the advantage of the selected maximizing action to zero, making the value stream estimate the state value and the other stream estimate advantage.For the maximizing action, Q equals V under this construction.
- Aggregation: An alternative mean-subtraction aggregator improves optimization stability, while preserving the relative ranking of advantage and Q values for greedy or ϵ-greedy policies.The experiments used the simpler average-based module after a softmax version produced similar results.
- Compatibility: Because the dueling architecture has the same input-output interface as standard Q-networks, it can be trained with DDQN, SARSA, and other Q-network improvements without algorithmic modifications.Training requires only back-propagation, with value and advantage estimates computed automatically.
4. Experiments
Experiments show that dueling networks accelerate policy evaluation as action spaces grow and substantially improve Atari performance over single-stream baselines. Across 57 games, the method remains stronger under human-start evaluations, while prioritized replay yields further gains.
- Policy evaluation: Policy evaluation uses a corridor environment with exactly computable Qπ(s, a) values, comparing single-stream and dueling architectures across 5, 10, and 20 actions.The behavior policy is ϵ-greedy with ϵ = 0.001, and performance is measured by squared error against true state-action values.
- Policy evaluation: With 5 actions, both architectures converge at similar speeds, whereas the dueling architecture performs better as the number of actions increases.Its value stream learns a general state value shared across similar actions, leading to faster convergence.
- Atari evaluation: The Atari evaluation covers 57 diverse games from raw pixels and rewards using a fixed algorithm, architecture, and hyper-parameter set.The dueling network is trained with DDQN and compared against single-stream Q-network results.
- Atari evaluation: Duel Clip outperforms Single Clip on 75.4% of games (43 out of 57) and the Single baseline on 80.7% (46 out of 57), achieving human-level performance on 42 out of 57 games.Among games with 18 actions, Duel Clip is better 86.6% of the time (26 out of 30).
- Robustness to human starts: Under the Human Starts metric, Duel Clip outperforms the Single baseline on 70.2% (40 out of 57) games and on 83.3% (25 out of 30) of 18-action games.Human Starts evaluates episodes from 100 starting points sampled from human expert trajectories, for up to 108,000 frames.
- Prioritized replay: With prioritized replay, the prioritized dueling agent performs significantly better than both the prioritized baseline and the dueling agent alone, reaching mean and median scores of 591% and 172%.These results are reported across all 57 Atari games using the 30 no-ops initialization.
5. Discussion
The dueling architecture improves learning by updating the state-value stream with every Q-value update, unlike single-stream architectures that update only one action’s value. Its separate advantage stream is robust to noise arising when action-value differences are small relative to overall state values.
- Learning efficiency: The value stream V is updated with every Q-value update, whereas a single-stream architecture updates only one action’s value.Values for the other actions remain untouched in the single-stream architecture.
- Scale of action differences: In Seaquest after DDQN training, the average action gap is roughly 0.04, compared with an average state value of about 15.The action gap is measured between the best and second-best action across visited states.
- Robustness: The separate advantage stream makes the dueling architecture robust to noise caused by small action-value differences relative to state-value magnitude.The passage identifies this robustness as a benefit of separating the advantage stream.
6. Conclusions
The paper introduces a dueling architecture that decouples value and advantage estimation while sharing feature learning, achieving dramatic improvements and new state-of-the-art results in Atari deep reinforcement learning.
- 6. Conclusions: The dueling architecture decouples value and advantage in deep Q-networks while sharing a common feature learning module.It is combined with algorithmic improvements for deep reinforcement learning.
- 6. Conclusions: The combined architecture and algorithmic improvements produce dramatic improvements over existing deep reinforcement learning approaches in the challenging Atari domain.The results are reported as the new state-of-the-art in this popular domain.
A. Double DQN Algorithm
The Double DQN algorithm uses online and target network parameters with replay-buffer sampling to construct targets and update the online network. It selects the next action with the online network but evaluates that action with the target network.
- Initialization: The algorithm initializes an empty replay buffer D, online parameters θ, copied target parameters θ−, and replay and training-size controls.Inputs include replay-buffer maximum size Nr, minibatch size Nb, and target-network replacement frequency N−.
- Interaction: For each episode, it initializes a frame sequence x and repeatedly selects actions from policy πB, observes rewards and next frames, and forms transitions.The current frame sequence defines state s, while the updated sequence defines s′.
- Replay sampling: The algorithm stores transitions in D, replaces the oldest tuple when the buffer reaches Nr, and samples Nb tuples uniformly for training.Each stored tuple has the form (s, a, r, s′).
- Target construction: It selects amax(s′; θ) with the online network and evaluates Q(s′, amax(s′; θ); θ−) with the target network before gradient descent on the squared loss.For terminal states, the target is r; otherwise, it is r + γQ(s′, amax(s′; θ); θ−).