A Brief Survey of Deep Reinforcement Learning

Kai Arulkumaran; Marc Peter Deisenroth; Miles Brundage; Anil Anthony Bharath

Source-linked AI summary

A Brief Survey of Deep Reinforcement Learning

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, Anil Anthony Bharath

arXiv:1708.05866v2Aug 19, 2017cs.LGcs.AIcs.CVstat.ML

TL;DR

Deep reinforcement learning addresses the challenge of scaling reinforcement learning to high-dimensional decision-making and visual-control problems. The survey synthesises foundational RL concepts and major value-based, policy-based, and actor-critic methods, highlighting applications from Atari games to robotics and identifying unresolved generalisation challenges.

Problem
Earlier reinforcement-learning approaches were limited in scalability and low-dimensional settings, motivating methods for high-dimensional visual and decision-making problems.
Method
The survey reviews RL foundations and synthesises value-based, policy-search, actor-critic, and neural-network methods across visual and robotic applications.
Results
DRL has enabled learning from raw visual inputs, including Atari gameplay and robotic control from camera inputs, while extending RL to varied machine-learning tasks.
Takeaways & Limitations
DRL provides a framework for learning through interaction across games, physical tasks, visual attention, and other optimisation problems.
Takeaways & Limitations
Some causal-model approaches have shown superior generalisation to standard DRL algorithms on some benchmarks, indicating unresolved generalisation challenges.

Abstract

from arXiv · show

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policy-based methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep $Q$-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

I. INTRODUCTION

Deep reinforcement learning combines reinforcement learning with deep neural networks to address high-dimensional decision-making and visual-control problems. This survey introduces RL, reviews major DRL methods, and connects them to applications ranging from Atari games to robotics and other learning tasks.

Earlier reinforcement-learning approaches lacked scalability and were limited to fairly low-dimensional problems.
The survey covers seminal and recent DRL developments, including neural-network uses that support autonomous agents.
Deep learning enables RL to scale to previously intractable problems with high-dimensional state and action spaces.
DRL learned to play Atari 2600 games directly from image pixels and at a superhuman level.
DRL policies can be learned directly from real-world camera inputs for robotic control.
DRL is presented as a general trial-and-error approach spanning physical automation and machine-learning tasks beyond video games.

A. Markov Decision Processes

Reinforcement learning is formalised as a Markov decision process with states, actions, transition dynamics, rewards, and discounting. The objective is to learn a policy that maximises expected cumulative reward, with extensions for continuing and partially observable settings.

A Markov decision process consists of states, actions, transition dynamics, rewards, and a discount factor.
The policy maps states to action distributions, while discounted rewards define the return accumulated along an episode.
The goal of RL is to find an optimal policy that achieves the maximum expected return from all states.
For non-episodic MDPs, γ < 1 prevents an infinite reward sum, while finite-transition methods remain applicable.
The Markov property makes the future conditionally independent of the past given the current state, although partially observable settings require state estimation.

B. Challenges in RL

RL must infer policies through reward-driven interaction while handling action-dependent observations, temporal correlations, and delayed consequences. These challenges appear concretely in visual robotic navigation, where actions affect both progress and future observations.

The agent must infer an optimal policy through trial-and-error interaction using reward as its only learning signal.
Agent actions influence observations, which can contain strong temporal correlations.
Long-range time dependencies create a temporal credit-assignment problem because action consequences may appear only after many transitions.
In indoor visual navigation, knowing the goal or remaining distance does not reveal the action sequence needed to reach it.
The survey distinguishes value-function, policy-search, and hybrid actor-critic approaches for solving RL problems.

A. Value Functions

Value-function methods estimate expected returns for states or state-action pairs and use these estimates to improve policies. Bellman bootstrapping underlies Q-learning and SARSA, which differ in how their updates use behavioural-policy transitions.

The state-value function V π(s) estimates expected return from state s when policy π is followed thereafter.
The Q-function incorporates an initial action and follows π from the succeeding state because transition dynamics are unavailable.
Given Qπ(s, a), a greedy policy selects argmaxa Qπ(s, a), and V π(s) can be obtained by maximising Qπ(s, a).
Bellman bootstrapping improves Qπ estimates using current value estimates and forms the foundation of Q-learning and SARSA.
SARSA uses on-policy transitions, whereas off-policy Q-learning uses rt+γ maxa Qπ(st+1, a) to approximate Q∗.
Generalised policy iteration alternates policy evaluation and improvement, with evaluation reducible to minimising temporal-difference errors from experienced trajectories.

B. Sampling

RL algorithms vary in how they estimate returns, whether they bootstrap, and whether they sample actions or evaluate all choices. Policy search directly optimises parameterised policies, while actor-critic methods combine policies with learned value-function feedback.

Sampling: Monte Carlo methods estimate expected returns by averaging complete policy rollouts, whereas TD methods bootstrap from intermediate estimates.Pure Monte Carlo methods can apply in non-Markovian environments but require episodic tasks; TD(λ) combines TD and Monte Carlo evaluation.
Policy Search: Policy search directly optimises a parameterised policy to maximise expected return using gradient-based or gradient-free optimisation.Policies may output distribution parameters, such as Gaussian means and standard deviations for continuous actions or multinomial probabilities for discrete actions.
Sampling: The figure’s dimensions distinguish bootstrapping from non-bootstrapping and sampled actions from expectation over all actions.One-step TD and Monte Carlo methods sample actions, while dynamic programming and exhaustive search take expectations over choices.
Policy Gradients: Policy-gradient methods estimate gradients over sampled trajectories, increasing the probability of sampled actions in proportion to their returns.The REINFORCE estimator provides this gradient when stochastic samples block ordinary backpropagation.
Policy Gradients: Trajectory-based policy gradients have high variance, which can be reduced by subtracting a baseline and weighting updates by an advantage.The baseline may be the average return across several episodes or a learned value estimate.
Actor-Critic: Actor-critic methods combine an explicit policy with a value function that supplies learning feedback, trading policy-gradient variance reduction against value-function bias.The critic uses value estimates as a baseline for policy-gradient updates.

D. Planning and Learning

Planning uses an environment model to produce or improve a policy, while model-free reinforcement learning learns directly from interactions. Learned models can improve sample efficiency but introduce model-error risks and added complexity.

Planning: Planning is any method that uses an environment model to produce or improve a policy.With a model, agents can use dynamic programming, heuristic trajectory search, or exhaustive search.
Learning: Model-free RL learns directly from environment interactions, whereas model-based RL simulates transitions using a learned model.Model-based methods are especially useful when real interactions are expensive because simulation can increase sample efficiency.
Learning: Learning an environment model adds complexity and exposes the learned policy to model errors.Model predictive control partly addresses this risk by replanning after short action sequences in the real environment.

E. The Rise of DRL

Deep reinforcement learning applies neural-network representation learning and function approximation to high-dimensional RL problems, including visual game playing and robotics. The survey reviews these developments through value-based methods beginning with DQN and related stabilisation techniques.

E. The Rise of DRL: Deep neural networks help DRL handle high-dimensional problems by learning compact feature representations and approximating functions.This representation learning helps address the curse of dimensionality relative to tabular and traditional non-parametric methods.
E. The Rise of DRL: The survey focuses on state-of-the-art and historical DRL techniques, especially methods whose state spaces are conveyed through images and video.Its value-function discussion begins with the deep Q-network.
The DQN: DQN achieved Atari scores comparable to those of a professional video-game tester using four stacked greyscale frames processed by convolutional and fully connected layers.The convolutional layers extract spatiotemporal features, while later layers encode action effects.
The DQN: DQN was demonstrated to operate directly from raw visual inputs across a wide variety of environments, selecting the best discrete action from one network forward pass.Its learned features jointly encode objects, their movements, and interactions.
The DQN: Saliency analysis can reveal which visual elements a trained DQN attends to by backpropagating the training signal into image space.In the illustrated Space Invaders frame, salient regions include a recently fired laser and an anticipated target.
The DQN: DQN uses experience replay and a frozen target network to address instability from function approximation in reinforcement learning.The target network is periodically updated to match the policy network, while replay stores and resamples prior transitions.

B. Q-Function Modifications

Q-function modifications address estimation bias, represent the distribution of returns, or decompose values into state and advantage components. Other adaptations use learned action representations or structured policies for challenging action spaces.

B. Q-Function Modifications: Double-Q learning reduces the overestimation caused by using a single estimator and the maximum action value in Q-learning.It replaces the single estimator with a double estimator for a better expected-return estimate.
B. Q-Function Modifications: Categorical DQN learns the full value distribution rather than only its expectation, exposing skewed or multimodal reward information.The distributional approach can potentially benefit other algorithms that use learned value functions.
B. Q-Function Modifications: Duelling DQN decomposes Qπ into state-value Vπ and advantage Aπ components, making relative action values easier to learn.The state-value function provides a baseline while the advantage function represents action differences.
B. Q-Function Modifications: Action embeddings and autoregressive policies address large or combinatorial action spaces by representing actions compactly or predicting them conditionally.Sequential DQN discretises large action spaces and is reported to outperform NAF, whose action representation is limited by its quadratic form.

V. POLICY SEARCH

Policy search methods optimize policies directly, using gradient-free or gradient-based approaches. Deep reinforcement learning commonly relies on backpropagation, trust-region constraints, actor-critic value estimates, and asynchronous parallelism to improve stability, efficiency, and scalability.

Policy-search foundations: Policy search directly optimizes policies with gradient-free or gradient-based methods, including evolutionary algorithms that evaluate populations of agents.Evolutionary methods support arbitrary non-differentiable models and broader parameter-space exploration but can be expensive for large populations.
Backpropagation through stochastic functions: Backpropagation remains the workhorse of deep reinforcement learning, including for stochastic policies and stochastic computation graphs.REINFORCE and reparameterisation allow neural networks to be optimized through stochastic functions.
Trust-region methods: Trust regions constrain policy updates so new policies do not deviate excessively from previous ones, reducing the chance of catastrophically bad updates.TRPO uses a surrogate objective and has been shown to be relatively robust with high-dimensional inputs.
Actor-critic methods: Actor-critic methods combine policy search with learned value functions that can use full returns or temporal-difference errors.They can benefit from techniques such as generalized advantage estimation and target networks, and have scaled to robotic visual navigation.
Asynchronous methods: Asynchronous advantage actor-critic trains policy and value networks in parallel across multiple processing threads and independent environments.Parallel computation distributes updates across cores or machines and can improve performance while reducing training time.
Actor-critic extensions: Experience replay and off-policy corrections extend actor-critic methods by improving sample complexity while combining on-policy stability with off-policy data efficiency.Retrace(λ) bias correction enables experience replay in a Q-value-based A3C variant.

VI. CURRENT RESEARCH AND CHALLENGES

Current DRL research addresses model-based learning, exploration, hierarchical structure, and successor representations. These approaches seek greater data efficiency, better use of imperfect models, and more structured decision-making, while deep models remain sample-intensive.

Model-based RL: Model-based reinforcement learning learns transition models that simulate environments, reducing real-environment interactions and trading off exploration with exploitation.Prior knowledge, such as physics-based models, can be incorporated to speed learning.
Model-based RL: Neural-network models can provide rollout information while learning to downplay predictions when the model is inaccurate.This approach may be more efficient, though less principled, than Bayesian uncertainty propagation.
Model-based RL: Deep models typically require many samples to tune their large parameter sets, making data efficiency an important challenge for model-based DRL.Locally linear models have been used with NAF to improve sample complexity when robotic interactions are expensive.
Successor representations: Successor representations replace transition-model learning with expected discounted future occupancies that can be linearly combined with rewards to calculate optimal actions.This decomposition is more robust than model-free methods when reward structures change, but remains fallible when transitions change.

B. Exploration vs. Exploitation

DRL exploration research addresses the tension between trying uncertain actions and exploiting known good ones. Proposed solutions include bootstrapped value estimates, uncertainty bonuses, intrinsic motivation, hierarchical policies, and learning beyond expert trajectories.

Exploration versus exploitation: Exploration versus exploitation requires agents to balance trying potentially useful actions against exploiting actions currently judged optimal.DQN commonly uses ε-greedy exploration, selecting a random action with probability ε.
Exploration strategies: Bootstrapped DQN maintains multiple Q-value heads trained with different initializations and replay samples to support exploration.Policy-space exploration can instead add noise to a single output head and use zero-order gradient estimates.
Exploration strategies: Upper confidence bound exploration selects actions by maximizing E[R] + κσ[R], favoring regions with uncertainty and moderate expected return.UCB implements optimism in the face of uncertainty and can be viewed as intrinsic motivation.
Hierarchical reinforcement learning: Hierarchical reinforcement learning organizes policies into higher-level and lower-level components, allowing top-level policies to pursue goals while subpolicies provide fine control.The options framework treats multi-timestep policies as actions alongside primitive actions.
Imitation learning: Behavioural cloning can fail to adapt to new situations because small deviations from demonstrations may compound until the policy cannot recover.RL fine-tuning and active learning provide more generalisable alternatives by learning from states away from optimal trajectories.
Inverse reinforcement learning: Inverse reinforcement learning estimates an unknown reward function from observed trajectories and can be combined with RL to improve demonstrated behaviour.Deep neural networks enable complex, nonlinear reward functions for inverse reinforcement learning.

E. Multi-agent RL

DRL extends beyond single-agent and feedforward settings through communication, recurrence, attention, memory, auxiliary tasks, and transfer learning. These methods address non-stationarity, temporal dependencies, visual processing, and the high sample demands of real-world learning.

Multi-agent RL: Multi-agent reinforcement learning studies multiple learning agents and the non-stationarity caused by other agents changing their behaviours.DRL research emphasizes differentiable communication to enable cooperation.
Memory and attention: Recurrent and attention-based DQN variants integrate temporally distant observations and selectively focus on inputs for partially observable or long-horizon tasks.DARQN beat DQN and DRQN on games requiring longer-term planning, while DQN performed better on games requiring quick reactions.
Memory and attention: Differentiable memory adds separately addressable storage with differentiable read and write operations, extending recurrent processing for flexible information handling.RNNs have also been applied to policy-search and actor-critic methods.
Transfer learning: Real-world visual RL is often impractical because training directly from visual inputs requires many samples, motivating transfer, multitask, and curriculum learning.These approaches exploit knowledge from related tasks to speed learning.
Transfer learning: Auxiliary tasks can train features that make the RL objective easier to optimize, including pixel control, reward prediction, value learning, and depth prediction.An ablation found predicting depth more useful than receiving depth as an additional input.
Transfer learning: Multitask and transfer learning can improve DRL sample efficiency and robustness while supporting policies that are more data- and parameter-efficient.Distillation transfers policies from large or multiple DQNs to smaller or unified networks.

H. Benchmarks

DRL benchmarks evolved from small control tasks toward richer environments that test high-dimensional perception and continuous control. Shared platforms such as OpenAI Gym support standardisation and reproducibility, while substantial generalisation and theoretical challenges remain.

Cartpole and Mountain Car emerged as standard benchmarks, but their small state spaces poorly represented realistic complexity.
The Atari 2600 environment broadened evaluation through more than 50 varied video-game tasks with challenging human-relevant objectives.
MuJoCo supports relatively realistic multi-joint continuous-control problems, alongside efforts to standardise such evaluations.
OpenAI Gym improves standardisation and reproducibility by enabling shared access to RL domains and public result dissemination.
Despite DRL successes, broad deployment in complex real-world settings remains constrained by generalisation and limited theoretical understanding.Causal-model approaches achieved superior generalisation over standard DRL in some benchmarks, while deeper integration with traditional AI may improve sample complexity, generalisation, and interpretability.
RL's interaction with the environment provides a way for agents to choose experiments while learning about changing, complex worlds.