PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning: An Overview

Yuxi Li

arXiv:1701.07274v6cs.LG

TL;DR

The paper addresses the need for a broad account of deep reinforcement learning and its rapidly expanding achievements. It surveys foundational elements, mechanisms, applications, and resources, highlighting breakthroughs including DQN and AlphaGo. The overview also identifies instability concerns and boundaries around applying deep RL in settings such as finance.

  • Problem

    The overview addresses the need to organize and explain recent deep reinforcement learning achievements across foundations, mechanisms, applications, and resources.

  • Method

    The paper provides a structured survey of reinforcement learning fundamentals, core elements, learning mechanisms, applications, unresolved topics, and supporting resources.

  • Results

    The overview reports breakthroughs including Deep Q-Network and AlphaGo, along with advances in architectures, learning methods, and applications.

  • Takeaways & Limitations

    Deep reinforcement learning spans value-based and policy-based methods, emerging learning mechanisms, and applications across games, robotics, language, vision, and other domains.

  • Takeaways & Limitations

    Combining off-policy learning, function approximation, and bootstrapping can produce instability and divergence, known as the deadly triad issue.

Abstract

from arXiv · show

We give an overview of recent exciting achievements of deep reinforcement learning (RL). We discuss six core elements, six important mechanisms, and twelve applications. We start with background of machine learning, deep learning and reinforcement learning. Next we discuss core RL elements, including value function, in particular, Deep Q-Network (DQN), policy, reward, model, planning, and exploration. After that, we discuss important mechanisms for RL, including attention and memory, unsupervised learning, transfer learning, multi-agent RL, hierarchical RL, and learning to learn. Then we discuss various applications of RL, including games, in particular, AlphaGo, robotics, natural language processing, including dialogue systems, machine translation, and text generation, computer vision, neural architecture design, business management, finance, healthcare, Industry 4.0, smart grid, intelligent transportation systems, and computer systems. We mention topics not reviewed yet, and list a collection of RL resources. After presenting a brief summary, we close with discussions. Please see Deep Reinforcement Learning, arXiv:1810.06339, for a significant update.

1 INTRODUCTION

Deep reinforcement learning combines deep neural networks with reinforcement learning and has recently produced breakthroughs such as DQN and AlphaGo. This overview organizes the field around core elements, mechanisms, applications, and supporting resources while remaining broad rather than complete.

  • Deep reinforcement learning combines deep neural networks with reinforcement learning, benefiting from recent advances in data, computation, algorithms, software, and financial support.
  • Recent breakthroughs include Deep Q-Network and AlphaGo, alongside novel architectures and applications across games, robotics, language, and other fields.
  • Representation learning enables automatic feature engineering and end-to-end gradient-based learning, reducing reliance on manually designed domain knowledge.
  • The overview covers background, six core RL elements, six mechanisms, applications, unresolved topics, and a collection of books, surveys, courses, and other resources.
  • The overview focuses on contemporary work from recent years, is not complete, and plans deeper analysis in future refinements.

2 BACKGROUND

The section introduces machine learning, deep learning, and reinforcement learning, then outlines their objectives, representations, and optimization foundations. It also highlights RL convergence results and the instability known as the deadly triad.

  • Machine Learning: Machine learning learns from data to make predictions or decisions, using datasets, loss functions, optimization procedures, and models.
  • Machine Learning: Supervised learning uses labeled data, unsupervised learning uses unlabeled data, and reinforcement learning uses evaluative feedback without supervised signals.
  • Machine Learning: Unsupervised learning extracts information from unlabeled data, while representation learning seeks simpler representations that preserve information from the original data.
  • Deep Learning: Deep learning composes multiple hidden-layer transformations and can serve as a function approximator within reinforcement learning.
  • Reinforcement Learning: Reinforcement learning models an agent interacting with an environment, selecting actions from states, receiving rewards, and transitioning according to environment dynamics.
  • Reinforcement Learning: TD-learning, Q-learning, and SARSA converge under certain conditions, but combining off-policy learning, function approximation, and bootstrapping can cause instability and divergence.

3 CORE ELEMENTS

The overview presents core reinforcement-learning elements and surveys deep-RL methods that use neural networks to approximate value functions, policies, or models. It highlights DQN and extensions that stabilize learning, reduce domain-knowledge requirements, and improve performance across tasks.

  • Core elements: Deep RL uses deep neural networks to approximate value functions, policies, or environment models within reinforcement-learning problems.The overview contrasts this with shallow RL, which uses approximators such as linear functions, decision trees, or tile coding.
  • Deep Q-Network: DQN stabilizes nonlinear action-value approximation with experience replay and a target network, using pixels and game scores as inputs.Its end-to-end design requires only minimal domain knowledge.
  • Deep Q-Network: DQN used one algorithm, architecture, and hyperparameter setting across 49 Atari games, outperforming previous algorithms and performing comparably to a human professional tester.The reported result concerns broad cross-task performance rather than a single game.
  • DQN extensions: Double DQN addresses Q-learning’s over-estimation problem by separating action selection from value evaluation, and it found better policies than DQN on Atari games.The standard max operator can select over-estimated values because the same values both select and evaluate actions.
  • DQN extensions: Prioritized experience replay samples transitions more frequently according to TD errors and improved DQN and Double DQN performance on Atari games.Importance sampling is used to avoid bias in the update distribution.
  • DQN extensions: Dueling networks separately estimate state value and action advantage before combining them into action values; with Double DQN and prioritized replay, they improved prior Atari results.The architecture uses separate fully connected streams after a convolutional layer.

4 IMPORTANT MECHANISMS

The overview presents attention and memory, unsupervised learning, GANs, and neural architecture search as mechanisms that extend deep reinforcement learning beyond standard reward-driven learning. These mechanisms provide memory-based computation, auxiliary training signals, generative modeling, and RL-guided architecture design.

  • Attention and Memory: Attention focuses computation on salient parts, while memory stores data over long periods and provides an addressable substrate.The overview describes attention as a mechanism for focusing and memory as long-term data storage; attention can address memory.
  • Attention and Memory: Differentiable neural computers use neural controllers with external read-write memory to represent and manipulate complex data structures end-to-end.DNCs support long-term storage and were applied to question answering, shortest-path finding, family-tree inference, and a moving-blocks puzzle.
  • Unsupervised Learning: Unsupervised and auxiliary learning exploit non-reward signals, such as pixel changes or pseudo-rewards, to improve representations and learning efficiency, especially when extrinsic rewards are rare.UNREAL combines an A3C base agent with pixel control, reward prediction, value-function replay, and a shared representation.
  • Generative Models: GANs train a generative model and discriminator in an adversarial minimax process, and sufficient capacity can recover the data-generating distribution.The generator captures the data distribution while the discriminator estimates whether samples come from training data rather than the generator.
  • Neural Architecture Design: Neural architecture search uses an RL-trained RNN controller to generate variable-length architectures, rewarding validation accuracy.The method achieved competitive CIFAR-10 results and better Penn Treebank language-modeling results than state-of-the-art methods.

5 APPLICATIONS

The overview surveys deep RL applications across games, robotics, NLP, and many industrial and scientific domains. AlphaGo illustrates how deep networks, reinforcement learning, and Monte Carlo tree search can achieve superhuman Go performance, while also exposing scope limitations.

  • Applications: Deep RL applications span games, robotics, NLP, computer vision, business, finance, healthcare, Industry 4.0, smart grids, transportation, and computer systems.
  • Games: Computer Go is difficult because its search space is enormous and position evaluation is hard.
  • AlphaGo: AlphaGo defeated Lee Sedol 4 games to 1 and later defeated Ke Jie 3:0, establishing a landmark in AI.
  • AlphaGo: AlphaGo combines convolutional neural networks, supervised and reinforcement learning, and Monte Carlo tree search in training and search phases.
  • AlphaGo Zero: AlphaGo Zero learns from random self-play without human data, uses raw board stones without manual features, and combines policy and value in one network.
  • Limitations: Directly applying AlphaGo algorithms to healthcare, robotics, and self-driving is nontrivial because those settings lack abundant data and sufficiently accurate models.

DEEPSTACK

The overview places DeepStack among game applications addressing imperfect information through recursive reasoning and focused computation. Other game examples use deep RL for difficult, partially observable, or multi-agent environments.

  • DeepStack: DeepStack defeated professional poker players by using recursive counterfactual reasoning to handle information asymmetry.
  • DeepStack: DeepStack focuses computation on decision-specific situations and trains value functions automatically with little domain knowledge or human expert data.
  • Video games: Video games are presented as potential testbeds for artificial general intelligence.
  • Video games: A3C with CNNs trained an agent in partially observable 3D Doom using recent frames, game variables, curriculum learning, and auxiliary prediction of actions and values.
  • Multi-agent games: A multi-agent actor-critic framework used dynamic grouping and parameter sharing to improve scalable coordination among StarCraft agents.

5.2 ROBOTICS

The robotics examples use reinforcement learning to connect perception, control, navigation, and data-efficient representation learning. Guided policy search converts policy search into supervised learning, while auxiliary tasks help navigation under sparse rewards.

  • Guided policy search: Guided policy search jointly trains perception and control end-to-end, mapping raw images directly to robot motor torques.
  • Guided policy search: Guided policy search alternates trajectory-centric reinforcement learning with supervised learning to improve data efficiency under unknown dynamics.
  • Guided policy search: Guided policy search achieved good performance on real-world manipulation tasks involving localization, visual tracking, and complex contact dynamics.
  • Navigation: Navigation learning maximizes cumulative reward while adding unsupervised depth reconstruction and self-supervised loop-closure classification.
  • Navigation: Navigation emerges as a by-product of goal-directed reinforcement learning rather than explicit position inference and mapping used by conventional SLAM.

5.3 NATURAL LANGUAGE PROCESSING

The NLP survey covers dialogue, translation, and text generation, emphasizing reinforcement learning for long-term objectives, user feedback, and sequence-level evaluation. It also situates RL alongside broader deep-learning and linguistically informed approaches.

  • Scope: NLP applications discussed include dialogue systems, machine translation, text generation, semantic parsing, question answering, summarization, and information retrieval.
  • Perspective: The survey presents NLP as involving synergy between deep and non-deep methods and between end-to-end approaches and linguistic knowledge.
  • Dialogue systems: Dialogue systems use reinforcement learning for task completion, knowledge access, policy learning, and learning from user feedback.
  • Dialogue systems: Deep RL dialogue generation addresses myopia, reward misalignment, and repetitive responses by optimizing a designed long-term reward with policy gradients.
  • Machine translation: Dual learning for machine translation uses feedback between opposite translation directions and policy gradient with language-model likelihood as reward.
  • Text generation: Text-generation models face exposure bias because training uses ground-truth prefixes while testing feeds generated words back into the model.
  • Text generation: MIXER aligns sequence-level training with testing objectives such as BLEU by combining REINFORCE and cross-entropy losses.

5.4 COMPUTER VISION

The overview presents deep reinforcement learning as a tool for computer vision tasks involving recognition, localization, tracking, scene understanding, physical reasoning, and vision-language integration. Across these tasks, RL supports sequential attention, search, decision-making, and interaction under computational or perceptual constraints.

  • Computer vision applications span recognition, motion analysis, scene understanding, NLP integration, and visual control.
  • Interactive and multimodal vision: RL also supports interactive perception and vision-language tasks, including object dynamics, grasp planning, manipulation learning, image captioning, and reward-based sequence training.
  • Recognition and localization: RL can improve image-classification and detection efficiency by selecting salient regions and stopping searches once localization confidence is sufficient.
  • Recognition and localization: The recurrent attention model uses REINFORCE to train nondifferentiable attention over selected image or video regions for classification and object detection.
  • Recognition and localization: Deep RL methods formulate object localization as sequential search, using DQN actions or tree-structured paths to maximize long-term localization accuracy.
  • Tracking and prediction: Tracking methods use deep RL policies in partially observable settings to decide where to look, when to reinitialize, and when to update object appearance models.
  • Scene understanding and physical reasoning: Scene-understanding approaches combine structured representations with rendering or physics engines, using REINFORCE variants to handle nondifferentiable graphics operations.

5.5 BUSINESS MANAGEMENT

The overview describes business-management applications of RL in recommendation, advertising, and customer interaction. These formulations optimize sequential, long-term user outcomes rather than only immediate responses.

  • RL applications in business management include advertising, recommendation, customer management, and marketing.
  • Recommendation: Personalized news recommendation is formulated as a contextual bandit that uses user and article context plus click feedback to maximize long-run total clicks.
  • Advertising and customer value: Personalized advertising can be formulated as RL to maximize customer lifetime value, contrasting with myopic supervised or contextual-bandit optimization of click-through rate.
  • Advertising and customer value: The overview notes concurrent reinforcement learning for customer interaction and related attempts to maximize customer lifetime value.

5.6 FINANCE

The overview surveys RL applications in finance and economics, including pricing, portfolio optimization, trading, risk management, and adaptive-market analysis. It also highlights institutional resistance to neural-network black boxes and broader machine-learning opportunities.

  • Financial applications: RL has been applied to option pricing and multi-period portfolio optimization, including value-function-based methods.
  • Financial applications: Policy search and deep neural networks have been used for trading, while deep RL is described as offering potential solutions for some risk-management issues.
  • Financial applications: The overview connects reinforcement learning with behavioral finance through the adaptive markets hypothesis, which may be approached using RL.
  • Challenges and opportunities: A key adoption boundary is that finance and economics academia finds black-box neural-network methods difficult to accept.
  • Challenges and opportunities: Machine learning opportunities in finance include fraud detection and consumer credit-risk assessment, while healthcare includes personalized medicine and dynamic treatment regimes.

5.10 SMART GRID

The supplied passages describe smart-grid demand response as adaptive control under changing electricity prices and environmental signals. They also place RL within broader optimization applications spanning transportation, industrial systems, cloud computing, and computer systems.

  • Smart-grid foundations: Smart grids use information technologies for electricity generation, transmission, distribution, consumption, and control, with adaptive control as an important aspect.
  • Demand response: Demand-response systems dynamically adapt electricity use to grid signals such as prices, temperature, and weather to improve efficiency, reduce costs, and reduce risks.
  • Adjacent control applications: The overview situates RL applications alongside adaptive traffic-signal control, where multi-agent RL addresses a multiple-player stochastic game.
  • Adjacent control applications: Multi-agent RL combines single-agent RL with game theory but faces stability, nonstationarity, and dimensionality challenges.
  • Computer-system optimization: Deep RL has been used for multi-resource cluster scheduling, cloud resource allocation, server power management, and device placement in computational graphs.
  • Computer-system optimization: A hierarchical cloud-computing framework separates virtual-machine allocation from server power management, reducing state/action space and enabling distributed operation.
  • Computer-system optimization: Google deployed machine learning for data-centre power management and reported a 40% reduction in energy consumption.

6 MORE TOPICS

The overview identifies numerous important deep learning and reinforcement learning topics that it does not cover, leaving them as pointers for future study and work.

  • Unreviewed topics: The overview leaves topics including compression, optimization, normalization, curriculum learning, and new Q-value operators for future study.These topics are listed among the areas not discussed in the overview.
  • Unreviewed topics: Additional omitted topics cover interpretability, deep probabilistic programming, active learning, program learning, relational reasoning, proving, and music generation.The list includes both methodological topics and application-oriented directions.
  • Unreviewed topics: Unreviewed directions also include large action spaces, predictive state representations, safe RL, agent modelling, and semi-supervised learning.The list spans both reinforcement learning-specific issues and broader machine learning directions.
  • Unreviewed topics: The overview further points to neural episodic control, continual learning, symbolic learning, PathNet, evolution strategies, capsules, and DeepForest.These entries extend the future-work list across memory, transfer, representation, and alternative learning approaches.
  • Unreviewed topics: The remaining pointers include retrosynthesis, quantum RL, usable machine learning, expressivity, testing, and deep learning efficiency.The authors explicitly state that completing these reviews would be desirable but leave them as future work.

7 RESOURCES

The overview assembles learning materials, communities, publications, blogs, benchmarks, software frameworks, and algorithm implementations for studying and applying deep reinforcement learning.

  • Resource types: The resource collection spans books, surveys, reports, courses, tutorials, conferences, journals, workshops, blogs, testbeds, and open-source implementations.The authors note that the collection is not complete.
  • Books and courses: Sutton and Barto’s reinforcement learning textbook, together with courses by Rich Sutton and David Silver, is recommended as an entry point.The textbook covers RL fundamentals and newer progress including DQN, AlphaGo, and policy gradients.
  • Books and surveys: The bibliography includes dedicated resources for deep learning, theoretical RL, Markov decision processes, machine learning, AI, NLP, semi-supervised learning, and game theory.It also lists resources on deep reinforcement learning and related application areas.
  • Online resources: Community resources include research blogs, newsletters, and video lectures from researchers and organizations such as DeepMind, Google Brain, OpenAI, Sutton, Silver, and Levine.The listed blogs include The Wild Week in AI and several research group sites.
  • Courses and venues: The overview points readers to courses, summer schools, workshops, and conferences such as NIPS, ICML, ICLR, RLDM, EWRL, AAAI, IJCAI, ACL, and ICRA.These venues and events provide additional educational and research materials.
  • Tools and implementations: Practical tools include ALE, RLlib, OpenAI Gym, RLGlue, RLPy, code repositories, DQN implementations, OpenAI Baselines, and TensorFlow or Keras projects.These resources support experimentation, comparison, education, and implementation.

8 BRIEF SUMMARY

The brief summary organizes reinforcement learning issues alongside representative approaches, emphasizing methods for reward, exploration, planning, attention, memory, transfer, hierarchy, adaptation, search, stability, and efficiency.

  • Reward and exploration: The summary maps unavailable reward functions to imitation learning, inverse RL, demonstrations, GAN-based imitation, and jointly trained dialogue reward models.These approaches address learning when explicit reward signals are unavailable.
  • Reward and exploration: Exploration–exploitation is addressed through count-based exploration, intrinsic motivation, bootstrapped DQN, and variational information maximization.The listed methods represent multiple strategies for improving exploration.
  • Models and planning: Model-based learning and model-free planning are represented by Dyna-Q, hybrid model-free/model-based RL, value iteration networks, and the predictron.The overview presents these as proposed approaches to model and planning issues.
  • Mechanisms: Attention, external memory, unsupervised learning, transfer learning, semi-supervised learning, and hierarchical RL address salience, long-term storage, auxiliary signals, cross-domain knowledge, mixed labels, and abstraction.The summary pairs each issue with a corresponding mechanism or representative approach.
  • Adaptation: Learning to learn targets rapid adaptation through flexible recurrent models, learned optimization, and one-, few-, or zero-shot learning.The listed approaches concern families of tasks and rapid adaptation to new tasks.
  • Search, stability, and efficiency: AlphaGo addresses gigantic search spaces by integrating supervised learning, reinforcement learning, and Monte Carlo tree search, while neural architecture search addresses architecture design.The summary also lists DQN stabilization techniques and guided policy search for end-to-end perception and control.

9 DISCUSSIONS

The authors review deep RL’s rapid progress and broad scope while acknowledging that the overview is incomplete and that important technical and practical challenges remain.

  • Scope boundary: The overview is incomplete in both depth and breadth because deep RL has been developing rapidly.The authors present the work as a summary of important achievements and potential directions rather than a complete account.
  • Scope and contribution: The overview covers six core elements, six mechanisms, twelve applications, background material, and RL resources.Its organization spans value functions, policies, rewards, planning, exploration, learning mechanisms, and applications.
  • Achievements: Deep RL breakthroughs include DQN and AlphaGo, while novel architectures and applications have received best-paper recognition at major conferences.Examples include dueling networks, spoken dialogue systems, information extraction, and value iteration networks.
  • Methods: The authors characterize value functions as central while noting growing use of policy optimization across robotics, language, attention, and learning-to-learn applications.They also describe transfer, unsupervised, and semi-supervised learning as emerging mechanisms.
  • Open challenges: The authors identify stability, convergence, accuracy, data efficiency, scalability, speed, simplicity, interpretability, robustness, and safety as essential issues for learning models.They also call for investigation of cognitive-science concerns such as intuitive physics, causal models, compositionality, and real-time action.
  • Applications and outlook: RL has many potential applications but few products so far, and may still need better algorithms, products, and market validation.The discussion frames this as a current development and commercialization boundary.

ACKOWLEDGEMENT

The authors acknowledge comments and seminar discussions that contributed to the work.

  • The authors thank numerous colleagues, seminar participants, and webinar attendees for their comments and discussions.
Loading 1701.07274v6…