PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning

Yuxi Li

arXiv:1810.06339v1cs.LGstat.ML

TL;DR

Deep RL combines deep neural networks with reinforcement learning, but instability remains an open issue when off-policy learning, function approximation, and bootstrapping are combined. This overview traces methods and applications, reporting a roughly tenfold sample-efficiency increase at each step across a progression of RL method families.

  • Problem

    When off-policy learning, function approximation, and bootstrapping are combined, instability and divergence may occur, and their root cause remains unclear.

    Source: §2.4.9 · p. 19

  • Method

    The overview traces deep RL methods and applications against the backdrop of the integration of reinforcement learning and neural networks.

    Source: §1 · p. 5, §4.1 · p. 31, ISSUES IN GAME THEORY · p. 64

  • Results

    Sample efficiency improves roughly ten times at each step from gradient-free methods through model-based “shallow” RL methods.

    Source: §21.1 · p. 103

  • Takeaways & Limitations

    Deep RL has achieved human-level or superhuman performance in many two-player games, while reinforcement learning still needs more effort to become practical.

    Source: §15 · p. 76, Research Direction 5: AutoRL · p. 108

    Combining off-policy learning, function approximation, and bootstrapping may cause instability or divergence, and the root cause is unclear.

    Source: §2.4.9 · p. 19

Abstract

from arXiv · show

We discuss deep reinforcement learning in an overview style. We draw a big picture, filled with details. We discuss six core elements, six important mechanisms, and twelve applications, focusing on contemporary work, and in historical contexts. We start with background of artificial intelligence, machine learning, deep learning, and reinforcement learning (RL), with resources. Next we discuss RL core elements, including value function, policy, reward, model, exploration vs. exploitation, and representation. Then we discuss important mechanisms for RL, including attention and memory, unsupervised learning, hierarchical RL, multi-agent RL, relational RL, and learning to learn. After that, we discuss RL applications, including games, robotics, natural language processing (NLP), computer vision, finance, business management, healthcare, education, energy, transportation, computer systems, and, science, engineering, and art. Finally we summarize briefly, discuss challenges and opportunities, and close with an epilogue.

6 Model

This section introduces deep reinforcement learning and its foundations, including deep-learning architectures, representation learning, and the setup and core concepts of reinforcement learning. It surveys and compares major policy-evaluation and improvement methods, model-based learning, function approximation, policy optimization, and resources, including the instability risk of combining off-policy learning, function approximation, and bootstrapping.

1 INTRODUCTION

  • Deep RL combines deep neural networks with trial-and-error decision making, with landmark successes including DQN, AlphaGo, and DeepStack, but retains interpretability, tuning, and reproducibility challenges.Deep representations enable automatic feature engineering and end-to-end learning, reducing reliance on domain knowledge for some problems.Source: §1 · pp. 5–6

2 BACKGROUND

  • Deep RL lies at the intersection of deep learning and reinforcement learning; deep learning is part of machine learning, which is part of AI.The fields evolve, and deep learning and deep RL also address classical AI problems.Source: §2 · p. 8

2.1 ARTIFICIAL INTELLIGENCE

  • AI is framed through four lenses—acting or thinking humanly or rationally—with rational-agent design presented as more amenable to scientific development.Its foundations include philosophy, mathematics, economics, neuroscience, psychology, computer engineering, control theory, and linguistics.Source: §2.1 · pp. 8–9

2.2 MACHINE LEARNING

  • Machine learning learns from experience; supervised, unsupervised, and reinforcement learning differ in labels, while RL uses evaluative feedback for sequential decisions and adds credit-assignment, stability, and exploration challenges.Unlike supervised and unsupervised learning, which are usually one-shot and myopic, RL considers long-term accumulative rewards.Source: §2.2 · p. 10

2.3 DEEP LEARNING

  • Deep learning uses hidden layers and nonlinear transformations to build representations, then backpropagates gradients to update weights against a loss function.It can be integrated with RL as a state representation or function approximator.Source: §2.2 · p. 10, §2.3 · pp. 11–12
  • Deep networks compose layers to learn representations from inputs; CNNs process structured signals, ResNets ease deep-network training, and LSTMs and GRUs use gates to retain sequential information.Dropout trains an ensemble of subnetworks, while batch and layer normalization aim to improve training efficiency.Source: §2.3 · p. 12

2.4 REINFORCEMENT LEARNING; 2.4.1 PROBLEM SETUP

  • In RL, an agent selects actions from states, receives rewards, and seeks to maximize discounted return; MDPs formalize Markov problems, while prediction evaluates policies and control seeks an optimal one.On-policy methods learn about the behavior policy, off-policy methods can learn a different target policy, and model-based methods use a known or learned transition-and-reward model.Source: §2.4 · pp. 12–13, §2.4.1 · p. 13

2.4.2 VALUE FUNCTION

  • Value functions predict expected discounted future reward for states or state-action pairs, while optimal value functions give the maximum achievable value under any policy.In a shortest-path problem, negative edge costs serve as rewards, and an optimal policy minimizes total distance to the destination.Source: §2.4.2 · pp. 13–14

2.4.3 EXPLORATION VS. EXPLOITATION

  • The exploration–exploitation dilemma balances trying uncertain actions against choosing the current best action; ϵ-greedy explores randomly with probability ϵ and otherwise acts greedily.Other approaches include optimistic initialization, upper confidence bounds, probability matching, and information state search (Silver, 2015).Source: §2.4.3 · p. 14

2.4.4 DYNAMIC PROGRAMMING

  • Dynamic programming assumes a known MDP model and uses Bellman backups: policy iteration alternates evaluation and improvement, while value iteration applies optimality backups to converge toward optimal values.Generalized policy iteration permits evaluation to stop before convergence; the backup operators’ contraction properties support convergence results (Silver, 2015; Sutton and Barto, 2018; Bertsekas and Tsitsiklis, 1996).Source: §2.4.4 · p. 15

2.4.5 MONTE CARLO

  • Monte Carlo methods estimate values from complete episodic returns without transition or reward models, and GLIE on-policy control converges to the optimal action-value function.Off-policy Monte Carlo can reuse behavior-policy experience to evaluate another policy, but importance-sampling corrections across an episode may greatly increase variance (Singh et al., 2000; Silver, 2015).Source: §2.4.5 · pp. 15–16

2.4.6 TEMPORAL DIFFERENCE LEARNING

  • Temporal-difference learning updates value estimates online from experience using bootstrapping; SARSA is on-policy control, whereas Q-learning is off-policy control.Bootstrapping can enable faster online and continual learning, but its target depends on the values being estimated, so the update is not true gradient descent.Source: §2.4.6 · pp. 16–17

2.4.7 MULTI-STEP BOOTSTRAPPING

  • Multi-step returns and eligibility traces extend one-step TD methods, with TD(λ) unifying TD(0) and Monte Carlo through a decay parameter λ.Eligibility traces provide short-term memory that helps with delayed rewards and non-Markov tasks; TD(1) is equivalent to Monte Carlo.Source: §2.4.7 · p. 17

COMPARISON OF DP, MC, AND TD

  • DP requires a model and bootstraps, whereas MC and TD are model-free; TD learns online from incomplete sequences, while MC uses complete episodes.TD generally has lower variance and some bias; MC has higher variance and zero bias, and is usually more effective in non-Markov environments.Source: COMPARISON OF DP, MC, AND TD · pp. 17–18

2.4.8 MODEL-BASED RL

  • Dyna-Q integrates learning, acting, and planning by updating from real environment experience and simulated trajectories generated by a learned model.The model predicts reward and next state for a state-action pair, which planning uses for additional Q updates.Source: §2.4.8 · p. 18

2.4.9 FUNCTION APPROXIMATION

  • Combining off-policy learning, function approximation, and bootstrapping can cause instability and divergence; the root cause remains unclear.All three elements serve useful purposes—scalability and generalization, computational and data efficiency, and separating behavior from target policies—so removing any one has costs.Source: §2.4.9 · p. 19

2.4.10 POLICY OPTIMIZATION

  • Policy-based methods directly optimize policy parameters by gradient ascent, often suit high-dimensional or continuous actions, and can learn stochastic policies, but may have high variance and local optima.REINFORCE uses returns, while actor-critic combines a critic that updates action values with an actor that updates the policy.Source: §2.4.10 · p. 20

2.4.11 DEEP RL

  • Deep RL uses deep neural networks for state or observation representation and/or value, policy, or model approximation, distinguishing it from shallow RL by the function approximator.The passage notes that DQN and AlphaGo stabilized learning and achieved outstanding results despite the instability risk.Source: §2.4.11 · p. 21

2.4.12 BRIEF SUMMARY

  • RL algorithms build on value functions, Bellman equations, and TD prediction, and vary by policy or value basis, model use, data policy, approximation, and backup type.The summary notes theoretical guarantees for linear-function-approximation methods including Gradient-TD and Emphatic-TD, while deep-network RL methods such as DQN and AlphaGo stabilize learning.Source: §2.4.12 · p. 22

2.5 RESOURCES

  • Recommended RL resources include foundational books and courses, Atari ALE and MuJoCo benchmarks, and toolkits or platforms such as OpenAI Gym, DeepMind Lab, and ELF.The passage also lists the DeepMind Control Suite, Dopamine, and ELF OpenGo among available resources.Source: §2.5 · p. 22

PART I: CORE ELEMENTS

The section frames RL around six core elements and distinguishes problem formulations and solution-method families. It contrasts value-based and policy-based methods while noting approaches that combine or bridge them.

PART I: CORE ELEMENTS

  • RL problems may be framed as prediction, control, or planning, with methods distinguished as model-free or model-based and value-based or policy-based.Source: PART I: CORE ELEMENTS · p. 23
  • TD methods such as Q-learning are value-based, direct policy search includes policy gradients and evolution methods, and actor-critic and maximum-entropy methods combine or bridge the families.The figure presents value- and policy-based RL methods.Source: PART I: CORE ELEMENTS · p. 23, Figure 3 · p. 23
  • RL’s six core elements are value function, policy, reward, model, exploration versus exploitation, and representation.Representation is relevant across RL elements.Source: PART I: CORE ELEMENTS · p. 23

3 VALUE FUNCTION

This section surveys value-based reinforcement learning, from value-function foundations to methods that stabilize, extend, and generalize value learning. It covers DQN and its extensions, distributional values, and goal-conditioned predictive learning.

3 VALUE FUNCTION

  • A value function predicts expected discounted future reward for a state or state-action pair; an optimal value function can be used to derive an optimal policy.Temporal-difference learning and Q-learning are classical methods for learning state and action values, respectively.Source: §3 · p. 24

3.1 DEEP Q-LEARNING

  • DQN’s deep convolutional Q-learning achieved outstanding results across many Atari games, outperforming previous algorithms and performing comparably to a human professional tester.The same algorithm, architecture, and hyperparameters were used across 49 games, but each game was trained separately.Source: §3.1 · pp. 24–25
  • DQN addresses instability in neural-network Q-learning with experience replay and a periodically updated target network, reducing data correlations and correlations between predictions and targets.Its end-to-end setup uses pixels and game score as inputs and requires minimal domain knowledge.Source: §3.1 · p. 24

DOUBLE DQN

  • Double DQN separates action selection by the online network from value evaluation by the target network to address Q-learning’s overestimation.Standard Q-learning uses the same values to select and evaluate an action, making overestimated values more likely to be selected.Source: DOUBLE DQN · p. 26

PRIORITIZED EXPERIENCE REPLAY

  • Prioritized replay samples transitions stochastically according to TD error, using importance sampling to reduce update-distribution bias; it improved DQN and Double DQN performance on Atari games.This contrasts with DQN’s uniform replay sampling.Source: PRIORITIZED EXPERIENCE REPLAY · p. 26

DUELING ARCHITECTURE

  • The dueling architecture estimates state value and action advantage in separate network streams, combines them into action values, and improved on DQN and Double DQN with prioritized replay on Atari.The authors propose replacing the maximum with an average for better stability.Source: DUELING ARCHITECTURE · p. 26

RAINBOW

  • Rainbow combines DQN, Double Q-learning, prioritized replay, dueling networks, multi-step learning, value distribution, and noisy nets, achieving better data efficiency and Atari performance.Removing prioritization or multi-step learning worsened performance for most games, while component contributions varied across games.Source: RAINBOW · p. 27

RETRACE

  • Retrace(λ) is a return-based off-policy control algorithm designed for low variance, safe use of samples from any behavior policy, and efficient use of samples from close behavior policies.Its convergence to the optimal action value is analyzed without assuming GLIE.Source: RETRACE · p. 27

4 POLICY

Policy optimization directly searches for a policy, unlike value-based methods that first optimize value functions. The overview traces policy gradients and actor-critic methods through trust-region and off-policy approaches, while noting their stability, data-efficiency, and reproducibility challenges.

4.1 POLICY GRADIENT

  • Policy gradients optimize policies directly, but Monte Carlo rollouts can make their estimates high-variance and sample-inefficient; critics or baselines can reduce variance, with possible added bias.DPG estimates gradients through backpropagation and is more efficiently estimated than stochastic policy gradients; its experiments report superiority particularly on high-dimensional tasks.Source: §4 · p. 30, §4.1 · p. 31

RECENT WORK

  • Off-policy learning can reuse trajectories and experience replay, while combining it with function approximation and bootstrapping can produce instability or divergence.Q-Prop, PGQL, and interpolated policy-gradient methods are presented as efforts to combine policy-gradient stability with off-policy sample efficiency; the section also surveys continuous-control benchmarks and the DeepMind Control Suite.Source: §4 · p. 30, RECENT WORK · p. 35, §4.5 · p. 35

4.1 POLICY GRADIENT

  • DDPG solved more than 20 simulated MuJoCo tasks with one algorithm and was competitive with model-informed planning, using 20 times fewer experience steps than DQN.It still required many training episodes, as do most model-free RL methods.Source: §4.1 · pp. 31–32

4.2 ACTOR-CRITIC

  • A3C ran much faster and performed better than or comparably with several DQN variants on Atari, and also succeeded in continuous motor control and visual navigation tasks.Parallel actors use different exploration policies; A3C does not use experience replay and can run on a multi-core CPU.Source: §4.2 · p. 33

TRUST REGION POLICY OPTIMIZATION (TRPO)

  • TRPO’s experiments report good performance on simulated swimming, hopping, and walking tasks, and on Atari from raw images.TRPO constrains gradient updates to stabilize policy optimization; its procedure optimizes a surrogate objective with theoretical monotonic-improvement guarantees before practical approximations.Source: TRUST REGION POLICY OPTIMIZATION (TRPO) · p. 33, §4.3 · p. 33

PROXIMAL POLICY OPTIMIZATION (PPO)

  • PPO alternates data sampling and optimization, aiming to retain TRPO’s stability with simpler implementation and better empirical sample complexity; it performs well on MuJoCo and Atari tasks.PPO can share policy and value parameters in a neural network and uses a truncated Generalized Advantage Estimator.Source: PROXIMAL POLICY OPTIMIZATION (PPO) · p. 34

4.4 POLICY GRADIENT WITH OFF-POLICY LEARNING

  • Maximum-entropy approaches combine policy gradients with off-policy learning; soft Q-learning uses a Boltzmann policy and a sampling network, while PCL learns multi-step path consistency from on- and off-policy traces.The overview associates maximum entropy RL with exploration, compositionality, and partial observability; soft Q-learning is described as a one-step special case of PCL.Source: §4.4 · p. 34, SOFT Q-LEARNING · p. 34, PATH CONSISTENCY LEARNING (PCL) · p. 34

4.5 BENCHMARK RESULTS

  • Deep RL results are sensitive to hyperparameters, random seeds, environments, and codebases, making reproducibility difficult.The reported influences include network architecture and reward scale.Source: §4.5 · p. 35

5 REWARD

Rewards evaluate an agent’s decisions, but may be sparse, unavailable, or misaligned with a designer’s intentions. The overview surveys reward learning, imitation from demonstrations, and reward decomposition as ways to address these challenges.

5 REWARD

  • When a reward function is unavailable, imitation learning can use expert trajectories through behavioral cloning or inverse reinforcement learning.Behavioral cloning maps expert state-action pairs to a policy without learning a reward function.Source: §5 · p. 36
  • A misspecified reward can omit important aspects of the designer’s intention and lead to poor behavior, as in the vacuum that ejects collected dust to earn more reward.The passage also illustrates the issue with King Midas’s wish that everything he touched turn to gold, despite his unstated needs for food and family.Source: §5 · p. 36

REWARD LEARNING

  • CIRL models reward alignment as a partial-information game in which a human knows the reward and a robot’s payoff is that human reward; its optimal solution can include active teaching and learning.The authors reduce the optimal policy-pair problem to a single-agent POMDP and report that optimality in isolation, as in apprenticeship learning and IRL, is suboptimal in CIRL.Source: REWARD LEARNING · p. 37

LEARNING FROM DEMONSTRATION

  • DQfD combines temporal-difference, supervised, and regularized losses, pretrains on demonstrations, then mixes self-generated samples with demonstration data; Atari experiments report better initial performance, more average rewards, and faster learning than DQN.GAIL instead learns policies directly from data, bypassing the intermediate reward-learning step of IRL.Source: LEARNING FROM DEMONSTRATION · pp. 37–38

REWARD MANIPULATING

  • HRA decomposes the reward into components and learns their value functions separately, enabling relatively low-dimensional representations for components; its Ms. Pac-Man experiments report above-human performance.The approach uses off-policy learning on sample trajectories in parallel.Source: REWARD MANIPULATING · p. 38

6 MODEL

Model-based RL uses learned environment models to support value or policy learning, while model-free methods avoid model identification but can require many samples. The overview surveys methods that combine learning and planning, including differentiable planning architectures.

6 MODEL

  • Model-based RL can improve data efficiency, but inaccurate learned models can limit performance; model-free RL may instead require many samples, which can be costly on physical systems.Dyna integrates learning and planning by using both real experience and simulated trajectories from a learned model.Source: §6 · p. 39

PLANNING

  • Planning methods include MCTS, which selects and expands nodes, evaluates a leaf, and backs up evaluations; learned planning architectures include MCTSnets, VIN, and the predictron.VIN is model-free and differentiable but its value iteration over the full state space with local transition dynamics limits its applicability; the predictron rolls an abstract model forward for value prediction in uncontrolled environments.Source: §6 · p. 39, PLANNING · p. 41

6 MODEL

  • SVG learns a model, value function, and policy jointly, using real environment trajectories for value gradients to mitigate compounded model errors; it uses the model for gradients, not prediction.Its formulation ranges from model-free methods with value functions to model-based methods without them.Source: §6 · p. 40
  • VPN predicts future values, rewards, and discounts in abstract states rather than predicting future observations, and uses lookahead planning to choose actions and compute target Q-values.TDMs offer another combination: model-free off-policy training with model-based control, aiming for sample efficiency while avoiding model bias.Source: §6 · p. 40
  • TreeQN uses a differentiable recursive tree with learned transitions, rewards, and values to estimate Q-values, encoding the MDP structure that ordinary fully connected DQN layers do not implement.The authors also propose the actor-critic variant ATreeC and evaluate both on box-pushing and Atari tasks.Source: §6 · p. 40

7 EXPLORATION VS. EXPLOITATION

The section surveys exploration–exploitation methods from bandits to deep RL, covering uncertainty-driven, count-based, curiosity-driven, and reward-directed approaches. It highlights different mechanisms for balancing information gathering with reward-seeking.

7 EXPLORATION VS. EXPLOITATION

  • Exploration trades current reward for information that can improve later decisions; bandit methods include optimism, probability matching, and information-state search.Source: §7 · p. 42
  • Optimistic initialization encourages early exploration by starting action values high, but may leave the agent selecting a suboptimal action.Source: §7 · p. 42
  • UCB achieves logarithmic asymptotic total regret, compared with linear total regret for ϵ-greedy and optimistic initialization (Auer, 2002).Source: §7 · p. 43
  • Thompson sampling implements probability matching by sampling a reward distribution from the posterior and choosing the action with the highest sampled value.Source: §7 · p. 43

COUNT-BASED METHODS

  • Pseudo-counts use a density model to extend count-based exploration to function approximation, linking information gain with intrinsic motivation and supporting exploration bonuses.A later study reports mixed Monte Carlo updates can facilitate exploration in sparse-reward settings such as Montezuma’s Revenge.Source: COUNT-BASED METHODS · p. 44

INTRINSIC MOTIVATION

  • VIME combines external reward optimization with intrinsic surprise, measuring information gain through variance inference and approximating beliefs about dynamics with Bayesian neural networks.Source: INTRINSIC MOTIVATION · p. 44

MORE WORK

  • Bootstrapped DQN uses parallel action-value estimates with separate target networks to implement Thompson-sampling exploration with deep neural networks.Source: MORE WORK · p. 45
  • UREX directs exploration toward action-sequence regions whose policy log-probability underestimates reward, balancing mode-seeking and mean-seeking terms (Nachum et al., 2017).Source: MORE WORK · p. 45

8 REPRESENTATION

Representation spans features, problem descriptions, knowledge, and reasoning across an RL task. The section surveys classical and neural representations while arguing that RL may require representations designed jointly for perception and control.

GENERAL VALUE FUNCTION (GVF)

  • The section frames representation as central to encoding and using knowledge, including predictive knowledge in general value functions and temporal abstraction in hierarchical RL.Relational RL integrates reinforcement learning with statistical relational learning for knowledge representation and reasoning.Source: §8 · p. 46, GENERAL VALUE FUNCTION (GVF) · pp. 50–51

8.1 CLASSICS

  • For large problems, function approximation replaces tabular representations; linear methods combine basis functions, while deep networks provide nonlinear approximations.Source: §8 · p. 46, §8.1 · p. 47

RL PROBLEM DESCRIPTION

  • POMDPs represent uncertainty over hidden states, PSRs represent states through observable action–observation predictions, and CDPs provide a framework for rich observations and systematic exploration.CDPs introduce Bellman rank and establish PAC learnability for several low-Bellman-rank problem classes.Source: RL PROBLEM DESCRIPTION · p. 47

STATE AND STATE-ACTION DISTRIBUTION

REPRESENTATION LEARNING

  • Representation learning seeks features that capture factors such as causality, sparsity, and temporal coherence; the section presents these as potential regularization principles for generalization.Source: REPRESENTATION LEARNING · p. 48

NEURAL NETWORK ARCHITECTURES

  • CNNs encode locality and translation invariance, while RNNs share recurrent-cell weights across time; recent work questions whether recurrence is best for sequence modeling.CNNs empirically outperform RNNs across a wide range of tasks in Bai et al. (2018); self-attention is proposed as an alternative for sequence transduction.Source: NEURAL NETWORK ARCHITECTURES · p. 48, CHALLENGES TO CNN AND RNN · p. 49

MEMORY

  • Memory-equipped networks support storage, inference, or reasoning, while recurrent networks trained for path integration develop grid-like representations that improve deep-RL navigation in challenging environments.Source: MEMORY · p. 49, GRID-LIKE REPRESENTATION · p. 49

HUMAN INTELLIGENCE

  • The discussion connects causal models, neural reasoning, and human-like learning, including compositionality and rapid adaptation to new tasks without retraining from scratch.Source: CAUSALITY · p. 51, HUMAN INTELLIGENCE · p. 51

PART II: IMPORTANT MECHANISMS

The overview identifies attention and memory, unsupervised learning, hierarchical RL, multi-agent RL, relational RL, and learning to learn as important mechanisms. It also notes that Bayesian RL and semi-supervised RL are not covered.

PART II: IMPORTANT MECHANISMS

9 ATTENTION AND MEMORY

The section surveys attention and memory designs alongside unsupervised-learning mechanisms and applications. It ranges from memory-equipped models and generative scene representations to imitation from unaligned videos.

9 ATTENTION AND MEMORY

  • Attention weights all memory locations or selects one by pointer; Weston et al. (2015), Graves et al. (2016), and Wayne et al. (2018) extend memory for question answering, structured problems, and partial observability.Source: §9 · p. 54, ATTENTION · p. 55

ATTENTION

  • Vaswani et al. (2017) use self-attention in Transformer for sequence transduction; it achieves good BLEU results on two translation tasks and is more time-efficient and parallelizable than recurrent models.Source: ATTENTION · pp. 54–55

10 UNSUPERVISED LEARNING

  • Unsupervised learning includes non-probabilistic and generative models; self-supervised learning creates labels from data, and RL approaches can use non-reward training signals.Source: §10 · p. 56

UNSUPERVISED AUXILIARY LEARNING

  • Jaderberg et al. (2017)'s UNREAL improves A3C on Atari and performs well on 3D Labyrinth; Mirowski et al. (2017) learn maze navigation using auxiliary tasks and stacked-LSTM memory.Auxiliary signals may help maximize cumulative rewards, including when extrinsic rewards are rarely observed.Source: UNSUPERVISED AUXILIARY LEARNING · pp. 56–57

GENERATIVE ADVERSARIAL NETS

  • Goodfellow et al. (2014) train a generator and discriminator adversarially; GANs can recover the data distribution with enough capacity, but are hard to train, and tuned models often perform similarly.Lucic et al. (2018) report that fine-tuning hyperparameters and random restarts make most evaluated models perform similarly.Source: GENERATIVE ADVERSARIAL NETS · p. 57

GENERATIVE QUERY NETWORK

  • Eslami et al. (2018)'s GQN learns 3D scene representations from 2D views without human semantic labels; its representations aid data efficiency and robust control in robot-arm reaching.Experiments report viewpoint invariance, compositionality, factorization, scene algebra, and decreasing Bayesian surprise with more observations.Source: GENERATIVE QUERY NETWORK · p. 58

PLAYING GAMES BY WATCHING YOUTUBE

  • Aytar et al. (2018) learn common representations from unaligned videos through self-supervision, derive a reward from one YouTube video, and report breakthrough results on three hard Atari games.Source: PLAYING GAMES BY WATCHING YOUTUBE · p. 58

11 HIERARCHICAL RL

The section surveys hierarchical RL and multi-agent RL, from temporal abstraction and learned options to decentralized execution and game-theoretic challenges. It highlights multi-agent Capture the Flag as a human-level result and describes agents learning structured representations of game situations through RL.

11 HIERARCHICAL RL

  • Hierarchical RL uses temporal abstraction to organize learning, planning, and knowledge across levels, addressing sparse rewards or long horizons through high-level goals.Its modular structure is usually conducive to transfer and multi-task learning; hard goals can be decomposed into easier sub-goals.Source: §11 · p. 59

HIERARCHICAL DQN

  • h-DQN separates goal selection from raw-action control, with a meta-controller choosing sub-goals and a controller pursuing each one.The meta-controller maximizes extrinsic reward, while the controller maximizes intrinsic reward; an internal critic rewards goal achievement.Source: HIERARCHICAL DQN · p. 59

FEUDAL NETWORKS

  • Feudal networks divide control between a long-timescale Manager setting latent-space sub-goal directions and a Worker choosing actions to follow them.A dilated LSTM supports backpropagation across long time steps.Source: FEUDAL NETWORKS · p. 60

OPTION-CRITIC

  • Option-critic learns intra-option policies and termination conditions alongside the policy over options, without requiring rewards or sub-goals.A related approach decouples behavior and target termination conditions and reports benefits from learning short options from longer ones.Source: OPTION-CRITIC · p. 60

OPTION DISCOVERY WITH A LAPLACIAN FRAMEWORK

CENTRALIZED TRAINING FOR DECENTRALIZED EXECUTION

  • Multi-agent RL faces nonstationarity and conceptual questions about its objective; centralized training can use joint information, but centralized execution faces scaling and decentralization challenges.A centralized critic can train decentralized actors, while COMA uses a counterfactual baseline to support multi-agent credit assignment.Source: §12 · p. 62, CENTRALIZED TRAINING FOR DECENTRALIZED… · p. 63

ISSUES IN GAME THEORY

  • Matrix-game treatments can miss temporally extended and partially informed social dilemmas, while sequential social dilemmas distinguish games with different patterns of cooperation.The section also describes Capture the Flag agents learning game-situation representations through RL rather than explicit models.Source: ISSUES IN GAME THEORY · p. 64
  • Human-level performance in Capture the Flag was reported as the first such result for multi-agent RL.The approach used population-based training, internal rewards, and temporally hierarchical RL; experiments also reported generalizable skills on random maps.Source: ISSUES IN GAME THEORY · pp. 64–65

13 RELATIONAL RL

This section surveys relational representations and graph-based reasoning in reinforcement learning, including methods for adapting, transferring, and learning algorithms across tasks. It also covers RL for graph combinatorial optimization and AutoML, with emphasis on learned graph-construction policies, neural architecture search, and limits of automated component discovery.

13 RELATIONAL RL

  • Relational RL connects reinforcement and relational learning to knowledge representation and reasoning, addressing challenges such as object generalization, transfer, planning, and prior knowledge.Dˇzeroski et al. (2001) proposed relational RL; Guestrin et al. (2003) introduced relational MDPs, and Diuk et al. (2008) introduced the closely related OO-MDPs.Source: §13 · p. 66

RELATIONAL LEARNING

  • Battaglia et al. (2018) propose graph networks with explicit entity nodes and relation edges to support combinatorial generalization over arbitrary relational structures.Node and edge permutations are invariant; relational RL work includes self-attention over entities in BOX-World and six StarCraft II mini-games, while SOORL achieved positive reward in Pitfall! but was outperformed there by concurrent self-supervision plus RL.Source: RELATIONAL LEARNING · pp. 66–67

14 LEARNING TO LEARN

  • Meta-learning spans learning what to learn, which model to learn, and how to learn, with the last category learning an algorithm that trains a base model.Methods include recurrent models, metric learning, and learned optimizers; MAML seeks an initialization that can be fine-tuned from a few examples.Source: §14 · p. 68

14.1 FEW/ONE/ZERO-SHOT LEARNING

  • MAML optimizes an initial representation so a few gradient updates adapt it to new tasks, with good results in few-shot classification, regression, and policy-gradient RL.Its meta-optimization trains the old parameters across tasks using outcomes computed with the adapted parameters; the method works for supervised learning and reinforcement learning.Source: §14.1 · p. 69

14.2 TRANSFER/MULTI-TASK LEARNING

  • Distral trains multiple task policies around a shared distilled policy to capture common behavior and reduce negative gradient interference, outperforming A3C baselines in grid worlds and complex 3D environments.Transfer learning covers knowledge transfer across domains; multitask learning shares representations across related tasks to improve generalization and learning across tasks.Source: §14.2 · p. 70

14.3 LEARNING TO OPTIMIZE

  • Li and Malik (2017) formulate optimization as RL: the current iterate and trajectory features are state, the update step is action, and the policy is the optimization algorithm.The RL agent lacks access to the state-transition model.Source: §14.3 · p. 71

14.4 LEARNING REINFORCEMENT LEARN

  • Xu et al. (2018) learn return functions by online gradient-based meta-learning of parameters such as γ and λ, reporting good results on Atari games.Unlike much recent meta-learning work framed as transfer or multi-task learning, the approach tunes the return while interacting with the environment and may extend to other return components, update rules, and hyperparameters.Source: §14.4 · p. 71

14.5 LEARNING COMBINATORIAL OPTIMIZATION

  • Dai et al. (2017) automate graph combinatorial optimization by learning heuristics from problem instances sampled from D that generalize to unseen instances from D.The method uses deep RL with graph embedding.Source: §14.5 · p. 72
  • Dai et al. (2017) use structure2vec embeddings and n-step fitted Q-learning to learn greedy graph-construction policies, reporting good performance on MVC, MAXCUT, and TSP across synthetic and real graphs.States are partial node selections, actions add nodes, and rewards reflect changes in the cost function.Source: §14.5 · p. 72
  • Vinyals et al. (2015) use attention-based pointer networks to select sequence elements for combinatorial outputs; unlike Dai et al. (2017)’s graph embedding, pointer networks are graph-agnostic.Their evaluated tasks include convex hulls, Delaunay triangulations, and planar TSP.Source: §14.5 · p. 73

14.6 AUTOML

  • Zoph and Le (2017) train an RNN controller with REINFORCE to search variable-length network architectures, achieving competitive CIFAR-10 image-classification results and better Penn Treebank language-modeling results than state-of-the-art methods.The controller generates architecture tokens, and validation accuracy supplies the reward.Source: §14.6 · p. 73
  • Related approaches transfer learned blocks (Zoph et al., 2017), use Q-learning or Bayesian search (Jin et al., 2018), and automate compression, augmentation, and tensor optimization.Cai et al. (2018a) study AutoML under limited computation resources of 200 GPU hours.Source: §14.6 · pp. 73–74
  • Neural architecture design still relies on manually designed components, leaving open whether AI can discover novel ones; manually devised architectures are also identified as possible automation targets.The text cites attention-based, multi-task, dueling, planning, and integrated learning-and-planning architectures as manually designed examples.Source: §14.6 · pp. 73–74

PART III: APPLICATIONS

This part surveys deep RL across classical and newer application areas, emphasizing games as testbeds for sequential decision-making and advances in video-game environments. It highlights the potential reach of game-playing techniques beyond games.

PART III: APPLICATIONS

  • The application survey spans games, robotics, NLP, vision, finance, management, healthcare, education, energy, transportation, computer systems, and science, engineering, and art; RL may fit problems framed as sequential decisions.Application areas can overlap, and operations research is treated as implicit in areas such as energy and transportation rather than listed separately.Source: PART III: APPLICATIONS · p. 75

15 GAMES

  • Games offer strong simulators and unlimited data, and deep RL has reached human-level or superhuman performance in many two-player games, with multiplayer games a frontier.The section cites achievements in Quake III Arena Capture the Flag and Dota 2 5v5.Source: §15 · p. 76

15.1 BOARD GAMES / COMPUTER GO

  • Board games are classical RL/AI testbeds; AlphaGo (Silver et al., 2016a) defeated Lee Sedol 4 games to 1 and Ke Jie 3:0, while AlphaGo Zero learned a superhuman Go program without human knowledge.Computer Go poses a vast search space and difficult position evaluation.Source: §15.1 · p. 76, COMPUTER GO · p. 77

ALPHAGO: TRAINING PIPELINE AND MCTS

  • AlphaGo combines deep convolutional networks, supervised and reinforcement learning, and MCTS in a training pipeline followed by lookahead search.The pipeline trains expert-move and rollout policies, an RL policy, and an RL value network; search expands policy-guided leaves, evaluates them, and backs up values.Source: ALPHAGO: TRAINING PIPELINE AND MCTS · p. 77

ALPHAGO ZERO

  • AlphaGo Zero (Silver et al., 2017) learned through self-play without human data or supervision, using raw board inputs and one network for policy and value, with no Monte Carlo rollouts.Its training loop uses MCTS for policy improvement and self-play outcomes for policy evaluation.Source: ALPHAGO ZERO · p. 78

ALPHA ZERO

  • Alpha Zero (Silver et al., 2017) generalized AlphaGo Zero to perfect-information, two-player zero-sum games and achieved superhuman performance in chess, shogi, and Go.It learns using only game rules, without human knowledge; the reported training required substantial compute, including around 2000 TPUs for data generation.Source: ALPHA ZERO · p. 78

15.2 CARD GAMES

  • Card games pose imperfect-information problems; Heads-up Limit Hold’em was essentially solved with counterfactual regret minimization.The section identifies security and medical decision support as potential applications of imperfect-information game methods.Source: §15.2 · p. 80

DEEPSTACK

  • DeepStack (Moravˇc´ık et al., 2017) became the first program to defeat professional players in Heads-up No-Limit Hold’em Poker.It uses recursive counterfactual regret minimization for information asymmetry and automatically trained value functions, with little domain knowledge or expert games.Source: DEEPSTACK · p. 80

15.3 VIDEO GAMES

  • The survey also covers Atari 2600, StarCraft and StarCraft II, Quake III Arena, Labyrinth, Minecraft, and Super Smash Bros. research.It discusses DQN for Atari, built-in-AI victories in StarCraft II, and a human-level Quake III Arena Capture the Flag agent.Source: §15.3 · p. 81
  • In Doom, Wu and Tian (2017) trained A3C from four raw frames and game variables with a simple-to-hard curriculum, winning ViZDoom Track 1 by a large margin.The agent predicted its next action and value function in a partially observable 3D environment with sparse, long-term rewards.Source: §15.3 · p. 81

16 ROBOTICS

Robotics offers broad applications for RL, but real-world learning faces challenges including dimensionality, model uncertainty, and reward specification. The section surveys learning approaches for robotics and highlights autonomous driving’s safety and evidence boundaries.

16 ROBOTICS

  • Robotics challenges include dimensionality, under-modeling, model uncertainty, and reward specification; RL approaches use representation, prior knowledge, and models.Examples include demonstrations and directed exploration, value-function approximation, and simulation-based rehearsal.Source: §16 · p. 82

16.1 SIM-TO-REAL

  • Peng et al. (2017c) transferred a dynamics-randomized pushing policy from simulation to a physical robot; OpenAI (2018) transferred dexterous manipulation to a physical hand.Other routes include progressive network columns for simulation-to-reality transfer and visual servoing adapted with realistic images; the latter assumes no obstacle-planning policy.Source: §16.1 · pp. 82–83

16.2 IMITATION LEARNING

  • Imitation approaches span neural inverse-cost learning, attention-based one-shot imitation, and meta-imitation policies adaptable from one human demonstration.The one-shot meta-imitation studies test in simulation and on physical robots.Source: §16.2 · p. 83

16.3 VALUE-BASED LEARNING

  • Successor-feature navigation transfers knowledge to new robot navigation tasks without localization, mapping, or planning, while prior GVFs can speed construction of new GVFs.The navigation work experiments in simulation and on physical robots.Source: §16.3 · pp. 83–84

16.4 POLICY-BASED LEARNING

  • Guided policy search trains CNN policies for direct image-to-torque control and reports strong performance on real-world manipulation tasks with complex contact dynamics.Other policy-based work includes asynchronous guided policy search for door opening with four robots and target-driven navigation in the AI2-THOR simulator.Source: §16.4 · p. 84

16.5 MODEL-BASED LEARNING

  • Model-based robotics work combines continuous-task Q-learning with local linear models, video prediction with model predictive control, and LQR with policy improvement.Other examples combine object-classification features, predictive dynamics, and fitted Q-iteration for visual servoing.Source: §16.5 · p. 84

16.6 AUTONOMOUS DRIVING VEHICLES

  • The authors argue current science and technology are not ready for fully autonomous road tests, citing simulator mismatch and insufficient road-test data for statistically significant claims.They propose closed-environment tests with robotic pedestrians until systems have capabilities such as scene understanding and common sense.Source: §16.6 · p. 85

17 NATURAL LANGUAGE PROCESSING

The NLP overview surveys language tasks and the interaction of deep learning with linguistic knowledge, then reviews sequence generation, machine translation, and dialogue systems. It highlights both training-objective mismatches and RL approaches that address them.

17 NATURAL LANGUAGE PROCESSING

  • In NLP, deep-learning and non-deep-learning approaches coexist in a field characterized more by synergy than competition, and methods may incorporate linguistic knowledge explicitly or implicitly.Melis et al. (2018) observed that standard LSTM outperforms recent models in neural language-model evaluation; Bai et al. (2018) found CNNs outperform RNNs over a wide range of tasks.Source: §17 · p. 86

17.1 SEQUENCE GENERATION

  • Sequence generation faces exposure bias from sequential word generation and a mismatch between word-level training losses and sequence-level evaluation metrics.MIXER aligns training with sequence-level objectives such as BLEU; actor-critic methods and SeqGAN are additional RL-based approaches to sequence prediction.Source: §17.1 · p. 87

17.2 MACHINE TRANSLATION

  • Dual learning uses feedback between translations in both directions, matching prior full-bilingual-data performance on English-to-French with 10% bilingual warm-start data plus monolingual data.He et al. (2016a) use language-model likelihood as the reward signal and suggest the mechanism may extend to other tasks with dual forms.Source: §17.2 · p. 88

17.3 DIALOGUE SYSTEMS

  • Dialogue systems include task-oriented agents and chatbots, and RL appears in deep-learning systems for policy learning and user-feedback-driven training.KB-InfoBot learns dialogue acts and a soft posterior over knowledge-base entries with RL; Su et al. (2016) jointly learn dialogue policy and reward using active learning, reducing manual annotations and mitigating noisy feedback.Source: §17.3 · p. 88

18 COMPUTER VISION

The overview surveys deep RL across computer vision and finance and business management, covering recognition, motion, scene understanding, language integration, visual control, and interactive perception. It also describes financial decision problems and applications in business.

18 COMPUTER VISION

  • Deep learning has driven computer-vision progress, while RL is used for classification, detection, captioning, and interactive perception.Recognition methods use sequential or active search to localize objects, with a trade-off between sampling more regions for accuracy and stopping once confidence is sufficient.Source: §18 · p. 90, §18.1 · pp. 90–91

18.2 MOTION ANALYSIS

  • Deep RL tracking policies can decide where to look, when to reinitialize, and when to update an object's appearance model under ambiguous frames or constrained computation.DARKO uses inverse RL to infer states, transitions, and rewards from streaming first-person video to model and forecast long-term goals.Source: §18.2 · p. 91

18.3 SCENE UNDERSTANDING

  • Scene-understanding approaches infer structured scene descriptions by inverting a renderer or recover physical representations that can reconstruct visual data.Wu et al. (2017b) use a REINFORCE variant to address non-differentiable graphics rendering; Wu et al. (2017a) report results with differentiable and non-differentiable physics engines.Source: §18.3 · p. 91

18.4 INTEGRATION WITH NLP

  • Vision-language work includes attention-based image captioning, RL-trained visually grounded dialogue, cooperative visual dialogue agents, and inverse-RL visual storytelling.Strub et al. (2017) optimize goal-driven dialogue for the GuessWhat?! game end to end.Source: §18.4 · p. 91

18.5 VISUAL CONTROL

  • Visual control derives policies from visual inputs in games, robotics, and self-driving, but the overview distinguishes computer-vision control from merely feeding images to a neural network.The authors state that a visual-control problem should include ingredients of, by, and for computer vision.Source: §18.5 · p. 92

18.6 INTERACTIVE PERCEPTION

  • Active visual completion uses look-around decisions rewarded by prediction accuracy for unobserved views, illustrated on panoramic scenes and 3D object shapes.The approach is task-agnostic, and the authors discuss generalization and transfer to new tasks and environments.Source: §18.6 · p. 92

19 FINANCE AND BUSINESS MANAGEMENT

  • RL is presented as a fit for sequential finance problems such as option pricing and multi-period portfolio optimization, while this overview selects only a couple of finance and business papers.American option pricing is described as an optimal-stopping problem centered on continuation value, with least squares Monte Carlo a standard finance approach.Source: §19 · p. 93, §19.1 · p. 94

19.2 PORTFOLIO OPTIMIZATION

  • Portfolio work includes direct-reinforcement trading and risk control, while business applications include sequential news and ad recommendation, customer value, and e-commerce decisions.The overview notes that continuous state and action spaces are needed for dynamic portfolio optimization and describes policy evaluation approaches for customer lifetime value and recommendation.Source: §19.2 · p. 94, §19.3 · p. 95

20 MORE APPLICATIONS

This section surveys reinforcement-learning applications across healthcare, education, energy, transportation, computer systems, security, science, engineering, and art. It also covers music and sequence generation and computational molecular generation, highlighting methods and challenges.

20.1 HEALTHCARE

  • Healthcare applications include personalized care and sequential treatment decisions, with Q-learning used in dynamic treatment regimes and deep RL studied for sepsis policy evaluation.Work also addresses missing and censored data, policy evidence, patient phenotypes, stroke treatment, and diagnosis.Source: §20.1 · p. 96

20.2 EDUCATION

  • Education applications span student, content, and course modeling, while RL methods address engagement, experimental allocation, and personalized learning.Mandel et al. (2014) combine importance sampling with cross-validation for offline policy evaluation; Liu et al. (2014) propose UCB-Explore to balance learning condition effectiveness against test performance.Source: §20.2 · p. 97

20.3 ENERGY

  • Demand response uses RL to shift or reduce peak electricity use in response to grid signals, with model-free device-level control and batch RL studied.Wen et al. (2015) decompose the formulation across devices, giving complexity linear in device count, while Ruelens et al. (2016) use batch RL.Source: §20.3 · pp. 97–98

ADAPTIVE CONTROL

  • Transportation research applies multi-agent RL to adaptive traffic-signal control, coordinating neighboring intersections and validating the approach with simulations and Toronto traffic data.El-Tantawy et al. (2013) formulate the problem as a multiple-player stochastic game; Belletti et al. (2018) study multi-task deep RL for ramp metering.Source: ADAPTIVE CONTROL · p. 98

20.5 COMPUTER SYSTEMS

  • Computer-systems applications include learned indexes and RL-based resource, scheduling, and power management, while direct combinatorial search can impose computational burdens.Liu et al. (2017a) use hierarchical global and local tiers for cloud resource allocation and server power management; the approach is validated on Google cluster traces.Source: §20.5 · p. 98, PERFORMANCE OPTIMIZATION · p. 99

PERFORMANCE OPTIMIZATION

  • 40% lower data-center energy consumption is reported for Google’s deployed machine learning, while RL device placement yields shorter execution times than human-designed placements.Mirhoseini et al. use a sequence-to-sequence model and REINFORCE, with execution time as the reward, to place TensorFlow operations.Source: PERFORMANCE OPTIMIZATION · pp. 98–99

SECURITY

  • Security applications face adversarial threats during both training and inference, and RL has been used for black-box malware evasion, though performance needs improvement.Anderson et al. (2018) generate functional evasive malware samples intended to help improve anti-malware engines.Source: SECURITY · p. 99

20.6 SCIENCE, ENGINEERING AND ART

  • RL and AI connect with science, engineering, and art through applications such as retrosynthesis and PDE control, while ideas from these fields also inform AI development.Segler et al. (2018) combine Monte Carlo tree search with learned networks and symbolic rules for retrosynthesis; Pan et al. (2018) use action descriptors to encode spatial regularities in high-dimensional PDE control and show theoretical sample-efficiency gains over conventional RL that ignores those regularities.Source: §20.6 · p. 100, §20.6.1 · p. 101, §20.6.2 · p. 101

20.6.3 MUSIC

  • Jaques et al. (2017) combine maximum likelihood estimation with off-policy reinforcement learning, using domain-specific rewards while penalizing divergence from a pretrained Reward RNN policy.The Reward RNN is pretrained with MLE; another RNN is then trained with off-policy RL to generate high-quality sequences while retaining information and sample diversity.Source: §20.6.3 · p. 101
  • Sequence Tutor incorporates music-theory rules to produce more pleasant-sounding, subjectively pleasing melodies than alternative methods.The authors also conduct experiments with Sequence Tutor for computational molecular generation.Source: §20.6.3 · p. 102

21 DISCUSSIONS

The discussion reviews deep RL’s core issues and approaches, outlines research directions across evaluation, multi-agent learning, representation, automation, and applications, and considers the field’s progress alongside unresolved challenges. It also discusses RL’s potential as a general learning and decision-making paradigm while noting the need for better algorithms and broader product validation.

21.1 BRIEF SUMMARY

  • Off-policy data, auxiliary rewards, reward shaping, hierarchy, shared goal representations, exploration, models, and transfer across tasks are among the approaches to credit assignment, sparse rewards, and sample efficiency.The overview also notes that incorporating inductive bias, structure, and knowledge can improve representations and problem formulation.Source: §21.1 · p. 103

21.2 CHALLENGES AND OPPORTUNITIES

  • Henderson et al. (2018) show that hyperparameters, random seeds, environments, and codebases affect deep RL results, complicating reproducibility and motivating detailed reporting and rigorous evaluation.They recommend reporting implementation and setup details, matching baseline implementations to original code, averaging results across many random seeds, and using significance tests.Source: §21.2 · p. 105

Research Direction 2: ”solve” multi-agent problems

  • Multi-agent learning remains difficult because of non-stationarity and theoretical infeasibility, despite strong results in games; the authors call for solving complex multi-agent problems.They also propose learning from entities rather than raw inputs, adding an intelligence component with knowledge, principles, and algorithms to end-to-end systems to improve efficiency and interpretability.Source: §21.2 · p. 105, Research Direction 2: ”solve”… · pp. 106–107

Research Direction 4: design an optimal representation for RL

Research Direction 5: AutoRL

  • The authors propose AutoRL to select state representations, function approximators, algorithms, and hyperparameters automatically.They also argue that deep learning supplies mechanisms while RL defines and pursues objectives for computable tasks, while noting hidden-layer size may be infeasible and computability can be an issue unless P = NP.Source: Research Direction 4: design an… · p. 108, Research Direction 5: AutoRL · p. 108

Research Direction 6: develop killer applications for (deep) RL

  • The authors argue that reinforcement learning needs broader practical applications with commercial value and recommend applying AlphaGo techniques, including deep learning, RL, Monte Carlo tree search, and self-play.Potential applications include robotics, industrial control, recommendation, protein folding, energy reduction, and materials discovery, although these techniques may require a good or perfect simulator.Source: Research Direction 5: AutoRL · p. 108, Research Direction 6: develop killer… · pp. 108–109

21.3 EPILOGUE

  • The authors describe deep RL as rapidly advancing through breakthroughs in algorithms, architectures, and applications, while acknowledging that their overview is incomplete and that the field’s progress depends on continued creativity.They point to DQN, AlphaGo, and DeepStack as breakthroughs and suggest RL may help when tasks can be framed as sequential decision-making.Source: §21.3 · p. 109
  • Deep learning has achieved many advances and market-validated products, whereas reinforcement learning still needs better algorithms and widespread product validation despite its promising applications.The authors argue that this is a timely moment to nurture, educate, and lead the RL market, and expect both fields to prosper.Source: §21.3 · p. 110
  • Reinforcement learning is framed as a general learning and decision-making paradigm that will influence deep learning, machine learning, and artificial intelligence broadly.The passage notes that Rich Sutton named his University of Alberta lab RLAI—Reinforcement Learning and Artificial Intelligence—in 2003.Source: §21.3 · p. 110
Loading 1810.06339v1…