Source-linked AI summary
Reinforcement Learning
Olivier Buffet, Olivier Pietquin, Paul Weng
TL;DR
Reinforcement learning addresses policy learning when an agent interacts with an environment whose dynamics are unknown, using observations and reward feedback. This chapter reviews the framework and its value-based and policy-search approaches, and notes that sample and computational requirements remain too large for many practical domains.
Problem
RL concerns learning a good policy through interaction when the environment model is unknown, while the system designer must still specify the reward function.
Source: §2 · p. 4Method
The chapter reviews the RL framework, value-based and policy-search approaches, and extensions for unknown rewards and risk-sensitive criteria.
Source: §1 · p. 2Results
The chapter’s overview distinguishes value-based learning from direct policy search and identifies actor-critic methods as policy search guided by learned policy value.
Source: §1 · p. 2Takeaways & Limitations
The chapter concludes that RL’s sample and computational requirements remain too large for many practical domains, including robotics.
Source: §6 · p. 20RL still has excessive sample and computational requirements for many practical domains, such as robotics.
Source: §6 · p. 20
Abstract
from arXiv · showhide
Reinforcement learning (RL) is a general framework for adaptive control, which has proven to be efficient in many domains, e.g., board games, video games or autonomous vehicles. In such problems, an agent faces a sequential decision-making problem where, at every time step, it observes its state, performs an action, receives a reward and moves to a new state. An RL agent learns by trial and error a good policy (or controller) based on observations and numeric reward feedback on the previously performed action. In this chapter, we present the basic framework of RL and recall the two main families of approaches that have been developed to learn a good policy. The first one, which is value-based, consists in estimating the value of an optimal policy, value from which a policy can be recovered, while the other, called policy search, directly works in a policy space. Actor-critic methods can be seen as a policy search technique where the policy value that is learned guides the policy improvement. Besides, we give an overview of some extensions of the standard RL framework, notably when risk-averse behavior needs to be taken into account or when rewards are not available or not known.
1 Introduction
Reinforcement learning (RL) treats autonomous-agent control as sequential decision-making, with agents learning good actions from observations and reward feedback. The chapter outlines the basic framework, two main policy-learning families, and extensions for unknown rewards and risk-sensitive behavior.
1 Introduction
- An RL agent learns to choose actions from observations and numeric reward feedback while interacting with an environment whose dynamics may be unknown.The loop proceeds from observing the current situation to choosing an action, receiving a reward, and moving to a new state.Source: §1 · p. 2
- The chapter focuses on a single learning agent with full observability, while noting partial observability, multi-agent learning, and Bayesian approaches as extensions or related work.Bayesian approaches are omitted for space reasons.Source: §1 · p. 2
- Value-based methods estimate an optimal policy’s value function, whereas policy search searches directly in policy space.The chapter also covers actor-critic methods and extensions for unknown rewards and risk-sensitive criteria, as stated in its overview.Source: §1 · p. 2
2 Background for RL
The chapter formulates sequential decision-making with Markov decision processes (MDPs), then describes RL as trial-and-error policy learning when the environment is unknown. It introduces value functions and classic algorithms for evaluating and improving policies.
2 Background for RL
- An MDP specifies states, actions, transition probabilities, rewards, a discount factor, and a horizon; its environment is assumed stationary.The reward function gives the immediate scalar reward, and the designer defines the model so an optimal policy performs the intended task.Source: §2 · pp. 2–3
- Planning in an MDP seeks a policy that maximizes expected discounted rewards, with policies allowed to be deterministic or randomized, stationary or time-dependent.A policy specifies which action to take in every state.Source: §2 · p. 3
- Policy iteration alternates policy evaluation and improvement, while value iteration repeatedly updates values toward the optimal value function.The chapter also introduces action-value functions, which measure the value of choosing an action and then following a policy.Source: §2 · p. 4
- RL addresses settings where the complete environment model or task description is unavailable, learning a policy through trial and error while retaining the standard expected-reward criterion.The reward function may be unknown to the agent, but the system designer still has to specify it.Source: §2 · p. 4
- SARSA combines on-policy action-value evaluation with policy improvement, while Q-learning uses an optimality backup to estimate the optimal action value off-policy.The chapter notes convergence conditions for these updates, including sufficient exploration for SARSA; constant learning rates are often used in practice.Source: §2 · p. 5
3 Value-Based Methods with Function Approximation
Value-based RL with function approximation estimates policy value functions from transition data when state-action spaces are too large for exact representation. The section surveys bootstrapped, residual, least-squares, projected fixed-point, and deep-learning approaches, along with their constraints and mechanisms.
3 Value-Based Methods with Function Approximation
- Function approximation parameterizes value functions and learns their parameters from transitions when state-action spaces are too large for exact representation.Linear approximations use basis functions; neural networks are another popular scheme.Source: §3 · p. 6
3.1.1 Bootstrapped Methods
- Bootstrapped methods replace unavailable policy values with Bellman-based targets, yielding TD updates for value functions and SARSA updates for action values.The Q-learning variant instead uses the Bellman optimality target to estimate the optimal policy's value.Source: §3.1.1 · pp. 6–7
3.1.2 Residual Methods
- Residual stochastic gradient descent can produce a biased value-function estimate because successor-state estimates are correlated.Baird et al. [1995] address this by drawing two different next states from the same state; this requires an on-demand generative model or simulator.Source: §3.1.2 · p. 8
3.2 Least-Squares Methods
- Least-squares methods include LSTD, which minimizes the projected Bellman residual and has a closed-form solution with linear function approximation.FPKF applies recursive least squares to bootstrapping-error minimization, while GPTD and KTD apply it to Bellman-residual minimization.Source: §3.2 · p. 9
3.3 Iterative Projected Fixed-Point Methods
- Iterative projected fixed-point methods alternate sampled Bellman updates with approximate projection onto the function space, converging when the composition remains a contraction.The exact projection is often unavailable and must itself be approximated from samples.Source: §3.3 · p. 10
3.4 Value-Based Deep Reinforcement Learning
- Deep Q-Networks combine replay-buffer sampling and a slower target network to address correlated, changing training data and reduce correlations between successive value estimates.The target-network construction also avoids direct differentiation through the max operator; later variants include prioritized replay and Double-DQN.Source: §3.4 · pp. 11–12
4 Policy-Search Approaches
Policy search directly optimizes parameterized policies, with model-free and model-based approaches differing in how they use trajectories and learned dynamics. The chapter surveys exploration, policy evaluation and update strategies, as well as model learning and prediction.
4 Policy-Search Approaches
- Parameterized policy search directly optimizes controllers, avoiding issues with discontinuous value functions and accommodating high-dimensional state and action spaces.Policies can also incorporate expert knowledge through their structure or example trajectories.Source: §4 · p. 12
4.1 Model-Free Policy Search
- Model-free policy search uses sampled trajectories to update policy parameters through exploration, policy evaluation, and policy updates.The chapter organizes its discussion around these three steps.Source: §4.1 · p. 13
4.1.1 Policy Exploration
- Policy exploration samples policy parameters or perturbs actions; an upper-level distribution over parameters is another option.A non-diagonal Gaussian can model parameter correlations, offering more stability but requiring more samples.Source: §4.1.1 · p. 13
4.1.2 Policy Evaluation
- Step-based evaluation uses state-action pairs, while episode-based evaluation uses complete trajectories and can apply criteria such as final distance to a target.Step-based estimates can have low variance but may be biased or noisy; episode-based estimates can have high variance, especially with stochastic dynamics.Source: §4.1.2 · p. 13
4.1.3 Policy Update
- Policy updates include gradient, inference-based, information-theoretic, stochastic-optimization, and path-integral approaches.Likelihood-ratio gradients require stochastic policies; information-theoretic methods constrain changes, while PI2 optimizes movement primitives using Monte Carlo rollouts.Source: §4.1.3 · pp. 13–15
4.2 Model-Based Policy Search
- Model-based policy search alternates real-world trajectory sampling, forward-model learning, and policy optimization, without explicitly handling exploration versus exploitation.The chapter considers model learning, reliable long-term prediction, and policy updates as its three main dimensions.Source: §4.2 · p. 15
- Probabilistic models can represent uncertainty from sparse data or model mismatch, as well as stochastic dynamics.Long-term policy evaluation can reduce sampling variance by using the same random initial states and sequences across policies; policy updates may use gradients or gradient-free methods.Source: §4.2 · pp. 15–16
5 Extensions: Unknown Rewards and Risk-sensitive Criteria
The chapter reviews extensions for unknown rewards, preferences, and risk-sensitive decisions beyond expected cumulative reward. It describes learning rewards from demonstrations, comparing policies through preferences, and incorporating risk measures, while noting a challenge for Bellman optimality.
5 Extensions: Unknown Rewards and Risk-sensitive Criteria
- The extensions address settings where rewards are non-scalar, unknown, or non-numeric, or where expected cumulative reward is unsuitable.The chapter focuses on reward learning, preference-based RL, and risk-sensitive RL.Source: §5 · p. 16
5.1 Reward Learning
- Reward design is difficult: suitable values may accelerate learning, while incorrect values may produce unwanted behavior.When rewards are unknown, demonstrations can support direct policy learning or inverse RL, which first recovers a reward function.Source: §5.1 · pp. 16–17
- Inverse RL is ill-posed because a constant reward makes every policy equivalent and optimal; maximum entropy is one proposed response to this degeneracy.Reward functions may be represented parametrically, including as linear combinations of features.Source: §5.1 · pp. 17–18
5.2 Preference-Based Approaches
- Preference-based approaches can compare histories without assuming scalar numeric rewards, but their decision model can produce preference cycles.Mixed solutions interpreted as Nash equilibria are one proposed way to guarantee an optimal solution.Source: §5.2 · p. 19
5.3 Risk-Sensitive Criteria
- Risk-sensitive criteria matter when policies are used only a few times or must avoid bad states; quadratic expected utility penalizes reward variance.Other approaches optimize measures such as CVaR, while risk-constrained problems maximize expected return subject to a risk bound.Source: §5.3 · pp. 19–20
- The Bellman optimality principle generally fails for risk-sensitive criteria, since an optimal policy's sub-policy may not itself be optimal.In most cases, a state-augmented MDP that includes cumulative rewards can recover the principle.Source: §5.3 · p. 20
6 Conclusion
The chapter surveys value-based and policy-search methods for large RL problems, alongside approaches for unknown rewards and risk sensitivity. It concludes that sample and computational requirements remain too high for many practical domains.
6 Conclusion
- The chapter reviews value-based and policy-search methods for large RL problems that require function approximation.It also surveys approaches to unknown rewards and risk-sensitive decision-making.Source: §6 · p. 20
- The chapter also covers approaches to unknown rewards and risk sensitivity in decision-making.These topics are presented alongside the main RL solution techniques.Source: §6 · p. 20
- Sample and computational requirements remain too high for many practical domains, including robotics.Research directions include exploiting problem structure or prior knowledge and reusing experience through transfer, lifelong, multi-task, or curriculum learning.Source: §6 · p. 20