Source-linked AI summary
A Tour of Reinforcement Learning: The View from Continuous Control
Benjamin Recht
TL;DR
The paper examines how reinforcement learning can safely and reliably control complex, uncertain systems, and surveys the relationship between learning and control. It uses optimization-based analysis and unknown-dynamics LQR to compare solution paradigms, finding that models matter and that model-based methods can outperform model-free approaches. The survey connects these findings to broader continuous-control applications and discusses remaining challenges at the control-learning interface.
Problem
Deploying data-driven reinforcement learning in demanding interactive systems requires safety and reliability, while the field must use complex trajectory and reward information efficiently.
Method
The survey casts RL in an optimization framework, reviews model-free and model-based methods, and compares them through an unknown-dynamics LQR case study before relating the findings to nonlinear applications.
Results
The survey reports that model-free methods are considerably less effective than simple model-based schemes on LQR, with related observations continuing on more challenging nonlinear applications.
Takeaways & Limitations
Combining learning theory with control can provide finite-data performance and stabilization guarantees, while models can improve effectiveness and support safer interaction with uncertain systems.
Takeaways & Limitations
Model-free approaches can use samples inefficiently in continuous control, and large discount factors can produce brittle methods requiring hyperparameter tuning.
Abstract
from arXiv · showhide
This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best-studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and control might be combined to approach these challenges.
1 Introduction
The survey frames reinforcement learning and control as closely related but practically separate fields, and proposes connecting them through continuous-control analysis, especially by comparing model-free and model-based approaches.
- Reinforcement learning uses past data to improve future manipulation of dynamical systems, overlapping substantially with control theory despite the communities’ practical separation.
- Deploying data-driven RL in demanding systems requires methods that are safe and reliable because failures can cause severe societal and economic consequences.
- The survey provides a shared language between control and RL by contrasting model-based action design with model-free prediction while emphasizing their common focus on planning, adaptation, and safeguards.
- Using unknown-dynamics LQR as a case study, the survey compares RL and control techniques and finds model-based schemes substantially more effective than model-free methods in theory and practice.
2 What is reinforcement learning?
Reinforcement learning formulates control as choosing policies that use observed trajectories and rewards to maximize long-term return under unknown dynamics. Its value comes from intervening in systems, but those interventions create complex feedback, sample-efficiency, and safety challenges.
- RL studies optimal control when dynamics are unknown, seeking input sequences that maximize an objective while beginning with minimal knowledge of system responses.
- The control problem models state transitions, disturbances, rewards, and actions chosen from previously observed states and inputs.
- Policies replace fixed action sequences because observing the state before each action enables feedback to continually mitigate uncertainty.
- In the standard RL interaction loop, an oracle returns trajectories and rewards after each policy deployment, and the learner seeks high reward with few samples.
- Unlike supervised prediction, RL permits the practitioner to vary control actions, affecting both system learning and reward optimization.
- RL is more challenging than supervised learning because interventions create complicated feedback loops, while failures can have catastrophic consequences.
3 Strategies for solving reinforcement learning problems
The survey organizes reinforcement-learning methods for unknown dynamics into model-based, model-free, approximate-dynamic-programming, and policy-search approaches. It uses control and optimization concepts to compare their assumptions, computational procedures, and trade-offs, emphasizing that model-free methods still rely on modeling assumptions and can be sample-inefficient in continuous control.
- Model-based reinforcement learning: Model-based reinforcement learning fits a model from observed data and uses it to approximate the optimal-control solution.Nominal control applies the estimated model as though it were the true dynamics.
- Model-free reinforcement learning: Model-free reinforcement learning directly maps observations to actions, with policy search improving episode rewards and approximate dynamic programming approximating optimal control costs.Approximate dynamic programming uses Bellman’s principle of optimality and recursive dynamic programming.
- Model-based reinforcement learning: System identification requires deliberately chosen inputs because dynamical outputs correlate over time with parameters, inputs, and disturbances.After data collection, supervised-learning tools can fit the transition model.
- Model-based reinforcement learning: Model-based control can solve the wrong problem when the estimated dynamics are incorrect, although it may work well when the estimate is close to the true dynamics.The formulation also requires a plausible noise model.
- Model-free reinforcement learning: Even model-free methods embed modeling assumptions through their chosen function parameterizations, while approximate dynamic programming uses one equation per step instead of the model’s d equations.The survey associates this difference with inefficient sample use in continuous control and notes that discount factors can make methods brittle.
- Policy search: REINFORCE is easy to implement and broadly applicable, but derivative-free noisy optimization and high-variance stochastic gradients can require many samples.Its gradient is not an explicit function of the underlying dynamics because optimization burden shifts to the sampling procedure.
4 Simplifying theme: The Linear Quadratic Regulator
The LQR provides a simple baseline for comparing reinforcement-learning and control methods under unknown dynamics. Its analysis highlights trade-offs between model-based estimation, uncertainty-aware robust control, and model-free methods.
- The LQR baseline: LQR combines linear dynamics with convex quadratic costs, making the control problem analytically tractable while retaining salient features of optimal control.Linear dynamics also guarantee a convex constraint set, and quadratic costs interact cleanly with noise.
- Known dynamics: When dynamics are known, dynamic programming computes linear state-feedback control through a matrix recursion based on A, B, Q, and R.Over an infinite horizon, the policy is static: π_t(x_t) = −Kx_t, with K determined by the Riccati equation.
- Unknown dynamics: Unknown dynamics turn LQR into a learning problem: the controller must identify the system while quickly obtaining effective control.This question motivates using LQR as a guiding baseline for connecting reinforcement-learning styles to control ideas.
- Model uncertainty: Nominal model-based control can fail when least-squares estimation understates an unstable eigenvalue, motivating robust treatment of model uncertainty.The survey notes that robust control can mitigate this uncertainty when only a small number of samples are available.
- Model-based methods: Model-based control estimates a nominal system from data, quantifies uncertainty around that estimate, and solves a robust optimization problem for execution.Coarse-ID Control uses supervised learning, probabilistic uncertainty guarantees, and uncertainty-penalizing robust optimization.
- Coarse-ID Control: If the true dynamics lie in the estimated uncertainty set, Coarse-ID Control guarantees a performant controller and bounds end-to-end performance by observed samples.The approach explicitly accounts for least-squares uncertainty and can extend beyond the LQR baseline to other optimal-control problems.
- Model-free methods: Model-free methods face dimension- and reward-scale-sensitive complexity, with noisy convex optimization attaining O((d^2B^2/T)^−1/3) convergence and strongly convex optimization O((d^2B^2/T)^−1/2) evaluations.The stated rates assume correctly chosen algorithm parameters, and constant reward offsets can substantially increase running time.
5 Numerical comparisons
The numerical comparisons move from a simple double integrator to unstable Laplacian dynamics, exposing how model quality, robustness, and algorithm choice affect control performance and stabilization. Model-based and robust approaches perform better as the dynamics become harder and safety becomes more important.
- 5.1 A Double Integrator: Nominal control and LSPI were indistinguishable from the optimal controller in the double-integrator experiment.
- 5.1 A Double Integrator: One 10-sample simulation produced a nominal model accurate to 3 digits and a nearly optimal control policy on the double integrator.The experiment used noise with covariance 10^-4I and training episodes of length L = 10.
- 5.1 A Double Integrator: Static linear policy search is nearly as effective as a time-varying policy for this two-state LQR problem, with only two decision variables.The policy-gradient implementation used Adam and a baseline based on previous-iterate rewards.
- 5.2 Unstable Laplacian dynamics: Robust LQR compares performance using the true model distance with performance when uncertainty is learned from data, while also tracking the fraction of stabilizing controllers.The robust optimization problem is solved through a small semidefinite program; uncertainty can be estimated with bootstrap simulation.
- 5.2 Unstable Laplacian dynamics: The unstable Laplacian instance models three coupled heat sources and makes accurate system estimation imperative for near-optimal control.The open-loop dynamics are unstable, so incorrect estimates of diagonal entries can make a mode appear stable and reduce cooling effort.
- 5.2 Unstable Laplacian dynamics: Nominal control frequently produces controllers that fail to stabilize the true system, whereas robust optimization helps provide stabilizing solutions.Stabilization requires all eigenvalues of A + BK to have magnitude less than 1.
- 5.2 Unstable Laplacian dynamics: Model-free methods remain far behind model-based methods on the unstable Laplacian instance, with approximate dynamic programming failing to find a decent solution despite 10 times more iterations.LSPI performs worse than random search, while the authors attribute its difficulty to the need for stabilizing controllers at every iteration and careful discount-factor tuning.
6 Beyond the Linear Quadratic Regulator
Beyond LQR, the survey contrasts derivative-free random search with model-based receding-horizon control on complex continuous-control tasks. These examples expose both the variability of model-free methods and the benefits of combining models, feedback, learned terminal costs, and safety constraints.
- 6.1 Derivative Free Methods for Optimal Control: Random search with linear controls and state whitening achieved state-of-the-art results on all MuJoCo benchmark tasks.The method used a simple random-search procedure with small algorithmic enhancements.
- 6.1 Derivative Free Methods for Optimal Control: MuJoCo benchmarks may be easier than believed and can reward exploitation of simulation approximations rather than reasonable walking policies.The survey presents LQR with unknown dynamics as an easier-to-understand benchmark with readily specified instances and interpretable performance limits.
- 6.1 Derivative Free Methods for Optimal Control: Model-free methods exhibited high variance across random seeds, with humanoid training slow almost a quarter of the time despite apparently good parameters.The resulting policies sometimes produced peculiar gaits, raising questions about the reliability of direct policy search for reasonable control tasks.
- 6.2 Receding Horizon Control: Receding-horizon control repeatedly replans after each action, using observed feedback to mitigate modeling errors.Longer horizons reduce dependence on Q-function accuracy but increase the importance of transition-model accuracy, creating a trade-off between learning models and value functions.
- 6.2 Receding Horizon Control: Better Q-function models can shorten the simulation horizon and move receding-horizon control closer to real-time operation while retaining high performance.Learning MPC instead improves the terminal cost iteratively, constraining terminal states to previously observed states.
- 6.2 Receding Horizon Control: Blended receding-horizon approaches can encode state and control constraints and disturbance models directly, supporting a balance between safety and performance during learning.The survey identifies safe learning while expanding knowledge of system capabilities as a central control challenge.
7 Challenges at the control-learning interface
The survey identifies several unresolved challenges at the control-learning interface, including end-to-end control, partial observability, adaptive control, and human interaction. It argues that progress will require advances in both machine learning and control.
- Models and control: Continuous-control systems may benefit more from learned models than direct model-free control, especially when model uncertainty is incorporated into control.The survey also reports learning models and value functions together for impressive results on real embodied agents.
- End-to-end control: End-to-end pixel-to-action control remains difficult because nonlinear output feedback and partial observability create severe intractability and sample-complexity challenges.Early results show promise, but pixel-trained methods have worse sample complexity than state-trained methods.
- End-to-end control: Guided Policy Search offers a promising route by using state-feedback controllers to generate training data for sensor-to-action mappings.The proposed coupling gradually reduces reliance on a precise state estimator.
- Partial observability: Moving from fully observed to partially observed scenarios makes control exponentially more difficult, while using diverse sensors safely and reliably remains an active challenge.
- Adaptive control: Adaptive control is substantially harder than episodic reinforcement learning because the policy must be learned online from a single trajectory rather than repeated simulator access.Even for LQR, the best adaptive-control approach remains unsettled, and earlier guarantees were asymptotic.
- Human interaction: When humans are in the loop, modeling their behavior can require inverse optimal control or inverse reinforcement learning to estimate their unknown objectives.Game-theoretic formulations treat human actions as disturbances, but assume humans optimize an objective that may itself need to be learned.