Source-linked AI summary
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, Peter Stone
TL;DR
Real-world robotics makes reinforcement learning difficult because physical interaction is costly and unsafe, while evidence of deep RL success remains uneven across competencies. This survey evaluates real-world DRL progress across robotic domains and identifies recurring trends, open challenges, and directions for broadly deployable solutions.
Problem
Real-world robotics limits reinforcement-learning evaluation because physical trial-and-error is costly or unsafe and differs from simulation.
Method
The survey categorizes DRL studies by robotic competency, problem formulation, solution approach, and validated level of real-world success.
Results
The analysis reviews and categorizes many real-world successes, revealing cross-domain trends, common open challenges, and avenues for algorithmic, procedural, and holistic improvements.
Takeaways & Limitations
DRL has demonstrated meaningful robotic capabilities, but broadly capable real-world systems require further advances in real-world learning and competency integration.
Takeaways & Limitations
Safety-critical navigation applications have few real-world successes because current RL methods lack sufficient robustness, explainability, and generalization.
Abstract
from arXiv · showhide
Reinforcement learning (RL), particularly its combination with deep neural networks referred to as deep RL (DRL), has shown tremendous promise across a wide range of applications, suggesting its potential for enabling the development of sophisticated robotic behaviors. Robotics problems, however, pose fundamental difficulties for the application of RL, stemming from the complexity and cost of interacting with the physical world. This article provides a modern survey of DRL for robotics, with a particular focus on evaluating the real-world successes achieved with DRL in realizing several key robotic competencies. Our analysis aims to identify the key factors underlying those exciting successes, reveal underexplored areas, and provide an overall characterization of the status of DRL in robotics. We highlight several important avenues for future work, emphasizing the need for stable and sample-efficient real-world RL paradigms, holistic approaches for discovering and integrating various competencies to tackle complex long-horizon, open-world tasks, and principled development and evaluation procedures. This survey is designed to offer insights for both RL practitioners and roboticists toward harnessing RL's power to create generally capable real-world robotic systems.
1. Introduction
Deep reinforcement learning has demonstrated strong capabilities for complex decision-making and potential for controlling challenging robotic systems, but its most notable milestones remain concentrated in simulation and games. This survey evaluates DRL’s progress in real-world robotics, identifying success factors, maturity across domains, underexplored areas, and common open challenges.
- Motivation: DRL combines reinforcement learning with deep neural networks to solve complex decision-making problems involving high-dimensional settings.RL agents learn through trial and error to maximize accumulated return using scalar rewards based on states and actions.
- Motivation: DRL could control robotic systems with high-dimensional observations and highly nonlinear dynamics for tasks conventional approaches cannot handle effectively.These approaches include classical control, optimal control, and sampling-based planning.
- Survey scope: The survey evaluates DRL’s progress in real-world robotic applications across domains, identifying success factors, maturity differences, broadly applicable techniques, and open challenges.It also highlights under-explored areas that must be addressed to advance DRL’s applications in robotics.
2. Why Another Survey on RL for Robotics?
This survey addresses gaps in prior RL-for-robotics reviews by focusing on demonstrated real-world DRL successes, introducing a comprehensive taxonomy, and refreshing analysis of the field’s progress and limitations.
- Real-world success: The survey focuses on DRL work demonstrating at least some real-world success, assessing the field’s current state and open challenges for robotic applications.It contrasts this focus with existing surveys that generally do not explicitly address real-world success.
- Real-world success: Its evaluation is grounded in a comprehensive assessment of DRL successes in robotics, including the level of real-world deployment.This distinguishes the discussion from case studies drawn only from the authors’ own research.
- Taxonomy: The survey introduces a taxonomy spanning learned robot competencies, problem formulation, solution approach, and level of real-world success.This contrasts with prior reviews focused on specific tasks or techniques and supports coverage of the broader DRL solution landscape.
- Renewed analysis: A renewed analysis is needed because DRL has progressed rapidly, while earlier surveys predated deep learning or primarily covered simulation accomplishments.The survey therefore provides an updated overview of the field’s successes and limitations.
3. Taxonomy
The survey introduces a taxonomy for analyzing DRL literature and assessing solution maturity across robotics applications. It spans four axes: learned robot competencies, RL problem formulation, solution approach, and level of real-world success.
- Taxonomy overview: The taxonomy categorizes DRL robotics papers along four axes: learned robot competencies, problem formulation, solution approach, and real-world success.This structure supports analysis of literature maturity and lessons from successes and failures.
- Robot competencies: Robot competencies are classified by whether DRL enables single-robot abilities, mobility, manipulation, or interaction with other agents.Mobility covers movement, manipulation covers moving or rearranging objects, and multi-agent competencies address interaction-level learning.
- Problem formulation: The problem-formulation axis describes the RL control problem, including action-space characteristics, within POMDP or Dec-POMDP models.Single-agent RL is typically modeled as a POMDP, whereas multi-agent RL uses Dec-POMDPs.
- Solution approach: The solution-approach axis classifies RL algorithms and enabling techniques, including simulator use through zero-shot or few-shot sim-to-real transfer and direct real-world or offline learning.Simulator usage is categorized by how simulators support practical solutions to robotic problems.
- Levels of real-world success: Real-world success is rated across six validation levels, from Level 0 simulation-only validation to Level 5 deployment on commercialized products.Intermediate levels range from limited or diverse laboratory conditions to confined or diverse representative real-world operations.
4. Competency-Specific Review
DRL has achieved its most mature real-world results in quadruped locomotion, while capabilities in bipedal locomotion, flight, navigation, and autonomous driving remain more limited or condition-dependent. Across these competencies, progress depends on sim-to-real methods, specialized controllers, and improved robustness for complex, long-horizon tasks.
- Quadruped locomotion: DRL has enabled mature real-world quadruped locomotion control, including applications in inspection, delivery, and rescue.Reported systems include ANYbotics, Swiss-Mile, and Boston Dynamics, with early validation using proprioceptive sensing for blind walking on flat indoor surfaces.
- Quadruped locomotion: Jumping and fall recovery extend quadruped competencies, but both have so far been validated only on flat surfaces.DRL policies have generated jumping trajectories with model-based tracking controllers and have been trained for fall recovery, including automatic reset applications.
- Quadruped locomotion: The most mature quadruped solutions predominantly use on-policy model-free RL, such as PPO, with zero-shot sim-to-real transfer.This approach is noted for robustness to hyperparameters, although on-policy RL may be less favorable for temporally extended action spaces or deterministic control actions.
- Bipedal locomotion: Bipedal locomotion has a sparser literature and more limited demonstrated real-world capabilities than quadruped locomotion.The reviewed work focuses on freely moving 3D bipeds and spans walking on flat indoor surfaces and varied indoor and outdoor terrain.
- Flight and wheeled navigation: DRL has achieved robust UAV flight control and off-road navigation, while autonomous-driving success remains limited to a few specific-condition examples.Quadrotor work demonstrated waypoint tracking and recovery from harsh initialization, while off-road methods include model-based, offline, and vision-based model-free RL; one cruise-control policy was deployed on 100 vehicles.
- Navigation: Real-world navigation results are uneven: DRL supports agile locomotion-integrated navigation and off-road driving, but visual-transfer limitations and lower-than-human aerial-racing robustness remain.Reported successes include high-speed obstacle avoidance, challenging-terrain navigation, kilometer-scale navigation, and zero-shot monocular-camera transfer, whereas many visual-navigation results remain simulation-limited.
5. General Trends and Open Challenges
More mature DRL solutions often use zero-shot sim-to-real transfer for locomotion and navigation, while demonstrations support real-world manipulation learning. Open challenges include sample-efficient real-world learning, long-horizon skill integration, principled system design, standardized evaluation, and leveraging datasets and foundation models.
- General trends: More mature solutions (L3-4) often use zero-shot sim-to-real transfer, particularly for locomotion and navigation, while human demonstrations support real-world learning in feasible manipulation tasks.Successful examples are much sparser when accurate simulation and real-world rollouts are prohibitive or stable, scalable RL algorithms are missing.
- Sample-efficient real-world learning: On-policy RL offers hyperparameter robustness, but its data demands hinder real-world learning, including long-horizon mobile manipulation and agile legged navigation.Long horizons, large operational spaces, sparse rewards, and complex contact dynamics contribute to poor sample efficiency.
- Sample-efficient real-world learning: Real-world learning is crucial when high-fidelity simulation is impractical, including open-world and contact-rich manipulation, lightweight quadrotor navigation, and physical HRI.The survey notes particular progress in manipulation, while safe exploration may also benefit from adaptive, less fragile hardware and passive safety mechanisms.
- Long-horizon task integration: Long-horizon tasks require directed exploration and temporal credit assignment while integrating diverse abilities, yet most reviewed RL successes remain short-horizon.Representative successes include quadruped velocity control and manipulator in-hand object rotation.
- Principled design and evaluation: Principled RL system design and standardized benchmarks are needed because heuristic choices can harm performance and varying testing conditions and metrics impede direct comparison.End-to-end and hybrid modular solutions vary by problem, and neither approach is universally superior.
- Emerging opportunities: Large-scale robot datasets and foundation models offer open opportunities for real-world RL, with foundation models showing promise for cross-domain reasoning and decision-making.The survey identifies these advances as potential tools for addressing several DRL challenges in robotics.
6. Conclusion
The survey finds that deep reinforcement learning has contributed to many robotic capabilities and real-world successes. It categorizes these successes by competency, problem formulation, and solution approach, revealing trends and avenues for future work.
- Deep reinforcement learning has played an important role in developing many robotic capabilities, producing numerous real-world successes.
- The survey reviews and categorizes these successes according to robotic competency, problem formulation, and solution approach.
- Analysis across these axes reveals general trends and avenues for future work, including algorithmic and procedural improvements and ingredients for real-world learning.
A. Term Definition · A.1. Problem Formulation
The survey organizes DRL robotics literature along four axes and defines problem formulations through action space, observation space, and reward function. These dimensions distinguish control abstraction, sensory representation, and feedback density.
- A. Term Definition: The taxonomy classifies literature by learned robot competencies, problem formulation, solution approach, and level of real-world success.
- A.1. Problem Formulation: Problem formulations are categorized by action space, observation space, and reward function.
- A.1.1. Action Space.: Low-level actions directly control joint-space variables, require minimal domain knowledge, and provide fine-grained behavioral control.
- A.1.1. Action Space.: Mid-level actions control workspace variables, such as end-effector pose or center-of-mass velocity, and are commonly converted to torques by external controllers.
- A.1.1. Action Space.: High-level actions represent temporally extended skills that shorten task horizons and keep exploration task-relevant, but require designing an appropriate skill set.
- A.1.2. Observation Space.: Low-dimensional observations compact proprioceptive information, object locations, and task information into a state vector.
- A.1.2. Observation Space.: High-dimensional observations use exteroceptive sensor data, including lidar readings, camera images, and point clouds.
- A.1.3. Reward Function.: Sparse rewards provide non-trivial feedback only occasionally, whereas dense rewards provide abundant feedback directly or through shaping components.
A.2. Solution Approach
The survey classifies solution approaches according to simulator usage, model learning, expert usage, and policy optimization.
- Solution Approach: Solution approaches are categorized by simulator usage, distinguishing zero-shot transfer, few-shot sim-to-real transfer, and direct learning offline or in the real world.The classification also asks whether and how simulators are used.
- Solution Approach: The survey records whether any part of the transition-dynamics model is learned from robot data.This perspective is termed model learning.
- Solution Approach: The classification identifies whether expert data, such as human or oracle-policy data, facilitate learning.This perspective is termed expert usage.
- Solution Approach: Policy optimization is included as a separate perspective for classifying solution approaches.The supplied passage introduces policy optimization but does not provide its subcategories.
A.2.1. Simulator Usage.
The section distinguishes simulator-use regimes in robotic reinforcement learning by whether policies transfer directly from simulation, receive limited real-world fine-tuning, or are trained entirely in the real world.
- Zero-shot sim2real trains entirely in simulation and deploys the resulting policy directly in the real world without additional learning.
- Few-shot sim2real pre-trains the robot in simulation and fine-tunes it in the real world using limited additional interactions.
- No Simulator conducts training entirely in the real world without using a simulator.
A.2.2. Model Learning. · A.2.3. Expert Usage.
The section contrasts tabula rasa RL with learning assisted by external experts. In robotics, expert input can expedite learning and includes demonstrations, planners, or oracle actions.
- A.2.3. Expert Usage.: Tabula rasa RL starts from random initialization and learns entirely through trial and error.
- A.2.3. Expert Usage.: Robotics learning can sometimes use an external expert to accelerate the learning process.
- A.2.3. Expert Usage.: External experts may provide human demonstrations, trajectory planners, or oracle actions.
- A.2.3. Expert Usage.: The survey classifies methods using external experts offline or online as works “with experts”.
- A.2.3. Expert Usage.: Expert-assisted methods are distinguished from methods that do not use external experts.
- A.2.3. Expert Usage.: Using an external expert gives learning methods an advantage over methods without expert assistance.
A.2.4. Policy Optimization.
Policy optimization in robotics can be organized into planning, offline, on-policy, and off-policy approaches, distinguished by how policies are derived and which data support learning. Representative methods include A*, MPPI, CQL, DT, PPO, TRPO, SAC, and DQN.
- Planning: Planning derives the robot policy online by solving an optimal control problem with a learned world model.Representative algorithms include A* and MPPI (Model Predictive Path Integral).
- Offline: Offline learning avoids environment interaction and learns a policy, optionally with a value function, directly from offline data.Representative algorithms include CQL (Conservative Q-Learning) and DT (Decision Transformer).
- On-policy: On-policy learning interacts with the environment and updates the policy only with transitions collected by the current policy.Representative algorithms include PPO (Proximal Policy Optimization) and TRPO (Trust Region Policy Optimization).
- Off-policy: Off-policy learning interacts with the environment and updates the policy using transitions collected by current and other or previous policies.Representative algorithms include SAC (Soft Actor-Critic) and DQN (Deep Q-Network).
A.2.5. Policy/Model Representation.
Policy and model representations span MLPs, CNNs, RNNs, and transformers, with architectures selected according to input structure, temporal dependencies, and multimodal information. MLPs target low-dimensional vectors, CNNs preserve spatial coherence, RNNs process time series using memory, and transformers fuse tokenized modalities.
- MLP-only models take 1D vector inputs and consist solely of fully connected layers, making them widely used for low-dimensional observations.
- CNNs preserve local spatial coherence and process images, 1D lidar readings or observation memory, and 3D point clouds.
- RNNs, including LSTM and GRU variants, use internal memory and are suitable for time-series data such as trajectories over time.
- Transformers process sequences of vector tokens with multi-head self-attention and support time series, natural-language instructions, visual information, and multimodal fusion.
B. Additional Tables
This section provides complete supplementary categorization of the reviewed papers across all four taxonomy axes, organized by problem formulation and solution approach for each robot competency. The tables also indicate levels of real-world success and mark papers spanning multiple categories.
- Taxonomy tables: Tables 1–2 categorize reviewed papers by problem formulation for each robot competency, while Tables 3–6 categorize them by solution approach.Together, the tables present the complete categorization across all four axes of the taxonomy.
- Real-world success: The color map identifies five levels of real-world success: Sim Only, Limited Lab, Diverse Lab, Limited Real, and Diverse Real.These success levels are used as in the tables in the main article.
- Multiple-category papers: A superscript ∗ marks papers appearing in multiple columns because they jointly adopt two different taxonomy elements.Examples include hierarchical policies producing low-level and mid-level actions, or policy networks combining CNNs and RNNs.