PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned

Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, Sergey Levine

arXiv:2102.02915v1cs.ROcs.LG

TL;DR

Real-world robotic deep RL must learn from raw observations under limited supervision, uncertain rewards, and physical constraints that simulated benchmarks often omit. This review synthesizes three prior case studies and related experience to assess how deep RL challenges have been addressed in robotics. The cases show successful learning of walking, grasping, and complex manipulation, including generalization, while substantial robustness, reward-specification, and sim-to-real challenges remain.

  • Problem

    Deep RL research often emphasizes games and simulated control, leaving limited connection to the constraints of learning complex behaviors from raw observations in real environments.

  • Method

    The article reviews robotic deep RL through three prior case studies, discusses selected challenges and empirical practices, and identifies outstanding real-world robotics problems.

  • Results

    Deep RL has been applied successfully to quadrupedal walking, grasping novel objects, and varied complex manipulation, with effective generalization reported in robotic grasping.

  • Takeaways & Limitations

    Deep RL can learn directly on real robots from raw sensory modalities and handle physically challenging tasks within the demonstrated case-study settings.

  • Takeaways & Limitations

    The experiments were conducted in constrained laboratory environments, with limited robustness to realistic visual perturbations and object-specific training in general.

Abstract

from arXiv · show

Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time,real world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn; as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.

1 Introduction

Robotic learning connects machine learning and robotics, exposing realistic constraints absent from simulation while offering robots learning-based flexibility. The article surveys these challenges through prior case studies and empirical recommendations.

  • Motivation: Robotics exposes learning constraints because robots receive raw sensory observations and little detailed supervision beyond those readings.Robots must also develop an internal representation of progress because they do not readily receive task-shaped rewards.
  • Motivation: Learning-based techniques can help robots operate in less structured environments, handle unknown objects, and learn state representations suitable for multiple tasks.
  • Challenges: Entering robotics presents a significant barrier for machine learning researchers, while robotics researchers face numerous algorithmic and robot-design choices.These choices include initialization, safety, behavior, and goal or reward definition.
  • Article scope: The article provides a high-level robotics-focused overview of deep RL, summarizes how prior work addressed key challenges, and identifies outstanding challenges.Many of the remaining challenges are not yet active research topics in the broader RL community.
  • Article scope: Rather than another comprehensive literature review, the discussion centers on three prior case studies and experience-based topics grounded in empirical evidence and alternative options.The intended audience includes roboticists using deep RL and machine learning researchers working with robots.

2 Background

The background formalizes robotics problems through states, actions, dynamics, observations, goals, and rewards. When dynamics are unknown, solving the problem falls into reinforcement learning rather than classical planning or optimal control.

  • Problem formulation: A robotics problem specifies state and action spaces plus dynamics describing how actions influence system state.The state includes internal robot variables and the world state intended for control.
  • Problem formulation: Because states are often not directly observable, robots use sensor observations to infer the system state.
  • Classical control and RL: Classical planning and optimal control use known dynamics to search for action sequences that reach goals or maximize reward.
  • Classical control and RL: When the dynamics model is unknown, the problem falls within reinforcement learning.

3 Case Studies in Robotic Deep RL

The case studies span manipulation, model-free skills, generalizable grasping, and locomotion, illustrating both the capabilities and limits of deep RL on real robots. Across tasks, performance depends on assumptions about state access, resetting, data collection, visual variability, and simulation-to-real transfer.

  • Learning manipulation skills: Guided policy search efficiently learns image-based manipulation by distilling multiple local policies into a global policy.Its assumptions become difficult when low-level state is unavailable or resetting the environment is challenging.
  • Model-free skill: Model-free methods can integrate experience across initial states and goals with sub-linear sample growth, while guided policy search generally scales linearly with initial-state variability.Model-free methods also apply to image-defined tasks without an explicit representation-learning phase.
  • Model-free skill: Model-free deep RL has learned real-world door opening, assembly, stacking, peg insertion, throwing, and dexterous manipulation from low-dimensional state observations.Recent off-policy methods have also enabled learning directly from raw images, although this remains particularly difficult.
  • Challenges and limitations: Real-world model-free skills were generally learned in constrained laboratories and had limited robustness to visual perturbations and unseen objects.Open-world data collection remains challenging, especially outside specialized setups that support unattended collection and restricted actions.
  • Learning to grasp with deep RL: QT-Opt frames grasping as multi-step closed-loop control, using self-supervised grasp outcomes to acquire strategies such as pregrasp manipulation, retrials, and disturbance recovery.It operates from monocular RGB observations with flexible three-dimensional gripper control.
  • Learning to grasp with deep RL: 86% grasp success was achieved using offline data, increasing to 96% after joint finetuning with 28,000 additional online grasps.The results support reusing previously collected experience and combining offline with online training for scalable robotic learning.

4 Outstanding Challenges in Deep RL and Strategies to Mitigate Them

Real-world deep RL faces challenges in sample efficiency, optimization stability, visual representation learning, and data collection. The article surveys strategies including off-policy reuse, model-based learning, learned visual representations, offline training, and simulation.

  • Challenges: Real-world deep RL must address optimization instability, local optima, delayed rewards, hyperparameter sensitivity, and the high interaction cost of robotic training.Common RL methods may require millions of gradient steps and interactions, making unmodified training prohibitive on physical robots.
  • Sample efficiency: Off-policy methods improve sample efficiency by reusing experience across updates, while model-based methods can learn from trajectories generated by an environment model.Off-policy samples may be reused hundreds or thousands of times, whereas model-based learning can require substantially less real-world data.
  • Visual representations: Input remapping and unsupervised representation learning reduce high-dimensional visual observations to lower-dimensional features for policy learning.Spatial autoencoders can summarize images with feature points learned without supervision.
  • Offline training: Offline training reuses data collected across experiments and enabled grasping policies to generalize to unseen objects with 500,000 trials.Offline training can scale policy learning to very large real-world datasets, but becomes unstable when the latest policy differs too much from the data-collecting policy.
  • Simulation: Simulation and sim-to-real transfer can reduce physical interaction requirements by generating experience faster than real time and transferring learned policies to robots.Simulation is proposed as a way to address the slow, expensive, and safety-constrained collection of real-world data.

4.3 Use of Simulation

Simulation offers fast, parallel, and safer experience generation, but policies trained in simulation can fail on physical robots because of the reality gap. The article discusses modeling, randomization, and adaptation strategies for transfer.

  • Benefits of simulation: Simulation can generate robotic experience orders of magnitude faster than real time, run many instances simultaneously, and collect data without human intervention.These properties make simulation useful for prototyping and reducing the cost and safety burden of physical data collection.
  • Reality gap: Direct deployment from simulation can fail catastrophically because modeling errors and visual discrepancies create a reality gap between simulated and real robots.The gap includes differences in robot dynamics and rendered images, while simulation benchmarks are easier than real-world equivalents.
  • Reality-gap sources: Reality-gap sources include incorrect physical parameters, unmodeled dynamics, stochastic environments, actuator dynamics, and missing latency modeling.Experiments with legged robots identified actuator dynamics and latency modeling as major contributors to model error.
  • Modeling: Accurate actuator and latency models narrowed the reality gap and enabled agile locomotion gaits learned in simulation to run on a real robot without robot-collected data.The transfer succeeded without requiring additional data collected on the physical robot.
  • Domain randomization: Domain randomization trains policies across varied dynamics, visual, and rendering parameters to improve robustness to simulation-to-reality variation.The approach randomizes robot and environment dynamics as well as textures and lighting, analogous to data augmentation.
  • Adaptation: A real-to-sim adapter achieved 70% real-world grasp success without real-world data and 91% after fine-tuning on 5,000 real-world grasps.The latter result previously required over 500,000 grasps.

4.4 Side-Stepping Exploration Challenges

Real-robot exploration is difficult because many tasks provide sparse or binary rewards, so demonstrations, scripted policies, model-based data aggregation, and reward shaping can guide learning. These aids improve access to useful experience but introduce limitations such as compounding errors, forgetting, bias, and engineering burden.

  • The exploration problem: Sparse-reward robotics tasks make exploration difficult because successful behavior can be a needle in a zero-reward haystack.Binary rewards are natural for many practical tasks, while more informative rewards require engineer-provided shaping.
  • Demonstration initialization: Imitation-learning pretraining can guide exploration, but policies may suffer compounding errors and have their initialization erased by subsequent random exploration.Dynamic movement primitives and guided policy search can provide more stable demonstration initialization than standard policy-gradient or actor-critic methods.
  • Demonstration data: Adding demonstrations to an off-policy replay buffer exposes learning to high-reward behavior, but value functions may learn demonstrated states without learning the actions that reach them.This approach is generally more effective when combined with joint behavioral-cloning and policy-gradient training.
  • Joint training: Jointly training policy-gradient and behavioral-cloning losses keeps policies closer to demonstrations, but can bias learning when demonstrations are suboptimal.The behavioral-cloning loss may prevent the policy from improving beyond the demonstrations.
  • Model-based aggregation: Model-based RL can use demonstrations to learn dynamics in important state-space regions and, with planning, enable complex behaviors such as tool use.Figure 7 describes combining unsupervised interaction and teleoperated demonstrations to learn visual dynamics and action-proposal models for unseen tools.
  • Scripted policies: Scripted policies provide initialization or large datasets, while residual RL retains the scripted component during learning rather than relying only on initialization.QT-Opt used 200,000 scripted-policy grasp attempts with 15-30% success to bootstrap a policy that later reached 96% success.
  • Reward shaping: Reward shaping can accelerate exploration by supplying guidance such as negative distance-to-goal rewards, but this guidance is difficult to design from image-only inputs and across many tasks.The required state configuration can grow exponentially when multiple manipulation tasks and variations must be handled.

4.5 Generalization

Generalization to new skills, environments, and tasks remains unsolved, but restricted forms can succeed when training data covers the desired variation and evaluation separates training from testing.

  • 4.5 Generalization: Generalization to arbitrary new skills, environments, or tasks remains unsolved and is necessary for robots operating across varied real-world scenarios.The review identifies data diversity and a correct train-test evaluation protocol as central requirements.
  • 4.5.1 Data Diversity: QT-Opt collected data from more than 1,000 object types to support generalization to objects unseen during training.Using only a small object set might not provide the needed generalization capability.
  • 4.5.1 Data Diversity: Greater environment diversity can reduce policy performance, often requiring larger and better neural networks to preserve performance on test objects.Kalashnikov et al. (2018) required a larger and deeper-than-usual Q-function for its broad object variety.
  • 4.5.2 Evaluation Protocol: Generalization requires tuning the entire system, including hyperparameters, against separate training and evaluation MDPs.The training and evaluation MDPs should differ along the variation the policy is intended to generalize across.

4.6 Avoiding Model Exploitation

Model-based RL can select poor actions by exploiting inaccurate regions of a learned dynamics model. Broad data, iterative data aggregation, multi-step losses, shorter horizons, and replanning are described as mitigation strategies.

  • 4.6 Avoiding Model Exploitation: Model exploitation occurs when action optimization finds state-space regions where an imperfect dynamics model is erroneously optimistic.This can result in poor action selection.
  • 4.6.1 Broad Data: Broad action and state distributions can make optimization under the learned model more successful.This strategy is especially relevant when data collection can cover broad distributions.
  • 4.6.2 Data Aggregation: Data aggregation interleaves real-world collection and model learning, adding data when model inaccuracies are exploited.The procedure resembles DAGGER and retrains the model with newly collected experience.
  • 4.6.3 Long-Horizon Error: Small single-step model errors can accumulate over long-horizon plans, especially after contact events cause predicted and real trajectories to diverge.Multi-step losses, shorter horizons, and replanning are reported as effective ways to limit accumulation and recover from exploitation.

4.7 Robot Operation at Scale

Scaling robot operation requires substantial data, careful engineering for unattended execution, and safeguards against wear, failures, and environmental changes. Hardware degradation can also confound policy comparisons over time.

  • 4.7 Robot Operation at Scale: Deep RL remains data-hungry, particularly when images are part of the observation, while one-human-per-robot monitoring has been a common safety practice.The operator can reset scenes, stop unsafe behavior, and support continuous operation.
  • 4.7.1 Experimental Setup: Successful experiments often depend on engineered setups or task choices that let robots reset scenes for unattended, potentially round-the-clock operation.The experimental setup is described as an important but often overlooked part of success.
  • 4.7.2 Reliability: Round-the-clock operation wears experimental setups through repeated unintended contact, requiring higher mean-time-between-failure and fail-safe redundancies.Useful training data must be preserved while the root causes of interventions are addressed.
  • 4.7.3 Non-Stationarity: Policies can fail when lighting or hardware dynamics change after training, including changes caused by battery level, wear, or hardware failure.Evaluation results may therefore differ across times or environmental conditions.
  • 4.7.3 Non-Stationarity: A hardware degradation of the fingers caused a consistent 5% performance drop after as few as 800 grasps on one robot.The review recommends proper A/B testing protocols to mitigate evaluation confounds.

4.8 Asynchronous Control: Thinking and Acting at the Same Time

Real robots execute asynchronously, so sensing-to-action latency violates the MDP assumption that the observed state remains fixed until action application. The section also highlights reward specification as a separate perceptual challenge, with current approaches lacking generality.

  • Asynchronous execution: Latency makes the next state depend on an unobserved delayed state, violating the MDP assumption and potentially causing RL algorithm failures.The delay spans sensing, transmission, computation, and actuation.
  • Asynchronous execution: Model-based planning can worsen latency because computationally expensive trajectory rollouts add tens of milliseconds before action execution.The cross-entropy method is given as an example of a planner whose computation alone can take tens of milliseconds.
  • Reward specification: Real-world reward assignment is difficult because task success often requires perceptual measurement rather than direct access to simulator or game state.The authors use additional sensors or simple heuristics in many case studies.
  • Reward specification: Sensor instrumentation and heuristics can provide task rewards, including door angles, walking speed, gripper encoders, and image comparisons.These methods supply task-specific success information during learning.
  • Reward specification: Reward methods based on sensors, heuristics, or goal images do not necessarily generalize to every robot task.The passage frames general reward specification as an unresolved breadth problem.

4.10 Multi-Task Learning and Meta-Learning

Multi-task and meta-learning aim to reuse experience so robots can learn future tasks efficiently, but task selection, conflicting optimization signals, and scaling remain substantial challenges.

  • Motivation: Multi-task learning trains several tasks simultaneously, while meta-learning trains across tasks so future tasks can be learned efficiently.Both approaches seek to avoid training each new task completely from scratch.
  • Task collection: Representative training tasks are difficult to specify because defining rewards for tens or hundreds of tasks is harder than defining one task reward.The task collection must represent tasks encountered during generalization or adaptation.
  • Optimization: Different learning rates and conflicting gradient signals create optimization challenges that are exacerbated in reinforcement learning.These difficulties arise when multiple tasks are learned together.
  • Optimization: Optimization challenges are less severe for similar tasks but become major challenges for more distinct tasks.The passage contrasts task similarity as an important condition for multi-task optimization.
  • Scaling: Scaling to many tasks retains challenges involving reward specification, relevant environment resets, robot operation at scale, and non-stationarity.The section presents these difficulties as becoming more complicated as task counts increase.

4.11 Safe Learning

Safe real-robot learning must limit damaging exploration and unexpected actions while preserving useful policy flexibility. The discussed approaches combine workspace restrictions, impact detection, action smoothing, trajectory priors, safety constraints, and robust training.

  • Safety motivation: Unsafe exploration can cause falls, self-collisions, jerky actuation, or obstacle collisions that damage robots and their surroundings.Repairs and manual interventions may then be required.
  • Action restrictions: Restricting the action space can enforce safety in semi-static grasping and manipulation tasks, but is often too restrictive for broader applications.Workspace limits are especially suitable when the relevant environment can be modeled.
  • Impact handling: Force-torque sensing complements modeled safety limits by detecting impacts and stopping motion when unmodeled objects are encountered.The RL agent experiences the stopped action as having a truncated effect.
  • Smooth actions: Random action noise can produce jerky motions, so reward penalties, smooth reference trajectories, or low-pass filters can reduce exploration-induced damage.These techniques may require additional manual tuning or user-specified data.
  • Smooth actions: PMTG combines a smooth periodic open-loop trajectory generator with a learned feedback policy that modulates walking speed, direction, and style.The authors report that this incorporates locomotion priors and regularizes the learned gait for safer deployment or direct real-robot learning.
  • Safety monitoring: Safety checks can detect impending collisions or excessive power and torque, but they require careful tuning and rich onboard sensing.The checks support deploying a recovery policy or shutting the robot down.
  • Safety constraints: Constrained MDPs model safety through hard constraints; roll and pitch constraints can guarantee that a robot remains upright when satisfied throughout training.In the locomotion projects, these constraints provide a rough measure of balance.
  • Robustness: Domain randomization and perturbation forces train policies to react to varied physical parameters and unseen situations.Perturbations may be random or adversarial.

4.12 Robot Persistence

Robot persistence requires maintaining physical capability and repeatedly completing tasks with minimal human intervention so learning can collect large, diverse datasets. In real environments, workspace failures, difficult resets, irreversible tasks, and limited scalability remain major barriers.

  • Persistence concept: Robot persistence is the capability to keep collecting data and training with minimal human intervention, supporting the large and diverse datasets needed by deep neural networks.The authors divide persistence into self-persistence and task persistence.
  • Self-persistence: Self-persistence means retaining the robot’s full range of motion while performing a task despite possible collisions or damage.Damage can remove abilities and require human intervention.
  • Task persistence: Task persistence requires repeatedly accomplishing a range of tasks, including hundreds of thousands of grasping trials for learning.Retrying tasks is tightly coupled to the environment and remains unsolved for many tasks.
  • Recoverability: Limited workspaces can leave objects unreachable, requiring exploration strategies that avoid unrecoverable states and human intervention.In image-based settings, even identifying states such as an object falling out of a bin is difficult.
  • Environment reset: Environment reset is difficult because completing another trial may require a sequence of robotic tasks as hard as the task being learned.Examples include unscrewing a bottle cap before trying to screw it again and resetting pouring or assembly tasks.
  • Irreversible tasks: Physically irreversible tasks may require other robots to deliver new objects, and that supply operation can be harder than the learning task itself.Examples include welding, cutting, and writing.
  • Open challenges: Task persistence remains mostly open because diverse initial states and object configurations are difficult to cluster or reset automatically.Restricting tasks to bins can enable millions of trials, but many tasks lack those favorable properties.
  • Open challenges: Human-performed resets made data collection time-consuming and physically burdensome, yielding only a few hours and fewer than 1,000 trials.The cited work stopped when operators began experiencing back pain.

5 Discussion and Conclusions

The review finds that deep RL can learn physically demanding robotic skills directly in the real world, but efficiency, stability, generalization, and autonomous data collection remain important challenges. Addressing these challenges could shift robot capabilities toward being limited by available learning time rather than engineering time.

  • Deep RL has been applied successfully to quadrupedal walking, grasping novel objects, and varied complex manipulation skills.
  • These case studies show that deep RL can learn directly from raw sensory modalities, including camera images, while handling substantial physical challenges.
  • Policies trained with deep RL can generalize effectively, including in robotic grasping experiments involving novel objects.
  • Efficiency and stability remain major challenges despite evidence that appropriate algorithms and hyperparameter choices can make deep RL less inefficient than commonly believed.
  • Robotic deep RL requires substantial robot data collection and engineering practices that keep robots operational with minimal human intervention and without persistent oversight.
  • Resolving these challenges could enable robots to learn more tasks, with capabilities increasingly constrained by available robot learning time rather than engineering time.
Loading 2102.02915v1…