Source-linked AI summary
Challenges of Real-World Reinforcement Learning
Gabriel Dulac-Arnold, Daniel Mankowitz, Todd Hester
TL;DR
Real-world RL remains difficult because assumptions common in research settings—such as plentiful simulation data and separable training environments—often fail in practice. The paper defines nine practical challenges, surveys approaches and evaluation criteria for them, and presents a modified control-suite testbed. Its central conclusion is that progress toward deployable RL requires addressing these challenges together, with human experts helping formulate rewards and safety constraints and interpret actions.
Problem
Real-world systems often lack suitable simulators and separate training environments, while safety violations may be rare in logs and safety constraints may remain unspecified.
Method
The paper defines nine practical RL challenges, surveys literature approaches and challenge-specific evaluation criteria, and modifies a control-suite task to present them together.
Results
The paper presents an example environment and evaluation criteria for measuring progress on practical RL challenges, while reporting little work that addresses all nine together.
Takeaways & Limitations
Deploying RL requires collaboration with product owners or other experts to formulate rewards and safety constraints, obtain demonstrations, and build confidence through explanations.
Takeaways & Limitations
The framework assumes the standard Markov Decision Process formulation, with later modifications for safety constraints, robustness, non-stationarity, and partial observability.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.
1. Introduction
Real-world RL differs from simulated research settings because practical systems are poorly defined, difficult or costly to simulate, stochastic, and safety-constrained. The paper organizes these difficulties into nine challenges and proposes approaches, evaluation criteria, and a modified control-suite testbed for studying them.
- Real-world RL adoption is slower because experimental setups leave a wide divide from poorly defined real-world systems.
- The paper identifies nine practical challenges, including offline learning, limited samples, continuous spaces, safety, partial observability, reward complexity, explainability, real-time inference, and delays.
- For each challenge, the paper reviews approaches and evaluation methods, then modifies a control-suite task to present the challenges together.
- Many real systems lack good simulators, are stochastic and non-stationary, impose strong safety constraints, and are expensive or slow to run.
- Production deployment requires evaluation beyond average return, including worst-case performance, safety, and separate reward components.
2. Practical Challenges
The paper frames real-world RL as a set of practical challenges spanning data collection, system complexity, safety, evaluation, and deployment constraints. It organizes existing approaches and challenge-specific evaluation criteria toward algorithms applicable across many real-world problems.
- Challenge framework: The paper presents nine challenges, including offline learning, limited samples, high-dimensional spaces, safety, partial observability, reward ambiguity, explainability, real-time inference, and large or unknown dynamics.It notes that research has addressed these challenges individually, while the paper seeks to guide evaluation and development of methods addressing them together.
- Data and training constraints: Real-world RL often requires learning from fixed logs and limited real-system samples because direct training and exploration are costly or unavailable.The paper describes batch off-line, off-policy training and emphasizes data efficiency for slow, fragile, or expensive systems.
- Data and training constraints: Batch RL iteratively trains an initial policy from behavior-policy data, gathers new experience, and retrains subsequent policies.The framework leaves the training algorithm open and allows different batches of experience across iterations.
- Evaluation and deployment: Off-policy evaluation estimates a new policy’s performance without deploying it, which is important when production regressions are costly.Evaluation becomes harder as the target and behavior policies, and their resulting state distributions, diverge; importance sampling, direct, and doubly robust methods are discussed.
- Evaluation and deployment: Warm-start performance and data efficiency can be measured by the policy’s return from behavior-policy data and the data required to exceed a desired performance threshold.Initial performance may determine whether system owners grant access, while threshold-based evaluation tests learning efficiency.
- System complexity: Real-world systems commonly combine large continuous state and action spaces with expensive operation, creating difficulties for traditional RL algorithms.Proposed responses include nearest-neighbor action selection, action elimination, and relevance networks for large or continuous action spaces.
2.4. Satisfying Safety Constraints
Real-world RL must treat safety as a first-class constraint during both operation and exploration, using formalisms and evaluations that expose violations across individual constraints. Robustness to hidden state, changing dynamics, and perturbations is likewise necessary for deployment.
- Safety constraints: Safety constraints can protect physical systems and their environments from hazards such as excessive temperatures, contact forces, battery depletion, obstacles, and excessive velocities.These constraints matter during normal operation and exploratory learning, and may be enforced by a fallback watchdog controller.
- Safety constraints: Constrained MDPs maximize cumulative reward subject to per-constraint cost limits, supporting state-action restrictions ranging from simple bounds to dynamic collision avoidance.The framework can represent total, discounted, or average cumulative costs, with constraints specified by c_k(s, a).
- Safety constraints: Budgeted MDPs treat constraint levels as unknown and learn policies parameterized by those levels, allowing users to inspect return–constraint trade-offs when limits are not absolute.This setup better matches real-world cases where small violations may be tolerated rather than categorically forbidden.
- Evaluation: Safety evaluation should count violations globally and separately for each constraint, then visualize their evolution during training and deployment.Individual counts reveal which requirements drive policy behavior; K denotes the number of safety constraints in the CMDP.
- Robustness: Real systems are often partially observable, stochastic, noisy, and non-stationary, so robustness can be evaluated across perturbed test environments and changing parameterizations.Relevant perturbations include sensor and action noise, delays, and variations in physical parameters; robust formulations can become overly conservative.
2.6. Unspecified and Multi-Objective Reward Functions
Real-world RL often faces reward functions that are unspecified, multi-objective, risk-sensitive, and uneven across task instances or users. The paper therefore emphasizes decomposed and distribution-aware evaluation, alongside approaches for inferring intended objectives and handling trade-offs.
- Unspecified and multi-objective rewards: Real systems commonly have multiple costs and unclear optimization goals, so a single global reward may fail to capture the objectives stakeholders need maintained.Examples include jointly reducing time-to-target and energy use, making reward formulation a substantial part of deployment work.
- Risk and distribution: Policy quality should be assessed across task instances and cohorts, not only by expected cumulative reward, because factories and recommender systems must serve every robot or user.Cohort-level analysis also supports fairness evaluation, while CVaR focuses on lower-tail outcomes.
- Unspecified rewards: Inverse reinforcement learning and related methods infer intended reward functions from demonstrations, supplied rewards, or training MDPs to improve alignment in new scenarios.These approaches target cases where the stated reward does not fully specify the desired behavior.
- Multi-objective evaluation: For multi-objective rewards, the paper proposes evaluating the per-component reward vector when the global reward is a linear combination of sub-rewards.Tracking objectives separately exposes policy trade-offs so stakeholders can choose preferred compromises.
- Explainability: Human operators need explanations of policy intent and failure cases, especially when a learned controller may discover unexpected ways to control a system.Explainability can be evaluated through human judgments, including A/B experiments measuring understanding of policy expressions.
3. Example Environment
The paper modifies the DeepMind Control Suite humanoid task into a testbed containing all nine real-world RL challenges. It pairs these modifications with challenge-specific metrics intended to assess practical applicability.
- Environment construction: The example environment starts from the Control Suite humanoid task and modifies it to present all nine real-world RL challenges.The humanoid already supplies high-dimensional continuous state and action spaces.
- Purpose: The environment is intended both to drive research on practical RL and to evaluate whether candidate algorithms address real-world applicability challenges.It serves as a common testbed for researchers developing algorithms across the challenge set.
- Environment construction: Batch RL is simulated by training an initial behavior policy, generating its dataset, and evaluating algorithms across datasets with different size, quality, and coverage.This represents taking over control from an existing controller.
- Challenge modifications: The task enforces online-only interaction, real-time inference, real-time physics simulation, safety constraints, changing domain parameters, multi-objective rewards, and actuator delays.Actuator delays are implemented by storing requested actions in an n-step queue before passing them to the simulator.
- Evaluation: Evaluation includes warm-start performance, samples needed to reach a performance threshold, safety-violation counts, worst-case reward, separate objective rewards, and qualitative explainability judgments.The authors argue that these evaluators are needed alongside average trial rewards to assess real-world aptitude.
4. Related Work
Related work has identified subsets of practical RL difficulties and proposed more robust evaluation, while this paper targets a setting combining the challenges in a concrete testbed.
- Practical RL challenges: Hester and Stone identify four robot-focused challenges—sample efficiency, high-dimensional spaces, sensor and actuator delays, and real-time inference—that overlap with this paper’s broader set.Their scope is specifically reinforcement learning on robots.
- Evaluation: Henderson et al. study variability across existing RL setups and propose robust performance estimators for learning algorithms.Their work is restricted to existing environments and complements the evaluation criteria proposed here.
5. Conclusion
The paper identifies nine practical RL challenges and emphasizes that progress is needed on their combination, not only individually. It also highlights model-based methods, ensembles, and collaboration with human experts as recurring themes for deployment.
- Conclusion: Nine practical RL challenges are presented as a combined target for research and production deployment.The paper notes that existing work has largely addressed these challenges individually.
- Conclusion: Model-based RL may address sample efficiency while also supporting off-policy evaluation, robustness, and explainability.
- Conclusion: Ensembles, expert input, reward and safety design, demonstrations, and explanations are identified as recurring deployment themes.The paper connects these elements to building confidence in systems intended for real products.
Appendix
The appendix identifies a table devoted to safety-constrained control environments.
- Appendix: The appendix includes Table 3 on safety-constrained control environments.
- Appendix: The table is titled “Safety-constrained control environments.”
- Appendix: The supplied appendix passage provides the table title but no environment-level entries or comparisons.
Additional Environments
The paper modifies four DeepMind Control Suite environments with safety and non-stationarity constraints to illustrate practical RL challenges and evaluate algorithms.
- Additional Environments: Four DeepMind Control Suite environments are extended with safety and non-stationarity constraints.The constraints can be considered independently or together.
- Additional Environments: The modified environments are intended to drive research in real-world RL and assess candidate algorithms’ applicability.
- Additional Environments: Cart-Pole, Quadruped, Reacher, and Humanoid were selected for increasing difficulty and closeness to real-world control systems.
Safety
For physical systems, safety constraints can involve static, kinematic, and dynamic aspects, which the paper presents for these environments in Table 3.
- Safety: Physical-system safety constraints may involve static aspects of the system.
- Safety: Physical-system safety constraints may involve kinematic aspects of the system.
- Safety: Physical-system safety constraints may involve dynamic aspects of the system.
Robustness
Table 4 lists possible perturbations for illustrating noise and non-stationarity across the proposed environments. Because perturbation choices can make environments impossible to learn, careful design is important for useful evaluation.
- Table 4 presents possible noise and non-stationarity perturbations for each proposed environment.
- The range of possible perturbations is effectively endless, so environments can be designed with different choices.
- Poorly chosen perturbations could make an environment impossible to learn, undermining its usefulness for evaluation.