Source-linked AI summary
A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models
Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, Hua Wei
TL;DR
RL policies are mainly learned in simulators because real-world data are limited and detrimental actions can have unacceptable consequences, creating a sim-to-real gap during deployment. The paper surveys this gap through an MDP-based taxonomy, covering classic and foundation-model techniques, domain-specific issues, and evaluation methods. It concludes that progress exists, but robustness, scalability, evaluation, and simulation fidelity remain important challenges.
Problem
Limited real-world data and the consequences of detrimental actions restrict RL policy learning to simulators, creating a sim-to-real gap during deployment.
Method
The paper provides a taxonomy organized by Observation, Action, Transition, and Reward, reviews classic and foundation-model techniques, and discusses domains and evaluations.
Results
The survey synthesizes progress from domain randomization, domain adaptation, and reward shaping through foundation-model approaches, while identifying robustness, scalability, evaluation, and simulation fidelity as continuing challenges.
Takeaways & Limitations
The paper serves as a reference for sim-to-real RL by organizing techniques, domain-specific insights, and evaluation methods for future realistic deployment research.
Takeaways & Limitations
Integrating perception-oriented error detection rules into an RL framework remains a promising future avenue, while simulation fidelity and environment complexity continue to constrain reliable transfer.
Abstract
from arXiv · showhide
Deep Reinforcement Learning (RL) has been explored and verified to be effective in solving decision-making tasks in various domains, such as robotics, transportation, recommender systems, etc. It learns from the interaction with environments and updates the policy using the collected experience. However, due to the limited real-world data and unbearable consequences of taking detrimental actions, the learning of RL policy is mainly restricted within the simulators. This practice guarantees safety in learning but introduces an inevitable sim-to-real gap in terms of deployment, thus causing degraded performance and risks in execution. There are attempts to solve the sim-to-real problems from different domains with various techniques, especially in the era with emerging techniques such as large foundations or language models that have cast light on the sim-to-real. This survey paper, to the best of our knowledge, is the first taxonomy that formally frames the sim-to-real techniques from key elements of the Markov Decision Process (State, Action, Transition, and Reward). Based on the framework, we cover comprehensive literature from the classic to the most advanced methods including the sim-to-real techniques empowered by foundation models, and we also discuss the specialties that are worth attention in different domains of sim-to-real problems. Then we summarize the formal evaluation process of sim-to-real performance with accessible code or benchmarks. The challenges and opportunities are also presented to encourage future exploration of this direction. We are actively maintaining a repository to include the most up-to-date sim-to-real research work to help domain researchers.
1 Introduction
Sim-to-real deployment remains difficult because policies trained in simulators face a gap when applied to reality. This survey unifies prior work through an MDP-based taxonomy, reviews classic and foundation-model techniques, and discusses domain-specific challenges and evaluations.
- Motivation: RL policies trained in simulators can suffer severe real-world performance drops and potential safety hazards under unseen scenarios.The gap is introduced during policy training and magnified during deployment execution.
- Existing approaches: Prior sim-to-real research addresses transition dynamics, perception, and execution through techniques such as domain randomization, domain adaptation, and grounded learning.
- Taxonomy: The survey frames sim-to-real techniques around four MDP elements: Observation, Action, Transition, and Reward.
- Scope: The review covers classic methods, foundation-model approaches, domain-specific challenges, and evaluation categories for understanding policy performance.Its evaluation discussion distinguishes sim-to-real validation from Policy Evaluation.
- Contributions: The paper contributes a comprehensive literature review that connects causes of the sim-to-real gap with emerging foundation-model techniques and domain-specific prospects.
2 The RL and Sim-to-Real Issue
RL learns policies through interaction and reward within an MDP, but policies trained in simulation may fail to generalize to reality because observation, action, transition, and reward differ across environments. These discrepancies can reduce real-world performance and create safety risks, motivating an MDP-based organization of Sim-to-Real causes and solutions.
- Reinforcement Learning: RL iteratively improves a decision policy from environment interaction and reward feedback to maximize accumulated expected return.The paper describes RL through MDP components including state, action, transition, reward, and discount factor.
- Sim-to-Real Issue: A policy learned in simulator M_s may not generalize well to real-world M_r, producing a measurable Sim-to-Real performance gap.The gap compares an evaluation metric applied consistently in simulation and reality.
- Observation Gap: Observation gaps arise from incomplete perception and mismatched representations caused by factors such as perception resolution and sensor noise.These mismatches make the observations available during deployment differ from those used for training.
- Action Gap: Action gaps reflect differences between simplified or discretized simulator actions and continuous real-world control, compounded by inevitable mechanical latency.Simulators may assume instantaneous action execution, whereas real mechanical components introduce delays.
- Transition Gap: Transition gaps occur when simulator and real-world dynamics produce different next-state distributions for the same state and action.The paper expresses this as P_s(s_t+1|s_t,a_t) ≠ P_r(s_t+1|s_t,a_t), attributing it to system-dynamics differences.
- Reward Gap: Reward gaps arise from mismatched reward design and action delay or granularity, which can impair deployment performance and create safety concerns.The paper notes that simulator-designed rewards may omit real-world cases or encourage unexpected actions.
- Survey Organization: The survey organizes Sim-to-Real solutions around Observation, Action, Transition, and Reward while also covering foundation-model developments and domain-specific evaluation.It frames the issue through MDP elements and discusses foundation models as an emerging direction.
3 Techniques
The survey introduces techniques for addressing Sim-to-Real discrepancies in MDP components, beginning with observational mismatches caused by differing sensor modalities. Observation-based domain randomization targets the state element by varying visual inputs, distinct from methods targeting transition dynamics.
- Observation Techniques: Sim-to-Real techniques address discrepancies in observational data arising from sensor-modality differences such as cameras and tactile sensors.The paper presents these strategies as methods for mitigating observation-related gaps.
- Observation Techniques: Observation-based domain randomization targets the MDP state element by introducing variability into visual observations.It is distinguished from transitional domain randomization, which targets transition dynamics.
action
Sim-to-real methods address mismatches in observations, actions, transition dynamics, and model grounding through complementary techniques. The surveyed approaches include randomization, adaptation, uncertainty handling, foundation-model assistance, and robust transition modeling.
- Observation: Domain randomization varies simulation visuals and sensors so policies learn from diverse scenarios and become more robust to real-world variation.Examples include textures, lighting, object positions, colors, camera parameters, and sensor noise.
- Observation: Domain adaptation aligns simulated and real observation features, using methods such as adversarial training, embedding alignment, and realistic depth transformations.The aligned inputs may include images, sensors, or LiDARs.
- Foundation models: Foundation models provide task-relevant semantic information and can improve observation grounding or predict real-world dynamics for more reliable actions.Language descriptions can act as semantic anchors, while LLM-enhanced forward models improve next-state prediction accuracy.
- Action: Action-related methods address uncertainty and sample efficiency through action advising, robust action selection, and language-model-generated preliminary controllers.RCMP estimates confidence from the variance of multiple value-function estimates, while LLM-generated controllers guide exploration.
- Transition: Transition-focused methods combine traditional randomization, adaptation, and grounding with LLM-enhanced strategies to improve policy robustness under dynamics discrepancies.Distributionally robust approaches also study regularization and performance bounds under model uncertainty and distribution shifts.
4 Research Focus and Simulation Environments
Sim-to-real research spans multiple domains, each with specialized challenges, simulators, benchmarks, and evaluation priorities. The survey organizes these domains and highlights large-scale and generative simulation approaches for producing diverse training interactions.
- Domain-specific focus: Sim-to-real research addresses domain-specific complexities through specialized simulators, benchmarks, and research focuses.The survey discusses robotics, transportation, recommender systems, and broader applications.
- Domain-specific focus: Robotics prioritizes safety, multi-task capability, and accuracy, while transportation emphasizes multi-agent coordination, smooth traffic transitions, and real-time decisions.These priorities reflect physical hazards in robotics and rapidly changing, interactive traffic environments.
- Domain-specific focus: Recommender systems focus on continuously adapting policies to user preferences and use off-policy evaluation and counterfactual analysis to assess simulated-to-live discrepancies.These methods aim to quantify and mitigate differences between simulated user models and actual behavior.
- Simulators and benchmarks: The survey catalogs simulators and benchmarks including Gazebo, MuJoCo, PyBullet, RoboSuite, RoboSumo, Meta-World, SUMO, CARLA, RecSim, and S2R-Rec.These resources support robotics, transportation, and recommender-system research, with additional platforms for finance, healthcare, energy, and games.
- Generative and large-scale simulation: Large-scale simulation can expose policies to massive experience, including 1.6 billion km of self-play driving data associated with emergent realistic and robust behaviors.Generative approaches also target synthetic environments, physics dynamics, and training data that are realistic and diverse.
- Generative and large-scale simulation: Physics-grounded and diverse simulations move beyond hand-engineered routines by providing richer training interactions for policy learning.The survey presents these approaches as a way to capture broader physical behavior.
5 Evaluation of Sim-to-Real
The survey compares sim-to-real, sim-to-scale-down-real, and sim-to-sim evaluation settings across cost, safety, and realism. Direct real-world testing is definitive but risky, while simulated alternatives provide safer and cheaper preliminary assessments whose predictivity depends on the metrics used.
- Evaluation settings: Sim-to-real, sim-to-scale-down-real, and sim-to-sim are evaluated along the dimensions of cost, safety, and realism.The three settings provide distinct ways to assess transfer from simulation toward real-world deployment.
- Evaluation settings: Sim-to-real deploys simulation-trained policies on physical systems, providing real-time feedback but exposing experiments to unexpected learned behavior and deployment risk.It is especially relevant to robotics and autonomous vehicles.
- Evaluation settings: Sim-to-scale-down-real uses specialized physical testbeds to improve safety, cost, and fault tolerance during evaluation.Examples include robotic environments with motion capture and safety measures.
- Evaluation settings: Sim-to-sim tests policies in different, often more realistic or varied simulations to assess robustness and generalization without real-world expense or risk.It is commonly used as a preliminary evaluation step.
- Metrics and assessment: Combining evaluation settings with robust metrics enables broader assessment, but simulation-to-real predictivity still depends substantially on the specific metrics.Sim-to-real provides direct applicability evidence, whereas sim-to-sim offers a cost-effective proxy.
- Metrics and assessment: Transfer metrics quantify performance discrepancies between E_sim and E_real, with metric choices tied closely to domain tasks.The survey summarizes relevant domain metrics and works for robotics, transportation, and recommender systems.
6 Open Challenges and Opportunities
The survey identifies persistent sim-to-real challenges in simulation fidelity, safety, and foundation-model integration. It presents opportunities involving richer simulations, stronger testing and offline evaluation, grounding and uncertainty methods, and more efficient models.
- Existing sim-to-real challenges: Simulation fidelity remains limited because simplified models may omit subtle physical interactions and reduce transferred policy performance.The survey points to incorporating real-world dynamic data and developing more advanced simulators as possible responses.
- Existing sim-to-real challenges: Real-world deployment can create safety risks when policies are insufficiently tested against unknown situations.Proposed responses include comprehensive testing, training-time safety constraints, and offline policy evaluation for worst-case understanding.
- Foundation-model challenges: Foundation-model hallucinations can produce outputs inconsistent with real-world data and lead to suboptimal or unsafe RL decisions.Retrieval-augmented generation and uncertainty quantification are described as preliminary mitigation directions, but the issue remains open.
- Foundation-model challenges: Larger, more capable language models increase inference time and resource demands, especially when inference is performed at every RL training step.Task-specific distillation and lighter foundation models are identified as ways to reduce these costs.
- Opportunities: The survey concludes that foundation models could improve the effectiveness and safety of sim-to-real transfer if these challenges are handled properly.The opportunity is presented together with unresolved constraints rather than as an established outcome.
7 Conclusion
The survey organizes sim-to-real RL methods within an MDP framework and reviews progress from traditional techniques to foundation-model approaches. It also synthesizes domain insights and evaluation methods while identifying robustness, scalability, and evaluation as continuing challenges.
- Contributions: The paper categorizes sim-to-real solutions within the MDP framework and covers traditional methods alongside foundation-model advances.The framework supports a unified survey of techniques across the field.
- Contributions: The survey identifies domain randomization, domain adaptation, and reward shaping among techniques used to mitigate the sim-to-real gap.It presents these techniques as part of substantial progress rather than a complete solution.
- Open challenges: Robustness, scalability, and evaluation remain critical challenges for realistic RL deployment.The paper positions its synthesis of techniques, domain-specific insights, and evaluation methods as a reference for future work.