PaperlayerView exact revision on arXiv

Source-linked AI summary

Provably Safe Sim-to-Real Transfer

Tingting Ni, Maryam Kamgarpour

arXiv:2609.01418v1cs.LGcs.AI

TL;DR

Safe sim-to-real transfer must handle simulator mismatch while collecting real-world data safely and supporting varying reward functions. The paper formulates this as reward-free safe RL and develops a computationally efficient algorithm that identifies mismatch, reuses reliable simulator transitions, and corrects errors with real data. Its guarantees provide safe exploration and feasible ϵ-accurate planning, with real-world sample complexity concentrated on the mismatch region when simulator errors are localized.

  • Problem

    Safe sim-to-real transfer must reduce costly real-world interaction despite simulator mismatch, while ensuring safe exploration and planning for varying reward functions.

  • Method

    The paper formulates the task as reward-free safe RL and uses an algorithm that identifies mismatch regions, reuses reliable simulator transitions, and corrects them with real-world data.

  • Results

    The algorithm guarantees safe exploration and feasible ϵ-accurate planning with high probability, with sample complexity tied to the mismatch region and separation gap.

  • Takeaways & Limitations

    When the mismatch region is small and well separated, simulator access reduces required real-world samples by concentrating learning where simulator knowledge is insufficient.

  • Takeaways & Limitations

    The analysis is limited to tabular CMDPs and assumes separable simulator shifts; extending it to continuous settings and more general simulator bias remains future work.

Abstract

from arXiv · show

To mitigate the sample complexity of real-world reinforcement learning (RL), a common practice is to first train a policy in a simulator, where samples are cheap, and then deploy the learned policy in the real world with the hope that it generalizes effectively. Such direct sim-to-real transfer is not guaranteed to succeed: simulator-trained policies can be suboptimal in the real world due to sim-to-real mismatch. Correcting this mismatch requires collecting data from the real system, but in many applications, such as robotics and healthcare, this data-collection process is itself subject to safety constraints. This gives rise to the problem of safe sim-to-real transfer: how can an agent exploit an imperfect simulator while ensuring safe real-world data collection and learning a near-optimal feasible policy for the target system? We address this problem by formulating safe sim-to-real transfer within the framework of reward-free safe RL. We design a computationally efficient algorithm that exploits simulator information to provably reduce real-world interaction while ensuring safe exploration and enabling the computation of a near-optimal feasible policy for any potential reward function. Our real-world sample complexity bound characterizes the benefit of using the simulator in terms of the sim-to-real mismatch.

1 Introduction

Safe sim-to-real transfer addresses costly, safety-constrained real-world interaction caused by simulator mismatch. The paper proposes a reward-free approach that uses simulator information to reduce interaction while supporting safe exploration and near-optimal planning.

  • Motivation: Real-world RL is challenging because interactions are costly and must satisfy safety requirements.Examples include collision avoidance in robotics and operational constraints in healthcare.
  • Motivation: Simulator mismatch can make policies that appear safe in simulation unsafe or suboptimal on the real system.Physical models and calibration provide cheap simulator interactions but cannot perfectly model the real world.
  • Prior approaches: One-shot transfer and safe meta RL depend on coverage assumptions that may fail when the simulator is biased.Worst-case or distributional optimization can also produce conservative policies for the specific real system.
  • Proposed approach: The paper formulates safe sim-to-real transfer as reward-free safe RL without requiring simulator coverage.The goal is safe exploration and reward-free planning for varying reward functions while using limited real-world interaction.
  • Contributions: The algorithm identifies reliable simulator regions, reuses certified transitions, and corrects mismatch with real-world data.Its guarantees include safe exploration and accurate planning, with sample complexity characterized by the mismatch region and separation gap.
  • Results: When mismatch is large, the framework recovers the fully online setting and improves its sample-complexity bound.Experiments in a safety-critical gridworld show larger simulator benefit when sim-to-real mismatch is smaller.

2 Problem setting

The problem is formulated on a real CMDP and a simulator sharing spaces, horizon, utilities, and safety constraints but differing in transition dynamics. The objective is safe data collection followed by reward-free feasible planning under explicit baseline and mismatch assumptions.

  • CMDP formulation: A CMDP specifies finite state and action spaces, horizon, transition dynamics, an initial state, utility functions, and a safety constraint.The constraint function encodes the safety requirement, while utility functions measure policy performance.
  • Objective: Varying rewards motivate reward-free planning, while a common constraint function represents unchanged physical safety specifications across deployments.Collecting new real-world data separately for every reward function would be sample inefficient.
  • Sim-to-real setting: The real system and simulator differ only in transition dynamics, while the agent has full simulator knowledge and can roll out policies in the real world.Real-world rollouts generate trajectories used during learning.
  • Objective: The agent must ensure every deployed data-collection policy is feasible in the real system.This requirement formalizes safe exploration because constraint violations such as collisions are not permitted.
  • Objective: After data collection, the algorithm must compute a feasible and ϵ-accurate policy for every reward function in the family without further real-world interaction.Efficiency is measured by the number of real-world state-action pairs sampled.
  • Assumptions: Safe exploration assumes a known strictly feasible baseline policy with margin ξ, which provides an exploration buffer and enables finite-sample safety certification.If ξ = 0, estimation error could prevent reliable distinction between feasible and infeasible behavior.
  • Assumptions: The separability assumption requires each real-simulator transition pair to be identical or separated by a known magnitude σs.This permits finite-sample identification of mismatch triples and makes simulator knowledge reliably exploitable.
  • Assumptions: The mismatch region B contains time-state-action triples where real-world and simulator transition kernels differ.When the simulator accurately models reality, B is small.

3 Algorithm Design

The algorithm progressively identifies where the simulator is unreliable, builds a hybrid model using real data only there, and selects safe exploration policies that reduce reward-free planning uncertainty.

  • Mismatch estimation: The algorithm maintains confidence bounds and shrinks an estimated mismatch region initialized as the full time-state-action space.A triple is removed only when its empirical real transition model is sufficiently close to the simulator.
  • Mismatch estimation: With high probability, the estimated mismatch region decreases monotonically while retaining every true mismatch triple.Thus, mismatch triples are not removed during the iterative certification process.
  • Hybrid model: The hybrid model uses empirical real-world transitions on possible mismatch triples and simulator transitions elsewhere.This avoids relearning transition dynamics in regions certified as non-mismatch.
  • Safe exploration: The exploration objective targets estimated mismatch-region areas where transition uncertainty can affect reward-free planning.A computable bonus certificate upper-bounds the unknown value-estimation error.
  • Safe exploration: The exploration problem is solved as a standard CMDP with bonus reward and safety constraint.Existing constrained policy optimization or occupancy-measure linear programming can solve it efficiently.
  • Safe exploration: When the empirically selected exploration policy may violate real-world constraints, the algorithm mixes it with the strictly feasible baseline policy.The mixture coefficient uses the baseline margin to compensate for possible constraint violation.
  • Termination: The algorithm stops when the uncertainty certificate is at most ε/2, indicating sufficient accuracy for reward-free planning.The same certificate also bounds possible constraint violation when transferring from the hybrid model to the real system.

4 Theoretical Guarantees

Under the baseline-feasibility and separability assumptions, the algorithm provides high-probability safe exploration and feasible ϵ-optimal planning. Its sample complexity concentrates real-world learning on the mismatch region and improves over the fully online benchmark.

  • Main guarantees: With probability at least 1 −δ, Theorem 4.1 bounds real-world samples while guaranteeing safe exploration and ϵ-accurate planning.The theorem sets ε = ξϵ/(2H) under the stated assumptions.
  • Main guarantees: Every executed policy is feasible in the real-world CMDP.This is the theorem’s safe-exploration guarantee.
  • Main guarantees: For any r ∈F, the output policy is feasible and ϵ-optimal for the true CMDP.The guarantee applies after data collection through reward-free planning.
  • Sample complexity: The sample-complexity bound separates learning real dynamics on B from certifying the remaining H|S||A| −|B| triples as non-mismatch.The first term scales with |B|, while certification depends polynomially on log σs^-1.
  • Comparison: When B = [H] × S × A, the method recovers the online setting and improves over Huang et al. [2023] by H/ξ2 for ϵ ≤min{1, ξ}.When |B| ≪H|S||A| and σs is well separated, the saving is of order e O(H2|S||A|/(ξ2|B|)).
  • Proof strategy: The proof combines reliable mismatch-region estimation, a value-estimation uncertainty certificate, and the Slater margin for safe mixtures.These ingredients support the theorem’s safety and planning guarantees.

5 Computational experiment

The gridworld experiments compare simulator-assisted safe transfer with reward-free safe and unconstrained baselines, showing a balance between safety and real-world sample efficiency. State visitation reveals that the proposed method concentrates interaction on mismatch regions while avoiding unsafe areas relative to unconstrained exploration.

  • Experiment setup: The 5 × 5 gridworld uses horizon H = 12 and compares reward-free safe RL, unconstrained sim-to-real RL, and the proposed method.The reward-free baseline isolates simulator access, while the unconstrained baseline isolates the cost of safe exploration.
  • Experiment setup: The environment contains three windy cells near an unsafe wall, where real transitions can push the agent opposite its chosen direction.This makes simulator mismatch safety-critical because simulator-optimized motion away from the wall can become motion toward it.
  • Safety versus sample complexity: The unconstrained baseline violates the safety constraint in 68.5% of learning iterations, while reward-free safe RL requires 7.5× as many real-world samples as the unconstrained baseline.All methods achieve accurate planning, but safety and sample complexity differ substantially.
  • Safety versus sample complexity: The proposed method maintains safe exploration using only 2× more samples than the unconstrained baseline.It therefore lies between the unconstrained method’s sample efficiency and the reward-free safe baseline’s safety guarantee.
  • State visitation: Sim-to-real safe RL concentrates visits on mismatch regions and spends less time in unsafe regions than the unconstrained baseline.The reward-free safe baseline instead spreads visits almost uniformly, relearning dynamics already modeled correctly by the simulator.
  • Assumption verification: The safety margin of the baseline policy can be estimated by rolling it out in the real system, while the experiments manually construct that policy from gridworld structure.Additional implementation details and ablations are provided in Appendix B.

6 Conclusion and future work

The paper concludes that simulator-guided safe exploration can reduce real-world interaction while supporting reward-free planning, with benefits tied to mismatch size and separation. Its scope is currently limited to tabular CMDPs and relies on separability, while several comparison and implementation details qualify the broader context.

  • Conclusion: The proposed algorithm identifies simulator mismatch, corrects it with real-world data, and guarantees safe exploration and ϵ-accurate planning with high probability.Its sample-complexity benefit scales with the mismatch region rather than the full state-action-time space when mismatch is small and well separated.
  • Comparison with related work: The method is the only compared approach combining simulator mismatch, constraints, safe exploration, and reward-free planning for arbitrary rewards.The paper describes it as the first provable algorithm with all four properties to the authors’ knowledge.
  • Comparison with related work: Prior bounds stated for stationary transitions and online iterations are converted to state-action sample counts using factors involving H and HSA.This conversion accounts for time-dependent transition kernels in the paper’s non-stationary finite-horizon setting.
  • Comparison with related work: Unlike a cited prior result, the analysis does not require σr-reachability and therefore has no dependence on σr.The prior comparison reports polynomial dependence on 1/σr.
  • Implementation: The experiments use linear programs over the occupancy-measure polytope, with the largest instance containing H|S||A| = 1200 decision variables and no GPU.The computations ran on a single desktop machine using HiGHS through SciPy.
  • Baselines: The reward-free safe RL baseline treats every state-action-time triple as mismatched and therefore does not exploit simulator information.This setting provides a simulator-free comparison for the proposed algorithm.
  • Baselines: The unconstrained sim-to-real baseline constructs exploration policies without enforcing safety, whereas the reward-free safe baseline uses a strictly feasible baseline policy with margin ξ.These modifications isolate simulator access and safe exploration, respectively.

B.2 Sample complexity versus the size of the mismatch region |B|

The experiment varies the mismatch-region size by changing the number of windy cells and finds that iterations grow approximately linearly with |B|. When all state-action pairs mismatch, simulator benefits disappear.

  • Experimental design: The study varies |B| across {4, 8, 12, 24, 40, 64, 84, 100} out of |S||A| = 100 by changing the number of windy cells.Each windy cell affects four actions, so n windy cells correspond to |B| = 4n; the default is |B| = 12.

B.3 Sample complexity versus the value of the separation parameter σs

The experiment studies how separation between matching and mismatching transitions affects sample complexity. Smaller σs makes classification harder and increases iterations toward the reward-free safe RL baseline.

  • Experimental design: The study varies σs by changing wind strength pwind while fixing |B| = 12.On windy cells, the real transition mixes intended and opposite-action simulator transitions.
  • Experimental design: The total-variation gap from the simulator scales linearly with pwind.The real transition is P real(· | s, a) = (1 − pwind)P sim(· | s, a) + pwindP sim(· | s, ā).
  • Results: Algorithm 1 requires more iterations when σs is smaller because non-mismatch pairs become harder to distinguish from mismatch pairs.Figure 4 varies log10(1/σs) and reports min–max ranges across 10 seeds while ensuring safe exploration and accurate planning.
  • Results: When separation is sufficiently small, the total iteration count approaches the reward-free safe RL baseline.The separation parameter enters the mismatch-region deletion rule through the threshold σs/2.

B.4.1 Algorithm for estimating σs and its guarantees

The procedure estimates the minimum simulator–real transition gap using safe baseline rollouts and lower confidence bounds. With high probability, it conservatively identifies mismatch triples and estimates σs from them.

  • Estimation procedure: Safe baseline rollouts in Mreal provide the data used to estimate σs while ensuring safe exploration.The procedure uses N episodes under the strictly feasible policy π0, then estimates the empirical real transition model and confidence bounds.
  • Guarantees: With probability at least 1 −δ, every triple in C is a true mismatch triple, and ˆσs is a conservative lower estimate over C.The guarantee follows from the confidence bounds and implies C ⊆ B together with the stated lower-bound property.
  • Guarantees: If all mismatch triples are identified, then ˆσs satisfies σs −2 max(h,s,a)∈B ρh(s, a) ≤ˆσs ≤σs.When B ⊆ C and B is nonempty, C = B, yielding the upper bound ˆσs ≤σs.
  • Guarantees: With sufficient real-world data, ˆσs approaches σs with high probability, and this conservative estimate supports safe exploration and accurate planning at increased sample cost.The estimate becomes exact on the identified mismatch set, while conservatism may increase sample complexity.

B.4.2 Sample complexity of estimating σs

The experiments measure how many safe-baseline samples are needed to estimate σs across three gridworld instances with different separation parameters. Larger separation makes estimation easier, while smaller separation is more sample-demanding.

  • Experimental setup: The sample complexity is the number of Mreal samples collected by rolling out the safe baseline policy π0.The experiments use the gridworld with |B| = 64 and vary wind-strength profiles across the mismatch region.
  • Experimental setup: The three instances are homogeneous with pwind = 0.8, homogeneous with pwind = 0.35, or heterogeneous with pwind varying from 0.2 to 0.8.The pwind = 0.35 instance has a small σs and is therefore more demanding to estimate.
  • Results: Instance I is easiest to estimate because it has the largest σs, whereas Instances II and III are harder because their separation parameters are smaller.Thus, estimation difficulty varies with the separation between simulator and real transition dynamics.

B.4.3 Effect of misspecifying σs

The experiments assess how using an estimated rather than true σs affects sample complexity, safe exploration, and planning accuracy. Underestimation preserves safety and accuracy but costs more samples, whereas overestimation can make planning infeasible in Mreal.

  • Performance effects: Algorithm 1 ensures safe exploration for all tested values of ˆσs because of its mixture-policy design.The experiments also evaluate sample complexity and whether the output policy is near-optimal and feasible in Mreal.
  • Underestimation: Underestimating σs preserves safe exploration and accurate planning but requires more real-environment samples to identify mismatch triples.The worst case is ˆσs = 0, which reduces Algorithm 1 to an online reward-free RL algorithm.
  • Overestimation: Overestimating σs may reduce real-environment samples but can incorrectly remove true mismatch triples, leaving an insufficiently accurate transition model for planning.For Instances II and III, infeasibility occurs when ˆσs is 2.5× the true value.
  • Implication: A conservative estimate of σs is important because underestimation increases sample complexity, while overestimation may cause inaccurate planning.The table reports samples, unsafe iterations, lost mismatch pairs, false deletions, output optimality, and output feasibility.

C.3 Proof of Lemma 3.1

The proof derives a tighter recursive empirical Bernstein bound for value estimation in the empirical model. It combines recursive error control, total-variance reasoning, and backward induction to obtain state- and action-value guarantees.

  • Final guarantee: The resulting Lemma C.5 bound is tighter than Lemma 3.1 because it removes the explicit dependence on the empirical variance term highlighted in the remark.A modified policy that selects a fixed action at (h, s) transfers the state-value bound to action values.
  • Proof setup: The proof conditions on simultaneous high-probability concentration events and the invariant that all true mismatch triples remain in ˆBt.The events control transition-model estimation and pseudo-count accuracy.
  • Recursive bound: For tuples outside ˆBt, mismatch exclusion and Definition 2.4 provide the starting point for the recursive empirical Bernstein bound.For tuples inside ˆBt, the proof decomposes estimation error and bounds the resulting terms recursively.
  • Recursive bound: Backward induction propagates the recursive bound across timesteps to control the relevant value estimates uniformly.The proof uses clipping and inequalities for square-root terms to keep the recursively defined quantities finite.
  • Variance propagation: The law of total variance and empirical occupancy distributions convert local transition uncertainty into cumulative value uncertainty under a policy.The argument recursively expands the variance terms and applies Cauchy–Schwarz.

C.4 Proof of Theorem 4.1

The proof establishes that the algorithm’s output is feasible for the true CMDP and near-optimal, while bounding its real-world interaction through safe-policy and mixture-policy iterations.

  • Proof strategy: The proof uses a lemma showing that sufficiently small estimation error makes a tightened empirical CMDP yield a feasible, near-optimal policy for the true CMDP.This lemma is then used to prove Theorem 4.1.
  • Optimality: The returned policy is feasible for the true CMDP after transferring feasibility from the tightened empirical constraint using the estimation-error bound.The argument explicitly concludes that πout is feasible for the true CMDP.
  • Safety: Safe exploration is ensured because the initial safe policy or adaptive mixture policy satisfies the required safety guarantee.The proof conditions on high-probability events and uses the assumptions and estimation lemma to establish this property.
  • Optimality: Termination at empirical gap ∆t ≤ ε/2, together with the tightened-problem lemma, yields the desired near-optimality guarantee for every reward function in F.The proof compares the returned policy with a feasible mixture and uses occupancy-measure equivalence to relate mixture policies to Markov policies.
  • Sample complexity: The sample-complexity proof splits iterations into Eπ0 and Erest, separately controlling fallback iterations and adaptive-mixture iterations before combining the bounds.Eπ0 is controlled by initial-policy estimation error, while Erest is controlled by the stopping rule and pseudo-count growth induced by αt.
  • Technical scope: The analysis uses an inequality for stochastic-policy occupancy distributions because policy randomness introduces additional variance terms absent for deterministic policies.The corresponding inequality suffices for the proof, whereas the related equality does not hold generally for stochastic policies.
Loading 2609.01418v1…