PaperlayerView exact revision on arXiv

Source-linked AI summary

SEVI: Safe Evidential Real-World Reinforcement Learning

Shaekh Mohammad Shithil, Lantao Liu

arXiv:2610.31829v1cs.ROcs.AIcs.LG

TL;DR

Online learning on physical robots risks unsafe interactions, while existing real-world RL pipelines do not explicitly determine when to reject a task action and invoke recovery. SEVI assesses risk and evidential support before execution, and across its simulated and hardware tasks reports fewer unsafe events alongside strong task performance.

  • Problem

    Real-world RL can train contact-rich manipulation on hardware, but unsafe online exploration remains a challenge and existing pipelines do not explicitly decide when to reject a proposed action for recovery.

    Source: §I · p. 1

  • Method

    SEVI uses a single-pass Evidential Safety Critic to estimate future constraint-violation risk and epistemic uncertainty, then substitutes offline-trained recovery when an action is unsafe or insufficiently supported by evidence.

    Source: §I · p. 2

  • Results

    30.0% on Block Plucking, 13.4% on Transport Nut, and 57.1% on USB Insertion: SEVI reduces unsafe events relative to SQRL, with the highest success rate and lowest unsafe-event count on all three tasks.

    Source: D. Safe Online Policy Learning · p. 8

  • Takeaways & Limitations

    SEVI decouples task-policy learning from execution-time intervention, reducing unsafe interactions while maintaining task performance across two simulated tasks and real-world USB insertion.

    Source: §VI · p. 8

Abstract

from arXiv · show

Real-world reinforcement learning has enabled contact-rich manipulation tasks to be trained directly on physical robots, but unsafe interactions during online learning remain a key barrier to practical deployment. We propose Safe Evidential Real-World Reinforcement Learning (SEVI), a framework for reliability-aware safety intervention during real-world policy learning. SEVI introduces an Evidential Safety Critic that estimates both future constraint-violation risk and epistemic uncertainty in a single forward pass. During online learning, SEVI evaluates each task-policy action before execution; if the action is predicted to be unsafe or the prediction is insufficiently supported by evidence, SEVI overrides it with an offline-trained recovery policy that guides the robot toward a lower-risk region. We evaluate SEVI on two simulated manipulation tasks and a real-world USB insertion task under in-distribution and distribution-shifted conditions. Across task-learning and safety-critic baselines, SEVI achieves the strongest trajectory-level safety prediction, the highest final task success, and the lowest unsafe-event counts. SEVI also reduces unsafe events by 13.4 to 57.1 percent relative to recovery-based safe RL while preserving low-latency execution-time filtering. The project page is available at: https://sevi-safe.github.io/

I. INTRODUCTION

Real-world RL can train contact-rich manipulation skills on hardware, but online exploration still risks unsafe interactions. SEVI addresses the intervention gap by assessing task actions and invoking recovery when risk is high or the safety prediction lacks evidence.

I. INTRODUCTION

  • Online real-world RL remains vulnerable to unsafe exploration, including excessive contact forces, object damage, workspace violations, and failures requiring human intervention.Existing pipelines support online learning with prior data and off-policy updates but do not explicitly decide when to reject a task action and switch to recovery.Source: §I · p. 1
  • SEVI evaluates task-policy actions for both predicted risk and evidential support, replacing unsafe or unsupported actions with an offline-trained recovery policy.The Evidential Safety Critic estimates future constraint-violation risk and epistemic uncertainty in one forward pass; recovery guides the robot toward a lower-risk region.Source: §I · p. 2
  • SEVI is evaluated on two simulated manipulation tasks and a real-world USB insertion task, with reported improvements in safety prediction under distribution shift and fewer unsafe events while maintaining task performance.The reported evaluation also finds better-calibrated uncertainty than dropout and ensemble baselines.Source: §I · p. 2
  • SEVI is designed to intervene without modifying the underlying off-policy RL objective.Its modular architecture decouples task-policy learning from execution-time safety intervention and is compatible with standard off-policy RL pipelines.Source: §I · p. 2

II. RELATED WORK

  • Recovery-based methods trigger from predicted risk, while SEVI also considers whether the safety estimate is reliable.Sampling-based uncertainty methods such as ensembles and Monte Carlo dropout require multiple models or repeated forward passes, increasing latency for real-time control; evidential prediction estimates uncertainty in a single pass.Source: §II · p. 2, §I · p. 2

III. PROBLEM FORMULATION

The problem is safe task learning through physical interaction, where actions can cause unsafe events. The formulation models future constraint violations as risk and treats safety intervention as execution-time action filtering.

III. PROBLEM FORMULATION

  • The task is modeled as an MDP with states, actions, initial-state distribution, transition dynamics, rewards, and a discount factor.Source: §III · pp. 2–3
  • A binary constraint signal marks safety violations, and the risk value is the discounted probability of a future violation after an action and subsequent policy execution.Source: §III · p. 3
  • Risk includes actions that may cause future violations, not only actions that are immediately unsafe.Source: §III · p. 3
  • Violations are treated as absorbing in the risk backup, which propagates failure information backward along trajectories.Source: §III · p. 3
  • Safety is implemented by filtering each task-policy action using predicted risk and prediction reliability, rather than direct constrained policy optimization.Source: §III · p. 3

IV. METHOD

SEVI combines an Evidential Safety Critic (ESC), an offline-trained recovery policy, and sample-efficient off-policy task learning. The ESC estimates future violation risk and epistemic uncertainty to support execution-time filtering.

IV. METHOD

  • SEVI evaluates each task-policy action before execution and replaces actions when predicted risk or epistemic uncertainty exceeds a threshold.Source: §IV · p. 3

A. Evidential Safety Critic (ESC)

  • The ESC models a temporal-difference risk target with an evidential predictive distribution whose parameters represent the predictive mean and evidence supporting it.Source: A. Evidential Safety Critic (ESC) · pp. 3–4
  • The critic trains on bounded risk Bellman targets transformed to logit space and modeled as noisy observations with unknown mean and variance under a Normal–Inverse–Gamma prior.Source: A. Evidential Safety Critic (ESC) · p. 4
  • Its evidential regression loss includes a regularizer that penalizes high evidence when the transformed target is far from the predictive mean.Source: A. Evidential Safety Critic (ESC) · p. 4
  • The ESC produces future violation risk and epistemic uncertainty in one forward pass, allowing the safety layer to reject high-risk or poorly supported actions.Source: A. Evidential Safety Critic (ESC) · p. 4

B. Recovery Policy

The recovery policy supplies corrective actions when the task policy proposes an action deemed unsafe or insufficiently supported. It is trained offline from human recovery demonstrations and acts as a fallback during online learning.

B. Recovery Policy

  • The recovery gate evaluates each proposed task action and replaces it with a recovery action when risk or epistemic uncertainty violates calibrated thresholds.Source: B. Recovery Policy · p. 4
  • Recovery is trained offline from teleoperated near-failure demonstrations, with an ESC-shaped reward based on predicted risk to guide behavior toward low-risk regions.Source: B. Recovery Policy · pp. 4–5
  • Epistemic uncertainty controls when recovery activates, while the recovery reward depends only on predicted mean risk.Source: B. Recovery Policy · p. 5
  • TD3+BC trains recovery from the demonstrations; behavior-cloning regularization keeps actions close to corrective examples and reduces extrapolation beyond that dataset.Source: B. Recovery Policy · p. 5
  • Control returns to the task policy once its proposed actions satisfy the risk and uncertainty thresholds.Source: B. Recovery Policy · p. 5

C. Integration with Online Real-World RL

SEVI integrates with the existing SERL task-learning pipeline by filtering executed actions without changing the task-learning objective. The approach aims to reduce unsafe interactions but does not guarantee zero violations.

C. Integration with Online Real-World RL

V. RESULTS

The evaluation asks whether the Evidential Safety Critic predicts safety reliably under distribution shift, whether uncertainty improves recovery gating, and whether SEVI reduces unsafe interactions while preserving task performance.

V. RESULTS

  • The evaluation is organized around safety prediction under shift, uncertainty-aware recovery gating, and safe online task learning.It asks whether epistemic uncertainty improves gating beyond predicted risk alone and whether unsafe interactions can be reduced without compromising task success.Source: §V · p. 5
  • SEVI is assessed first for safety prediction, then for uncertainty quality and risk-based recovery gating, and finally in closed-loop online RL.The last stage covers both simulation and a real robot.Source: §V · p. 5
  • In the Block Plucking illustration, a potentially destabilizing task action triggers recovery before task execution resumes successfully.The figure depicts task–recovery switching, with the unsafe action in red and successful resumed execution in green.Source: Figure 3 · p. 5

A. Evaluation Setup

The evaluation combines independent safety-critic tests under in-distribution and shifted conditions with closed-loop task-learning comparisons. SEVI has the strongest reported trajectory-level safety prediction, while also improving detection speed with low-latency uncertainty estimation.

A. Evaluation Setup

  • The closed-loop and critic baselines distinguish task learning without filtering, constraint-penalized learning, point-estimate recovery, uncertainty-free SEVI, and full SEVI.Using the same offline-trained recovery policy across recovery-based variants isolates the safety critic and gating rule.Source: A. Evaluation Setup · p. 5
  • Evaluation covers Block Plucking and Transport Nut in simulation and USB Insertion on a Franka robot, with visual and task-configuration perturbations used to test distribution shift.The USB task also varies distractor objects and cable layouts.Source: A. Evaluation Setup · pp. 5–6
  • Simulation uses five seeds and 100 final-policy rollouts per method and task; hardware uses three 3-hour training sessions and 50 final evaluation rollouts per run.Unsafe events and interventions are counted throughout online training, separately from final-policy success.Source: A. Evaluation Setup · p. 6
  • The ESC uses a twin-critic evidential head, while offline TD3+BC recovery policies and SERL task policies use task-specific data and demonstrations.The same recovery policy is shared across recovery-based variants; task policies use 50 successful demonstrations as prior data.Source: A. Evaluation Setup · p. 6

B. Safety Prediction under Distribution Shift

  • SEVI achieves the highest balanced accuracy across evaluated settings, with OOD gains over SQRL-style critics and improved unsafe-trajectory detection while retaining strong safe-trajectory rejection.The reported OOD balanced accuracy rises from 0.66 to 0.80 on Block Plucking and from 0.58 to 0.74 on USB Insertion.Source: B. Safety Prediction under… · pp. 6–7, Table I · p. 7
  • SEVI has the lowest normalized detection time on Block Plucking and USB Insertion, with latency comparable to SQRL while providing epistemic uncertainty in one forward pass.MC Dropout and Deep Ensembles estimate uncertainty at substantially higher inference cost.Source: Figure 4 · p. 6, B. Safety Prediction under… · p. 7, Table II · p. 7

C. Uncertainty-Aware Recovery Gating

SEVI combines evidential risk and uncertainty estimates with recovery gating, and its closed-loop evaluations report stronger safety–performance results across simulation and hardware. The section also describes hardware learning outcomes, framework limitations, and supplementary reproducibility material.

C. Uncertainty-Aware Recovery Gating

D. Safe Online Policy Learning

  • After 175 minutes on USB Insertion, SEVI reached 0.84 success with a 7.6 s cycle time, versus 0.69 success and 11.4 s for SQRL.SEVI's intervention rate declined during training, while SQRL showed larger intervention fluctuations and slower success improvement.Source: Figure 6 · p. 8, D. Safe Online Policy Learning · p. 8
  • Across all three tasks, SEVI achieved the highest success rate and lowest unsafe-event count; versus SQRL, it reduced unsafe events by 30.0% on Block Plucking, 13.4% on Transport Nut, and 57.1% on USB Insertion.Source: Table V · p. 8, D. Safe Online Policy Learning · p. 8

VI. CONCLUSION

  • SEVI's recovery behavior is not optimized for local or motion-efficient correction, switching may recur near failure boundaries early in learning, and the learned safety components do not guarantee zero violations.Source: §VI · p. 8

Supplementary Material

  • Supplementary material documents experimental protocols, implementation, reproducibility, metric definitions, coverage analysis, repeated hardware evaluation, and recovery-behavior evaluation.Source: Supplementary Material · p. 10

VII. SAFETY LABELS AND SUCCESS CRITERIA

The paper defines task-specific success and safety labels, constructs separate distribution-shift evaluations, and calibrates fixed recovery-gating thresholds before online learning. It also reports that task success changes only modestly under threshold perturbations.

VII. SAFETY LABELS AND SUCCESS CRITERIA

  • Safety supervision propagates unsafe labels from the first failure timestep onward, while evaluation counts a continuous failure segment as one unsafe event unless a reset precedes a new failure.Success and violation criteria are task-specific: they include object and workspace outcomes, insertion success, pose thresholds, keep-out breaches, and excessive contact force.Source: §VII · p. 10

VIII. OOD EVALUATION DETAILS

  • OOD tests vary camera viewpoints and object geometry in simulation, and distractors, cable configurations, and visual layout on hardware; USB OOD rollouts yield M > 8.5 versus an ID average of M ≈1.2.The perturbations preserve task objectives and safety definitions while changing the visual and geometric context.Source: §VIII · p. 10

IX. THRESHOLD CALIBRATION AND SENSITIVITY

  • Thresholds are calibrated once using 50 successful and 50 failure validation rollouts and held fixed; ±20% threshold perturbations change final success by less than 8% on Block Plucking, 9% on Transport Nut, and 5% on USB Insertion.Source: §IX · pp. 10–11, Table VI · p. 10

X. BASELINE AND ABLATION DETAILS

The implementation compares task-learning and safety-critic baselines while holding the task learner and recovery controller consistent across recovery-based variants. The evidential critic filters actions, the off-policy task policy learns from prior data, and an offline policy performs local recovery.

X. BASELINE AND ABLATION DETAILS

  • Task-learning baselines share observations, action spaces, rewards, and interaction budgets; SERL and SAC omit recovery filtering, SAC-C adds a constraint penalty, and the SQRL-style baseline shares SEVI's recovery policy.Source: §X · p. 11
  • Recovery variants share one controller and differ in action evaluation: SQRL-style uses point-estimate risk, SEVI without uncertainty uses mean risk, and full SEVI gates on risk and epistemic uncertainty.MC Dropout uses T = 10 stochastic passes and Deep Ensembles use M = 5 critics, while ESC predicts risk and uncertainty in one pass.Source: §X · p. 11

XI. IMPLEMENTATION DETAILS

  • The implementation-details section identifies reproducibility details for SEVI as its purpose.Source: §XI · p. 11

A. Evidential Safety Critic

B. Task Policy Learning

C. Recovery Policy

  • The recovery policy is trained offline with TD3+BC on teleoperated near-failure trajectories to return the robot to safer regions, using a value-based risk objective regularized toward demonstrated behavior.Epistemic uncertainty triggers recovery at execution time but is not an explicit recovery-reward penalty; each simulation task uses 150 recovery trajectories and USB uses 200.Source: C. Recovery Policy · pp. 11–12, Table IX · p. 12

XII. EVALUATION METRIC DEFINITIONS

The evaluation reports trajectory-level classification, detection timing, and inference latency alongside uncertainty-quality and closed-loop metrics. The definitions specify how detection timing is normalized and how uncertainty scores are interpreted.

  • Trajectory-level safety prediction uses TPR, TNR, and balanced accuracy, with TPR and TNR calculated from true-positive, false-negative, true-negative, and false-positive counts.Source: §XII · p. 12
  • Normalized detection time is the first detection timestep divided by episode horizon, so lower values indicate earlier detection; inference latency is wall-clock time per state–action safety evaluation.Sampling-based baselines include all stochastic passes or ensemble evaluations in latency.Source: §XII · p. 12
  • Uncertainty quality is assessed with AUROC, APR, ECE, and Brier score; lower ECE indicates better calibration, while closed-loop measures include final success, unsafe events, interventions, and cycle time.Source: §XII · p. 12

XIII. EPISTEMIC UNCERTAINTY UNDER TRAINING-DATA COVERAGE

The coverage experiment links greater training-data support with lower epistemic uncertainty. Across repeated USB hardware runs, SEVI retains its success and safety gains, while uncertainty-aware gating improves outcomes with two recovery behaviors.

XIII. EPISTEMIC UNCERTAINTY UNDER TRAINING-DATA COVERAGE

  • With 10%, 50%, and 100% training coverage, epistemic uncertainty was highest, intermediate, and lowest, respectively, supporting its interpretation as a signal of limited data support.The same failure region and held-out trajectories were used across conditions, while training coverage varied; SEVI gates recovery on epistemic, not aleatoric, uncertainty.Source: §XIII · pp. 12–13

XIV. REPEATED HARDWARE EVALUATION

  • SEVI achieved 0.83 ± 0.03 success and 15 ± 4 unsafe events, versus SQRL’s 0.68 ± 0.04 success and 35 ± 3 unsafe events across repeated hardware runs.The results are means ± standard deviations across three independent 3-hour hardware training runs; SEVI had the highest final success and lowest unsafe-event count across evaluated methods.Source: §XIV · p. 13, Table X · p. 13

XV. ADDITIONAL RECOVERY-BEHAVIOR EVALUATION

  • Risk-plus-uncertainty gating improved task success and reduced unsafe events under both Hit-and-Recover and Stuck-and-Recover behaviors.The gate determines when recovery is invoked; the recovery behavior determines how the robot moves away from an unsafe or unreliable configuration.Source: §XV · p. 13, Table XI · p. 13
Loading 2610.31829v1…