PaperlayerView exact revision on arXiv

Source-linked AI summary

SAFTER: Safety-Aware Filtering Through Explicit Recoverability for Robust Reinforcement Learning under Distribution Shift

Benedictus C. G. Cinun, Bin Hu

arXiv:2610.29149v1eess.SYcs.LGcs.SY

TL;DR

Under bounded actuation, an action can be instantaneously safe yet eliminate future recovery options, while sampled backup-CBF constraints are costly to enforce inside RL training. SAFTER combines backup recoverability and input admissibility in a differentiable, closed-form filter embedded in SAC. It improves transition-based sample efficiency and navigation performance under geometric shift, with success reported on two real-world layouts.

  • Problem

    Under bounded actuation, conventional instantaneous CBF filters do not ensure future recoverability, while sampled backup-CBF enforcement introduces multiple online constraints into RL training.

    Source: §1 · p. 2

  • Method

    SAFTER embeds a differentiable closed-form backup-CBF filter in SAC, combining current, sampled future, terminal, and input-admissibility conditions in a composite certificate.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Results

    SAFTER shows improved transition-based sample efficiency and strong performance under geometric distribution shift; it achieves 73.3% and 76.7% success on two real-world layouts.

    Source: §6 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    Experiments support recoverability-aware filtering as a runtime safety mechanism and a learning mechanism for sample-efficient, robust policies.

    Source: §6 · p. 9

    The result applies to the continuous safety model and backup safety at selected horizon samples; finite-rate implementation, inter-sample behavior, sensing updates, and reduced-model assumptions remain relevant boundaries.

    Source: §4 · p. 6

Abstract

from arXiv · show

Learning safe policies that generalize to unseen environments is difficult under bounded actuation, where an unsafe decision may become unrecoverable. Conventional control barrier function (CBF) filters enforce only instantaneous safety and ignore future recoverability. We propose SAFTER (Safety-Aware Filtering Through Explicit Recoverability), which embeds a differentiable, closed-form backup-CBF filter into Soft Actor-Critic training. SAFTER unifies current safety, finite-horizon backup safety, terminal recoverability, and input admissibility into a single differentiable certificate. Its CBF condition yields an exact minimum-intervention projection without iterative online optimization. In quadruped navigation, SAFTER reaches 90% training success with 30.6% fewer transitions than Vanilla SAC and 34.1-35.1% fewer than current-state CBF-filtered SAC baselines. Across unseen simulated environments, SAFTER improves success by up to 42.4 and 61.4 percentage points over the state-of-the-art methods CBF-RL and SEA-Nav, respectively, and achieves 73.3% and 76.7% success on two real-world navigation maps. Ablations identify accurate differentiation through the safety filter as a key contributor to robustness, in which replacing the projection Jacobian with the identity approximation reduces success rate in a High-Shift environment from 61.4% to 6.2%. These results show that recoverability-aware filtering serves as both a runtime safety mechanism and a learning mechanism for sample-efficient, robust policies.

1 INTRODUCTION

Safe reinforcement learning must protect robots during interaction while supporting policy improvement from safety interventions. SAFTER addresses the limits of instantaneous filtering by combining backup recoverability with a differentiable filter used in training.

1 INTRODUCTION

  • Reward penalties offer only an indirect learning signal and do not ensure that each executed action satisfies a prescribed safety condition.Trial-and-error exploration can generate unsafe actions during training and deployment.Source: §1 · p. 1
  • Under bounded actuation, instantaneous safety may still leave too little control authority to prevent a future violation.Backup CBFs ask whether a prescribed recovery maneuver remains feasible over a finite horizon and reaches a terminal safe set.Source: §1 · p. 1
  • Making predictive safety practical in RL requires handling multiple backup and input constraints while exposing the exact filter Jacobian to the task policy.The formulation must reduce the constraint collection to an analytical minimum-intervention map.Source: §1 · p. 2
  • SAFTER combines current-state, sampled backup-horizon, terminal, and bounded-input conditions into a composite certificate with a closed-form minimum-intervention filter.Its actor differentiates through the filter, while the critic evaluates the executed filtered action.Source: §1 · p. 2

2 RELATED WORK

Prior work develops safety filtering, backup recoverability, and composite filters through distinct approaches. SAFTER combines finite-horizon backup recoverability with an explicit projection differentiated during off-policy task-policy learning.

Safety-filtered reinforcement learning

  • Safety-filtered RL includes shields, constrained and safety-critic methods, reward shaping, reachability approaches, and training-time filters, but these do not directly provide sampled finite-horizon backup recoverability through the task-policy action map.Examples include SEA–Nav for safe and agile LiDAR-based locomotion and CBF–RL, which combines closed-form CBF filtering with barrier-inspired reward shaping.Source: §2 · p. 2

Backup CBFs and predictive safety

  • Backup-CBF work spans active-set invariance, time-varying backups [Singletary et al., 2022], learned backups in RLBUS, and differentiable backup-CBF projection in PS2-RL [Yun et al., 2026].SAFTER instead combines an augmented composite backup certificate with an explicit projection in surrogate-control space during off-policy task-policy learning.Source: §2 · p. 2

Composite and explicit safety filters

  • Prior composite filters include LSE composition for current-state CBFs, optimization-based smooth aggregation of backup conditions, and bounded-input interpolation between nominal and backup controls [van Wijk et al., 2026].SAFTER uses a single composite affine constraint and exact minimum-Euclidean correction in the full surrogate control space.Source: §2 · p. 3

3 RECOVERABILITY-AWARE SAFE RL: PROBLEM FORMULATION

The formulation places a safety filter between a SAC policy and the environment, using a reduced state for safety evaluation. It extends instantaneous CBF safety to finite-horizon backup recoverability, whose sampled constraints motivate SAFTER’s analytical filter.

Safety-filtered Soft Actor-Critic RL

  • In SAC, the actor proposes a nominal command, but the executed filtered command generates the transition, is stored in replay, and is evaluated by the critics.During actor updates, SAFTER differentiates through the safety map so the nominal policy can adapt to the intervention.Source: §3 · p. 3

Quadruped-navigation example

  • Safety is evaluated on a reduced continuous state x with local dynamics, distinct from the full environment state and the actor’s observation.For quadruped navigation, the safety model uses planar pose and high-level velocity commands; LiDAR returns define obstacle barriers.Source: §3 · p. 3

From current-state safety to backup recoverability

  • A conventional CBF filter minimally modifies a nominal input to maintain instantaneous safety, but bounded actuation can leave a state without enough authority to prevent future violation.Backup recoverability requires the prescribed backup trajectory to remain safe and reach a terminal set invariant under that backup controller.Source: §3 · p. 3
  • Sampling backup safety at K future times adds NhK conditions beyond current and terminal conditions, making direct enforcement a multi-constraint bCBF–QP.The quadruped example evaluates 2400 future geometric barriers with Nh = 120 rays and K = 20 positive-time samples, before terminal and input-admissibility conditions.Source: §3 · pp. 3–4

Recoverability-aware safe RL: problem formulation

  • SAFTER’s formulation seeks an executed command that satisfies encoded safety conditions when feasible, minimally changes the nominal command, and remains efficient and differentiable during SAC training.It targets the online optimization bottleneck by combining backup prediction, a composite certificate, and an analytical projection.Source: §3 · p. 4

4 RECOVERABILITY-AWARE DIFFERENTIABLE SAFETY FILTERING

SAFTER augments bounded commands and combines current, sampled backup, terminal, and input-admissibility margins into a differentiable recoverability certificate. Its CBF condition reduces these margins to one affine constraint, enabling an exact minimum-intervention projection and differentiation through the safety map during SAC training.

Analytical minimum-intervention projection

  • The composite certificate reduces sampled multi-constraint backup-CBF enforcement to one affine constraint and an exact minimum-intervention projection that can be differentiated through during SAC training.The number of backup samples and primitive barriers does not determine the number of online projection constraints.Source: §4 · pp. 4–5

Bounded-command augmentation and current safety

  • Command-state augmentation represents bounded physical commands as state constraints and lifts current-safety margins so they can be combined with predictive and input-admissibility conditions.The augmented model uses z = [xᵀ, uᵀ]ᵀ and surrogate dynamics u̇ = −Au + Aû.Source: §4 · p. 5

Predictive backup recoverability

  • Sampled backup margins test future safety along the prescribed recovery trajectory, while terminal margins test whether its final state reaches the lifted recovery set.For the quadruped specialization, the backup command drives translational and yaw-rate commands toward zero, testing whether obstacle clearance can dissipate commanded motion.Source: §4 · p. 5

Composite recoverability certificate

  • SAFTER uses an LSE smooth minimum to combine safety margins; nonnegativity of the resulting certificate implies nonnegativity of each encoded current, sampled backup, terminal, and input-admissibility condition.The certificate remains one differentiable scalar, although individual margins must still be evaluated.Source: §4 · p. 5

Safety and recoverability guarantee

  • Under the stated assumption and continuous enforcement, the augmented safe set is forward invariant and the backup trajectory satisfies encoded safety at sampled times and reaches the terminal recovery set.The guarantee is for the continuous safety model and selected backup-horizon samples; finite-rate implementation and inter-sample behavior are discussed separately.Source: §4 · p. 6

SAFTER–SAC integration

  • In SAC, the filtered command generates the environment transition and is stored in replay and evaluated by critics, while actor gradients propagate through the finite-rate safety map away from switching boundaries.The nominal policy’s entropy term remains defined on the nominal action.Source: §4 · p. 6

5 EXPERIMENTS AND ANALYSIS

SAFTER is evaluated for learning efficiency, filter runtime, nominal-policy behavior, and robustness under geometric shift. Its results include improved transition efficiency, lower filter latency, and strong simulated and real-world navigation success, with the projection-Jacobian ablation highlighting the role of differentiating through the filter.

Experimental setup

  • The study uses a Unitree Go2 with a frozen 50 Hz low-level policy, 10 Hz high-level control, 256 parallel environments, five seeds, and up to 20.48M transitions per seed.The evaluation questions cover backup recoverability and learning efficiency, geometric-shift robustness, and the effect of the projection Jacobian.Source: §5 · pp. 7–8

RQ1: Learning Efficiency and Policy Internalization

  • 4.06±0.08M transitions to 90% training success; SAFTER uses 30.6% fewer than Vanilla SAC and 34.1%–35.1% fewer than current-state CBF baselines.Similar convergence with bCBF–LSE–QP indicates the gain is primarily associated with predictive backup recoverability rather than the analytical realization alone.Source: §5 · pp. 7–8
  • 12.24× and 990.43× lower latency than direct bCBF–QP at batch sizes 1 and 100; SAFTER is 2.30× and 2.80× faster than matched bCBF–LSE–QP.Total latency includes certificate construction and synthesis.Source: Table 2 · p. 8, §5 · p. 8
  • 95.1% success without the runtime filter and 99.7% with it on training maps suggest recoverability-aware training shapes nominal behavior compatible with the backup strategy.Source: §5 · p. 8

RQ2: Robustness under Geometric Distribution Shift

  • 98.8%, 88.6%, and 61.4% success on Low-, Medium-, and High-Shift; real-world success is 73.3% and 76.7%, beating strongest baselines by 10.0 and 60.0 percentage points.Representative Medium-Shift trajectories show wider detours with greater obstacle clearance, consistent with preserving room for backup recovery.Source: Figure 4 · p. 8, Table 3 · p. 9, §5 · p. 9

RQ3: Why Does SAFTER Learn Differently?

  • Replacing the Exact projection Jacobian with Identity reduces success by up to 85.4 percentage points across evaluated distribution shifts, while removing the correction reward has mixed effects.Exact and Identity retain the same forward safety-filter map and differ only in backward differentiation through the analytical projection.Source: Table 4 · p. 9, §5 · p. 9

6 CONCLUSION

The conclusion presents SAFTER as a differentiable training map combining command-state augmentation, hierarchical LSE composition, and exact minimum-Euclidean projection. Experiments show improved transition-based sample efficiency and strong performance under geometric shift, while uneven terrain and dynamic obstacles remain future-work targets.

6 CONCLUSION

  • SAFTER combines command-state augmentation, hierarchical LSE composition, and exact minimum-Euclidean projection in a differentiable training map.Source: §6 · p. 9
  • Experiments show improved transition-based sample efficiency and strong performance under geometric distribution shift in simulated and real-world environments.Source: §6 · p. 9
  • Future work will consider uneven terrain and dynamic obstacles.Source: §6 · p. 9

A.1 TRAINING AND EVALUATION MAPS

The simulated map protocol uses five hand-crafted training maps and predefined start–goal scenarios in a fixed workspace. Unseen evaluation environments vary obstacle geometry and passage width relative to training maps.

A.1 TRAINING AND EVALUATION MAPS

  • The training bank contains five hand-crafted maps with fixed obstacle configurations and predefined initial and goal regions.The maps use a 15 m × 15 m workspace, cylindrical obstacles of radius 0.6 m, and 200 predefined scenarios per map.Source: §A.1 · p. 12, Figure 5 · p. 12
  • The policy uses local planar LiDAR with a 5 m maximum range, so navigation decisions respond as previously unseen geometry enters sensing range.Source: §A.1 · p. 12
  • Low-Shift preserves row structure while increasing obstacle count and reversing start and goal regions; Medium-Shift narrows passages, while High-Shift changes obstacle shapes, radii, and clearances.Source: §A.1 · p. 12

A.2 PATH LENGTH AND NAVIGATION TIME OF FILTER-ENABLED EVALUATIONS ON UNSEEN ENVIRONMENTS

Table 5 reports path length and navigation time for successful episodes in unseen static environments. SAFTER’s success-rate gains do not imply uniformly shorter or faster successful trajectories, and its backup-CBF projection composes safety conditions differently from the direct sampled baseline.

A.2 PATH LENGTH AND NAVIGATION TIME OF FILTER-ENABLED EVALUATIONS ON UNSEEN ENVIRONMENTS

  • Path lengths are broadly comparable across SAC/CBF methods, while SAFTER generally takes longer than current-state baselines and less time than bCBF–LSE–QP or CBF-RL.These path-length and navigation-time statistics cover successful episodes only, so interpret them alongside Table 3 success rates.Source: §A.2 · p. 12, Table 5 · p. 13

B.1 DIRECT SAMPLED BCBF PROJECTION

  • The direct baseline enforces the augmented-state CBF conditions separately, with class-K functions associated with the respective conditions.The supplied passages do not establish additional LSE bounds or projection details.Source: §B.1 · p. 13
  • The direct bCBF–QP baseline keeps sampled safety conditions as separate QP constraints, whereas SAFTER composes them into one affine condition before projection.The methods use the same command-state augmentation, backup maneuver, horizon samples, terminal set, and physical command limits.Source: §B.1 · pp. 13–14

B.3.1 BACKUP-FLOW SENSITIVITY

  • The backup-flow sensitivities of future and terminal terms contribute to the Lie derivatives used by the safety condition.At finite policy rate, sampling the backup horizon does not certify safety between adjacent samples.Source: §B.3.1 · p. 14, §B.1 · p. 14

B.5 CONTINUOUS SAFETY ASSUMPTIONS AND PROOF

The continuous safety result is conditional on the stated differentiability, model, initialization, terminal-set, and feasibility assumptions. Under continuous enforcement, the composite safe set is forward invariant, with consequences for the reduced safe-model state, sampled backup margins, terminal recoverability, and input bounds.

B.5 CONTINUOUS SAFETY ASSUMPTIONS AND PROOF

  • Under the stated assumptions, continuous enforcement of the projected CBF condition makes the composite safe set forward invariant.The assumptions include differentiable augmented dynamics and certificate terms, a representative augmented model, an invariant terminal backup set, safe initialization, and feasibility.Source: §B.5 · p. 15
  • The theorem’s consequences include a reduced state remaining safe, nonnegative sampled backup margins, a terminal backup state in the terminal set, and bounded command state.The terminal-set consequence relies on the prescribed backup remaining in the terminal set and projecting into the original safe set.Source: §B.5 · p. 15

C.1 POLICY OBSERVATION, ACTION, AND SAFETY-FILTER INPUTS

  • The actor observation and safety-filter input contain related but nonidentical information; the filter uses LiDAR and command history, not goal displacement or measured body velocity.The actor observation includes measured body-frame velocities, while the filter evaluates the analytical projection using the local augmented model.Source: §C.1 · pp. 15–16

C.2.1 SCAN-FIXED LOCAL GEOMETRY

  • The navigation certificate uses an augmented state containing planar position, yaw, and the three applied command components.A scan-fixed local frame holds LiDAR obstacle points fixed during one filter computation.Source: §C.2.1 · p. 16

C.2.2 ELLIPTICAL SAFETY BARRIER

  • The obstacle barrier represents the robot footprint and clearance with an inflated body-aligned ellipse, whose anisotropy lets yaw affect the barrier.The supplied Go2 semi-axes are 0.58945 m longitudinally and 0.39445 m laterally.Source: §C.2.2 · pp. 16–17

C.2.3 CURRENT-TIME HOCBF CONDITION

  • Under augmented command-input dynamics, the primitive barrier has no first-order dependence on the surrogate input, so the current-time condition uses a lifted HOCBF constituent.The current-time gradient expression supplies the code-aligned gradient used in the hierarchical certificate.Source: §C.2.3 · p. 17

C.2.4 BACKUP ROLLOUT AND POSITIVE-TIME BARRIER

  • The backup rollout uses a smooth numerical approximation of the exact flow, differentiating a fixed eight-point Gauss–Legendre quadrature map analytically.The implementation stores 21 rollout nodes, with 20 strictly positive-time samples entering the LSE certificate over a 4.0 s horizon.Source: §C.2.4 · p. 18

C.2.5 FORWARD-INVARIANT ELLIPTICAL TERMINAL BACKUP SET

  • The terminal backup set intersects geometric-clearance and residual-motion conditions and is forward invariant under the continuous zero-surrogate backup for the fixed-scan model.Its projection lies within the original safe set because the terminal ellipse is larger than the ordinary safety ellipse.Source: §C.2.5 · p. 19

C.2.6 IMPLEMENTATION-SPECIFIC LSE SCALING

The implementation uses fixed positive scales to normalize current-time and input-admissibility constituents before hierarchical LSE aggregation. Inactive safety projection yields exponential tracking of the nominal command, while the executed command supplies the environment transition and critic action.

C.2.6 IMPLEMENTATION-SPECIFIC LSE SCALING

  • The current-time HOCBF constituent is normalized by fixed code-defined scaling, while positive-time and terminal constituents enter without separate tunable normalization.The terminal scales are one, leaving those constituent values unchanged.Source: §C.2.6 · p. 20
  • Input-admissibility constituents use positive per-axis reference scales, preserving their signs while affecting their numerical weighting in LSE aggregation.For the stated bounds, the scales are [2, 1, 1], corresponding to 2 m s−1, 1 m s−1, and 1 rad s−1.Source: §C.2.6 · p. 20

C.2.7 SAFETY-FILTER AND IMPLEMENTATION PARAMETERS

  • The filter and policy run at 10 Hz while the frozen low-level controller runs at 50 Hz; the backup rollout uses fixed eight-point Gauss–Legendre quadrature.The implementation distinguishes 21 stored rollout nodes from the 20 positive-time samples entering the certificate.Source: §C.2.7 · p. 21

D.1 NOMINAL SURROGATE TRACKING

  • When the safety projection is inactive, the applied command exponentially tracks the nominal command.The executed command generates the environment transition and is stored in replay for critic evaluation.Source: §D.1 · p. 22

D.2 FINITE-RATE EXPERIMENTAL REALIZATION

The implementation projects commands in surrogate space, propagates and clips them at finite rate, and trains SAC on executed filtered actions while differentiating through the filter. Its continuous-time guarantee is limited to the reduced model and does not certify the clipped command between high-level updates.

D.2 FINITE-RATE EXPERIMENTAL REALIZATION

  • The finite-rate implementation applies the analytical projection, propagates the command, then clips it to physical limits; the executed command becomes the augmented model’s next input state.The same command limits are included in the certificate, and clipping is also applied to the executed command.Source: §D.2 · p. 22

D.3 GUARANTEE SCOPE

  • The continuous-time guarantee is limited to the reduced safety model: backup safety is sampled at selected τk, and filtering is recomputed only at 10 Hz.The theorem does not guarantee that the clipped command satisfies the composite CBF condition between high-level updates; the guarantee also assumes the local scan model represents motion over the prediction horizon.Source: §D.3 · p. 22

E.1 CRITIC AND ACTOR UPDATES

  • Critics evaluate the filtered action that generated each transition, while the actor differentiates through the analytical projection and finite-rate command map.The projection Jacobian removes the actor-gradient component normal to the active safety half-space; the finite-rate physical-action Jacobian includes command dynamics and the local clipping derivative.Source: §E.1 · p. 23

E.2 ENTROPY TEMPERATURE AND CRITIC PARAMETERIZATION

  • The entropy temperature is tuned using the nominal policy’s tractable tanh-Gaussian log-density, while the critic is only an observation-conditioned approximation, not an exact Markov action-value representation.Distinct nominal actions can yield the same executed action but incur different correction costs; “exact” refers to the local analytical projection derivative, not an exact policy gradient for the complete objective.Source: §E.2 · pp. 23–24

E.3 TIME-LIMIT TRUNCATION

  • For time-limit truncations, replay stores the final pre-reset observation and the SAC target continues bootstrapping; true task termination instead masks the target.Source: §E.3 · p. 24

E.4 HIGH-LEVEL REWARD FUNCTION

  • At each 10 Hz policy step, SAC-based methods share the task reward terms, while the correction term is disabled for Vanilla SAC and correction-reward-off ablations.The correction weight is −0.05 by default and zero for those variants.Source: §E.4 · p. 24, Table 7 · p. 24

E.5 SAC NETWORK ARCHITECTURE AND OPTIMIZATION

The SAC-based methods use shared actor and critic network sizes and common optimization settings. The actor outputs a three-dimensional stochastic command policy.

E.5 SAC NETWORK ARCHITECTURE AND OPTIMIZATION

  • The actor is a tanh-squashed diagonal Gaussian with three means and three log-standard deviations, clamped to [−5, 2].Source: §E.5 · p. 24
  • Actor and critic hidden layers have 512, 256, and 128 units with ELU activations, and twin Q-networks use separate Polyak-averaged targets with τP = 0.005.Source: §E.5 · p. 24
  • Training uses Adam at 3 × 10−4, discount γ = 0.99, target entropy −3, replay capacity 2 × 10^6, and minibatches of 1024.Updates begin after 20,000 environment transitions, with one update per environment step.Source: §E.5 · p. 24

Related papers

Loading 2610.29149v1…