Source-linked AI summary
FAITH: Feasibility-Aware Safety-Filtered RL for High-Dimensional Systems
Songyuan Zhang, Baljeet Singh, Sarthak Ranjeet Kaingade, Chuchu Fan, Bryan Trinh
TL;DR
Safety and task objectives can compete within policy optimization, while conventional filters have limitations in high-dimensional action spaces and when no safe action exists. FAITH learns a safety value and feedforward filter, training the task policy through filtered dynamics. Simulations and humanoid experiments demonstrate safety, performance, harm reduction, and hardware execution.
Problem
Safety and task objectives can compete in policy updates, while conventional filters can be costly in high-dimensional action spaces and do not specify least-harm behavior when no action is admissible.
Source: §I · p. 1, C. Enforcing Safety on the Action · p. 3Method
FAITH learns a safety critic and feedforward filter, trains the task policy through filtered rollouts using task reward, and uses the filter to prioritize lower predicted harm when no action meets its learned safety condition.
Source: §I · p. 2, B. Feedforward Safety Filter · p. 3Results
Simulations demonstrate safety, performance, and harm reduction, while humanoid experiments demonstrate hardware execution.
Source: §VII · p. 7Takeaways & Limitations
Under global optimization over a policy class containing an optimizer, the method recovers the constrained objective described in the paper.
Source: C. Task Policy and Training Loop · p. 4Discounting, finite λ, and learning errors leave the learned controller without formal safety or optimality guarantees; evaluation excludes impact severity and human injury, and hardware demonstrations are qualitative.
Source: §VII · p. 8
Abstract
from arXiv · showhide
Safe reinforcement learning commonly places safety and task performance in the same policy objective, where they can introduce competing updates. Safety filters separate them at action execution, but classical designs require an analytic safety function and dynamics model, and standard minimal-intervention filters are myopic to long-horizon task return because they minimize only instantaneous action deviation. Hard projections are also undefined when no safe action exists. We present FAITH, a feasibility-aware, model-free framework that approximates the optimal state-action safety value and amortizes minimal-intervention filtering with a feedforward network. The task policy optimizes the task return through the filtered dynamics, which recovers the feasible constrained problem without a competing safety term in the task-policy update. When no action satisfies the learned safety condition, the same filter approaches the action with minimum predicted peak harm. On a double integrator example and a Safety Gym environment, FAITH achieves the highest return among methods with no feasible-start violations and matches the lowest harm from infeasible starts. On a 29-DoF humanoid, it reaches a 99.95% safety rate while retaining 97% of the unfiltered return in Walking-Avoid, and obtains the highest measured safety rate in Push-Avoid by learning to sacrifice balancing and fall away from the protected region. The same policies are also demonstrated on a real-world Unitree G1 humanoid.
I. INTRODUCTION
FAITH separates safety enforcement from task-policy optimization, addressing competing objectives and limitations of conventional safety filters. It distinguishes states where safety is feasible from those where the controller should minimize harm.
I. INTRODUCTION
- Safety and task performance can favor opposing policy updates; penalty weights can permit violations or induce conservatism, while primal-dual methods introduce an unstable saddle-point problem.Feasibility-aware methods distinguish states where safety is possible and guide optimization toward harm reduction otherwise,,,,.Source: §I · p. 1
- Classical safety filters require an analytic safety function and dynamics model, while learned filters can retain online sampling or optimization that becomes difficult at high action dimensions.A fixed task policy may also fail to adapt to long-horizon return after filtering, and hard projection does not specify a least-harm action when none is admissible.Source: §I · pp. 1–2
- FAITH uses learned safety values and a feedforward filter, trains the task policy through filtered dynamics using task reward, and approaches minimum predicted peak harm when no action is accepted.The same filter handles the infeasible case without a feasibility classifier, mode switch, or separate recovery controller.Source: §I · p. 2
- Across a double integrator, Safety Gym, and a 29-DoF humanoid, experiments demonstrate high task performance and safety in feasible settings, reduced harm in infeasible settings, and qualitative physical-humanoid operation.The results cover simulated settings and a real humanoid demonstration.Source: §I · p. 2
A. Feasibility-Aware Safe Reinforcement Learning
- The feasible set comprises states from which some policy can avoid the unsafe set indefinitely; being currently outside that set does not alone establish feasibility.The paper distinguishes maximizing return subject to safety within the feasible set from minimizing peak harm outside it.Source: A. Feasibility-Aware Safe… · p. 2
B. Enforcing Safety in the Objective
Objective-based safety methods incorporate constraints into task-policy optimization, so policy updates must balance safety and performance.
B. Enforcing Safety in the Objective
- Penalty, primal-dual, and primal methods all enforce safety within task-policy optimization rather than only on executed actions.Penalty methods modify reward; primal-dual methods jointly optimize a multiplier and policy; primal methods alternate reward improvement and constraint reduction.Source: B. Enforcing Safety in the Objective · p. 2
- Penalty methods use a safety weight, which can permit violations when small or make the policy conservative when large.Primal-dual methods adapt the weight but introduce a saddle-point optimization problem.Source: B. Enforcing Safety in the Objective · p. 2
- When safety and performance favor opposing update directions, improving safety can counteract task progress.Objective-based methods weight reward improvement against constraint reduction or alternate between their updates.Source: B. Enforcing Safety in the Objective · p. 2
C. Enforcing Safety on the Action
Safety filters enforce constraints on executed actions while leaving the task objective unchanged. The section identifies limitations of conventional minimal-intervention filtering, especially when no admissible action exists.
C. Enforcing Safety on the Action
- A minimal-intervention filter corrects a task policy’s nominal action to satisfy an action-level safety condition, and a valid condition preserves the feasible set when admissible actions are repeatedly applied.CBF-based filters commonly use a known control-affine model, while learned safety values can enable filtering without an explicit dynamics model.Source: C. Enforcing Safety on the Action · p. 2
- Instantaneous action-deviation minimization does not optimize task return, so a fixed nominal policy can be suboptimal after filtering.Online optimization or sampling with a learned nonlinear safety function can also be costly in high-dimensional action spaces.Source: C. Enforcing Safety on the Action · p. 3
- If no action satisfies the safety constraint, projection does not define an action that minimizes harm.The filter’s least-harm behavior is therefore not specified by hard projection alone.Source: C. Enforcing Safety on the Action · p. 3
III. METHOD
FAITH combines a learned safety critic, a backup actor, and a learned action filter, while training the task policy on filtered rollouts using only task reward.
III. METHOD
- FAITH uses a backup actor to supply low-harm actions for safety-critic targets, and trains the task policy on filtered rollouts using only reward.It requires neither known dynamics nor a given barrier, and deployment uses the task policy and filter.Source: §III · p. 3
A. Safety Q-Function
- The state-action safety critic evaluates whether a candidate action admits a persistently safe trajectory, unlike a state value that cannot evaluate actions without accessing dynamics.The backup actor approximates the minimization in the critic target, and the critic is fitted from replay-buffer samples.Source: A. Safety Q-Function · p. 3
- The critic update targets a discounted safety Bellman fixed point, which can differ from the optimal safety value when the safety discount is below one,.Together with approximation errors, this makes the learned nonpositive critic value a practical safety estimate rather than a formal certificate.Source: A. Safety Q-Function · p. 3
B. Feedforward Safety Filter
FAITH uses a learned safety critic to define admissible actions and a feedforward filter to approximate minimal-intervention projection. The same filter remains defined when no action is admissible, prioritizing lower predicted harm.
B. Feedforward Safety Filter
- The learned acceptance condition is Qh,ϕ(s, a) + δ ≤0, and its corresponding nonlinear projection can be expensive online in high-dimensional action spaces.A feedforward residual network amortizes the projection.Source: B. Feedforward Safety Filter · p. 3
- When the nominal action is admissible, the ideal filter preserves it; otherwise, it prioritizes satisfying the learned constraint while minimizing action deviation.The parameterized filter approximates this objective over replay states and replayed or randomly sampled nominal actions.Source: B. Feedforward Safety Filter · p. 3
- When no action satisfies the learned condition, the filter loss reduces to action deviation plus λ times predicted harm, so increasing λ prioritizes lower predicted peak harm.Under compact actions, continuous critic values, and global per-input minimization, the loss approaches minimum predicted harm at O(1/λ), up to learning errors.Source: B. Feedforward Safety Filter · p. 3
C. Task Policy and Training Loop
The task policy proposes actions that the filter converts into executed actions, defining filtered dynamics on which the policy optimizes return. Under a perfect filter and global optimization, this recovers the constrained optimum on feasible states, but the learned filter only approximates that ideal.
C. Task Policy and Training Loop
- For a fixed filter, task-policy proposals and filter-produced executed actions define the filtered dynamics, and the task policy maximizes return in that system.PPO updates the policy with the filter held fixed; safety affects trajectories through filtering, not through an added safety term in the task reward.Source: C. Task Policy and Training Loop · p. 3
- Under a perfect filter and the stated optimization conditions, globally optimizing filtered-system return recovers the constrained optimum on feasible states.The learned filter only approximates a perfect filter, so this result is not a guarantee for the deployed controller.Source: C. Task Policy and Training Loop · p. 4
- The task policy, safety critic, backup actor, and filter learn from replay, while PPO uses on-policy rollouts; backup-policy trajectories are stored but excluded from PPO updates.Retaining unsafe and near-boundary transitions preserves safety-relevant coverage as the task policy changes and mitigates catastrophic forgetting.Source: C. Task Policy and Training Loop · p. 4, Figure 2 · p. 4
- The task-policy objective contains no competing safety term, while training through the filter lets the policy adapt to safety interventions.Safety and task return are optimized in different modules: the filter for safety and PPO for return.Source: C. Task Policy and Training Loop · pp. 3–4
IV. RELATED WORK
FAITH combines learned peak-harm estimation, amortized action correction, and task-policy training under filtered dynamics. Its single model-free filter addresses both feasible avoidance and infeasible harm reduction.
IV. RELATED WORK
- Unlike prior feasibility-aware formulations that incorporate learned safety values into task-policy optimization, FAITH enforces safety on actions while the policy optimizes return under filtered dynamics.The cited approaches include multipliers, reachability-dependent objectives, and feasibility-guided generation,,.Source: §IV · p. 4
- FAITH combines learning the filter function with direct correction and filtered-rollout policy training, using a feedforward filter that needs neither a dynamics model nor an online deployment solve.When its learned acceptance set is empty, the same loss prioritizes lower predicted harm.Source: §IV · p. 4
- For humanoids, FAITH uses one model-free filter for avoidance and harm reduction, allowing protected-region clearance to take precedence over balance without a fall-away reward or separate recovery controller.The task policy learns under filtered dynamics when safety is feasible; otherwise, the same filter prioritizes lower harm.Source: §IV · p. 4
V. SIMULATION EXPERIMENTS
In the double-integrator experiment, FAITH combines perfect feasible-start safety with the highest return among safe methods, and attains the lowest peak harm from infeasible starts. The experiments compare it with feasibility-aware, penalty, and barrier-based approaches.
V. SIMULATION EXPERIMENTS
A. One-dimensional Double Integrator
- The double integrator uses bounded acceleration, a safe set |p| ≤1, and a reward maximized outside that set, creating a conflict between safety and performance.The study trains three seeds for 2,000 iterations and evaluates 4,096 episodes, selecting the safest checkpoint among iterations 1,800, 1,900, and 2,000.Source: A. One-dimensional Double Integrator · p. 5
- 100% safety and the highest return among safe methods on feasible starts, plus the lowest peak harm on infeasible starts, are reported for FAITH.For infeasible starts, FAITH and PPO(100) attain the lowest mean peak harm; FAITH does so by braking immediately.Source: A. One-dimensional Double Integrator · p. 5
- RC-CRPO crosses the feasibility boundary, PPO exhibits a safety-performance trade-off, and ROCBF produces suboptimal trajectories because it ignores the acceleration limit.RCRL shows large variance between seeds, potentially due to its unstable saddle-point problem.Source: A. One-dimensional Double Integrator · p. 5
B. Safety Gym
In Safety Gym, FAITH is safe from feasible starts where the evaluated baselines violate safety. On the humanoid, it retains 97% of unfiltered return in Walking-Avoid and achieves the highest safety rate in Push-Avoid.
C. 29-DoF Humanoid Near Protected Regions
- The 29-DoF Unitree G1 is evaluated in Walking-Avoid and Push-Avoid over 4096 trajectories, using protected-region LiDAR observations and locomotion rewards.Push-Avoid adds a start-of-episode push; the evaluation reports safety rate, return, and peak penetration on common-failure episodes.Source: C. 29-DoF Humanoid Near Protected… · p. 5
- Safety is defined as human protection, so falling itself is not unsafe; the reward gives greater contact-force penalties to the head and hands than the torso.This qualifies the Push-Avoid result: the robot may sacrifice balance while protecting its head and hands.Source: C. 29-DoF Humanoid Near Protected… · p. 5
B. Safety Gym
- 100% safety rate and high mean return: FAITH outperforms the evaluated baselines on SafetyCarCircle1-v0, where all initial states are feasible.The figure caption reports min–max error-bar ranges.Source: B. Safety Gym · p. 5, Figure 5 · p. 6
C. 29-DoF Humanoid Near Protected Regions
- FAITH matches the highest safety rate in Walking-Avoid while retaining 97% of unfiltered return, and achieves the highest safety rate in Push-Avoid.In Push-Avoid, its higher safety comes with lower return as it sacrifices balance to preserve clearance.Source: Table I · p. 6, C. 29-DoF Humanoid Near Protected… · p. 6
- In common-failure Push-Avoid episodes, FAITH obtains the lowest harm without a fallback controller or explicit mode switch.The reported peak-harm evaluation is limited to common-failure episodes.Source: C. 29-DoF Humanoid Near Protected… · p. 6
D. Ablation Studies
The ablations examine filter design, policy adaptation, replay data, numerical search, and penalty scale. They show that filtered-dynamics training and backup replay support performance, while numerical search is slower and penalty scale affects safety and return.
Ablation Studies
- FAITH closely follows the analytical optimal solution produced by Oracle and PPO+GTCBF, outperforming PPO trained through ROCBF.The comparison is reported for the Double Integrator ablations.Source: Table III · p. 6, D. Ablation Studies · p. 6
- Fixing FAITH’s filter and training PPO through it performs similarly to FAITH in both the Double Integrator and Car-Circle environments.Source: D. Ablation Studies · p. 6
- Removing filtered-dynamics training degrades task return because PPO does not adapt to the filter, and slightly degrades safety.The authors suggest more unsafe policy actions in the replay buffer may make filter learning harder.Source: D. Ablation Studies · p. 6
- Replacing backup rollouts with filtered task-policy rollouts lowers safety, potentially because the safety critic receives out-of-distribution values without backup replay.Source: D. Ablation Studies · p. 6
- Numerical search is 18× slower in both environments, performing similarly to FAITH in the Double Integrator but worse in Car-Circle.The comparison holds the policy and critic fixed.Source: Table IV · p. 7, D. Ablation Studies · p. 7
- Reducing the maximum penalty degrades safety and increases peak harm; doubling it matches the safety rate but yields slightly lower return.Results remain robust to a moderate change using 0.5λmax.Source: D. Ablation Studies · p. 7
VI. HARDWARE EXPERIMENTS
Hardware demonstrations show FAITH executing both upright recovery and protective fall-away behaviors on a Unitree G1. The paper also notes that its guarantees and evaluation scope are limited.
VI. HARDWARE EXPERIMENTS
- The 29-DoF Unitree G1 recovers balance under a lighter push and falls away onto its legs and torso under a stronger push.The hardware setup combines 10 Hz LiDAR scans with robot state for 50 Hz joint-position targets, tracked by a 1000 Hz PD controller.Source: §VI · p. 7
- The same controller supports upright recovery and falling away from the protected region.The fall-away behavior is consistent with the head and hand contact penalties used in training.Source: §VI · p. 7
VII. CONCLUSIONS
- Under stated assumptions, filtering preserves the constrained optimum; when the learned feasible set is empty, it approaches minimum predicted peak harm.The conclusion reports simulations for safety, performance, and harm reduction, alongside humanoid hardware execution.Source: §VII · p. 7
- Discounting, finite λ, and learning errors leave the learned controller without formal safety or optimality guarantees.Evaluation is limited to planar protected regions, excludes impact severity and human injury, and reports no hardware safety or harm statistics or quantified effects of LiDAR noise, latency, or communication delay.Source: §VII · p. 8