PaperlayerView exact revision on arXiv

Source-linked AI summary

Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

Ruihan Wu, Rui Yang, Donggeon David Oh, Duy Nguyen, Haimin Hu

arXiv:2609.27312v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Competitive robot policies must pursue task success without violating safety, yet existing approaches couple safety and task learning. S2C separates them with a learned safety filter retained during training and deployment; it outperforms eight safe RL baselines in simulated touchdown games.

  • Problem

    Safe RL commonly learns safety and task competence together, a coupling that complicates training and is difficult to tune in competitive robot interactions.

    Source: §I · pp. 1–2

  • Method

    S2C learns a robust safety filter with adversarial RL, embeds it in the task-policy training environment, and retains the same filter at deployment.

    Source: §I · p. 2, §IV · p. 4

  • Results

    Against eight safe RL baselines, S2C has the highest win rates; in symmetric games it averages 76.5%, scores Elo 139, and its strongest pool opponent achieves a net win rate of 0.11.

    Source: §I · p. 2, A. Simulated Touchdown Games · p. 7

  • Takeaways & Limitations

    Under perfect filtering, filtered-game equilibrium policies remain safe and non-exploitable against safety-aware opponents.

    Source: §IV · p. 4

    Formal guarantees for ϵ-Nash equilibria under approximation errors from imperfect filters and task policies remain an open research question.

    Source: §VII · p. 8

Abstract

from arXiv · show

Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated touchdown games, S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability. Hardware stress tests against a human opponent confirm S2C's competence.

I. INTRODUCTION

Competitive robots must balance task success with strict safety constraints, but coupling those objectives can leave policies difficult to tune and vulnerable to deliberate attack. S2C separates safety filtering from competitive task learning and evaluates that approach against safe RL baselines.

I. INTRODUCTION

  • Existing safe RL methods couple safety and task competence in one policy, making their trade-off difficult to tune and tending toward attackable overly conservative or aggressive strategies.Methods include reward penalties, expected cumulative safety-cost constraints, and episode termination on violations.Source: §I · pp. 1–2
  • Safety filters modify a task policy’s proposed action to prevent an upcoming failure, but single-agent guarantees do not directly extend to competitive settings where safety and performance depend on the opponent.Prior single-agent work proves that perfect, least-restrictive filtering preserves asymptotic optimal return in safety-constrained RL [16].Source: §I · p. 2
  • S2C first learns a robust safety filter, then embeds it in task-policy training; applying the same filter during training and deployment can substantially reduce exploitability.The paper formulates competitive robot interactions as safety-critical Markov games and shows that perfect filters yield non-exploitable Nash equilibria within the safe-policy class.Source: §I · p. 2
  • S2C achieves the highest win rates and, in symmetric games, the highest Elo rating and lowest empirical exploitability among eight safe RL baselines.Hardware tests also demonstrate robustness to human strategies unseen during training.Source: §I · p. 2, Figure 2 · p. 3

II. RELATED WORK

  • Safe RL baselines use reward penalties, expected-cost constraints, or early termination, but generally do not guarantee safety against competitive or unseen opponents.A positive expected-cost budget can permit rare catastrophic violations, and opponent-dependent trajectories can exceed a training-time cost budget at deployment.Source: §II · p. 2
  • Game-theoretic solvers can require known dynamics and differentiable objectives, while their runtime cost can limit use in high-dimensional systems; S2C uses IPPO for task-policy training.Centralized training, self-play, and population-based training are other multi-agent RL approaches described in the related work.Source: §II · p. 2

III. PROBLEM FORMULATION

The paper models competitive robot interactions with safety constraints as a Markov game, where violating safety triggers an adjudicated win or loss. Under its assumptions, the game is zero-sum and has a value, enabling Nash-equilibrium policies that are non-exploitable.

III. PROBLEM FORMULATION

  • The model treats two players’ time-varying task performance as competing objectives and makes a safety violation an immediate loss.The game represents states, actions, transitions, rewards, safety margins, and an adjudication map.Source: §III · p. 3
  • When both players’ safety specifications are violated, fault attribution determines the adjudicated outcome.In the touchdown example, an inter-robot collision is attributed to the robot moving faster toward its opponent.Source: §III · p. 3
  • The running example is a symmetric two-robot touchdown game on a 5.2 × 3.0 m field, with sparse rewards for reaching the opponent’s line first.Solo safety margins measure distance to the ground and field boundary; the coupled margin measures distance to the opponent.Source: §III · p. 3
  • Under the stated assumption, the safety-critical Markov game is zero-sum and admits a value at every state.The value follows from coincident max–min and min–max payoffs under standard minimax assumptions.Source: §III · p. 4
  • A stationary Nash equilibrium guarantees the game value against every opponent policy, and neither player can improve expected payoff by switching policies.At an equilibrium, both players’ exploitability is zero.Source: §III · p. 4

IV. ANALYSIS: NON-EXPLOITABILITY UNDER PERFECT SAFETY FILTERING

The analysis shows that perfect safety filters preserve non-exploitability for equilibrium policies in the safe-policy game. This result relies on filters that leave safe actions unchanged.

IV. ANALYSIS: NON-EXPLOITABILITY UNDER PERFECT SAFETY FILTERING

  • S2C uses the same filters during training and deployment so task policies adapt to the filtered game dynamics.Source: §IV · p. 4
  • Under perfect filtering, a filtered-game Nash equilibrium induces an equilibrium in the original game restricted to safe policies.The result is stated for initial states in the specified safe set.Source: §IV · p. 4
  • A perfect filter returns safe actions and leaves every safe action unchanged.Source: §IV · p. 4
  • Because safe actions pass unchanged, a payoff-improving safe deviation remains available in the filtered game, linking its equilibria to safe-game equilibria.Source: §IV · p. 4
  • The induced equilibrium payoffs coincide, yielding zero exploitability for both players in their respective games.Source: §IV · p. 4

V. METHOD: LEARNING MINIMALLY EXPLOITABLE ROBOT POLICIES WITH FILTERED RL

S2C separates robust safety-filter synthesis from task-policy learning, then retains the filter in deployment. The framework is modular, with specific filter-synthesis, task-training, and deployment methods instantiated in this paper.

V. METHOD: LEARNING MINIMALLY EXPLOITABLE ROBOT POLICIES WITH FILTERED RL

  • Algorithm 1 collects rollouts against the other task policy and, with probability p, against a policy from a pool of historical checkpoints.The pool is periodically expanded with the current task policies.Source: §V · p. 5
  • S2C sequentially learns a robust safety filter, embeds it in the task-training environment, and deploys the task policy with the filter.The pipeline overview is illustrated for the asymmetric touchdown game.Source: §V · pp. 4–5, Figure 3 · p. 5
  • The paper instantiates filter synthesis with MAGICS, task-policy training with IPPO, and filter deployment with Robust Q-CBF.Source: §V · p. 5

A. Synthesizing Robust Safety Filters

S2C synthesizes each player's safety filter independently of the task objective, using an adversarial safety game. The learned fallback policy and critic together form a robust filter for task actions.

A. Synthesizing Robust Safety Filters

  • The safety game has the robot maintain its safety while an adversary drives it toward failure.The resulting fallback policy is trained to protect the player against adversarial opponent behavior.Source: A. Synthesizing Robust Safety Filters · p. 5
  • MAGICS jointly trains a safety fallback, a worst-case adversarial policy, and a safety critic for high-dimensional interactive tasks.It provides a scalable approximation to HJ-based robust safety analysis that would yield a perfect filter.Source: A. Synthesizing Robust Safety Filters · p. 5
  • The fallback policy and critic form a filter that takes the state and task policy as inputs, simulates a worst-case opponent action, and returns a filtered task action.Source: A. Synthesizing Robust Safety Filters · p. 5

B. Learning Task Policies in the Filtered Environment

Task policies train in an environment that embeds each player's safety filter, and the simulator executes filtered rather than proposed actions. A historical policy pool supplies additional opponents during training.

B. Learning Task Policies in the Filtered Environment

C. Deploying Task Policies with Robust Safety Filters

At deployment, each task policy is paired with the same robust safety filter used during training. The filter passes safe proposed actions unchanged and corrects unsafe ones, allowing the task policy to learn to cooperate with its interventions.

C. Deploying Task Policies with Robust Safety Filters

  • The same robust safety filter is used during task-policy training and deployment, so the policy learns within the filtered environment.The paper identifies this consistency as critical to S2C and links it to minimizing exploitability by avoiding unnecessary overrides.Source: C. Deploying Task Policies with Robust… · p. 6
  • At each control step, the filter passes a candidate task action unchanged if it satisfies the learned safety constraint; otherwise, it corrects the action.Source: C. Deploying Task Policies with Robust… · p. 6
  • The filter searches for the action closest to the proposal that is certified safe, and uses the learned fallback policy if none is found.The search uses a fixed number of gradient-descent steps.Source: C. Deploying Task Policies with Robust… · p. 6

VI. EXPERIMENTS

The experiments test whether S2C can improve competitive performance without sacrificing safety and generalize to opponents not encountered during training. They evaluate simulated touchdown games and hardware play against a human, using safe RL baselines and measures of safety and competitive performance.

VI. EXPERIMENTS

  • The evaluation tests safe competence and generalization to out-of-distribution opponents across simulated games and hardware play against a human.The hypotheses are that S2C yields more competitive policies without sacrificing safety and generalizes better to out-of-distribution opponents.Source: §VI · p. 6
  • S2C is compared with four safe RL baselines—ET, Nom, CPO, and Lag—and filtered versions of those baselines.The nine methods use the same IPPO algorithm, network architecture, and training schedule.Source: §VI · p. 6
  • The symmetric-game evaluation tracks safe rate, win rate, empirical exploitability, and Elo; Fig. 4 frames safety against win rate.Source: §VI · p. 6, Figure 4 · p. 6

A. Simulated Touchdown Games

In symmetric simulated touchdown games, S2C leads the round-robin results while maintaining a high safe rate. Ablations show that a filter alone does not confer competitive competence, while S2C’s training is associated with earlier emergence of competitive tactics.

A. Simulated Touchdown Games

  • 76.5% average win rate, Elo 139 versus Nom† at 37, and a 0.11 net win rate for the strongest pool opponent against S2C lead the round-robin results.S2C also achieves a 90.0% safe rate, close to Lag†; the strongest pool opponent achieves 0.54–0.75 net win rates against the baselines.Source: Table I · p. 7, A. Simulated Touchdown Games · p. 7
  • Filtering baselines at runtime raises their safe rates by up to 22.4 percentage points, but changes win rates by at most 7.4 points while exploitability remains high.The baselines were trained without the filter and did not learn to cooperate with it; the paper concludes that competence must be co-learned in the filtered environment.Source: A. Simulated Touchdown Games · p. 7
  • S2C’s feints and abrupt stop-and-turn tactics emerge earlier during training; the authors attribute this to the filter braking or stabilizing the robot near danger.They contrast this with baseline runs that can become overly conservative or stereotyped and exploitable.Source: A. Simulated Touchdown Games · p. 7

B. Hardware Stress Tests Against Human Opponents

In hardware attack–defense trials against a human, S2C achieves the highest task completion while retaining high safe rates. The trials expose policies to demanding human movements that were not encountered during training.

B. Hardware Stress Tests Against Human Opponents

VII. CONCLUSION

S2C separates safety filtering from competitive task-policy learning, with a non-exploitability guarantee under perfect filtering. In quadruped touchdown evaluations, it achieved the strongest task performance and lowest exploitability across the evaluation pool while maintaining high safe rates.

VII. CONCLUSION

  • S2C achieved the highest task performance and lowest exploitability across the evaluation pool while maintaining high safe rates.The framework was validated in simulation and hardware tests on a quadruped touchdown game.Source: §VII · p. 8
  • S2C decouples safety filtering from competitive task-policy learning in multi-robot interactions.Under perfect safety filtering, filtered-game equilibrium policies remain non-exploitable within the original game's safe policy class.Source: §VII · p. 8
  • The analysis assumes perfect safety filters; guarantees for approximate filters and task policies remain an open research question.The paper also identifies integration with latent-space safety filters and visual feedback as an opportunity for unstructured, open-world interactions.Source: §VII · p. 8

Related papers

Loading 2609.27312v1…