PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Bellman Operator for Safety-Critical Reinforcement Learning

Nishanth Arun Rao, Royina Karegoudra Jayanth, Benjamin Eysenbach, Jaime Fernández Fisac

arXiv:2610.12420v1cs.LG

TL;DR

Safe RL must balance task performance with strict safety, while existing approaches may trade joint learning against hard guarantees. This paper introduces a joint Bellman operator learned through two-timescale stochastic approximation. Its theory establishes safety and optimality properties, and neural continuous-control evaluations report stable learning with near-zero violations.

  • Problem

    Safe RL seeks near-optimal policies with strict safety assurances, but existing approaches often trade soft-constraint joint learning against hard safety guarantees.

    Source: §1 · pp. 1–2

  • Method

    A robust joint Bellman operator combines safety and task objectives at the value-function level, with a two-timescale stochastic approximation framework for learning its fixed point.

    Source: §1 · p. 2

  • Results

    The proposed neural approximation shows stable, low-variance training and near-zero safety violations in continuous-control evaluations.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The paper's theory states that the converged optimal joint policy maximizes task returns while maintaining safety.

    Source: §1 · p. 2, §8 · p. 10

    High-dimensional neural approximations violate strict forward invariance, convergence relies on a heavily scaled safety margin that may cause numerical instability, and the method assumes fully observed states with well-defined continuous safety margins.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints. Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average. In this work, we propose a novel Bellman operator that unifies performance and safety objectives into a joint value function. We show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework. On the fast timescale, the safety value of the learning joint policy is estimated, while the joint value is estimated on the slow timescale. Convergence is ensured by formulating the limiting dynamics as an occupation-averaged differential inclusion, and showing that it asymptotically converges to a set of limiting optimal safety-constrained task value functions. Theoretically, once converged, the resulting optimal policy maximizes task return while maintaining safety at all times. Empirical evaluations on continuous control tasks with neural approximations demonstrate stable convergence with near-zero safety violations at test time.

1 INTRODUCTION

Safe RL seeks to combine near-optimal task performance with strict safety, but existing approaches often trade off joint learning against strict guarantees. This paper proposes a joint Bellman operator and learning framework, with theoretical guarantees and continuous-control evaluations.

1 INTRODUCTION

  • Existing methods either optimize safety as a soft constraint or enforce hard guarantees, while model-free methods bound expected costs and model-based methods face model-bias and real-time computation challenges.The paper seeks a model-free approach that strictly adheres to safety specifications while optimizing task performance.Source: §1 · p. 1
  • Safety filters can produce erratic boundary interventions, while some unified policy architectures face destabilizing reward-cost gradient conflicts.The paper assumes no oracle safe action set and aims to learn safety from data.Source: §1 · p. 2
  • The paper combines safety and task objectives at the value-function level through a robust joint Bellman operator and a two-timescale learning framework.The operator's objective is derived from robust optimal stopping theory.Source: §1 · p. 2
  • The authors prove forward invariance of the safe set under the joint policy and convergence to optimal robust task performance within that set.Source: §1 · p. 2
  • Neural approximations on continuous-control tasks show stable training convergence and near-zero safety violations during evaluation.The experiments use tasks from Gymnasium and SafetyGymnasium under adversarial disturbances.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries establish the finite discounted control setting and introduce CMDPs and Hamilton-Jacobi safety analysis as background for the paper's formulation.

2.1 DEFINITIONS AND NOTATIONS

  • The setup uses finite state, action, and disturbance spaces, and defines the viability kernel as states where a policy can avoid failure under every adversarial disturbance.Source: §2.1 · p. 2

2.2 CONSTRAINED MARKOV DECISION PROCESSES

  • A CMDP adds auxiliary constraint costs to an MDP and seeks to maximize expected discounted return subject to a safety-cost threshold.The described formulation is typically solved with Lagrangian relaxation.Source: §2.2 · p. 3

2.3 HAMILTON-JACOBI SAFETY ANALYSIS

  • Hamilton-Jacobi reachability seeks a safe set through a zero-sum differential game; its robust safety value is characterized by an HJI variational inequality.The time-discounted Bellman equation enables TD learning because the infinite-horizon operator does not induce a contraction.Source: §2.3 · p. 3

3 LEARNING SAFETY-CONSTRAINED TASK ACTION-VALUE FUNCTION

The paper constructs a joint safety-constrained task value function under adversarial disturbances. Its safety critic and task-value estimate are coupled through the policy and a stopping-based objective.

3.1 THE JOINT FORMULATION

  • The joint formulation combines time-discounted task and safety values so task-reward accrual is capped by the policy's safety value.The joint policy is maximin: the disturbance observes the controller's action, and the controller optimizes against that disturbance.Source: §3.1 · pp. 3–4

3.2 SAFETY ACTION-VALUE OF THE JOINT POLICY

  • For a fixed joint policy, the safety action-value is evaluated by a time-discounted Bellman operator that is a γ-contraction, with its nonnegative safety value defining the policy's safe set.Source: §3.2 · p. 4

3.3 JOINT ACTION-VALUE FUNCTION

  • For fixed safety action-value Y, the joint Bellman operator is a γ-contraction and has a unique fixed point.The operator uses m(s, a, d) = min{r(s, a, d), g(s)}.Source: §3.3 · p. 4

3.4 OBJECTIVE FUNCTION OF JOINT ACTION-VALUE FUNCTION

  • The stopping time switches the objective from expected return to terminal safety value when a trajectory leaves the task-optimizing region.The paper defines task-optimizing and safety-optimizing regions by comparing task and safe value functions; task performance is optimized while trajectories remain in Ωψ.Source: §3.4 · pp. 4–5

3.5 COUPLED FIXED POINT

  • The coupled fixed point requires the safety critic to equal the safety value of the greedy policy derived from Q, while Q is fixed by its own Bellman operator truncated by that critic.Because Q-to-policy mapping can be discontinuous, the paper motivates learning the two value functions on separate timescales.Source: §3.5 · p. 5

4 TWO-TIMESCALE STOCHASTIC APPROXIMATION

The analysis studies coupled temporal-difference learning through fast and slow recursions, using a set-valued fast-timescale analysis to address policy discontinuities and an averaged differential inclusion for the slow dynamics.

4 TWO-TIMESCALE STOCHASTIC APPROXIMATION

  • The TD method uses bootstrapped updates with separate step-size sequences for its coupled value-function recursions.The analysis assumes bounded iterates and bounded conditional variance of total martingale noise.Source: §4 · p. 6

4.1 FAST TIMESCALE: SAFETY CRITIC TRACKING

  • When multiple maximin policies tie, the fast safety-critic dynamics are modeled as a differential inclusion; under the stated assumptions and bounded iterates, the coupled iterates approach its compact global attractor graph almost surely.Without ties, the inclusion reduces locally to the usual single-policy ODE behavior.Source: §4.1 · p. 6

4.2 SLOW TIMESCALE: AVERAGED DIFFERENTIAL INCLUSION

  • The slow recursion averages Bellman residuals over admissible occupation measures supported on the equilibrated fast-timescale attractor; its interpolation is almost surely an asymptotic pseudotrajectory of the resulting differential inclusion.Its omega-limit set is nonempty, compact, invariant, and internally chain-transitive.Source: §4.2 · pp. 6–7

5 SAFE CONFIGURATION SET AND FORWARD INVARIANCE

The safe configuration set contains joint Q-functions whose induced policies preserve the viability kernel. The limiting dynamics keep this set forward invariant and exponentially attract nearby configurations under the stated assumptions.

5 SAFE CONFIGURATION SET AND FORWARD INVARIANCE

  • The safe configuration set Qcfg consists of joint Q-functions whose induced policies choose kernel-preserving actions.Kernel-preserving actions keep the next state inside the viability kernel against every disturbance.Source: §5 · p. 7
  • The fast attractor's negative certification on unsafe actions and positive homogeneity support the stated margin condition across the coupled dynamics.The passage states a sufficient margin bound for the certification on J; if J is empty, the bound holds trivially.Source: §5 · p. 7
  • The limiting dynamics exponentially attract configurations near Qcfg and leave Qcfg forward invariant under Assumption 3 and the stated margin condition.The distance to Qcfg decays at rate e^{-(1−γ)t} when the initial distance is below ϵ0.Source: §5 · p. 7
  • From any initial q in Qcfg, the induced policy keeps trajectories inside Ω almost surely for all time and against every disturbance sequence.The result follows from the policy's nonnegative safety value on Ω.Source: §5 · p. 7

6 CONVERGENCE TO OPTIMAL SAFETY-CONSTRAINED TASK PERFORMANCE

The convergence result identifies safe joint value functions that attain the optimal task value within the viability kernel. It requires a sufficient safety-margin scale when δΩ is positive; when it is zero, the finite-K optimality gap vanishes as K grows.

6 CONVERGENCE TO OPTIMAL SAFETY-CONSTRAINED TASK PERFORMANCE

  • The robust safety-constrained task Bellman operator has a unique fixed point, and its value is the maximum expected return attainable while keeping the agent in Ω.Every q in Q* corresponds to a safe policy attaining the maximal return among safe policies.Source: §6 · p. 8, §B.7 · p. 33
  • Theorem 4 defines a sufficient margin scale K* for convergence to optimal safety-constrained task performance.The theorem's conclusion is subject to its stated conditions, including δΩ > 0 and the required bounds on K and the iterates.Source: §6 · p. 8
  • Every limiting controller attains the optimal task value while remaining safe, when the stated boundedness and margin conditions hold.The limiting set lies within Qcfg, and its restricted values on I equal Q*Ω.Source: §6 · p. 8
  • When δΩ is positive, a sufficiently large finite K yields convergence to the optimal safety-constrained task value; when δΩ is zero, the finite-K optimality gap vanishes as K tends to infinity.The sufficient margin scale prevents safety truncation when δΩ > 0; if δΩ = 0, no finite K is covered by that optimality guarantee.Source: §6 · p. 8

7 EXPERIMENTS

The experiments evaluate JointSAC, a neural approximation of the joint policy, on adversarial continuous-control tasks. Across 11 environments, it achieves stable task performance with near-zero safety violations in final evaluations.

7 EXPERIMENTS

  • The experiments ask whether neural two-timescale learning converges stably, how its policy compares with CMDP baselines, and how the safe set expands during training.The reported experiments focus on scaling the optimal joint policy to high-dimensional continuous spaces.Source: §7 · p. 8
  • JointSAC is a neural approximation built on SAC that trains controller and disturbance policies using finite-timescale gradient descent-ascent.The method targets continuous-control tasks where exact dynamic programming is intractable.Source: §7 · pp. 8–9

7.1 EXPERIMENTAL SETUP

  • The benchmark uses three Gymnasium locomotion tasks and eight SafetyGymnasium SafeVelocity tasks augmented with adversarial disturbances.Baselines include least-restrictive safety filters and CMDP methods PPOLag, SACLag, and PCPO.Source: §7.1 · p. 9

7.2 MAIN RESULTS

  • Across all 11 environments, JointSAC maintains near-zero violations, with a maximum cost of 0.03 ± 0.08 on Hopper-v4, while baselines show inconsistent and sometimes major cost violations.Final evaluations comprise 300 rollouts per training seed and 1,500 distinct evaluation seeds in total.Source: §7.2 · p. 9, Table 1 · p. 9
  • Training curves show stable convergence to high task rewards with near-zero costs, while baselines have higher variance and only some meet the permissible cost threshold.Figure 2 presents episodic returns and costs across five seeds.Source: §7.2 · p. 9, Figure 2 · p. 9

8 CONCLUSION AND LIMITATIONS

The paper presents a robust Bellman operator and two-timescale algorithm, with theoretical convergence to optimal safety-constrained task value functions and empirical near-zero violations. Its limitations include neural approximations that violate strict forward invariance and assumptions that may be difficult to meet in unstructured real-world settings.

8 CONCLUSION AND LIMITATIONS

  • The robust Bellman operator and two-timescale algorithm target a joint value-function fixed point; the theory establishes convergence and safety, while JointSAC's neural approximation shows stable, low-variance training with near-zero violations.The conclusion distinguishes the theoretical guarantee from empirical neural approximation results.Source: §8 · p. 10
  • High-dimensional continuous-space scaling requires neural approximations that violate strict forward invariance; the theory also relies on a heavily scaled safety margin K ≥ K*, which may cause numerical instability.In practice, the authors say K must be heuristically tuned as high as practically possible.Source: §8 · p. 10

A.1 DEFINITIONS AND NOTATIONS

  • The setup defines the viability kernel as states from which a policy can avoid failure for all time despite adversarial disturbances.The paper also defines state, action, and disturbance spaces, reward, safety margin, and constraint and failure sets.Source: A PRELIMINARIES · p. 14, §A.1 · p. 14

A.2 REINFORCEMENT LEARNING

  • The reinforcement-learning preliminaries frame learning as approximating an optimal value function or policy, with Bellman contraction supporting convergence to a unique fixed point.The stated contraction guarantee is for the Bellman operator under the supremum norm.Source: §A.2 · p. 15

A.3 DIFFERENTIAL INCLUSIONS, ASYMPTOTIC PSEUDOTRAJECTORIES, AND INTERNALLY CHAIN-TRANSITIVE SETS

  • The stochastic recursive-inclusion framework yields an asymptotic pseudotrajectory by averaging slow-timescale Bellman residuals over occupation measures of equilibrated fast dynamics.The preliminaries introduce differential inclusions, asymptotic pseudotrajectories, and internally chain-transitive sets used in this analysis.Source: §A.3 · pp. 15–16

A.4 CONSTRAINED MARKOV DECISION PROCESSES

  • A CMDP adds auxiliary constraint costs to an MDP and seeks maximum expected return subject to a safety threshold; Lagrangian relaxation is a standard solution approach.Under Slater's condition, the passage states that strong duality makes the unconstrained minimax problem equivalent to the primal CMDP.Source: §A.4 · p. 16

A.5 HAMILTON-JACOBI SAFETY ANALYSIS

The appendix reviews robust safety analysis through a Hamilton–Jacobi formulation and presents contraction results for the safety and joint Bellman operators.

A.5 HAMILTON-JACOBI SAFETY ANALYSIS

  • The safe set comprises initial states from which a policy can avoid failure at all times despite adversarial disturbance, and the safety value solves an HJI variational inequality.The HJI value is described as the unique viscosity solution.Source: §A.5 · pp. 16–17
  • For stochastic dynamics, discounting the safety Bellman equation with γ ∈ (0, 1) ensures contraction on the space of value functions.The undiscounted operator does not induce a contraction on safe value functions.Source: §A.5 · p. 17

B.1 SAFETY ACTION-VALUE OF THE JOINT POLICY (SEC. 3.2)

  • For any fixed joint policy, the time-discounted safety Bellman operator is a γ-contraction under the supremum norm.The stated result applies for γ ∈ [0, 1).Source: §B.1 · p. 17

B.2 JOINT ACTION-VALUE FUNCTION (SEC. 3.3)

  • The joint Bellman operator combines clipped reward m(s, a, d) = min{r(s, a, d), g(s)} with the next-state value and is a γ-contraction for fixed Y.The next-state value is VQ(s) = max_a min_d Q(s, a, d); the contraction is under the supremum norm for γ ∈ [0, 1).Source: §B.2 · p. 18

B.3 OBJECTIVE FUNCTION OF JOINT ACTION-VALUE FUNCTION (SEC. 3.4)

The section analyzes the joint action-value objective and its fast- and slow-timescale behavior, including safety-critic tracking and the averaged differential inclusion. It also establishes contraction and invariance properties of the safe configuration set and derives safety guarantees for policies induced by configurations in that set.

B.3 OBJECTIVE FUNCTION OF JOINT ACTION-VALUE FUNCTION (SEC. 3.4)

  • The safety value is bounded above by g(s), so it is negative in failure states; this bound holds for arbitrary policies, not only safety-greedy policies.A positive immediate margin does not guarantee viability: inevitable-failure states can still have negative long-term safety value.Source: §B.3 · pp. 19–20
  • Before switching regions, the joint formulation evaluates only the task objective while remaining implicitly upper-bounded by the safety margin g(s).The switching time marks the first entry into the safety-maximizing region.Source: §B.3 · p. 20
  • The joint policy’s objective is expressed as an optimal-stopping problem over the first switching time into the safety-maximizing region.The switching time is a valid stopping time, and the theorem states the corresponding joint objective for trajectories generated by the joint policy.Source: §B.3 · pp. 20–23

B.4 FAST TIMESCALE: SAFETY CRITIC TRACKING (SEC. 4.1)

  • Under the stated assumptions and almost-sure boundedness, the fast safety critic tracks a compact global attracting set, and the coupled iterates approach its graph almost surely.The attracting-set map is upper semicontinuous.Source: §B.4 · p. 24
  • The faster safety-critic updates provide the joint critic with a safety estimate for its current policy before the joint critic updates and changes that policy.This is the interpretation of the quasi-static joint critic.Source: §B.4 · p. 25

B.5 SLOW TIMESCALE: AVERAGED DIFFERENTIAL INCLUSION (SEC. 4.2)

  • Under Assumptions 1–2 and almost surely bounded iterates, the slow interpolation is an asymptotic pseudotrajectory whose ω-limit set is nonempty, compact, invariant, and internally chain transitive.The averaged differential inclusion is formed from long-run averages of the fast-timescale dynamics encoded by occupation measures.Source: §B.5 · p. 25

B.6 SAFE CONFIGURATION SET AND FORWARD INVARIANCE (SEC. 5)

  • Qcfg requires nonnegative values on kernel-preserving triples and values at most −κ on selected kernel-exiting triples, so its induced greedy policy is kernel-preserving.The exiting disturbance is selected to witness a transition outside the viability kernel; the corresponding action-value constraint is imposed at that selected coordinate.Source: §B.6 · p. 26

B.6.1 GEOMETRY OF THE SAFE CONFIGURATION SET

  • Qcfg is nonempty and each Cϵ is a closed convex polyhedron with a coordinatewise clipping projection; for q ∈ Cϵ0, greedy selection favors kernel-preserving actions by at least κ − 2ϵ.The strict gap follows from ϵ < ϵ0 < κ/2.Source: §B.6.1 · pp. 27–28

B.6.2 SAFETY CERTIFICATION OF THE FAST ATTRACTOR

  • For q ∈ Cϵ0, every fast-attractor safety critic is nonnegative on kernel-preserving triples and, under Assumption 3, negative on selected kernel-exiting triples.This asymmetry arises because kernel-preserving actions keep successors in Ω, where constraints apply, while exiting actions can lead outside Ω; Assumption 3 also yields the margin bound Θ = ((1−γ)Rmax + κ)/γ on J.Source: §B.6.2 · p. 29

B.6.3 UNIFORM VIOLATION CONTRACTION

  • For 0 ≤ ϵ ≤ ϵ0, the joint Bellman operator maps Cϵ into Cγϵ uniformly over critics in the fast attractor.This contracts distance to Qcfg, without requiring the moving composed map q ↦ T_Yq[q] itself to be a contraction.Source: §B.6.3 · p. 30

B.6.4 AVERAGING OVER THE FAST ATTRACTOR PRESERVES THE VIOLATION CONTRACTION

  • Every averaged slow-timescale vector at distance ϵ < ϵ0 from Qcfg has the form v = z − q, with z ∈ Cγϵ.The averaged target z is obtained by averaging Ty[q] over a probability measure supported on Λ(q).Source: §B.6.4 · p. 31
  • Averaging over occupation measures preserves violation contraction because Cϵ is convex, so conv(Cϵ) = Cϵ.The averaged target z lies in Cγϵ because that set is closed and convex.Source: §B.6.4 · p. 31

B.6.5 FORWARD INVARIANCE

  • If ρ(q(0)) < ϵ0, every solution satisfies ρ(q(t)) ≤ ρ(q(0))e^{−(1−γ)t}; Qcfg is forward invariant and exponentially attracting.The attraction basin contains the open neighborhood {q : ρ(q) < ϵ0}.Source: §B.6.5 · p. 31

B.6.6 THE SAFE CONFIGURATION SET CERTIFIES SAFETY

  • For q ∈ Qcfg, the safety value is nonnegative throughout Ω, and the policy µq keeps trajectories in Ω with probability one against every disturbance sequence.These are the two robust-invariance claims of Lemma 12.Source: §B.6.6 · p. 31
  • From any safe initial state, policies induced by configurations in Qcfg keep trajectories in Ω almost surely for all time, against every disturbance sequence.This guarantee concerns trajectories under a policy induced at a given configuration; safety throughout training additionally requires safe initialization and a safe exploration mechanism, neither assumed here.Source: §B.6.6 · p. 32

B.7 CONVERGENCE TO OPTIMAL SAFETY-CONSTRAINED TASK PERFORMANCE (SEC. 6)

Under a sufficient safety-margin condition, the limiting joint policy attains optimal safety-constrained task value; asynchronous updates preserve the limiting safety and optimality results, changing only effective convergence rates.

B.7 CONVERGENCE TO OPTIMAL SAFETY-CONSTRAINED TASK PERFORMANCE (SEC. 6)

  • When δΩ > 0 and K ≥ K*, the iterates converge to limiting controllers that are safe and attain the maximal return among safe policies.The theorem also requires the stated bounded-iterate and prior-theorem assumptions.Source: §B.7 · pp. 33–34
  • If δΩ = 0, the theorem no longer guarantees optimal safety-constrained task value for any finite K; as δΩ approaches zero, the sufficient K* grows without bound.Finite-K truncation can decrease task return; the stated conservatism vanishes as K grows if the margin condition holds.Source: §B.7 · pp. 34–35

B.8 EXTENSION TO ASYNCHRONOUS SETTINGS

  • Asynchronous coordinate updates preserve the limiting safety and optimality results, while positive diagonal rescaling changes effective convergence rates.The extension assumes recurrent updates, regular local step sizes and noise, and preserved two-timescale separation.Source: §B.8 · p. 36

C ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS

  • For neural approximations, the authors implement the framework as JointSAC, combining SAC with gradient descent-ascent timescale separation between the controller and adversarial disturbance.Exact dynamic programming is described as seldom feasible for high-dimensional continuous state spaces.Source: C ADDITIONAL EXPERIMENTAL DETAILS AND… · p. 36

C.1 LOW-DIMENSIONAL TABULAR EXPERIMENT

The tabular double-integrator experiment compares the joint value method with runtime compositions of separate task and safety critics, then illustrates how the learned safe set expands during training.

C.1 LOW-DIMENSIONAL TABULAR EXPERIMENT

  • With reward centered at xg = 1.2 (σx = 0.8, σv = 1.0) and control-cost weight λa = 1.0, the task encourages exiting the right boundary, while safety requires braking early enough to remain inside |x| < 1.The experiment uses full tabular dynamic programming to numerical convergence, with no neural network approximations.Source: §C.1 · p. 37
  • At large K, the joint framework recovers the viability-restricted optimum; its advantage over the hard mask and least-restrictive filter is attributed to propagating policy-conditioned safety truncation through Bellman updates.The safe set is verified exhaustively, so performance differences reflect admissible-action ranking rather than critic error or a mismatched feasible set.Source: §C.1 · pp. 37–38

C.2 SAFE SET EXPANSION

  • On InvertedPendulum-v4, the safe set expands toward positive cart velocity for higher reward before asymptotically reaching the maximal safe set.The figure shows the constraint boundary and the safety critic's 0-level set.Source: §C.2 · p. 38, Figure 3 · p. 38

C.3 CONTINUOUS CONTROL EXPERIMENTAL SETUP

The continuous-control setup tests safety-constrained reward maximization under adversarial disturbances, and the margin-scale ablation finds a reward–stability trade-off as K increases.

C.3 CONTINUOUS CONTROL EXPERIMENTAL SETUP

  • The evaluation spans three Gymnasium locomotion tasks and eight SafeVelocity tasks, with torso disturbances testing liveness constraints and conflicting reward-cost limits.The controller seeks forward locomotion while disturbances seek destabilization or speed-limit violations.Source: §C.3 · pp. 38–39
  • The baselines include least-restrictive safety filtering and CMDP methods using Lagrangian dual descent or trust-region projections, adapted to adversarial zero-sum settings.The evaluation tracks episodic returns and costs, prioritizing satisfaction of safety requirements.Source: §C.3 · p. 39

C.4 MARGIN SCALE K VARIATION

  • As K increases, episode reward generally improves while safety is generally preserved with small variations; however, excessive K may cause training instabilities, as observed in Walker2d.The ablation increased K in steps of 200 from 200 to 2000 across Hopper, Walker2d, SafetyHalfCheetah, and SafetyWalker2d.Source: §C.4 · p. 40

C.5 ADDITIONAL RESULTS

JointSAC maintains near-zero costs across the reported evaluation environments, while training-curve results show stable convergence and distinguish performance across task suites.

C.5 ADDITIONAL RESULTS

  • Across 11 environments, JointSAC maintains near-zero evaluation costs, with a maximum cost of 0.03 ± 0.08 on Hopper-v4; baseline cost violations vary by environment.Each environment has 1,500 evaluation episodes across five training seeds.Source: §C.5 · p. 41
  • Across three disturbed Gymnasium locomotion tasks, JointSAC converges stably, with near-zero episode costs on two of three tasks.Figure 5 plots episodic returns in the top row and costs in the bottom row across five seeds.Source: Figure 5 · p. 41, §C.5 · p. 41
  • Across eight disturbed SafeVelocity environments, JointSAC shows stable convergence with near-zero episode costs.Figure 6 plots returns in the top two rows and costs in the bottom two rows across five seeds.Source: Figure 6 · p. 42

Related papers

Loading 2610.12420v1…