PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning

Boyang Li, Matthew Kim, Sylvia Lee Herbert

arXiv:2609.33337v1cs.LGcs.RO

TL;DR

Average-cost safe RL can permit individual trajectory violations, while HJ-based hard-safety methods face action distributions that may be multimodal. SSM combines HJ-gated score regression with an online diffusion actor-critic. It reports best or near-best task performance with low false-safe rates on control benchmarks, and the lowest cost with competitive reward on Safety-Gymnasium velocity tasks.

  • Problem

    Expected-cost methods enforce safety only on average, while hard HJ constraints can produce disconnected viable action sets that challenge Gaussian or deterministic actors.

    Source: §1 · p. 2

  • Method

    SSM trains an online diffusion actor by routing a score target with HJ feasibility, using reward guidance for viable actions and recovery guidance for infeasible states.

    Source: §1 · p. 2

  • Results

    SSM attains best or near-best task metrics with low false-safe rates on quadrotor and fixed-wing benchmarks, and lowest cost with competitive reward on two Safety-Gymnasium velocity tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SSM provides an online actor-critic approach that combines diffusion-policy score regression with HJ reachability-based state-wise feasibility.

    Source: §1 · p. 2

    SSM relies on a learned safety critic, and the paper does not establish the critic-error bound used to characterize its gate; online accuracy under nonstationary replay remains open.

    Source: §7 · p. 10

Abstract

from arXiv · show

Online safe reinforcement learning (RL) seeks policies that maximize reward while satisfying safety constraints. A popular line of research in safe RL relaxes safety to a soft expected-cost constraint and solves the resulting Constrained Markov Decision Process via primal-dual Lagrangian updates that only enforce safety on average. To address this limitation, hard, state-wise constraints are introduced and often imposed through Hamilton-Jacobi (HJ) reachability. Yet such constraints require solving different objectives in the feasible and infeasible regions: reward maximization in the former, recovery toward the feasible regions in the latter. The resulting target action distributions are inherently multimodal, and this structure poses a fundamental challenge for the Gaussian or deterministic actors used in existing HJ-based safe RL, which often collapse onto suboptimal modes. Diffusion policies provide the expressiveness needed to represent such distributions, and recent work on Q-score matching offers a route to training them for online RL by score regression -- but has been applied only to reward maximization. We propose Safe Score Matching (SSM), an off-policy actor-critic method that adapts Q-score matching to hard-constrained safe RL by gating a two-branch score target with HJ reachability: inside the feasible set, the denoising process degenerates to Q-score matching on actions classified as viable by the HJ critic; outside, a recovery branch biases denoising toward regions with lower worst-case violation. On quadrotor and fixed-wing trajectory-tracking and stabilize-and-avoid benchmarks, SSM attains the best or near-best task performance with low false-safe rates, whereas the primal-dual baseline admits more unsafe behavior and reachability-based baselines tend to be more conservative; on Safety-Gymnasium velocity tasks, SSM attains the lowest cost with competitive reward.

1 Introduction

Online safe RL must reconcile hard, state-wise safety with reward optimization, but existing HJ-based actors can struggle with disconnected viable actions. SSM combines HJ-gated score targets with a diffusion policy, and reports strong task performance with low false-safe rates.

Motivation

  • Primal–dual methods enforce expected cumulative-cost budgets only on average, while HJ reachability imposes state-wise safety but can yield disconnected viable action sets.A unimodal actor may place probability in unsafe regions between disconnected components or collapse onto a local reward optimum.Source: §1 · p. 2
  • Online diffusion score-matching methods align denoisers to reward gradients, but provide no mechanism for hard, state-wise safety.Diffusion policies can represent multimodal action distributions, while existing HJ-based policies are Gaussian or deterministic.Source: §1 · p. 2

Contributions

  • SSM routes an action-level score target by HJ feasibility: viable actions receive reward guidance, while infeasible states receive recovery guidance that minimizes worst-case safety violation.The routing uses the learned safety critic, and the score field depends on learned reward and safety critics.Source: §1 · p. 2
  • SSM is presented as an online actor-critic method combining HJ reachability-based state-wise feasibility with diffusion-policy score regression for hard-constrained safe RL.The paper positions the method alongside existing safe RL families in Table 1.Source: §1 · p. 2, Table 1 · p. 2

Empirical validation

  • SSM attains the best or near-best task metric on Quad2D, Quad3D, and F16 with low false-safe rates, and the lowest cost with competitive reward on two Safety-Gymnasium velocity tasks.The evaluation spans trajectory tracking, regulation, stabilize-and-avoid, and velocity tasks.Source: §1 · p. 2

2 Related Work

Prior safe RL methods variously enforce average-cost constraints, state-wise constraints with Gaussian or deterministic actors, or use generative policies for reward optimization alone. SSM addresses the online setting at the intersection of reachability-based safety and diffusion policies.

Safe RL families

  • Online primal–dual methods enforce safety as an expected cumulative-cost budget, whereas state-wise and reachability-based methods target hard safety but use Gaussian or deterministic actors.Generative-policy methods include offline diffusion and flow policies and online score- or flow-matching methods, which are reward-only.Source: §2 · p. 3

Closest prior work

  • FISOR combines hard safety, HJ-style feasibility, and a diffusion policy offline; SSM studies the corresponding online actor–critic setting and converts HJ signals into a denoising score target.SSM’s online policy is updated from newly collected data.Source: §2 · p. 3

3 Problem Formulation

The formulation replaces aggregate expected-cost safety with a pointwise constraint and uses HJ reachability to reason about feasible initial states and recovery. Its reachability-value reformulation preserves reward maximization where safety is feasible and assigns a worst-case-violation objective elsewhere.

3 Problem Formulation

  • Most online safe RL methods solve a CMDP relaxation with a constraint aggregated over initial states and trajectories, which can leave individual trajectories violating the pointwise safety condition.The optimal Lagrange multiplier also varies across tasks.Source: §3 · p. 3
  • The hard-constrained formulation requires h to remain non-positive at every step, but this trajectory-level condition is not directly amenable to temporal-difference learning and is unsatisfiable from infeasible initial states.HJ reachability instead produces value functions that reason about initial states.Source: §3 · p. 3, §3.1 · p. 3

3.1 Reachability Reformulation

  • The HJ safety value captures worst-case violation along a trajectory and collapses the infinite-horizon constraint into a condition on the initial state.The paper defines both policy and optimal HJ safety value functions.Source: §3.1 · p. 4
  • The optimal HJ safety value being non-positive is equivalent to the existence of an optimal policy that enforces the hard constraints.This criterion motivates the optimal feasible region.Source: §3.1 · p. 4
  • HJ avoid state-action values track worst-case violation from a state-action pair, and the optimal feasible action set defines actions that preserve feasibility.The learned implementation approximates this support restriction and is not itself an oracle safety guarantee.Source: §3.1 · p. 4
  • The reachability-value reformulation matches the hard-constrained problem on feasible initial states and switches to minimizing worst-case future violation on infeasible states.This is the reachability-constrained formulation used by RCRL and also adopted by offline FISOR.Source: §3.1 · p. 4

4 Method: Safe Score Matching

SSM uses diffusion policies to represent action distributions that may be multimodal under reachability constraints, and adapts Q-score matching with a feasibility-gated target. The target guides viable actions toward reward improvement and infeasible states toward reduced worst-case violation.

4 Method: Safe Score Matching

  • The feasible action distribution may be non-convex and disconnected, making deterministic actors commit to one component and unimodal Gaussians average across components or unsafe regions.Diffusion policies can represent more complex action distributions; Figure 1 illustrates left- and right-route modes from nearly the same initial state.Source: §4 · p. 5, Figure 1 · p. 5
  • Q-score matching regresses the reward-Q action gradient as a time-independent denoising target, avoiding differentiation through the expensive, high-variance reverse chain.SSM adds a feasibility-dependent split and restricts the feasible branch to HJ-viable actions.Source: §4 · p. 5

4.1 HJ-Gated Local Policy Improvement

  • The feasible subproblem improves reward only over actions preserving the oracle feasible set, while the infeasible subproblem biases actions toward lower worst-case violation without guaranteeing recovery.These are fixed-critic actor targets, not an exact decomposition of the trajectory-level control problem.Source: §4.1 · p. 6
  • Entropy regularization yields a closed-form target whose feasible-state score is QSM’s reward score restricted to actions admitted by the safety critic, and whose infeasible-state score favors lower predicted violation.When the gate is inactive, the target reduces to the QSM reward score.Source: §4.1 · p. 6
  • The feasible target can be multimodal through reward-Q peaks, disconnected viable components, or their intersection; diffusion denoising preserves the closed-form target while allowing finite-step sampling.The passage contrasts this with an energy-based sampler that requires iterative MCMC.Source: §4.1 · p. 6

4.2 SSM Denoising Target

  • SSM trains a reverse-chain vector field by squared regression to the HJ-gated guidance field, without propagating gradients through that field.At the population optimum, the learned field equals −Mq times the guidance field under the training distribution.Source: §4.2 · p. 7

5 Practical Algorithm

SSM learns an HJ safety critic from off-policy Bellman targets and uses its feasibility estimate to route the score target. Its finite-candidate action minimization is conservative at a fixed critic, while the candidate-set ablation exposes a task–safety trade-off.

Safety critic and HJ gate

  • The safety critic is trained by mean-squared regression to a discounted HJ Bellman target, with an EMA target network for bootstrapping.The Bellman target requires minimizing Qh over actions.Source: §5 · p. 7
  • The gate estimates the inner action minimum using one policy sample and K −1 Gaussian perturbations, then uses cVh(s) ≤0 as the feasibility indicator.Because the candidate set is a subset of the action space, its minimum is at least the exact minimum at a fixed critic; this conservatism does not extend to the learned critic.Source: §5 · p. 7

Learned-critic error

  • With zero thresholds, bounded optimistic critic error can admit unsafe states or actions only when their exact safety value lies in (0, εQ].The bound εQ is assumed rather than established; replacing zero thresholds with −εQ prevents errors of that size from admitting unsafe states or actions to the reward branch.Source: §5 · p. 8

Candidate-set ablation

  • On Quad3D, replacing local Gaussian proposals with independent reverse-chain samples substantially increases terminal tracking ℓ1, and proposal scale has an interior optimum at ση = 0.3.Increasing K from 8 to 32 at ση = 0.3 eliminates observed episode violations but increases terminal tracking ℓ1, making K a task–safety operating knob.Source: §5 · p. 8

6 Experiments

Across the control benchmarks, SSM combines best or near-best task performance with safety-prediction strengths, with a Quad2D exception. It also achieves the lowest mean cost on both Safety-Gymnasium velocity tasks, while the experiments examine performance, safety, and computational trade-offs.

Benchmarks and baselines

  • The control suite covers Quad2D circular-reference tracking, Quad3D regulation, and F16 stabilization-and-avoid, while Safety-Gymnasium adds HalfCheetah and Swimmer velocity tasks.Methods are evaluated at their best control-task checkpoint; Safety-Gymnasium uses five seeds and a fixed endpoint.Source: §6 · p. 8

Results on the control benchmarks

  • SSM has the lowest Quad2D tracking error and Quad3D terminal tracking ℓ1, and the highest F16 stabilization rate; RESPO has slightly lower Quad3D final-50-step tracking ℓ1.On Quad3D and F16, SSM also has the highest precision; on Quad2D, it predicts the fewest states safe while RESPO has higher precision and coverage.Source: §6 · p. 9, Figure 3 · p. 9
  • On F16, RAC and EFPPO accept more unsafe states than SSM, while CAL accepts the most unsafe states on every control task.CAL constrains cost only in expectation; Figure 3 reports task performance above and safety prediction below, with precision, false-safe rate, and coverage as safety metrics.Source: §6 · p. 9, Figure 3 · p. 9

Safety-Gymnasium

  • SSM has the lowest mean cost on both Safety-Gymnasium velocity tasks and the highest reward among within-budget methods on HalfCheetah.On Swimmer, SSM's cost is 0.5, more than ten times lower than any other method, while RAC and ALGD achieve higher reward within budget.Source: §6 · p. 9, Table 2 · p. 10
  • SSM's cost peaks within the first 100k environment steps and remains low afterwards.The paper relates this pattern to its objective of targeting h(st) ≤0 at every step rather than an expected-cost budget.Source: §6 · p. 10

Additional studies

  • On CarGoal1, varying αr, β, or Mq from 0.25× to 4× default changes reward and the fraction of violation-free episodes only slightly.Under the same expected-cost formulation, a QSM diffusion actor achieves higher reward and lower cost than a Gaussian SAC actor.Source: §6 · p. 10

Computation

  • 4 to 6 times fewer environment steps let SSM reach its reported Quad3D performance than RAC and RESPO; training takes about one hour versus RESPO's about 2.5 hours.At deployment, SSM runs one five-step reverse chain per action.Source: §6 · p. 10

7 Conclusion and Limitations

SSM combines state-wise HJ reachability with diffusion-policy score regression, routing viable states toward reward and infeasible states toward recovery. It reports strong task performance with low false-safe rates, while relying on a learned safety critic whose online accuracy remains challenging.

7 Conclusion and Limitations

  • SSM attains the best or near-best task metric with low false-safe rates on quadrotor and fixed-wing benchmarks, and the lowest cost with competitive reward on Safety-Gymnasium velocity tasks.Its actor applies reward guidance to states and actions classified as viable, and recovery guidance to infeasible states.Source: §7 · p. 10
  • Accurate online learning of the safety critic from nonstationary replay remains an open challenge, and the analysis assumes deterministic dynamics.The cited critic-error proposition gives a gate margin if an error bound holds but does not establish that bound.Source: §7 · p. 10

A Extended Related Work

  • HJ-based safe-RL methods use Gaussian or deterministic actors, while SSM replaces that policy class and update rule with diffusion and HJ-gated score matching.Prior HJ approaches use a feasible/infeasible objective split, and their actors are limited for non-convex feasible-action distributions.Source: A Extended Related Work · p. 15
  • Online diffusion and flow methods including QSM are reward-only; SSM extends QSM’s score-target machinery to a reward/recovery target indexed by HJ feasibility.Related safe generative methods include offline FISOR and online ALGD with an expected cumulative-cost constraint.Source: A Extended Related Work · p. 15

B Q-Score Matching: Background and Endpoint Guidance

  • QSM uses a t-independent denoising target to avoid backpropagating through the full reverse sampling chain, and SSM extends its reward-only target using HJ feasibility.SSM’s score is for the clean-action target; the exact score of forward-smoothed intermediate marginals is generally intractable because the hard-feasibility indicator creates a discontinuous density.Source: B Q-Score Matching: Background and… · pp. 15–16
  • SSM’s reverse update is characterized as a diffusion-parameterized Langevin-style sampler on an HJ-gated energy landscape, not an exact multi-level noisy-score sampler.The guidance strength is treated as a hyperparameter rather than assigned a closed-form stationary density for the finite-step chain.Source: B Q-Score Matching: Background and… · p. 16

C Finite-Candidate HJ Gate

The finite-candidate HJ gate estimates feasibility by minimizing the safety critic over a sampled subset of actions. At a fixed critic, this yields a conservative inner approximation, but online critic error can break that guarantee.

C Finite-Candidate HJ Gate

  • Each gate evaluation uses one diffusion sample and K −1 local Gaussian-proposal candidates, with the same construction for target-state and current-state gates.The two gates differ in whether they query the EMA target or online safety critic.Source: C Finite-Candidate HJ Gate · p. 16
  • The finite-candidate gate’s minimum safety value is at least the continuous-action minimum, making its feasible-state set an inner approximation at a fixed critic.The proposition does not extend this conservatism to the learned critic in the online loop.Source: C Finite-Candidate HJ Gate · p. 16
  • The fixed-critic proposition certifies state-level feasibility classification, not that every action sampled by the diffusion policy is viable.Reward guidance is separately restricted to actions satisfying Qh(s, at) ≤0.Source: C Finite-Candidate HJ Gate · p. 16

D Proofs for Section 4

  • The appendix proves the gated-target theorems under measurability and finite partition functions, with stronger differentiability and continuity assumptions required only locally for later results.The stronger hypotheses apply to Theorem 2 and Proposition 5.Source: D Proofs for Section 4 · p. 17

D.1 Proof of Theorem 1

The appendix establishes the entropy-regularized HJ-gated target through a KL argument, then records its zero-temperature limits and the denoising-regression identity. The regression target aligns the update with the clean-action score only under stated conditions.

D.1 Proof of Theorem 1

  • The KL-divergence identity yields a unique feasible-branch maximizer equal to the target Boltzmann policy, and likewise a unique infeasible-state maximizer.The density and uniqueness arguments assume the stated partition-function and support conditions.Source: §D.1 · p. 18

Zero-Temperature Concentration (sanity check)

  • As αr →∞ the feasible target concentrates on reward maximizers, while as β →∞ the infeasible target concentrates on minimizers of the safety value.The limits recover the unregularized greedy subproblems under the stated regularity conditions.Source: Zero-Temperature Concentration (sanity… · pp. 18–19

D.3 Denoising Regression Identity and DDPM Guidance Coefficient

  • The population L2-optimal denoiser target is −Mq φ̄(s, at), independent of diffusion time.The regression identity follows because this target depends on (s, at) alone, not on latent noise or time.Source: §D.3 · p. 19
  • The reverse update aligns with the clean-energy score only on the target-support interior where projection is inactive; at action boundaries, projection can attenuate or eliminate guidance.This is guidance-component alignment, not an exact correspondence to the finite-step reverse chain’s target score.Source: §D.3 · p. 20

E Implementation and Pseudo Code

Appendix E gives the online SSM training loop, critic and denoiser updates, and an alternative safety-value estimator; the following appendices specify architectures, settings, and benchmark systems.

E Implementation and Pseudo Code

  • The online loop collects replay data, estimates target-state HJ values by finite-candidate minimization, updates reward and safety critics, then trains the denoiser with an HJ-gated score target.The algorithm also updates EMA target networks.Source: E Implementation and Pseudo Code · pp. 20–21
  • The reward critic uses clipped double-Q regression with bootstrap actions from the diffusion policy, and the denoiser update uses the same reverse chain for environment and bootstrap actions.On control tasks, reward guidance is normalized per sample; recovery uses the raw safety gradient, with a separate fallback when a feasible state has a non-viable queried action.Source: E Implementation and Pseudo Code · pp. 21–22
  • Quad2D uses an expectile-smoothed state-value estimate with τ = 0.1; other tasks use finite-candidate minimization, and any consistent surrogate for min_a Qh(s, a) is admissible.The expectile value is fit to a smoothed lower expectile over replay-buffer actions.Source: E Implementation and Pseudo Code · p. 22

F.1 Network Architecture

  • The safety and reward critics are two-hidden-layer MLPs; the denoiser uses Mish activations and Fourier time embeddings, with three hidden layers on Quad3D and F16.Quad2D uses two denoiser hidden layers, while the higher-dimensional Quad3D and F16 use three.Source: §F.1 · p. 22

F.2 Hyperparameters; F.3 System Dynamics and Safety Functions

  • The appendix specifies shared and task-specific hyperparameters and deterministic continuous-time dynamics, rewards, and safety functions for the three control benchmarks.The tasks cover Quad2D circular-reference tracking, Quad3D regulation toward the origin, and F16 stabilization in a low-altitude band.Source: §F.2 · p. 22, §F.3 · pp. 22–23, Table 3 · p. 23

F.4 Evaluation Protocol and Metrics

The evaluation appendix describes baseline training and checkpoint selection, then defines task-specific tracking and stabilization metrics.

F.4 Evaluation Protocol and Metrics

  • Every control-task baseline is rerun and trained for its recommended iterations or at least three hours, and each method is evaluated at its best checkpoint.SSM uses an RTX 5090; baselines use RTX 4090 or RTX 5090 GPUs.Source: §F.4 · p. 24
  • Quad2D tracking error averages squared six-coordinate state errors over steps and episodes, including episodes ending in crashes.Quad3D terminal tracking ℓ1 includes crashed episodes, while final-50-step tracking ℓ1 averages the same distance over each episode’s last 50 steps.Source: §F.4 · p. 24
  • F16 stabilization rate is the fraction of episodes that keep altitude within H ∈[50, 150] for at least 50 consecutive steps.The appendix refers to Section 6 for the safety metrics.Source: §F.4 · p. 24

F.5 Quad3D Ablation Study

The Quad3D ablation compares safety-value estimators and candidate configurations; local sampling choices trade tracking performance against observed violations.

F.5 Quad3D Ablation Study

  • Variants are evaluated on 500 safe, non-goal reset states; metrics include terminal tracking ℓ1, episode and step violation rates, and crash fraction.The evaluation reports initial h with mean −0.530 and 95th percentile −0.099.Source: §F.5 · p. 24
  • The expectile-regressed value baseline increases terminal tracking ℓ1 by an order of magnitude over the sampled gate.The sampled gate resolves min_a Qh(s, a) locally, whereas the expectile network learns a smoothed envelope over policy actions; the text relates local resolution to Quad3D’s wide reset distribution and disconnected feasible region.Source: §F.5 · p. 24
  • On 500 evaluation episodes, default K = 8, ση = 0.3 has the lowest terminal tracking ℓ1; K = 32, ση = 0.3 is the only sampled-gate cell with zero observed violations across all three safety metrics.The latter is the safety-clean cell with the lowest accompanying terminal tracking ℓ1.Source: §F.5 · p. 24, Table 5 · p. 25
  • Among no-Gaussian variants, K = 1 has the lowest terminal tracking ℓ1 with zero observed violation, while policy-only K = 8 has the worst terminal tracking ℓ1.Both drop local proposals; only K differs.Source: §F.5 · p. 25
  • Along ση = 0.3, raising K from 8 to 32 removes observed episode violations but increases terminal tracking ℓ1; off that ridge, the K–violation relationship is non-monotonic.The proposed explanation is that ση = 0.1 proposals stay near the policy mode, while ση = 0.8 can reach poorly calibrated Qh actions.Source: §F.5 · p. 25

F.6 Safety-Gymnasium Experiments and Additional Studies

The Safety-Gymnasium studies compare fixed-horizon velocity-task results and additional sensitivity and policy-class experiments; SSM uses reachability routing with a diffusion denoiser.

F.6 Safety-Gymnasium Experiments and Additional Studies

  • Velocity-task costs count threshold-exceeding steps in 1,000-step episodes; methods use five seeds, 50 evaluation episodes, and 1M steps, except RESPO’s 10M-step horizon.CAL and ALGD train against budget d = 25, which is also the feasibility criterion for all methods.Source: §F.6 · p. 25
  • For velocity tasks, SSM retains the HJ critic and finite-candidate gate but trains with a posterior-noise regression target and an adaptive reward coefficient.At test time it acts with one diffusion sample without Qh filtering; its gate candidates are used during training, not deployment.Source: §F.6 · pp. 25, 26, 28
  • SSM cost peaks within the first 100k steps and remains low on both velocity tasks, while RESPO’s reward at comparable step counts is far below the other methods.Figure 4 plots test reward and cost against environment steps on a log scale; the cost axis is truncated.Source: §F.6 · p. 26, Figure 4 · p. 28
  • Across 16× sweeps of αr, β, and Mq on SafetyCarGoal1-v0, defaults attain the highest reward and goal count while every setting retains low cost and many violation-free episodes.These coefficients rescale reward, recovery, or the whole denoising guidance without moving critic-defined gates.Source: §F.6 · p. 26
  • Under the same expected-cost formulation on SafetyCarGoal1-v0, QSM-Lag outperforms SAC-Lag in reward and goal count, has substantially lower cost, and is the only actor staying within budget d = 5.QSM-Lag uses a diffusion actor; SAC-Lag uses a Gaussian actor.Source: §F.6 · p. 27
  • On Quad3D, SSM reaches its reported performance with 4 to 6 times fewer environment steps than RAC and RESPO and trains in about one hour, versus about 2.5 hours for RESPO.CAL is more sample-efficient, converging within 5 × 10^5 steps; SSM deploys one five-step reverse chain per action.Source: §F.6 · pp. 27–28
Loading 2609.33337v1…