PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Mumuksh Tayal, Manan Tayal, Ravi Prakash

arXiv:2603.15136v2cs.LGcs.AI

TL;DR

Offline safe RL must optimize reward from static data while satisfying hard safety constraints, but existing approaches can rely on soft costs or slow iterative inference. SafeFQL combines reachability-inspired safety value learning with behavior-cloned flow policies and one-step actor distillation. Across navigation and Safety Gymnasium tasks, it reports competitive rewards, near-zero violations, and a 2.5× inference speedup over iterative action sampling.

  • Problem

    Offline safe RL seeks reward-maximizing policies from static datasets while satisfying strict safety constraints, avoiding risky online interaction and slow inference.

  • Method

    SafeFQL combines reachability-inspired safety value learning, behavioral-cloned flow policies, and Q-optimized one-step actor distillation without deployment rejection sampling.

  • Results

    SafeFQL matches or exceeds prior offline safe RL performance while reducing violations across navigation and Safety Gymnasium tasks, with a 2.5× inference time speedup.

  • Takeaways & Limitations

    SafeFQL supports real-time safety-critical control by retaining competitive rewards and near-zero violations with a single action proposal.

  • Takeaways & Limitations

    Hard indicator masking may produce a non-smooth loss landscape and can require hyperparameter tuning if replaced by softer masking or Lagrangian relaxation.

Abstract

from arXiv · show

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

1 Introduction

Safe offline RL addresses reward optimization from fixed data under strict safety constraints, where online failures and slow generative inference are problematic. SafeFQL combines reachability-inspired safety learning with one-step flow policy extraction and reports strong benchmark performance with lower deployment latency.

  • Motivation: Online constrained RL can be unsafe because it requires environment interaction and repeated rollouts, motivating offline policy synthesis from logged data.Training-time failures are costly in safety-critical domains, and faithful simulators may be unavailable.
  • Motivation: Classical HJ and CBF methods encode stronger state-wise safety notions but face dimensionality or model-learning challenges in practical offline settings.Grid-based HJ methods face the curse of dimensionality, while learned dynamics can introduce safety-estimation errors.
  • SafeFQL: SafeFQL combines reachability-inspired safety value learning with a one-step flow actor optimized by Q-learning and regularized toward behavior-supported actions.The approach avoids iterative generative sampling and rejection sampling at deployment while retaining expressive action modeling.
  • SafeFQL: SafeFQL delivers substantially lower inference latency than diffusion-style safe generative baselines despite potentially higher offline training cost.The claimed latency advantage targets real-time safety-critical control loops.
  • Evaluation: Across navigation and Safety Gymnasium benchmarks, SafeFQL reports lower safety violations while maintaining strong reward relative to prior constrained offline RL and safe generative baselines.The supplied contribution statement summarizes the cross-benchmark pattern without specifying a single aggregate metric.

2 Background and Problem Setup

The paper formulates offline safe RL as maximizing discounted reward while requiring trajectories to remain outside a failure set at every future time. It contrasts hard state-wise safety with expected-cost methods and motivates flow-based, distilled policy extraction for efficient deployment.

  • Problem Setup: The CMDP setup models states, actions, transitions, rewards, safety values, and discounting under hard state constraints.The safety function defines a failure set of unsafe states that must be avoided.
  • Problem Setup: The offline dataset contains state, action, reward, safety, and next-state tuples collected by an unknown behavior policy, with no further environment interaction permitted.Policies induce trajectories through the environment transition function.
  • Problem Setup: The objective is maximum discounted return subject to remaining outside the failure set for all future time steps.Only policies whose trajectories remain entirely outside the failure set are feasible.
  • Generative Policies: Diffusion policies require many discrete inference steps, whereas flow matching learns a velocity field enabling efficient sampling through a single ODE integration.The flow policy is a deterministic map from state and noise that induces a stochastic action policy through Gaussian noise.
  • Safe Offline RL: Expected-cost offline methods can degrade value estimation and generalization, while HJ-based methods identify states that can reach failure within a time horizon.HJ value signs distinguish states where safe policies exist from states where failure is unavoidable under the best policy.
  • Policy Extraction: Multi-step generative processes can be distilled into single-step policies to reduce computational cost while matching the outputs of their multi-step counterparts.This provides the policy-extraction direction used by efficient offline control methods.

3 Safe Flow Q-Learning

SafeFQL separates critic learning, behavior-flow policy distillation, and feasibility-gated actor optimization to obtain safe offline actions without deployment-time rejection sampling. Its reachability-inspired safety critic prioritizes feasibility while the distilled one-step actor preserves low-latency execution and dataset support.

  • Framework: SafeFQL decomposes training into critic learning, behavior-flow fitting and distillation, then feasibility-gated actor optimization.The phases are sequentially dependent, while networks within each phase are trained in parallel to convergence.
  • Reward and safety critics: Reward critics use IQL expectile regression, avoiding actor queries during critic updates and representing high-quantile in-dataset actions.The value function tracks a high expectile of the behavior-policy Q distribution, while target parameters are updated with EMA.
  • Reward and safety critics: SafeFQL replaces soft cumulative-cost penalties with a reachability-inspired safety critic trained by max-backup recursion.The recursion takes the maximum of immediate safety margin and discounted future safety value, propagating future hazards backward; Qc < 0 indicates predicted continued safety under behavior-like actions.
  • Behavior flow policy and one-step distillation: The flow teacher is behavior-cloned and distilled into a one-step actor, whose deployment cost is a single feedforward pass regardless of teacher integration steps.Distillation also regularizes the actor toward the offline-data support, reducing exploitation of critic extrapolation errors.
  • Feasibility-gated actor objective: The feasibility gate gives safety strict priority: infeasible actions ignore reward and recover feasibility, while feasible actions optimize reward.This exclusive ordering avoids the additive reward-safety tradeoff that can let reward gradients push actions into infeasibility near the boundary.

4 Experiments

SafeFQL is evaluated against constrained offline RL and generative baselines across navigation and Safety Gymnasium tasks, emphasizing safety, reward, sampling efficiency, and computation time. It achieves strong safety-reward trade-offs while reducing action-sampling and deployment costs.

  • Evaluation setup: SafeFQL is compared with BEAR-Lag, COptiDICE, CPQ, C2IQL, FISOR, and SafeIFQL using safety violations and cumulative episode reward.The evaluation covers fixed offline datasets and reports mean ± standard deviation across seeds.
  • Results: SafeFQL achieves the lowest costs across evaluated environments and the highest reward among frameworks with comparable costs.Figure 2 reports reward and cost as mean ± standard deviation across five seeds.
  • Results: SafeFQL maintains zero violations while significantly increasing reward over all baselines in Safe Boat Navigation.The reported navigation evaluation uses fixed randomly selected initial states.
  • Results: Across Safety Gymnasium MuJoCo tasks, SafeFQL achieves the lowest safety violations and highest reward among frameworks with comparable near-zero costs.The tasks include HalfCheetah, Hopper, Ant, Walker2D, and Swimmer.
  • Sampling efficiency: SafeFQL reaches comparable high safety with N=1 action sample, whereas FISOR and SafeIFQL require N=16 samples.The comparison concerns safety rate, defined as the percentage of episodes without collisions, in Safe Boat Navigation.
  • Computation time: SafeFQL requires longer training but achieves minimal inference latency by directly outputting one action instead of using rejection sampling or multi-step denoising.The computation-time comparison evaluates FISOR, SafeIFQL, and SafeFQL across five independent trial runs.

5 Conclusion, Limitations and Future works

SafeFQL combines reachability-inspired safety with a distilled one-step flow actor, achieving efficient deployment and near-zero violations across the reported benchmarks. The authors identify hard indicator masking as a potential source of non-smooth training behavior.

  • Conclusion: SafeFQL distills a multi-step flow policy into a one-step actor, achieving a reported 2.5× inference-time speedup.The framework maintains competitive rewards and near-zero violations using a single action proposal across navigation and Safety Gymnasium tasks.
  • Conclusion: SafeFQL combines generative flow-model expressivity with Hamilton-Jacobi reachability principles for scalable offline safe reinforcement learning.The method targets safe action selection without iterative deployment-time action sampling.
  • Limitations and future work: Hard indicator masking of Q-critic functions may yield a non-smooth loss landscape and affect training stability.The authors suggest continuous masking or soft Lagrangian relaxations, which would require hyperparameter tuning.

A Description of the Experiments

The experiments cover a custom drifting-river boat task and Safe Velocity MuJoCo environments, using explicit geometric or velocity-based safety definitions. Offline data, fixed evaluation states, and environment parameters define the evaluation setting.

  • Safe Boat Navigation: Safe Boat Navigation models a point-mass boat bypassing two obstacles in a drifting river toward a goal.The river drift varies with the boat’s y-coordinate, and evaluation uses 500 randomly selected initial states.
  • Safe Boat Navigation: The boat’s dense distance-based reward encourages progress toward the fixed goal [0.5, 0.0]ᵀ.The reward uses C = 0.1 and is designed so maximizing reward drives the boat toward the goal.
  • Safe Boat Navigation: Boat obstacles and failure states are encoded by ℓ(x), with ℓ(x) > 0 identifying states inside an obstacle.The super-level set {x : ℓ(x) > 0} defines the failure region.
  • Offline data generation: The custom boat dataset contains 2,500 uniformly sampled initial states and trajectories of 400 timesteps with Δt = 0.005 s.Control inputs are sampled uniformly from the admissible action set subject to the norm constraint.
  • Safety MuJoCo Environments: Safety Gymnasium evaluates Safe Velocity tasks in Hopper, HalfCheetah, Swimmer, Walker2D, and Ant, where velocity must remain below task-specific thresholds.Agents maximize reward while incurring cost whenever instantaneous velocity exceeds its threshold.
  • Safety MuJoCo Environments: MuJoCo safety uses a continuous function with ℓ(x) ≤ 0 denoting the safe set, replacing sparse binary costs with a dense signal.Threshold velocities, integration timesteps, and action-space bounds are taken from official documentation and summarized in Table 1.

B Additional Results

Figure 6 compares boat-navigation trajectories for generative-policy methods across candidate action-pool sizes. SafeFQL uses one candidate because it avoids rejection sampling, achieving zero collisions and trajectories closest to the goal among safe baselines.

  • SafeFQL uses N=1, whereas FISOR and SafeIFQL use N∈{1,4,8,16} to compare trajectory rollouts across candidate action-pool sizes.SafeFQL does not require action rejection sampling.
  • SafeFQL achieves zero collisions across its boat-navigation trajectories.
  • SafeFQL reaches closest to the goal among all safe trajectories produced by the baselines.
  • Avoiding rejection sampling removes the computational burden of multi-action sampling at inference time.

C Experimental Details

Experiments used a common hardware system for both training and evaluation, consisting of a 14th-Gen Intel Core i9-14900KS CPU, 64 GB RAM, and an NVIDIA GeForce RTX 5090 GPU.

  • All experiments used the same system for training and evaluation.The system included a 14th-Gen Intel Core i9-14900KS CPU, 64 GB of RAM, and an NVIDIA GeForce RTX 5090 GPU.

C.2 Network Architecture and Training Details of the Proposed Algorithm

The experimental setup documents SafeFQL and baseline hyperparameters, datasets, implementations, and selected optimization settings. MuJoCo experiments use DSRL dataset standards, with specified critic, target-network, and advantage-clipping procedures.

  • Training settings for all environments are detailed in Table 2.The MuJoCo environments use the widely accepted DSRL dataset.
  • Expectile regression uses τ=0.9, while reward and safety critics use clipped double Q-learning.The clipped method takes the minimum of two Q values.
  • Target Q networks are updated with exponential moving average using weight 0.005 for new parameters.
  • Exponential advantages are clipped to (−∞,100] in feasible regions and (−∞,150] in infeasible regions.
  • Baseline hyperparameters are listed in Table 3, with implementations sourced from the cited official or adapted repositories.The baselines use DSRL dataset standards.
Loading 2603.15136v2…