Source-linked AI summary
Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Juntao Dai, Yaodong Yang
TL;DR
SafeRL addresses the challenge of optimizing policies under multiple safety constraints, but existing benchmark coverage and usability remain limited. Safety-Gymnasium expands environments across agents, tasks, and visual inputs while SafePO provides 16 algorithms; the study reports faster CPU rendering than Safety Gym and improved safety–reward trade-offs, though real-world transfer remains limited.
Problem
SafeRL needs environments that support safety-critical policy optimization under multiple constraints across varied settings.
Method
The paper builds Safety-Gymnasium on Gymnasium and MuJoCo and releases SafePO, a single-file library containing 16 diverse SafeRL algorithms.
Results
Safety-Gymnasium expands agents, tasks, multi-agent coverage, and visual support, while CPU rendering runs twice as fast as Safety Gym’s former implementation.
Takeaways & Limitations
The benchmark supports algorithm verification and comparison across single-agent, multi-agent, and vision-based safety challenges.
Takeaways & Limitations
The study does not encompass all constraint types, including human-centric safety considerations, and its simulated results may transfer imperfectly to complex real-world applications.
Abstract
from arXiv · showhide
Artificial intelligence (AI) systems possess significant potential to drive societal progress. However, their deployment often faces obstacles due to substantial safety concerns. Safe reinforcement learning (SafeRL) emerges as a solution to optimize policies while simultaneously adhering to multiple constraints, thereby addressing the challenge of integrating reinforcement learning in safety-critical scenarios. In this paper, we present an environment suite called Safety-Gymnasium, which encompasses safety-critical tasks in both single and multi-agent scenarios, accepting vector and vision-only input. Additionally, we offer a library of algorithms named Safe Policy Optimization (SafePO), comprising 16 state-of-the-art SafeRL algorithms. This comprehensive library can serve as a validation tool for the research community. By introducing this benchmark, we aim to facilitate the evaluation and comparison of safety performance, thus fostering the development of reinforcement learning for safer, more reliable, and responsible real-world applications. The website of this project can be accessed at https://sites.google.com/view/safety-gymnasium.
1 Introduction
Safety-Gymnasium addresses limited SafeRL simulation coverage with an extensible benchmark spanning diverse agents, tasks, inputs, and constraints. SafePO complements it with 16 algorithms and analyses across 54 environments.
- SafeRL addresses policy optimization under multiple constraints for safety-critical applications, where unsafe actions can cause serious harm.
- Refactoring of the physics engine: Safety-Gymnasium directly supports MuJoCo and removes dependence on mujoco-py, enabling newer features and lowering installation barriers.
- Enhanced Visual Task Support: CPU rendering is twice as fast as the former approach, while visual components are more comprehensive.
- Environmental Components: Safety-Gymnasium expands SafeRL environments to single-agent, multi-agent, vision-based, and high-dimensional GPU-supported scenarios.
- Algorithm Components: SafePO provides 16 diverse SafeRL algorithms in individually decoupled files, covering single- and multi-agent, first- and second-order, Lagrangian, and projection-based methods.
- Insights and Analysis: The benchmark and library support analysis of 16 algorithms across 54 environments, with metadata available for verification and further research.
2 Related Work
SafeRL research relies on simulation environments and constrained-policy methods, but the field has comparatively few dedicated environments and faces algorithmic challenges in high-dimensional control.
- Safety Environments: Existing suites include AI-Safety-Gridworlds, Safety Gym, safe-control-gym, and MetaDrive, covering different safety properties and control settings.
- SafeRL Algorithms: SafeRL is commonly formulated through CMDPs, while CPO provides near-constraint-satisfaction guarantees but depends on Taylor approximations and high-dimensional Fisher-matrix inversion.
3 Preliminaries
The preliminaries formulate SafeRL as constrained decision-making for single agents and cooperative multi-agent systems. The objective is to maximize expected reward while satisfying cost constraints.
- Constrained Markov Decision Process: A CMDP specifies state and action spaces, transitions, rewards, cost functions, cost thresholds, an initial-state distribution, and a discount factor.
- Constrained Markov Decision Process: A parameterized policy maps states to action distributions and induces state-transition probabilities and discounted state distributions.
- Constrained Markov Decision Process: The CMDP objective maximizes infinite-horizon discounted reward over policies whose discounted costs satisfy all specified thresholds.
- Constrained Markov Game: A constrained Markov game extends the formulation to multiple agents with joint actions, shared transitions, rewards, and per-agent cost functions.
- Constrained Markov Game: In the fully cooperative setting, agents maximize common expected reward while satisfying each agent’s safety constraints.
4 Safety Environments: Safety-Gymnasium
Safety-Gymnasium provides accessible SafeRL environments spanning single-agent, multi-agent, vision-only, and high-dimensional robotic scenarios, with varied tasks and safety constraints.
- The Gymnasium-based environments include Point, Car, Doggo, racecar, ant, and multi-agent robot configurations.
- Supported tasks include Velocity, Run, Circle, Goal, Push, and Button objectives.
- Safety constraints include velocity limits and costs for contacts with pillars, hazards, sigwalls, and other environmental objects.
- Vision-only environments support more realistic visual inputs, including RGB and RGB-D observations.
- Safety-DexterousHands uses Isaac Gym for GPU-enabled parallel sampling in single-agent and multi-agent ball-transfer tasks.
- DexterousHands rewards combine ball-goal positional distance and angular distance, while safety costs penalize joint-angle violations.
5 Safe Policy Optimization Algorithms: SafePO
SafePO is a single-file library of SafeRL algorithms designed for extensibility, visualization, documentation, and validation against existing implementations.
- SafePO provides 16 diverse SafeRL algorithms in a single-file-style library.
- SafePO implementations follow original papers, are checked against open-source code, and achieve comparable or better performance than existing implementations.
- Its architecture allows new algorithms to be integrated by inheriting base algorithms and implementing only their unique features.
- SafePO supports TensorBoard and WandB logging, visualizing more than 40 training parameters and intermediate computation results.
- The library includes installation guidance, common-issue solutions, usage instructions, customization guidance, and responsible-use information.
- Table 1 compares implementations over 10 evaluation iterations and more than 3 seeds across 10 environments under cost_limit=25.00.
6 Experiments and Analysis
The experiments compare SafeRL algorithms across single-agent and multi-agent Safety-Gymnasium tasks, emphasizing reward–cost trade-offs, stochasticity, and optimization strategy. Results show distinct safety and performance patterns across Lagrangian, projection-based, and PID-controlled methods.
- Experimental setup: Figure 8 compares episodic reward and proportional episodic costs across 40 navigation, 6 velocity, and 8 multi-agent velocity tasks.The left graph reports reward, while the right graph reports costs under varying constraints.
- Reward and Cost: 98% lower cost with PPO-Lag than PPO in velocity tasks coincides with a 45% reward decrease.This illustrates the trade-off between maximizing reward and satisfying safety constraints.
- Randomness and Oscillation: All SafeRL algorithms remain within cost_limit on velocity tasks, whereas highly stochastic navigation tasks produce pronounced performance oscillations and more dispersed optimal policies.The navigation results show greater algorithmic divergence than the velocity results.
- Lagrangian vs. Projection: Lagrangian methods oscillate more around safety constraints than projection-based methods, with PPO-Lag showing more extreme Strongly Unsafe and Strongly Safe outcomes than CPO.PCPO has lower average costs and rewards than CPO across navigation and velocity tasks, while similar differences appear between MACPO and MAPPO-Lag.
- Lagrangian vs. PID-Lagrangian: CPPO-PID retains episodic rewards similar to PPO-Lag while entering the Strongly Unsafe region less often.Its training outcomes therefore contain a larger proportion of Safe states and improved safety performance.
7 Limitations and Future Works
The study does not cover all forms of safety constraints and evaluates policies only in simulated environments. Consequently, transfer to complex real-world safety-critical applications may be limited.
- Scope of constraints: Human-centric safety constraints important in human-AI collaboration are not fully integrated into the study.The authors identify this as an example of an uncovered constraint type.
- Simulation scope: The evaluation focuses on safety tasks within a simulated environment rather than physical real-world systems.The paper proposes transferring refined policies to physical robotic platforms as future work.
- Real-world transfer: Transferability of the reported results to complex real-world safety-critical applications may be limited.This boundary follows directly from the study’s simulation-focused scope.
A.1 Hyperparameters Analysis
The appendix describes SafePO’s network, optimization, and Lagrangian settings, then analyzes how hyperparameters and algorithm classes relate to observed performance. It reports that Lagrangian methods are useful but sensitive, while second-order methods trade velocity reward for navigation safety.
- Hyperparameter settings: SafePO estimates reward and cost advantages with GAE and learns neural-network parameters using Adam.These are the shared optimization components described for the disclosed settings.
- Network settings: Single-agent models use 3-layer Tanh MLPs with hidden sizes [64, 64], expanding to [256, 256] for Ant and Doggo navigation agents.Multi-agent models instead use 3-layer ReLU MLPs with hidden sizes [128, 128].
- Lagrangian settings: Lagrangian-based methods are sensitive to hyperparameter configuration, including the multiplier’s initial value and learning rate.Higher initial values promote safer exploration but may impede task completion, while high learning rates induce excessive oscillations and hinder convergence.
- Evaluation protocol: The performance tables evaluate SafePO algorithms under cost_limit=25.00 using 10 assessment iterations across multiple random seeds.The table marks reward and cost directions and distinguishes constraint breaches, compliance, and optimal policies.
- Performance analysis: Second-order algorithms obtain lower rewards than other methods in MuJoCo velocity tasks but perform better on navigation tasks requiring higher safety standards.Their navigation advantage is associated with similar rewards alongside fewer and smoother cost violations.
B.1 Single-agent Specification
The single-agent specification describes four robot embodiments—Point, Car, Racecar, and Ant—with increasing control and dynamics complexity. Their action and observation spaces are documented in the accompanying tables.
- Point: Point operates in 2D with separate rotation and forward/backward actuators, and a front square helps identify orientation and push boxes.The specification corresponds to Figure 9 and the Point information, observation, and action tables.
- Car: Car operates in 3D with two independently driven parallel wheels and a freely rolling rear wheel requiring coordinated steering and movement.Its overall information, action space, and observation space are listed in Tables 9–11.
- Racecar: Racecar uses one velocity servo and one position servo to control rear-wheel speed and front-wheel steering angle.Its dynamics model references the MIT Racecar project, and its spaces are documented in Tables 12–14.
- Racecar: Racecar must coordinate tire steering angle and speed to achieve its specified goal.This coordination is described as analogous to the control relationship involved in human driving.
- Ant: Ant is a quadrupedal robot with a torso and four two-limb legs connected by hinges.Movement requires coordinating moments applied to eight hinge drivers, with spaces listed in Tables 15–17.
B.2 Multi-agents Specification
Safety-Gymnasium specifies multi-agent and Gymnasium-based environments spanning decomposed robots, diverse tasks, safety constraints, and vision-only scenarios. The suite also extends task complexity through progressively difficult construction, racing, and FormulaOne environments.
- Multi-agent robots: 2-ant, 2-ant-diag, and 4-ant decompose the Ant robot into two or four agents controlling different leg partitions.The configurations define separate action spaces for each agent.
- Multi-agent robots: Safety-Gymnasium also provides other multi-agent versions of robots beyond those described in detail.Additional specifications are available in the project documentation.
- Gymnasium-based tasks: Gymnasium-based environments support Velocity, Run, Circle, Goal, Push, and Button tasks with distinct movement, navigation, manipulation, and constraint objectives.The tasks include coordinated locomotion, reaching goals, moving boxes, and activating buttons.
- Safety constraints: Safety constraints are represented through velocity limits, pillars, hazards, Sigwalls, vases, and gremlins that impose costs on unsafe actions or contacts.These mechanisms cover excessive speed, obstacle contact, restricted regions, fragile objects, and moving objects.
- Vision-only environments: Vision-only environments use more realistic MuJoCo tasks and support both RGB and RGB-D inputs.The environments target evaluation of vision-based SafeRL algorithms.
- Progressive task settings: Construction, Race, and FormulaOne tasks increase difficulty by adding restricted areas, hazards, obstacles, distant starts, and randomized checkpoints.BuildingButton, BuildingGoal, and BuildingPush likewise progress from basic objectives to environments with more robots, hazards, or obstacles.
B.6 Some Issues about Safety Gym
The paper identifies several issues in Safety Gym involving lidar coverage, cost representation, observation-space ranges, dependency maintenance, and limited robot specifications. Safety-Gymnasium addresses some representation and usability problems while documenting remaining limitations.
- Observation and sensing issues: Safety Gym’s original Natural Lidar cannot detect low-lying objects, potentially limiting comprehensive environmental observations.The paper presents this as a bug in the original implementation.
- Observation-space representation: Safety Gym’s default one-dimensional observation representation assigns all observation ranges to [−∞, +∞], which can mislead users.Safety-Gymnasium uses Gymnasium dictionary and flattening mechanisms to provide a more intuitive representation.
- Cost representation: Safety Gym represents costs by default as binary values, recording only whether a cost is incurred.The paper argues that this omits differences in violation severity and interactions among multiple costs.
- Cost representation: When multiple obstacles trigger costs, binary cost totals can reduce the learning information available for multiple constraints.The paper gives different vase-collision velocities as an example of behaviors that could warrant different costs.
- Dependency maintenance: Neglected dependency maintenance can create compatibility errors, including NumPy binary incompatibility and removal of the numpy.complex attribute.The paper specifically associates these problems with numpy =1.17.4.
C.2 Task Representation
Safety-DexterousHands defines dual-hand manipulation tasks with high-dimensional observations and actions, reward functions based on object-goal pose errors, and explicit joint-safety constraints. The task representation distinguishes physical state, actions, rewards, and constrained joint ranges.
- Hand Over: The Hand Over task uses two opposing Shadow Hands to grasp an object with one hand and transfer it to the other while both bases remain fixed.The object initially descends near the right hand before being passed across.
- Hand Over: Hand Over has 398-dimensional observations and 40-dimensional actions, with a reduced 24-dimensional hand observation when fixed bases are excluded.The action and observation spaces are documented separately for the task.
- Rewards: The Hand Over reward increases as translational and rotational object-goal discrepancies decrease.The reward is r = exp(−0.2(αdt + dr)), where α balances translational and rotational terms.
- Hand Over Catch: Hand Over Catch combines half Hand Over with Catch Underarm, requiring an object to be thrown from a vertical hand to a palm-up hand.This combined task has 422-dimensional observations and 52-dimensional actions.
- Safety constraints: Safety Finger constrains forefinger joints ②, ③, and ④ to [22.5°, 67.5°], [22.5°, 67.5°], and [−10°, 10°], respectively.The cost is triggered when any of the three joint angles leaves its permitted interval.