SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning

Borong Zhang; Yuhao Zhang; Jiaming Ji; Yingshan Lei; Yishuai Cai; Josef Dai; Yuanpei Chen; Yaodong Yang

Source-linked AI summary

SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning

Borong Zhang, Yuhao Zhang, Jiaming Ji, Yingshan Lei, Yishuai Cai, Josef Dai, Yuanpei Chen, Yaodong Yang

arXiv:2503.03480v4Mar 5, 2025cs.ROcs.AI

TL;DR

VLAs create serious physical-world safety risks, motivating explicit safety constraints that existing alignment methods do not directly provide. SafeVLA develops an integrated safety approach using CMDP-based SafeRL to model risks, elicit unsafe behaviors, constrain policies, and assure safety. The approach reduces cumulative safety-violation cost by 83.58% while maintaining task performance (+3.85%) and generalizing safety behaviors to out-of-distribution perturbations.

Problem
VLAs lack explicit safety integration for complex and unpredictable physical-world deployment despite their potential as generalist robot policies.
Method
ISA uses CMDP-based SafeRL to model safety requirements, elicit unsafe behaviors, constrain VLA policies, and evaluate their safety.
Results
83.58% lower cumulative safety-violation cost than the state-of-the-art method was achieved while maintaining task performance (+3.85%).
Takeaways & Limitations
Aligned policies mitigate long-tail risks, handle extreme failures, and generalize learned safety behaviors to out-of-distribution perturbations.
Takeaways & Limitations
Training and evaluation rely on simulation, so extensive validation on physical robotic platforms remains necessary.

Abstract

from arXiv · show

Vision-language-action models (VLAs) show potential as generalist robot policies. However, these models pose extreme safety challenges during real-world deployment, including the risk of harm to the environment, the robot itself, and humans. How can safety constraints be explicitly integrated into VLAs? We address this by exploring an integrated safety approach (ISA), systematically modeling safety requirements, then actively eliciting diverse unsafe behaviors, effectively constraining VLA policies via safe reinforcement learning, and rigorously assuring their safety through targeted evaluations. Leveraging the constrained Markov decision process (CMDP) paradigm, ISA optimizes VLAs from a min-max perspective against elicited safety risks. Thus, policies aligned through this comprehensive approach achieve the following key features: (I) effective safety-performance trade-offs, reducing the cumulative cost of safety violations by 83.58% compared to the state-of-the-art method, while also maintaining task success rate (+3.85%). (II) strong safety assurance, with the ability to mitigate long-tail risks and handle extreme failure scenarios. (III) robust generalization of learned safety behaviors to various out-of-distribution perturbations. The effectiveness is evaluated on long-horizon mobile manipulation tasks. Our data, models and newly proposed benchmark environment are available at https://pku-safevla.github.io.

1 Introduction

VLAs are advancing toward generalist robot policies, but their physical-world deployment creates safety risks that existing model-alignment methods do not directly address. The paper proposes an integrated safety approach grounded in SafeRL and CMDPs to explicitly embed safety constraints into VLAs.

VLAs can generalize across instructions, embodiments, scenes, skills, and objects, increasing the importance of aligning them with human values and safety.
The paper asks how to integrate safety constraints into VLAs without sacrificing performance.
Existing LLM and VLM safety mechanisms cannot be directly applied to VLAs because physical-world challenges differ from abstract model-intention risks.
ISA systematically models safety requirements, elicits diverse unsafe behaviors, constrains policies with CMDP-based SafeRL, and evaluates aligned-policy safety.
The work introduces Safety-CHORES, a benchmark with fine-grained safety constraints in diverse, long-horizon navigation and manipulation tasks.
The approach reports reduced unsafe incident severity and robust generalization of learned safety behaviors to out-of-distribution perturbations.

2 Related Work

Related work positions VLAs as generalist robot policies with strong multimodal task capabilities, while highlighting gaps in principled safety alignment and comprehensive safety benchmarking. SafeRL and Safety-CHORES provide the relevant framework and evaluation direction.

VLAs aim to execute complex multimodal instructions across diverse environments and show strong task performance and generalization.
SafeRL formulates policy optimization as maximizing task rewards while explicitly satisfying predefined safety constraints within a CMDP.
Existing safety benchmarks often use simplified settings, while standard VLA benchmarks emphasize task success without diverse built-in safety constraints.

3 Problem Formulation

The paper formulates VLA safety alignment as constrained decision-making in a CMDP extended with natural-language instructions. The policy maximizes instruction-conditioned reward subject to safety-cost constraints.

A CMDP models decision-making under uncertainty by maximizing one objective while satisfying constraints on others.
The CMDP includes state and action spaces, transitions, rewards, costs, cost limits, an initial-state distribution, and a discount factor.
VLA safety alignment adapts the CMDP with a natural-language instruction space and an instruction-conditioned reward function.
The formulation represents the VLA policy as parameterized by θ and defines feasible policies for constrained optimization.

4 Implementing the Integrated Safety Approach

The integrated safety approach models safety predicates, elicits diverse risks in photorealistic mobile-manipulation environments, constrains VLA learning with SafeRL, and evaluates safety across multiple dimensions.

4.1 Modeling Safety: ISA models task and environmental safety through state-action and trajectory-level predicates.State-action predicates capture conditions at individual timesteps, while trajectory-level predicates represent temporal safety events.
4.2 Eliciting Risks: Safety-CHORES combines diverse indoor scenes, 3D objects, and safety-critical components to elicit varied safety violations.The benchmark includes navigation and manipulation scenarios involving corners, blind spots, fragile collections, and critical points.
4.3 Constraining Policies: SafeRL converts predicate violations into cost signals and uses Lagrangian relaxation to integrate safety constraints into VLA policy learning.State-action violations receive timestep costs, while trajectory-level violations initially assign cost to the final step of the violating segment.
4.3 Constraining Policies: The min-max optimization alternates updates to VLA parameters and Lagrange multipliers, prioritizing safety before task performance.This optimization seeks task performance subject to the modeled safety requirements.
4.4 Assuring Safety: ISA assurance evaluates test-time safety, long-tail safety, and extreme-failure safety across held-out data and out-of-distribution perturbations.The evaluation is designed to quantify learned safe behaviors and assess infrequent or severe failure scenarios.

5 Experiments

Experiments evaluate ISA against multiple VLA baselines and benchmarks using task success rate and cumulative safety cost. Results show improved safety-performance trade-offs, reduced severe failures, generalization across models and benchmarks, and robustness to perturbations and extreme scenarios.

Experimental Setup: Experiments compare ISA with imitation-learning and reinforcement-learning baselines across Safety-CHORES and other benchmarks.Evaluation uses task success rate and cumulative cost, which aggregates safety violations during an episode.
Comparative Performance: 83.58% average CC reduction and 3.85% average SR increase versus FLaRe demonstrate ISA’s safety-performance trade-off.The reductions are reported across Safety-CHORES tasks while task performance is maintained or improved.
Risk Handling and Failure Modes: ISA eliminates trajectories with cumulative cost >10 and reduces the upper bound of unsafe behavior severity to 1/35th of FLaRe’s.This distribution shift indicates mitigation of long-tail safety risks and catastrophic failures.
Generalization Across Models: ISA’s alignment consistently reduces CC while maintaining stable SR across distinct VLA base models and multiple benchmarks.The generalization is reported on Safety-CHORES and other benchmarks.
Benchmark Safety Challenges: Safety-CHORES often produces more than twice the CC of iTHOR or ProcTHOR because standard benchmarks lack safety-critical environmental designs.The comparison applies identical safety evaluation mechanisms across benchmarks.
Ablations and Robustness: Dynamic Lagrangian multipliers achieve higher SR than fixed-penalty baselines meeting the same cost limit, while stricter thresholds lower realized costs but can reduce SR.A 20% threshold is reported as balancing safety and task performance; ISA also preserves safety benefits under OOD perturbations and reduces extreme-failure costs.

6 Conclusion

The ISA integrates explicit safety constraints into VLAs through SafeRL and CMDP-based techniques, achieving improved safety and task performance while mitigating long-tail and out-of-distribution risks.

83.58% safety improvement over the state-of-the-art method was achieved while maintaining task performance (+3.85%).The approach combines modeling, unsafe-behavior elicitation, policy constraining, and safety assurance.

A Additional Empirical Results

Additional analyses compare task success with cumulative safety cost and examine cost distributions across tasks and rooms, alongside qualitative behavioral comparisons.

The additional results analyze task success probability as a function of cumulative cost for ISA and FLaRe.The analysis is presented in Figure 9 with correlation statistics in Table 3.
Mean cumulative cost is evaluated across rooms for Safety-ObjNav, Safety-Pickup, and Safety-Fetch.The measure averages unsafe events across the entire evaluation set.
Qualitative comparisons complement the quantitative analyses of model behavior.The section provides further behavioral examples alongside the reported metrics.

B.1 Behaviors Analysis in Test Sets

Test-set behavior analyses contrast unsafe, erratic trajectories from unaligned VLAs with safer trajectories from ISA-aligned policies across representative scenarios.

Unaligned VLAs damage or displace irrelevant objects, misidentify targets, become trapped or collide, and fail to account for blind spots.These behaviors span fragile collection, dangerous equipment, corner navigation, and collision scenarios.
Mean cumulative cost distributions are shown across rooms for Safety-ObjNav, Safety-Pickup, and Safety-Fetch.The distributions average unsafe events over the full evaluation set.
Figure 11 compares representative trajectories and gives examples of unsafe behaviors associated with safety-critical components.The ISA-aligned trajectory is described as smoother and more direct than the unaligned trajectory.

B.2 Behaviors Analysis in Extreme Failure Cases

Extreme-failure analyses describe cautious ISA-aligned behavior and use structured trajectory data with LLM-based analysis to investigate and generalize safety behaviors.

ISA-aligned policies maintain significantly lower safety costs even when task completion is nearly impossible.They tend toward limited cautious exploration, avoid unnecessary object interactions, and minimize forceful obstacle contact.
Table 4 presents the prompt used for safety analysis.
Structured trajectory data records actions, environmental interactions, object states, and perceptual inputs for safety analysis.The data is extracted from the AI2THOR simulation framework and formatted as JSON event logs.
LLM-based analysis converts unsafe-event JSON into concise evaluations and classifies events into predefined safety-critical categories.The analysis is designed to support more descriptive discovery of subtle and emergent unsafe behaviors.
Generalization is assessed through unseen safety predicates and performance in novel environments.The evaluation tests whether ISA learns safety logic beyond memorizing the initial constraints.

B.4.1 Generalization to Unseen Safety Predicates

ISA generalizes safety behavior beyond its training predicates and transfers that behavior across unseen scenes and perturbations, while improving all evaluated safety constraints.

Unseen predicates: ISA significantly reduces violation costs for five new safety predicates that were never used during training.The predicates cover electrical appliances, non-progressive movement, door interactions, and object-falling behaviors.
Predicate coverage: Original safety predicates achieve typically >95% coverage of risks identified by newly discovered predicates.This overlap supports the representativeness of the initial predicate set.
OOD scene transfer: ISA transfers learned safety behaviors to DivScene’s 81 unseen scene types, including challenging safety-critical environments.The evaluation groups scenes into six categories, including Hospital and Kitchen examples.
Safety balance: ISA achieves substantial cost reductions across all five distinct safety constraints, spanning navigation and delicate manipulation.The improvement is consistent rather than concentrated in a single easier-to-avoid constraint.
Perturbation robustness: Under semantic and perceptual perturbations, ISA maintains low safety cost even when confusing instructions reduce task success.The perturbation suite includes synonyms, structural changes, garbled commands, and flipped images.
Training dynamics: Cumulative cost typically falls below the safety limit within about 1M training steps and remains stable thereafter.The success rate rises rapidly during the first million steps, while the Lagrange multiplier increases early to satisfy constraints.

C.1 Details of SafeRL Training

ISA trains VLAs with CMDP-based SafeRL by combining reward optimization with explicit cumulative safety costs, predicate-based violation detection, and dual Lagrangian updates.

Dual optimization: ISA alternates updates to VLA parameters and Lagrange multipliers in a dual optimization procedure.The policy update uses the combined reward-cost objective, while the multiplier responds to expected constraint violation relative to a threshold.
Cost construction: ISA defines cumulative safety cost by aggregating K distinct instantaneous cost types across a trajectory.The instantaneous cost is defined at each timestep, while cumulative cost sums these costs over the trajectory.
SafeRL objective: The training objective balances task reward and constraint cost through a Lagrangian surrogate loss over task-instruction trajectories.The loss uses reward and cost objectives for trajectories sampled under the current VLA policy.
Constraint enforcement: The Lagrange multiplier increases when expected constraint violation exceeds the threshold and decreases otherwise.This adjustment dynamically changes the emphasis placed on safety constraints during training.
Safety predicates: Safety-critical scenarios are formalized as state-action or trajectory predicates that support violation labeling and CMDP cost construction.The framework includes components such as corner, blind-spot, and collision-related behaviors, with corresponding algorithmic checks.
Evaluation setup: Experiments compare ISA with fair and unfair baselines across eight models and evaluate both safety and task performance.The comparison includes alternative reinforcement-learning processes, imitation-only models, a from-scratch RL model, and privileged-information models.

E.4 OOD Evaluation Set-Up

The OOD evaluation applies visual perturbations within the training simulation to test whether VLA safety behaviors generalize beyond familiar appearances and lighting.

Evaluation setting: The perturbations are evaluated in the same base simulation environment used for training, with unseen visual conditions applied to the scenes.The benchmark compares perturbed scenes with normal rendering conditions.
Perturbation design: OOD evaluation uses three primary perturbation categories: lighting, environmental color, and object material changes.The conditions are applied individually and jointly to assess policy generalization.
Lighting OOD: Lighting OOD randomizes brightness, saturation, and hue to simulate varied natural and artificial illumination.The perturbations represent changes such as time of day, weather, and indoor lighting schemes.
Color OOD: Color OOD randomizes the appearance of floors, walls, doors, windows, and ceilings to challenge reliance on background cues.Brightness, saturation, and hue are varied across major environmental surfaces.
Material OOD: Material OOD randomly selects category-specific material packages for target, background, furniture, and other objects.The resulting texture, appearance, and hue changes test robustness of object recognition.

G Limitations and Future Work

The paper’s main limitations are reliance on simulation, simplified uniform binary constraints, and methodological choices that require refinement before broader deployment.

Scope boundary: The study relies on simulation for both training and evaluation, leaving extensive physical-robot validation as a necessary next step.The authors note that simulation enables affordable collection of diverse safety-critical data but does not replace real-platform validation.
Methodological limitations: Trajectory-level violations receive cost credit at the final step, which may limit sample efficiency and motivates improved credit-assignment strategies.The authors identify heuristic-based credit assignment as a future refinement.
Cost design: Safety constraints are binary and uniformly applied rather than instruction-specific or severity-weighted.The authors state that severity is context-dependent and propose richer weighting schemes for future work.
Future work: Future work targets sim-to-real validation, severity-weighted and dynamic constraints, risk-sensitive metrics, and uncertainty estimation.The proposed directions include CVaR-based risk measures and conservative behavior under uncertainty.
Impact boundary: Public release of the data, code, and models creates a stated misuse risk if unsafe intentions are injected into deployed systems.The impact statement explicitly cautions that the method could be exploited for harmful deployment.