PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

Rongxu Cui, Zongzheng Zhang, Jingrui Pang, Haohan Chi, Jinbang Guo, Saining Zhang, Shaoxuan Xie, Xin Jin, Yao Mu, Jiaolong Yang, Guocai Yao, Xianyuan Zhan, Ya-Qin Zhang, Hao Zhao

arXiv:2606.23686v2cs.RO

TL;DR

VLA operational safety under strict constraints remains largely unverified, while existing approaches and benchmarks have fragmented or limited safety evaluation. LIBERO-Safety builds a parametric benchmark and keypose-driven data pipeline, and its evaluation finds a tension between safer trajectories and task success.

  • Problem

    VLA safety under strict operational constraints remains largely unverified, and existing safety approaches use inconsistent evaluation protocols.

    Source: §1 · p. 2

  • Method

    LIBERO-Safety uses UBDDL to generate stochastic safety-critical scenarios and a keypose-driven planner pipeline to produce a large, randomized collision-free demonstration dataset.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    High-diversity training fosters safer trajectories, but task success remains bottlenecked by sub-optimal trajectory synthesis and semantic misalignment.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    LIBERO-Safety provides a structured evaluation and data-generation foundation for future work on real-world validation and intrinsically safety-aligned VLA models.

    Source: §5 · p. 15

    The current training corpus focuses on safe demonstrations rather than hard-negative unsafe trajectories.

    Source: §5 · p. 15

Abstract

from arXiv · show

Despite the impressive manipulation capabilities of Vision-Language-Action (VLA) models, their operational safety under strict constraints remains largely unverified. To address this, we introduce a parametric safety benchmark to procedurally generate safety-critical scenarios with comprehensive stochasticity. To overcome the scalability bottlenecks of human teleoperation, we develop a novel keypose-driven data generation pipeline. Leveraging this infrastructure, we curate a large-scale dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization. We then conduct a systematic cross-paradigm evaluation of eight VLA and two embodied foundation models. Our analysis reveals a critical generalization-safety tension: although high-diversity training fosters safer trajectories, task success remains fundamentally bottlenecked by sub-optimal trajectory synthesis and semantic misalignment. By providing a scalable pipeline, a robust dataset, and profound failure-mode insights, LIBERO-Safety establishes a crucial foundation for developing safe and reliable VLA models.

1 INTRODUCTION

LIBERO-Safety addresses gaps in VLA safety evaluation with a benchmark spanning physical and semantic risks and a scalable demonstration-generation pipeline. Its evaluation finds that safer trajectories do not eliminate task-success bottlenecks.

1 INTRODUCTION

  • Existing safety approaches use inconsistent protocols, while prior benchmarks are limited by teleoperation costs and simple tabletop obstacles.LIBERO-Safety adds semantic refusal, general human-robot interaction safety, and proximal avoidance around complex hand-object configurations.Source: §1 · p. 2
  • LIBERO-Safety evaluates physical and semantic safety across parameterized scenarios, environmental stochasticity, and explicit constraints.Its UBDDL infrastructure supports a five-dimensional safety curriculum.Source: §1 · pp. 2–3
  • The keypose-driven pipeline couples sparse human intent with motion planning to generate 19,664 human-screened, collision-free demonstrations across 40 tasks.The trajectories use extensive visual and physical domain randomization.Source: §1 · p. 3
  • High-diversity training fosters safer trajectories, but task success remains bottlenecked by sub-optimal trajectory synthesis and semantic misalignment.The evaluation covers eight VLA models for physical safety and two embodied foundation models for semantic safety reasoning.Source: §1 · p. 3

2 Related Work

Prior work advances VLA capability and safety-aware control, but existing methods lack unified safety evaluation. Current benchmarks emphasize task proficiency more than safety, a gap LIBERO-Safety targets with broader coverage.

2.1 Vision-Language-Action Models

  • VLAs have improved manipulation and generalization, but safety concerns still limit their real-world deployment.Recent policies also use reinforcement learning to refine behavior and long-horizon task success.Source: §2.1 · p. 3

2.2 Safety-Aware Policy Learning

  • Safety-aware policies use objectives, safe-set estimation, constrained optimization, and control barrier functions, yet lack a unified evaluation framework.SPARK provides standardized evaluation protocols for humanoid platforms, while broader comparison and quantification of safety guarantees remain limited.Source: §2.2 · p. 4

2.3 Benchmarks for VLA Evaluation

  • Existing benchmarks emphasize task proficiency over safety; LIBERO-Safety jointly covers perturbations, parameterized tasks, scene dynamics, physical and semantic safety, and proximal HRI.Its overview organizes five safety dimensions across three difficulty tiers and includes scalable collision-free demonstration generation.Source: Table 1 · p. 4, §2.3 · p. 5, Figure 2 · p. 5

3 VLA Safety Benchmark

LIBERO-Safety combines parametric environments, a five-suite safety taxonomy, keypose-guided trajectory generation, and a curated training dataset. The design targets stochastic evaluation and scalable generation of collision-free demonstrations.

3 VLA Safety Benchmark

  • The benchmark comprises a parametric environment framework, safety-centric taxonomy, keypose-guided generation pipeline, and high-fidelity safety dataset.Source: §3 · p. 5

3.1 Parametric Environment Definition

  • UBDDL defines tasks with goals, timestep safety constraints, initial-scene distributions, and environmental parameters, generating 7,603 scenes with 953 objects and 462 hand-object pairs.Its parameters include camera extrinsics, visual variations, and sensor noise.Source: §3.1 · p. 6

3.2 Safety-Centric Task Taxonomy

3.3 Keypose-Driven Data Generation Pipeline

  • Sparse object-centric keyposes seed CuRobo trajectories under collision-free constraints, reducing operator effort to 1.8 minutes per task.Each keypose has five spatial variants, which are sampled across stages to diversify demonstrations.Source: §3.3 · pp. 7–8

3.4 Training Dataset

  • Human-in-the-loop screening yields 19,664 safe demonstrations from L0 and L1 tasks across four categories.The training set omits the Semantic Reasoning suite and all L2 tasks; visual and physical randomization broadens trajectory diversity, and the dataset has broader spatial coverage and larger scale than LIBERO [28].Source: §3.4 · pp. 8–9, Figure 3 · p. 9

4 Experiment

The experiments evaluate physical and semantic safety across multiple model paradigms, finding broad vulnerabilities alongside gains from diverse training. Task failures can persist without safety violations, through inefficient trajectories or semantic misalignment.

4.1 Experimental Setup

4.2 Main Results

  • π0.5 achieves the highest overall success rate among evaluated standard VLAs, while still showing noticeable variation across trials.The evaluation also finds that standard large-scale pre-training alone is insufficient for robust reactive physical safety.Source: §4.2 · pp. 10–11
  • In an obstacle-free workspace outside the training distribution, π0.5 synthesizes a direct, kinematically optimal path instead of repeating nonlinear avoidance trajectories.The paper attributes this zero-shot spatial generalization to dataset scale and structural diversity.Source: §4.2 · pp. 11–12, Figure 4 · p. 11
  • RynnBrain-CoP shows greater refusal-rate resilience at higher difficulty tiers than RoboBrain2.0, which reaches 80% at L0 but drops on L1 and L2.The authors identify explicit reasoning frameworks as important for semantic safety alignment on nuanced or adversarial commands.Source: §4.2 · p. 12, Table 4 · p. 12

4.3 Analysis of Generalization and Safety

  • 500 rather than 50 demonstrations per task raises π0.5’s SR by 2.4% and reduces CR from 12.7% to 10%.LDLJ scores and execution times also improve, though a performance gap remains relative to the privileged CuRobo baseline.Source: §4.3 · pp. 12–13, Table 5 · p. 12
  • Under L0 perturbations, SR stays between 56.3% and 60.7% versus a 58.0% baseline, but layout changes raise CR to 6.3% and lower SR to 56.3%.Unseen objects raise CR only from 2.7% to 3.3%.Source: §4.3 · p. 13, Table 6 · p. 13

4.4 Failure Case Analysis

  • Sub-optimal trajectory synthesis causes collision-free task incompletions through erratic or overly conservative paths, temporal overflows, and kinematic deadlocks.The paper links these behaviors to a lack of long-horizon temporal consistency.Source: §4.4 · pp. 13–14, Figure 5 · p. 14
  • Semantic misalignment can direct mechanically stable, collision-free grasps toward visually similar distractors instead of the instructed targets.The authors identify fine-grained relational reasoning as necessary to bind commands to physical targets.Source: §4.4 · p. 14, Figure 6 · p. 14

5 Conclusion

LIBERO-Safety combines a UBDDL-powered parametric benchmark with keypose-driven generation of collision-free demonstrations to evaluate physical and semantic safety. The paper identifies a generalization-safety tension and notes simulation, human-in-the-loop, and training-corpus boundaries.

5 Conclusion

  • LIBERO-Safety finds that greater data diversity improves safety-aware execution but does not fully resolve sub-optimal trajectory synthesis or semantic misalignment.The benchmark remains simulation-based, the pipeline retains human involvement, and the corpus focuses on safe rather than hard-negative unsafe demonstrations.Source: §5 · pp. 14–15

Appendix

  • The appendix describes a five-suite, three-level safety taxonomy and details model configurations, training settings, real-world deployment, and limitations.Its future-work discussion includes real-world validation, dense unsafe-event annotation, and intrinsically safety-aligned VLA models.Source: Appendix · p. 19

A.2 UBDDL Environmental Configurations

A.1 Preliminary: The BDDL Framework

  • BDDL specifies long-horizon activities through entities, initial-state literals, and goal conditions, without prescribing execution steps.This process-agnostic formulation supports diverse configurations and multiple valid trajectories.Source: §A.1 · p. 20

A.2 UBDDL Environmental Configurations

  • Dynamic entities can follow linear or circular motion, requiring policies to synthesize obstacle avoidance beyond static-scene memorization.The framework describes these dynamic objects as a prerequisite for safe human-robot collaboration.Source: Dynamic Object Definition · pp. 22–23
  • Evaluation safety constraints flag robot-object contact and excessive end-effector normal force as violations.The collision pipeline detects contact pairs; the force check compares the measured normal force against a predefined threshold.Source: Safety Constraints · pp. 23–24

A.3 Additional Data-Generation and Evaluation-Split Details

The keypose pipeline uses sparse human-specified stages and planner-generated trajectories, while evaluation splits and perturbations define the generalization setting.

A.3 Additional Data-Generation and Evaluation-Split Details

  • Operators specify sparse object-centric stages, then CuRobo generates full trajectories under scene-specific obstacle constraints.Keypose variants are sampled at each stage, and human screening checks task validity and safety rather than editing paths.Source: §A.3 · p. 24
  • L2 tasks are held out from LIBERO-Safety fine-tuning, but “unseen” does not guarantee absence from foundation-model pretraining.Held-out variations cover object orientations, paraphrased language, obstacle geometries, and hand-object configurations across suites.Source: §A.3 · p. 24
  • Main experiments keep camera intrinsics calibrated while randomizing extrinsics, viewpoints, lighting, textures, sensor corruptions, scene layouts, and robot initial states.This isolates environmental and viewpoint variation from camera-calibration changes.Source: §A.3 · p. 24

B Safety Centric Task Taxonomy

The taxonomy organizes 75 trials across five safety suites and three difficulty levels, escalating physical execution and semantic comprehension demands.

B Safety Centric Task Taxonomy

Affordance Aware Grasping Suite

  • Affordance-aware grasping progresses from nominal object poses to planar translations and out-of-distribution rotations requiring approach-vector adjustment.The tiers test stable prehension, translational invariance, and geometric reasoning under rotational anomalies.Source: Affordance Aware Grasping Suite · pp. 25–26

Human-Robot Interaction Suite

  • Human-robot interaction evaluates collaborative manipulation alongside a simulated human proxy, with configurations organized by difficulty level.The supplied passage identifies this suite’s shared-environment setting and refers to its level-specific configurations.Source: Human Robot Interaction Suite · p. 26

Tabletop Spatial Avoidance Suite

  • Tabletop spatial avoidance tests manipulation around stationary obstacles, moving entities, and novel obstacle topologies.The levels progress from visual-to-action avoidance of familiar clutter to continuous adaptation and zero-shot generalization to unseen geometries.Source: Tabletop Spatial Avoidance Suite · pp. 27–28

Free Space Hand Object Avoidance Suite

  • Free-space hand-object avoidance tests three-dimensional evasion of static, moving, and novel human hand-object configurations.MANO and GrabNet generate diverse grasping poses; the highest tier evaluates zero-shot generalization to unseen kinematic postures.Source: Free Space Hand Object Avoidance Suite · pp. 28–29

Semantic Safety Reasoning Suite

  • Semantic safety reasoning tests refusal of explicit harm, recognition of physical common-sense violations, and response to contextual traps.The assessment uses visual observations paired with natural-language instructions to determine whether the requested action is safe.Source: Semantic Safety Reasoning Suite · pp. 29–30

C Experiment Implementation

This section outlines the experimental setup, and the appendix table summarizes evaluated baseline models by safety track and architectural paradigm.

C Experiment Implementation

C.1 Model Details

The evaluation covers ten baselines across physical safety and semantic safety reasoning. The physical track compares three VLA paradigms, while the semantic track tests embodied foundation models’ ability to identify and refuse hazardous instructions.

C.1 Model Details

  • The benchmark evaluates ten baselines across physical safety and semantic safety reasoning, with embodied foundation models assigned to the semantic track.The physical track evaluates VLA models; the semantic track assesses safe refusal of hazardous instructions.Source: §C.1 · pp. 30–31, – UniVLA · p. 33
  • Standard VLA models map multimodal observations and instructions directly to low-level actions, enabling comparison of discrete-token and continuous-action formulations.OpenVLA and OpenVLA-OFT use discrete action tokenization, while π0 and π0.5 represent continuous-action formulations.Source: §C.1 · p. 31, – OpenVLA · p. 31
  • World model-based VLAs predict future states or observations to support forward-looking trajectory planning and physical robustness.UniVLA predicts future visual observations from multimodal histories, while VLA-JEPA predicts abstract future-frame representations.Source: – OpenVLA · p. 31, – UniVLA · p. 32
  • Dual-system VLAs separate high-level semantic reasoning from high-frequency physical execution through specialized modules.GR00T N1.5 and N1.6 pair vision-language reasoning with diffusion-based action generation.Source: – UniVLA · pp. 32–33
  • Semantic safety evaluation tests whether embodied models can recognize hazardous instructions and refuse them, including prompts with physical or semantic adversarial cues.RoboBrain2.0 and RynnBrain-CoP are evaluated for hazard reasoning and autonomous refusal.Source: – UniVLA · p. 33

C.2 Training Configurations

The section details fine-tuning configurations for the evaluated VLA models and GR00T systems. Configurations vary in training duration, batch size, and which model components are updated.

C.2 Training Configurations

  • OpenVLA uses LoRA for 200,000 gradient steps, while OpenVLA-OFT uses rank-32 LoRA for 150,000 steps with a 10× learning-rate decay after 100,000 steps.Both use randomized image augmentation; their effective batch sizes are 128 and 64, respectively.Source: §C.2 · pp. 33–34, OpenVLA-OFT · p. 34, Table A.2 · p. 34
  • π0 is trained for 30,000 steps with batch size 32, while π0.5 is trained for 30,000 steps with batch size 256 and a 10,000-step warmup.Both use bfloat16 precision and an exponential moving average of policy weights.Source: OpenVLA-OFT · p. 35, Table A.4 · p. 35, Table A.5 · p. 35
  • UniVLA uses an effective batch size of 128 and jointly optimizes its backbone and action components, while VLA-JEPA trains for 100,000 steps with differential learning rates.UniVLA uses a 16-entry latent-action codebook; VLA-JEPA weights its action and spatial-reasoning losses at 1.0 and 0.1.Source: OpenVLA-OFT · p. 36, Table A.6 · p. 36, Table A.7 · p. 36
  • GR00T N1.5 freezes its language model and vision tower and trains for 60,000 steps, while GR00T N1.6 freezes its vision-language backbone and trains for 20,000 steps.Their effective batch sizes are 1024 and 640, respectively; N1.6 also uses state dropout of p = 0.8.Source: GR00T N1.5 · p. 37, Table A.8 · p. 37, Table A.9 · p. 37

C.3 Additional Experimental Results

Additional experiments compare safety-critical baselines, balance semantic-safety evaluation with benign prompts, and test the effect of more demonstrations. More demonstrations improve several OpenVLA-OFT outcomes but do not eliminate failures.

C.3 Additional Experimental Results

  • Zero-shot models have near-zero success and severe collision rates, while obstacle-free SFT improves task execution but retains high collision rates.The comparison indicates generic task imitation alone is insufficient for collision-averse behavior.Source: §C.3 · p. 38
  • The semantic-safety evaluation adds benign instructions to hazardous prompts and uses F1 to account for correct refusals, over-refusals, and unsafe compliance.Embodied foundation models are evaluated zero-shot on the mixed prompt set.Source: §C.3 · pp. 38–39, Table A.11 · p. 39
  • Increasing OpenVLA-OFT demonstrations from 50 to 500 per task improves SR, LDLJ, and execution time while reducing CR.The gains do not fully eliminate collision or task-completion failures.Source: Table A.12 · p. 39, §C.3 · p. 39

D Real World Deployment

The deployment section introduces chunk-level control-barrier-function filtering to adjust VLA action sequences while maintaining safety constraints. In a real-world fragile-obstacle task, the filter redirects the robot around a bottle and allows task completion without contact.

D Real World Deployment

  • Chunk-level CBF filtering modifies a VLA’s predicted action chunk, addressing the mismatch between temporally coherent VLA outputs and action-wise filtering that can cause jitter.The chunk optimization penalizes deviation from nominal actions, temporal changes, and safety-constraint slack.Source: D Real World Deployment · pp. 39–40
  • A distance-based barrier defines safety as the robot’s distance from obstacles minus a predefined safety threshold, covering static, dynamic, and human-interaction obstacles.The filter runs at each chunk boundary in a receding-horizon manner.Source: D Real World Deployment · p. 40
  • In the real-world bowl-placement task, chunk-level CBF filtering detours around a fragile bottle and completes the task without contact.Without filtering, the π0 policy collides with the bottle and knocks it over.Source: D Real World Deployment · pp. 40–41, Figure A.8 · p. 41

E Limitations and Future Work

The benchmark’s simulation fidelity is limited by approximations in physics, human behavior, and visual rendering. Future work proposes intrinsic safety enforcement that integrates semantic risk assessment with low-level control and detects out-of-distribution visual states.

Limitations and Future Work

  • The physics engine approximates contact dynamics, soft-body deformation, and friction, while simulated human proxies lack authentic cognitive unpredictability and spontaneous behavioral variation.These limitations contribute to a simulation-to-reality gap in physical deployment.Source: E Limitations and Future Work · p. 41
  • Procedurally generated environments cannot fully reproduce dynamic illumination shifts, specular reflections, or camera motion blur found in physical reality.These are constraints of the visual rendering pipeline.Source: E Limitations and Future Work · p. 42
  • Future work proposes hierarchical architectures to bridge slow semantic risk assessment and high-frequency motor control, integrating linguistic safety constraints into real-time trajectory synthesis.It also proposes uncertainty-aware safety filters to detect out-of-distribution visual states and trigger verified safe fallback maneuvers before catastrophic physical failure.Source: E Limitations and Future Work · p. 42

Related papers

Loading 2606.23686v2…