PaperlayerView exact revision on arXiv

Source-linked AI summary

VLSA: Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer

Songqiao Hu, Zeyi Liu, Shuang Liu, Jun Cen, Zihan Meng, Shihefeng Wang, Xiang Li, Xiao He

arXiv:2512.11891v2cs.ROeess.SY

TL;DR

VLA models need explicit safety mechanisms for collision-prone, unstructured environments. AEGIS adds a plug-and-play CBF-based safety layer to existing VLA models, and experiments show substantially better obstacle avoidance and task success, while limitations remain in perception, unconstrained links, and safety-induced distribution shift.

  • Problem

    Existing VLA models provide broad manipulation generalization but often lack explicit safety constraints, while retraining-based approaches are costly and difficult to apply to existing models.

  • Method

    AEGIS adds a plug-and-play safety constraint layer that uses vision-language obstacle identification, spatial information, and CBF-based control to modify unsafe VLA actions without retraining.

  • Results

    AEGIS achieves more than 50% improvement in obstacle avoidance and nearly 10% higher task success than state-of-the-art baselines on SafeLIBERO.

  • Takeaways & Limitations

    The results support using a plug-and-play safety layer to add theoretically guaranteed safety while retaining existing VLA instruction-following capabilities.

  • Takeaways & Limitations

    Residual collisions arise from perception and spatial-grounding errors, unconstrained kinematic links can collide, and safety enforcement can shift the robot into out-of-distribution states that impair task completion.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in generalizing across diverse robotic manipulation tasks. However, deploying these models in unstructured environments remains challenging due to the critical need for simultaneous task compliance and safety assurance, particularly in preventing potential collisions during physical interactions. In this work, we introduce a Vision-Language-Safe Action (VLSA) architecture, named AEGIS, which contains a plug-and-play safety constraint (SC) layer formulated via control barrier functions. AEGIS integrates directly with existing VLA models to improve safety with theoretical guarantees, while maintaining their original instruction-following performance. To evaluate the efficacy of our architecture, we construct a comprehensive safety-critical benchmark SafeLIBERO, spanning distinct manipulation scenarios characterized by varying degrees of spatial complexity and obstacle intervention. Extensive experiments demonstrate the superiority of our method over state-of-the-art baselines. Notably, AEGIS achieves over 50% improvement in obstacle avoidance rate while substantially increasing the task success rate by nearly 10%. All benchmark datasets, code, and supplementary materials are publicly available at https://vlsa-aegis.github.io/.

I. INTRODUCTION

Existing VLA models generalize across manipulation tasks but often lack explicit safety guarantees, motivating AEGIS, a plug-and-play safety constraint layer that preserves instruction following while enforcing collision avoidance.

  • I. INTRODUCTION: Safety is essential for deployment because collisions in unstructured environments can cause hardware damage, human injury, or property loss.
  • I. INTRODUCTION: Retraining-based safety methods impose high computational and data-collection costs and are difficult to apply directly to existing pretrained VLA models.
  • I. INTRODUCTION: AEGIS translates instruction-relevant visual risks into physical-space avoidance requirements using open-set detection, depth information, and a CBF-based quadratic-programming solver.
  • I. INTRODUCTION: AEGIS integrates control barrier functions into VLA models through a plug-and-play safety constraint layer without retraining.The layer dynamically adjusts nominal actions only when potential safety violations are detected.
  • I. INTRODUCTION: On SafeLIBERO, AEGIS improves obstacle avoidance by more than 50% and task success by nearly 10% over state-of-the-art baselines.The benchmark contains 32 scenarios and 1,600 episodes with varied obstacle configurations.

II. RELATED WORK

Prior VLA safety approaches rely on costly retraining or soft objectives, while classical planners can discard model intent; this work motivates training-free, explicit constraints and formulates CBF-based filtering of VLA actions.

  • A. Vision-Language-Action Models: Existing VLA models can generalize well but may produce unsafe trajectories in out-of-distribution environments because safety is often overlooked.
  • A. Vision-Language-Action Models: Retraining-based methods are costly, difficult to apply to pretrained models, and generally treat safety as a soft reward objective rather than a hard inference-time constraint.
  • B. Safety-Critical Control: Classical motion planners may override semantic actions and lack rigorous theoretical safety guarantees, motivating CBF-based alternatives.
  • B. Safety-Critical Control: Integrating CBFs with VLA models must bridge raw visual inputs to precise geometry and distinguish task-relevant obstacles from other objects.
  • III. PROBLEM FORMULATION: The safety filter models the end-effector and relevant obstacles as ellipsoids and computes a safe control input that minimally deviates from the nominal VLA action.
  • III. PROBLEM FORMULATION: AEGIS constructs a CBF from language instructions and visual observations and solves the safety-control problem in real time.

A. Main Architecture

VLSA adds a safety-constraint layer after the base VLA output, combining vision-language obstacle assessment with control that modifies only potentially unsafe actions. The assessment identifies a task-relevant obstacle and localizes it in 3D using visual, language, detection, depth, and point-cloud processing.

  • A. Main Architecture: The safety-constraint layer receives visual features, linguistic features, and nominal VLA actions, modifying actions only when safety risks are detected.When no risk is identified, its output remains identical to the original VLA output.
  • A. Main Architecture: AEGIS combines vision-language safety assessment with safety-guaranteed control to convert task-relevant obstacle information into corrected actions.The assessment semantically identifies and spatially localizes an obstacle before control models collision geometry.
  • A. Main Architecture: The VLM identifies one critical non-robot obstacle from the instruction and agent-view RGB image, outputting a uniquely identifiable name containing its color and type.That textual name is used as the grounding query for spatial localization.
  • A. Main Architecture: GroundingDINO retains the highest-confidence 2D bounding box for the identified obstacle before projecting its region into 3D space.The retained box provides the basis for subsequent spatial reconstruction.
  • A. Main Architecture: Agent-view and back-view depth point clouds are fused in a unified world coordinate system, then filtered and clustered to extract the main obstacle body.The preprocessing applies workspace bounds, removes the farthest 20% of points, and isolates the most populous cluster.

C. Action-driven Safety-guaranteed Control Module

The action-driven control module represents the obstacle and end-effector as minimum-volume enclosing ellipsoids and constructs a distance-based control barrier function. A quadratic program adjusts nominal VLA actions while the barrier condition provides a collision-avoidance guarantee under stated representation and initialization assumptions.

  • C. Action-driven Safety-guaranteed Control Module: AEGIS models the processed obstacle point cloud with a minimum-volume enclosing ellipsoid and the end-effector with a fixed-size ellipsoid.The obstacle ellipsoid parameters are optimized to enclose the point cloud while minimizing ellipsoid volume.
  • C. Action-driven Safety-guaranteed Control Module: The augmented state combines end-effector position and orientation with a virtual sphere state that maps to a point on the end-effector ellipsoid.This construction supports the tangent-plane distance formulation between the two ellipsoids.
  • C. Action-driven Safety-guaranteed Control Module: Maximizing the signed distance over the virtual state yields the shortest distance between ellipsoids, which serves as the control barrier function for action correction.The nominal VLA actions are adjusted into safe commands through a quadratic program.
  • C. Action-driven Safety-guaranteed Control Module: Under accurate safety assessment, precise point-cloud filtering, and complete obstacle representation, AEGIS guarantees that the end-effector will not collide with the obstacle.The theorem assumes both generated ellipsoids strictly enclose the obstacle and end-effector, respectively.
  • C. Action-driven Safety-guaranteed Control Module: Starting from a safe configuration, the QP enforces the barrier differential constraint, preserving the safe set through forward invariance.The proof invokes Nagumo’s Theorem to establish h(x(t)) ≥ 0 for all subsequent times.

A. Setup

The evaluation uses SafeLIBERO, which introduces obstacle-intervention scenarios across 16 tasks from four LIBERO suites, and compares AEGIS with both its base policy and a cross-architectural VLA baseline. Performance is measured by collision avoidance, task success, and execution efficiency in translational-only and full action-space settings.

  • A. Setup: SafeLIBERO covers 16 tasks across four LIBERO suites, with Level I obstacles near targets and Level II obstacles directly blocking movement paths.The benchmark creates two safety-critical scenarios per task using diverse everyday objects as obstacles.
  • A. Setup: The study compares the base π0.5-LIBERO policy with AEGIS and OpenVLA-OFT under identical evaluation conditions.The comparisons measure direct plug-and-play safety improvements and cross-architectural performance, respectively.
  • A. Setup: Collision Avoidance Rate measures strictly collision-free episodes, while Task Success Rate measures completed tasks within the time limit.Collisions do not trigger early termination for Task Success Rate.
  • A. Setup: Execution Time Steps is the average episode length including timeouts, with lower values indicating greater efficiency and fewer futile interactions.The metric complements safety and success measurements with an efficiency measure.
  • A. Setup: The experiments use a Franka Emika Panda at 20 Hz in translational-only and full action-space settings.The translational setting focuses on positional collision avoidance, while the full setting includes translational and rotational dynamics.

1) Performance Analysis:

AEGIS substantially improves both collision avoidance and task execution on SafeLIBERO, outperforming the base π0.5 and OpenVLA-OFT baselines. Qualitative results show that it avoids collisions that cause baseline task failures.

  • SafeLIBERO is the benchmark used for the quantitative comparison of safety and task execution.
  • 77.9% / 68.9% CAR makes AEGIS roughly fourfold safer than π0.5 and OpenVLA-OFT across translational / full action spaces.The corresponding CAR values are 18.7% / 17.1% for π0.5 and 15.1% / 5.7% for OpenVLA-OFT.
  • AEGIS dynamically generates collision-free actions and succeeds on the bowl-placement task, whereas π0.5 collides and fails.
  • 68.1% / 67.5% TSR is the highest reported result for AEGIS, exceeding π0.5 at 50.9% / 57.8%.The text links preserved workspace integrity to preventing disruptive collisions in cluttered environments.

2) Behavioral and Constraint Analysis:

AEGIS maintains safety during obstacle approach while modifying nominal actions only when needed. Its constraint layer remains lightweight enough for real-time control.

  • Qualitative Visualization: AEGIS dynamically generates collision-free actions across obstacle geometries while maintaining goal progression.The QP minimizes deviation from the original VLA action subject to strict safety bounds.
  • Constraint Evolution: h(x) decreases as the robot approaches obstacles but remains strictly positive, supporting enforcement of the forward-invariance condition.The reported condition is ḣ ≥−α(h) at every time step.
  • Time Complexity: The SC layer acts as a lightweight piecewise intervention that modifies unsafe nominal actions through a convex QP with one linear constraint.
  • Real-world Experiments: Real-world experiments use a 7-DoF Panda arm, stereo cameras, 15 Hz VLA inference, and a 1 kHz low-level controller.The experiments cover two tasks evaluated across two obstacle levels.
  • Time Complexity: 0.356 ms per step is the SC layer's computation time, approximately 1.86% of total cycle latency.The reported setup is an RTX 4090 GPU, with the control loop maintaining 20 Hz execution.

C. Discussion and Limitations

Real-world tests show AEGIS can preserve collision avoidance with a zero-shot π0.5-DROID policy, but residual failures arise from perception limits and safety-induced distribution shift. The current formulation also leaves kinematic links unconstrained.

  • Limitations: Residual collisions primarily result from obstacle misidentification, inaccurate spatial grounding, aggressive point-cloud filtering, and unconstrained kinematic links.The current formulation guarantees collision avoidance only for the end-effector, not the robot’s other links.
  • Limitations: Safety enforcement can drive the robot into out-of-distribution states, such as rarely observed higher altitudes, where the policy may fail to recover toward the goal.The authors identify broader training coverage of safety-induced out-of-distribution regions as future work.
  • Evaluation scope: AEGIS’s 32-scenario SafeLIBERO evaluation is complemented by real-robot experiments using a zero-shot π0.5-DROID policy across two tasks and two safety levels.The physical platform matches the DROID dataset hardware configuration, so no fine-tuning is used.
  • Real-world experiments: The proposed method successfully identifies obstacles and modifies nominal π0.5-DROID actions for safety, matching simulation behavior in physical-robot experiments.The baseline policy directly collides with predefined obstacles, whereas AEGIS uses external-camera point clouds to construct CBFs.
Loading 2512.11891v2…