PaperlayerView exact revision on arXiv

Source-linked AI summary

WBAG: A Whole-Body and Attached-Geometry Safety Framework for Vision-Language-Action Manipulation

Samuel Zhen, Siwon Jo, Yanze Zhang, Wenhao Luo

arXiv:2610.01083v1cs.RO

TL;DR

Inference-time VLA safety methods often use end-effector-centered geometry without explicitly modeling the articulated robot or attached objects. WBAG adapts protected geometry after grasping and applies geometric constraints to the VLA’s native action. On SafeLIBERO, it achieves the highest aggregate Scene Safety and Safe Success among evaluated configurations, at 97.38% and 59.38%.

  • Problem

    Existing inference-time VLA safety frameworks typically use end-effector-centered geometry that does not explicitly represent the full articulated robot or attached-object geometry.

    Source: §I · p. 1

  • Method

    WBAG reconstructs scene and target geometry, adapts its protected set after grasping, and applies differentiable geometric CBF constraints to the VLA’s native operational-space action.

    Source: §I · p. 2

  • Results

    WBAG achieves 97.38% Scene Safety and 59.38% Safe Success on SafeLIBERO, the highest aggregate results among evaluated configurations.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The results support explicitly modeling the full robot–attached-object system rather than relying on end-effector-centered safety geometry for runtime VLA manipulation.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-language-action (VLA) policies have demonstrated impressive capabilities in generalizable robotic manipulation, but their deployment in the real world remains challenging due to potential collisions involving different parts of the robot, manipulated objects, and the surrounding environment. Existing inference-time VLA safety frameworks typically rely on simplified end-effector-centered representations that do not explicitly model the full articulated robot and attached-object geometry. In this paper, we present WBAG, a safety framework that models the robot's whole-body and grasp-dependent attached geometry. WBAG constructs a grasp-conditioned safe set that adapts the protected geometry as objects are grasped, then converts this evolving geometry into differentiable CBF constraints that minimally modify the VLA's native six-dimensional operational-space action for collision avoidance across robot, scene, and attached geometry. On the SafeLIBERO benchmark, a variant of LIBERO augmented with obstacles for safety evaluation, WBAG achieves the best overall safety and safe task success among the evaluated methods under a scene-level safety evaluator that monitors all eligible non-task objects, reaching 97.38\% aggregate Scene Safety and 59.38\% Safe Success.

I. INTRODUCTION

Existing inference-time VLA safety methods commonly center collision avoidance on the end effector, leaving articulated links and grasped objects outside the explicit geometric model. WBAG adapts protected geometry after grasping and applies geometric safety constraints to the VLA’s native action; on SafeLIBERO it leads the evaluated configurations in aggregate Scene Safety and Safe Success.

I. INTRODUCTION

  • End-effector-centered representations do not explicitly model the full robot or attached objects, so links or grasped geometry may collide while the end effector remains separated.The relevant collision geometry spans the articulated robot and changes when an object is grasped.Source: §I · p. 1
  • WBAG reaches 97.38% Scene Safety and 59.38% Safe Success on SafeLIBERO, the highest aggregate results among evaluated configurations.The evaluation uses a scene-level evaluator that monitors all eligible non-task scene objects.Source: §I · p. 2
  • WBAG’s grasp-conditioned safe set protects the articulated robot before grasping and adds the rigidly attached object after grasping.Differentiable geometric CBF constraints minimally modify the VLA’s native six-dimensional operational-space action.Source: §I · p. 2
  • WBAG reconstructs scene and target geometry from visual observations and enforces constraints without retraining the VLA or modifying its operational-space controller.The framework connects perception-derived, grasp-conditioned geometry to the policy’s native action interface.Source: §I · p. 2
  • The controlled SafeLIBERO evaluation isolates the effects of scene coverage, whole-body protection, and attached-geometry modeling on safe VLA manipulation.The supplied contribution passage continues across both excerpts.Source: §I · p. 2

II. PRELIMINARIES

The preliminaries establish control barrier functions as safe-set constraints and Bernstein-polynomial signed distance fields as differentiable geometric representations. Together, these provide clearance information that can enter optimization-based control.

A. Control Barrier Functions

  • A control barrier function constrains inputs so a controller satisfying its condition keeps the prescribed safe set forward invariant.For geometric collision avoidance, the safety function can be defined from robot–obstacle clearance.Source: A. Control Barrier Functions · p. 2

B. Bernstein Polynomial Signed Distance Fields

  • A BP-SDF approximates signed distance to a body’s surface, with positive values outside, zero on the surface, and negative values inside..Source: B. Bernstein Polynomial Signed… · p. 2
  • Composing the smooth BP-SDF representation with robot forward kinematics makes link clearance differentiable with respect to configuration.Its spatial distance gradients support optimization-based control and geometric CBF construction.Source: B. Bernstein Polynomial Signed… · p. 2
  • Figure 2 depicts the pipeline from RGB-D scene reconstruction and offline robot BP-SDF fitting to grasp-conditioned geometry and geometric CBF-QP constraints.The protected set expands to include attached geometry after grasp confirmation.Source: Figure 2 · p. 3

C. Problem Formulation

The problem formulation treats a frozen VLA’s nominal action as the command to be safety-adjusted. It defines which robot and scene geometries are protected and requires a minimally changed motion command while preserving the gripper command.

C. Problem Formulation

  • A frozen VLA maps an observation and language instruction to a nominal seven-dimensional action containing translation, axis–angle rotation, and a gripper command.The VLA is instantiated in the experiments with π0.5.Source: C. Problem Formulation · p. 3
  • The protected set changes from the articulated robot before grasp confirmation to the robot–object system afterward.The grasp mode σ distinguishes pre- and post-grasp states.Source: C. Problem Formulation · p. 3
  • Scene obstacles exclude objects requiring intentional task contact, and the attached target is not constrained against itself..Source: C. Problem Formulation · p. 3
  • The safety filter computes a minimally deviating six-dimensional motion command that maintains collision-free operation between every protected body and scene obstacle.It modifies only the operational-space motion command and preserves the VLA’s gripper command.Source: C. Problem Formulation · p. 3

III. GRASP-CONDITIONED GEOMETRIC BARRIER

The grasp-conditioned geometric barrier encodes pairwise clearance between protected bodies and scene obstacles. Its ideal closest-point definition relies on geometric regularity, while the paper notes that online use requires a finite-contact approximation.

III. GRASP-CONDITIONED GEOMETRIC BARRIER

  • The barrier section derives clearance using ideal closest-point evaluation, with a finite-contact approximation introduced later for online use..Source: §III · p. 3

A. Pairwise BP-SDF Clearance

  • For each protected-body–obstacle pair, the clearance barrier uses the closest point on an offset obstacle surface and a protected-body margin.A nonnegative barrier enforces the prescribed geometric separation.Source: A. Pairwise BP-SDF Clearance · p. 3
  • The pairwise clearance gradient relies on convex relevant level sets, a smoothly varying closest point, and nonzero BP-SDF gradients at the boundaries..Source: A. Pairwise BP-SDF Clearance · p. 3

B. Grasp-Conditioned Composite Barrier

The grasp-conditioned composite barrier combines clearance constraints across protected bodies, then maps the resulting condition to the VLA’s operational-space command. Its forward-invariance guarantee holds within a fixed grasp mode, not across the discrete attachment transition.

B. Grasp-Conditioned Composite Barrier

  • A soft minimum aggregates pairwise body–obstacle barriers into one constraint per obstacle, and nonnegative composite clearance implies nonnegative clearance for every protected-body pair.Larger β more tightly approximates the minimum.Source: B. Grasp-Conditioned Composite Barrier · p. 4

C. Action-to-Motion Mapping and Fixed-Mode Safety

  • The local action-to-motion mapping relates the six-dimensional operational-space command to joint motion so the barrier condition can constrain the command.The fixed-mode proposition states that satisfying the constraint for every obstacle, from an initial state in Cσ, makes Cσ forward invariant under its assumptions.Source: C. Action-to-Motion Mapping and… · p. 4
  • At attachment, the protected set expands to include the grasped object, adding clearance constraints that may exclude the robot’s current configuration from the new safe set.The fixed-mode guarantee does not cover the discrete transition itself; it applies after attachment if the configuration lies in C1.Source: C. Action-to-Motion Mapping and… · p. 4

IV. WBAG SAFE CONTROL FRAMEWORK

WBAG implements grasp-conditioned geometric barriers in a perception-to-control pipeline. It approximates ideal closest-point searches online and minimally adjusts the VLA command to satisfy geometric CBF constraints.

IV. WBAG SAFE CONTROL FRAMEWORK

  • The Section III barrier assumes exact continuous closest-point evaluation between each protected body and obstacle.This is the ideal formulation; the deployed pipeline approximates that search online.Source: §IV · p. 4
  • WBAG reconstructs scene and target geometry from multi-view RGB-D, updates the protected set upon grasp confirmation, and propagates attached-object geometry with the hand.Robot-link BP-SDFs are precomputed; ideal closest-point barriers are approximated using finite local contact search and mapped to the VLA command through a local action-to-motion model.Source: §IV · p. 4
  • A quadratic program minimally modifies the nominal VLA action while satisfying the grasp-conditioned geometric CBF constraints.The mapped constraints apply to the VLA’s six-dimensional operational-space command.Source: §IV · p. 4

A. Perception-derived Scene and Target Geometry

WBAG derives scene and target geometry from RGB-D observations, then adapts protected geometry after grasp confirmation. It refines the hand-relative attachment pose before propagating the object with the gripper.

A. Perception-derived Scene and Target Geometry

  • Multi-view RGB-D observations are segmented and fused into world-frame point clouds, with task roles identifying the target, goal, and support objects and determining obstacle exclusions.Observed points are converted to convex hulls and fitted with convexity-promoting BP-SDF constraints; retained non-task objects form the obstacle set, while the target is retained for post-grasp protection.Source: A. Perception-derived Scene and Target… · p. 4

B. Grasp-Conditioned Attached Geometry

  • After grasp confirmation, target RGB-D points are registered to the initial model with ICP to refine the hand-relative attachment pose before rigid propagation with the gripper.The initial target pose comes from perception; the pose may change as the object settles in the gripper, so WBAG refines the transform during a short post-grasp window and then holds it fixed.Source: B. Grasp-Conditioned Attached Geometry · p. 5, Figure 3 · p. 5
  • The propagated pose determines the attached object’s world-frame BP-SDF, which becomes a protected body after grasping and enters the pairwise and composite barriers.Its coordinates are expressed in the attached-object frame induced by the hand pose and hand-relative attachment transform.Source: B. Grasp-Conditioned Attached Geometry · p. 5

C. Online Geometric Barrier Approximation

WBAG approximates geometric barriers with locally searched contact candidates and maps the resulting constraint to the VLA’s operational-space action. The local action model treats the gripper as rigidly attached to the hand.

C. Online Geometric Barrier Approximation

  • WBAG approximates the continuous closest-point search by locally searching the obstacle level set for candidate points with small protected-body clearance, then aggregates them with a soft minimum.The candidate barrier uses the protected-body margin and the same soft-min sharpness parameter as the composite barrier.Source: C. Online Geometric Barrier… · p. 5

D. Action-to-Motion Mapping

  • A damped-pseudoinverse Jacobian-based mapping converts the VLA’s operational-space increment into an approximate local joint velocity for applying the deployed barrier.The action scales and effective action horizon parameterize the local action-to-motion map.Source: D. Action-to-Motion Mapping · p. 5
  • The local action-to-motion model treats gripper geometry as rigidly attached to the hand and omits finger motion induced by the scalar gripper command.Source: D. Action-to-Motion Mapping · p. 5

E. Grasp-Conditioned Geometric CBF-QP

WBAG uses a minimum-intervention QP to select a feasible command close to the nominal VLA action. Its deployed filter uses finite contact search and permits slack, unlike the exact closest-point, zero-slack formulation covered by Proposition 1.

E. Grasp-Conditioned Geometric CBF-QP

  • The minimum-intervention QP selects the feasible command closest to the nominal VLA action.The gripper command remains unchanged.Source: E. Grasp-Conditioned Geometric CBF-QP · pp. 5–6
  • W weights deviation from the nominal command, λδ penalizes CBF relaxation, and umin and umax impose native action bounds.δ is a shared slack variable; when δ = 0, the constraint has the CBF form applied to the finite contact barrier.Source: E. Grasp-Conditioned Geometric CBF-QP · p. 6
  • The deployed filter uses finite contact search and permits slack, so Proposition 1's guarantee for exact closest-point evaluation with zero slack does not apply to deployment.The figure compares designated-obstacle EEF-MVEE, scene-wide EEF-MVEE, EEF-only with attached geometry, whole-body without attached geometry, and full WBAG.Source: E. Grasp-Conditioned Geometric CBF-QP · p. 6, Figure 4 · p. 6

V. RESULTS

On SafeLIBERO, full WBAG achieves 97.38% Scene Safety and 59.38% Safe Success; broader scene coverage alone improves safety but reduces Safe Success. The evaluation compares configurations across 16 tasks using a scene-level displacement-based safety measure.

A. Evaluation Setup

  • The benchmark covers 16 LIBERO tasks—four each from Goal, Long, Object, and Spatial—and uses 50 randomized episodes per task and safety level.Two obstacle placements per task and level yield 1,600 episodes total.Source: A. Evaluation Setup · p. 6
  • Scene Safety monitors eligible scene bodies not required for task contact; an episode is unsafe if any moves more than 0.01 m from its settled initial position.Safe Success requires task completion with Scene Safety; Unsafe Success is task completion without Scene Safety.Source: A. Evaluation Setup · p. 6
  • All configurations use the same frozen π0.5 checkpoint and identical episode initializations, with simulator-provided object identities used to isolate safety representation from semantic identification errors.The evaluated configurations range from policy-only and EEF-MVEE baselines to scene-wide EEF protection and WBAG geometry variants.Source: A. Evaluation Setup · pp. 6–7

B. Benchmark Performance

  • 97.38% Scene Safety, 59.38% Safe Success, and 0.68% Unsafe Success are reported for full WBAG on SafeLIBERO.Scene Safety rises from 70.87% for AEGIS baseline through 72.94% for SAM3 EEF MVEE and 82.75% for SAM3 Scene EEF MVEE; scene-wide coverage reduces Safe Success from 55.56% to 46.25%.Source: Table II · p. 7, B. Benchmark Performance · p. 7

A. Evaluation Setup

  • WBAG protects modeled moving robot links, including the gripper and fingers, and adds reconstructed target geometry after grasp confirmation.Controller and evaluation settings are held fixed across configurations.Source: A. Evaluation Setup · p. 7

B. Benchmark Performance

  • WBAG has the highest Scene Safety and lowest Unsafe Success in all four suites, but Safe Success leadership varies by suite.WBAG leads Safe Success on Long and Object, while EEF MVEE configurations lead on Goal and Spatial.Source: B. Benchmark Performance · p. 7

C. Ablation and Qualitative Analysis

WBAG reports strong SafeLIBERO safety and safe-success results, while ablations support complementary benefits from whole-body and attached-object protection. Runtime analysis identifies contact search as the main filter cost, and the conclusion distinguishes the ideal-barrier guarantee from the deployed approximation.

C. Ablation and Qualitative Analysis

D. Runtime Analysis

VI. CONCLUSION

  • WBAG enforces BP-SDF clearance CBF constraints on the native operational-space action without retraining the VLA or modifying its controller.Source: §VI · p. 8
  • 97.38% Scene Safety and 59.38% Safe Success are reported for WBAG on SafeLIBERO.Source: §VI · p. 8
  • The stated fixed-mode forward-invariance result applies to the ideal barrier, while deployment uses finite contact search, local action mapping, and QP slack.Source: §VI · p. 8
Loading 2610.01083v1…