PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Link Safety Filtering for VLA Policies Around Moving Hazards

Yatharth Agarwal, Vijay Raghunathan

arXiv:2609.40007v1cs.ROcs.CV

TL;DR

Task completion alone does not establish that a VLA policy will leave unrelated moving hazards untouched. This paper adds a training-free, multi-link runtime shield that tracks a fitted hazard ellipsoid; across six simulated hazard-motion conditions, it reduces collisions and raises safe-success.

  • Problem

    A pretrained VLA policy can complete a task while disturbing an unrelated moving hazard, so task success alone does not establish safe behavior.

    Source: §I · p. 1

  • Method

    A training-free runtime shield filters a pretrained policy’s commands with five arm ellipsoids and transports a reset-fitted hazard ellipsoid using sparse optical flow.

    Source: §I · p. 1

  • Results

    Across six hazard-motion conditions, collisions fall from 65.62% to 27.27% and safe-success rises from 29.35% to 50.43%.

    Source: §I · p. 2

  • Takeaways & Limitations

    Runtime protection for pretrained VLA policies can extend beyond the end effector, stay current around moving hazards, and run on shared edge hardware without retraining the policy.

    Source: §VII · p. 8

Abstract

from arXiv · show

A vision-language-action (VLA) policy can finish a manipulation task while knocking over objects unrelated to it, so task success alone does not show that the policy is safe to deploy in clutter. We study how to keep a pretrained VLA policy clear of such hazards at run time without retraining it, which requires guarding more of the arm than the end effector, following the hazard as it moves, and sharing onboard compute with the policy. Our training-free shield covers the gripper, wrist, and forearm with five ellipsoids and filters every commanded motion through one barrier program against a keep-out ellipsoid fitted from RGB-D perception at reset. Sparse optical flow then carries that ellipsoid's center along with the hazard, with no repeated detection or refitting. Over six simulated hazard-motion conditions, the shield lowers collision from $65.62\%$ to $27.27\%$ and raises safe-success, task completion without collision, from $29.35\%$ to $50.43\%$. Ablations show that guarding the arm links protects beyond end-effector shielding, and that tracking recovers most of the protection lost when the hazard estimate is frozen at reset. On heterogeneous edge hardware, the five-ellipsoid barrier runs on the CPU in $2.2$~ms at the 99th percentile, and trimming the vision--language prefix and taking fewer flow-matching steps shortens each $\pi_{0.5}$ policy call on the integrated GPU from $343$ to $177.3$~ms. On a physical SO-101 arm across four tasks, the arm touched the hazard in 3 of 16 shielded episodes versus 11 of 16 unshielded ones. Project page: https://yathag.github.io/multilink-safety-filter/

I. INTRODUCTION

The introduction frames runtime safety as a gap for pretrained VLA policies: task success does not ensure unrelated hazards remain untouched. It presents a multi-link, motion-aware shield and evaluates its protection and edge deployment.

I. INTRODUCTION

  • A policy can complete its task while disturbing an unrelated object, so task completion alone does not establish safety around moving hazards.The introduction reports that π0.5 completes 62.93% of episodes but disturbs the hazard in 65.62%.Source: §I · p. 1
  • The training-free shield guards five arm ellipsoids and tracks the moving hazard while leaving the pretrained policy’s weights unchanged.Perception and tracking are mapped alongside policy inference on heterogeneous edge hardware.Source: §I · pp. 1–2
  • The evaluation covers static- and moving-hazard benchmarks, isolates coverage and tracking effects, and measures the shield’s edge-compute cost.The same hardware mapping is used on a physical SO-101 arm around a tracked moving hazard.Source: §I · p. 2
  • Across six hazard-motion conditions, collisions fall from 65.62% to 27.27% and safe-success rises from 29.35% to 50.43%.The shield covers the gripper, wrist, and forearm and transports the hazard estimate without refitting.Source: §I · p. 2

II. RELATED WORK

Related work spans learned safety behavior, runtime constraints for VLA policies, geometric protection, hazard tracking, and efficient edge inference. The paper distinguishes its approach by combining multi-link ellipsoids, motion updates, and heterogeneous deployment.

II. RELATED WORK

  • Existing VLA safety filters constrain end effectors, while manipulator filters guard links with spheres and Any-Body Guard uses a quasi-static scene representation.The paper instead guards five ellipsoids spanning the gripper, wrist, and forearm against a motion-updated hazard ellipsoid.Source: §II · p. 2
  • Per-step segmentation and promptable video segmentation update hazard locations but run a neural network at every update; sparse optical flow transports a reset-fitted ellipsoid instead.The ellipsoid retains its reset shape.Source: §II · p. 2
  • Fine-tuning can encode avoidance in policy weights, whereas runtime shielding imposes a newly designated geometric constraint without updating those weights.The stated scope is task-irrelevant hazards that should remain untouched; tasks requiring contact need constraints beyond a strict keep-out region.Source: §II · p. 2
  • Policy efficiency alone does not determine shielded-system performance because hazard estimation and filtering share compute, and policy approximations change the commands being filtered.The paper maps policy inference, perception, tracking, and filtering to heterogeneous hardware at their respective cadences.Source: §II · p. 2

III. SYSTEM DESIGN

The system places a training-free runtime filter between a pretrained VLA policy and the robot. A barrier program adjusts commands to protect modeled robot volumes against a shared hazard ellipsoid.

III. SYSTEM DESIGN

  • The shield filters each command from a policy action chunk immediately before execution, while perception and control exchange only the hazard ellipsoid.The policy is evaluated using π0.5, but the filter acts on commanded end-effector motion rather than depending on policy architecture.Source: §III · p. 2, Figure 2 · p. 3

A. The Barrier Program

  • The CBF-QP jointly adjusts commanded motion to satisfy keep-out constraints while minimizing deviation from the policy’s nominal command.The hazard is represented by an ellipsoid with a center, rotation, and positive semi-axes.Source: A. The Barrier Program · pp. 2–3
  • Each guarded ellipsoid contributes a tangent-plane barrier, with a nonnegative signed distance certifying separation from the hazard ellipsoid.The guarded ellipsoid is rigidly attached to its link, and the barrier is evaluated at the measured state.Source: A. The Barrier Program · p. 3
  • The link-motion map predicts each guarded volume’s motion from the commanded end-effector twist, but the simulated operational-space controller can execute different null-space joint motion.The controller executes the minimizing command and integrates the auxiliary barrier-vector rates over the control step.Source: A. The Barrier Program · p. 3

B. Multi-Link Coverage

The shield extends modeled protection beyond the end effector and updates a reset-fitted hazard ellipsoid as the object moves. Its coverage and tracking remain bounded by the modeled links, shape assumptions, and discrete updates.

B. Multi-Link Coverage

  • Five guarded ellipsoids cover the gripper, wrist, and forearm, including three overlapping ellipsoids along the forearm; the elbow and upper arm are not guarded.All five constraints share one hazard ellipsoid in a single barrier program.Source: B. Multi-Link Coverage · p. 3

C. Hazard Initialization and Tracking

  • At reset, a VLM names the hazard, an open-vocabulary detector grounds it, and registered depth supports a minimum-volume enclosing ellipsoid fit.The selected detection fixes object identity for the episode.Source: C. Hazard Initialization and Tracking · p. 3
  • For moving hazards, optical flow transports only the reset ellipsoid’s center, retaining its orientation and semi-axes.Anchored displacement cancels a fixed projection offset, but assumes no large hazard rotation or change in visible shape.Source: C. Hazard Initialization and Tracking · p. 3
  • Lucas–Kanade flow tracks persistent, depth-consistent image points, with displacement and appearance gates rejecting unsupported updates and prompting a bounded template search.The tracker measures every five control steps and holds the center between measurements.Source: C. Hazard Initialization and Tracking · p. 4
  • The deployed loop certifies barrier conditions only against the latest estimated pose, not continuous-time invariance for a moving hazard.If the program becomes infeasible, the implementation executes and logs the nominal action rather than invoking a certified fallback.Source: C. Hazard Initialization and Tracking · p. 4

D. Edge-Aware Policy Execution

The policy reuses cached vision–language prefix representations within each call, so prefix size affects both encoding and repeated attention. The deployment trims that prefix and shortens the integration loop without changing policy weights.

D. Edge-Aware Policy Execution

  • Prefix keys and values are computed once per call and reused across flow-matching steps, but the cache resets when a new observation changes the prefix.Reducing prefix slots cuts both one-time encoding and repeated prefix-action attention; fewer integration steps reduce only the repeated path.Source: D. Edge-Aware Policy Execution · p. 4
  • The deployed configuration removes a masked third camera slot and reduces the padded text budget from 200 to 64 tokens, shrinking the prefix from 968 to 576 positions.The longest benchmark instruction uses 21 tokens.Source: D. Edge-Aware Policy Execution · p. 4
  • Integration steps fall from ten to five, and the resulting approximation is evaluated with the shield active without modifying policy weights.Source: D. Edge-Aware Policy Execution · p. 4
  • The barrier and optical-flow update run on the CPU at their respective cadences, while dense policy inference runs once per ten-action chunk on an iGPU or NPU.The filter workloads are shorter than accelerator-dispatch overhead, motivating CPU execution.Source: D. Edge-Aware Policy Execution · p. 4

IV. BENCHMARKS AND METRICS

The evaluation uses static and commanded-motion hazard benchmarks with shared simulator, robot, policy interface, and metrics. It specifies hazard conditions, collision scoring, implementation details, and paired statistical comparisons.

IV. BENCHMARKS AND METRICS

  • The two simulated benchmarks share a simulator, robot, policy interface, and metrics, and estimate hazard geometry from rendered RGB and registered simulator depth rather than exact pose or shape.Source: §IV · p. 4
  • SafeLIBERO covers 32 static scenes and 1,600 episodes per configuration, comparing the shield with the unshielded policy, AEGIS, OSCBF, and end-effector-only and multi-link variants.Source: §IV · p. 4
  • The moving benchmark runs 624 episodes per condition across six hazard-motion conditions, including a stationary hazard, orbit, shuttle, and three one-way escapes at 50 mm/s.Each episode records true and commanded hazard pose and per-camera visibility at every step.Source: §IV · p. 4
  • Collision flags hazard displacement exceeding 1 mm at a control step, while safe-success means task completion without collision; robot–hazard contacts are also recorded separately.Moving-scene displacement is measured from the latest commanded position, excluding scripted motion.Source: §IV · p. 5
  • The simulated robot is a Franka Panda, and the barrier is solved with OSQP through CVXPY after converting normalized policy actions to end-effector twists.AEGIS and OSCBF use their published robot envelopes and gains, which differ from the shield's envelope.Source: §IV · p. 5
  • Paired tests use scene-level averages and permutation tests with Holm–Bonferroni correction; equivalence requires a 90% confidence interval within the prespecified ±3 percentage-point margin.Source: §IV · p. 5

V. EVALUATION

The evaluation first compares safety and task performance across static and moving hazards, then examines edge execution and policy optimizations with the shield active.

V. EVALUATION

  • The evaluation is organized around safety and task performance across static and moving hazards, followed by edge execution and policy optimizations with the shield active.Source: §V · p. 5
  • Table II reports edge-platform execution, including policy-call stages across CPU, iGPU, and NPU and the shield components.Source: Table II · p. 6
  • AEGIS and OSCBF serve as external reference points while retaining their own robot envelopes and gains, making the comparison one of practical system standing.Source: §V · p. 6

A. Safety and Task Performance

Across moving-hazard conditions, the shield reduces collision and raises safe-success, while task success shows neither significant change nor equivalence. The section also reports comparisons, edge costs, policy-configuration effects, and physical-arm episode counts.

A. Safety and Task Performance

  • Collision falls from 65.62% to 27.27% and safe-success rises from 29.35% to 50.43%; task success is 62.93% unshielded and 61.38% shielded, with neither significant change nor equivalence.Source: A. Safety and Task Performance · p. 6
  • Across all six conditions, OSCBF exceeds our shield’s collision rate by 8.2 to 36.7 percentage points; this gap does not establish that torque-level filtering protects less than action-level filtering because robot envelopes differ.Source: A. Safety and Task Performance · p. 6
  • Among episodes with recorded robot–hazard contact, collision is 53.85% unshielded, 31.86% for OSCBF, 23.61% for AEGIS, and 13.17% for the shield.Source: A. Safety and Task Performance · p. 6
  • On static SafeLIBERO, the shield collides in 27.00% of episodes versus 30.75% for AEGIS; the multi-link variants have the highest observed safe-success.With a frozen hazard estimate, the multi-link configuration collides in 25.19% versus 27.00% for the complete system.Source: A. Safety and Task Performance · p. 6

B. Edge Deployment

  • A barrier step costs 0.0189 J and an optical-flow update 0.132 J, compared with 7.86 J per policy call; detection and hazard naming add 19.7 J and 51.1 J once per episode.Source: B. Edge Deployment · p. 6
  • Trimming the prefix from 968 to 576 positions and halving integration steps reduces the iGPU policy call from 343 ms to 177.3 ms without changing policy weights.Source: B. Edge Deployment · p. 6
  • With the reduced policy configuration, static-hazard collision rises by 1.75 percentage points while task success and safe-success remain equivalent; on the 300 mm escape, task success and safe-success fall significantly.The reduced configuration lowers single-client call time by 32.7% at p50; shielded collision remains 22.92% versus 22.44%.Source: B. Edge Deployment · p. 6
  • On the physical SO-101 arm, the hazard was touched in 3 of 16 shielded episodes versus 11 of 16 unshielded episodes, while placements completed in 11 versus 13 episodes.The hardware episode budget was small, and the hazard was hand-carried; these counts were reported without paired tests.Source: B. Edge Deployment · pp. 6–7

VI. ABLATIONS AND ANALYSIS

The ablations and analysis isolate protection and task costs across coverage and motion, varying one policy factor while holding perception and the controller fixed.

  • The controlled comparisons hold perception and the controller fixed while varying one policy factor.This isolates the effects being tested along the coverage and motion axes.Source: §VI · p. 7
  • The studies examine multi-link coverage and its protection and task costs.Source: §VI · p. 7
  • The studies also examine perception, tracking, placement accuracy, and update frequency.These factors concern losses and requirements for maintaining protection.Source: §VI · p. 7

A. Multi-Link Protection and Its Task Cost

Multi-link coverage reduces collisions and their severity, while hazard estimation and tracking determine how much protection persists when the hazard moves.

A. Multi-Link Protection and Its Task Cost

  • Static collision falls from 31.31% to 25.19% with wrist and forearm coverage added to the end effector.Task success and safe-success do not change significantly; safe failures rise to 21.94% of episodes under multi-link coverage.Source: A. Multi-Link Protection and Its Task… · p. 7
  • Multi-link coverage reduces median hazard displacement from 147.7 mm in unshielded episodes to 32.5 mm for surviving disturbances.Its advantage over end-effector coverage is concentrated below 50 mm displacement.Source: A. Multi-Link Protection and Its Task… · p. 7

B. Hazard Estimation and Tracking Failures

  • On the 300 mm escape condition, collision falls from 30.29% with a frozen estimate to 22.44% with optical flow, versus 18.43% with exact geometry and true motion.Optical flow improves every measured rate over the frozen estimate, retaining most protection available with exact geometry and motion.Source: B. Hazard Estimation and Tracking… · p. 7
  • VLM-generated hazard naming keeps aggregate collision at 22.44%, but a plausible wrong phrase assigns a different object in six of 624 episodes.Aggregate rates conceal this grounding failure.Source: B. Hazard Estimation and Tracking… · p. 8
  • A detector call costs about 148× the latency and 149× the energy of an optical-flow update, while fixed-cadence refitting collides as often as a frozen estimate.Optical flow transports the identity established at reset without re-detection or association.Source: B. Hazard Estimation and Tracking… · p. 8

C. Placement Accuracy and Update Frequency

Placement sweeps show that transporting the hazard estimate matters as it moves, while cadence tests identify update rates that preserve protection; the conclusion highlights deployment costs and a remaining policy–shield interaction challenge.

C. Placement Accuracy and Update Frequency

  • Moving the hazard away from a frozen estimate retains 82.1% of collision-reduction benefit at 25 mm and 27.6% at 50 mm, versus 92.0% and 72.6% when only the ellipsoid is translated.The gap widens from 9.9 to 45.0 percentage points; median fitted-center error is 20.8 mm.Source: C. Placement Accuracy and Update… · p. 8
  • Collision stays between 22.76% and 24.84% at strides 1–20, but rises to 28.37% at stride 40 and 29.97% at stride 80.Strides 1 and 2 are equivalent to stride 5; longer strides approach the frozen-estimate rate of 30.29%.Source: C. Placement Accuracy and Update… · p. 8
  • Per-step flow measurement is equivalent to stride 5 on the stationary condition, so stride 5 is deployed at 3.1 ms per measurement on the edge CPU.Source: C. Placement Accuracy and Update… · p. 8

VII. CONCLUSION AND FUTURE DIRECTIONS

  • Multi-link shielding and sparse optical flow provide onboard protection while keeping the barrier and tracker workloads at millisecond scale.Policy inference remains the dominant deployed-system cost.Source: §VII · p. 8
  • For unbatched π0.5, lower accelerator power does not mean lower energy per call, making prefix-processing latency the more relevant optimization target.Prefix trimming and fewer flow-matching steps reduce policy latency.Source: §VII · p. 8
  • The fixed policy does not observe shield corrections within its current action chunk, which can limit recovery when avoidance requires replanning.The paper identifies feedback of shielded state and corrective actions, with safe fallbacks, as a path toward higher completion and stronger protection.Source: §VII · p. 8

Related papers

Loading 2609.40007v1…