PaperlayerView exact revision on arXiv

Source-linked AI summary

ShieldVLA: Feasibility-Aware Safety Alignment for Vision-Language-Action Models

Manan Tayal, Akshay Nambi

arXiv:2609.13231v1cs.ROcs.AI

TL;DR

VLA fine-tuning methods provide limited safety guarantees, while visual safety learning lacks dense per-step labels. ShieldVLA uses an HJ reachability critic to gate reward and recovery optimization and rubric-based VLM scores for supervision; across five benchmarks, it lowers safety cost and improves task success over SafeVLA.

  • Problem

    Existing VLA fine-tuning offers limited safety guarantees, and dense per-step safety labels are generally unavailable in visual domains.

    Source: §1 · pp. 1–2

  • Method

    ShieldVLA learns a model-free HJ reachability critic that gates reward optimization in feasible regions and recovery near unsafe states, using rubric-based VLM safety scores as structured supervision.

    Source: §1 · p. 2

  • Results

    ShieldVLA reduces cumulative safety cost by 57% and improves task success rate by +0.13 over SafeVLA across five navigation and manipulation benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework separates reward maximization from safety intervention and improves safety-performance trade-offs across multiple VLA backbones and embodiments.

    Source: §1 · p. 2

    The method assumes access to a per-step scalar safety margin for each visual observation, whose construction is addressed separately.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Vision-Language-Action (VLA) models demonstrate strong generalization in robotic manipulation and navigation, but existing fine-tuning methods provide limited safety guarantees. Current approaches primarily rely on Lagrangian optimization that enforces safety through soft penalties on expected cumulative cost, often resulting in residual constraint violations or overly conservative behavior. Moreover, learning safety in visual domains is challenging due to the absence of dense per-step safety annotations. We propose ShieldVLA, a safety-aligned fine-tuning framework for VLA models based on Hamilton-Jacobi (HJ) reachability. ShieldVLA learns a model-free approximation of the HJ reachability value function directly from visual observations to estimate the safe operating region. The learned safety critic gates policy optimization by separating reward maximization within feasible regions from recovery near unsafe states, avoiding persistent reward-cost trade-offs. To enable scalable supervision in visual environments, we introduce rubric-based VLM safety scores that convert semantic safety feedback into structured critic targets without requiring manual cost labels. Across five navigation and manipulation benchmarks spanning multiple VLA backbones, ShieldVLA reduces cumulative safety cost by 57% on average and improves task success rate by +0.13 over SafeVLA.

1 Introduction

ShieldVLA addresses safety and supervision gaps in VLA fine-tuning by using a model-free HJ reachability critic to gate policy optimization and rubric-based VLM scores to supervise it. Across five navigation and manipulation environments, it improves safety-performance trade-offs over SafeVLA.

1 Introduction

  • Lagrangian methods softly penalize expected cumulative cost, but can leave critical violations and bias policies toward conservative behavior, especially in long-horizon tasks.Visual safety supervision is also difficult because dense per-step labels are generally unavailable and free-form VLM judgments are noisy and inconsistent.Source: §1 · pp. 1–2
  • HJ reachability represents the safe set with a value function capturing worst-case future constraint violations, shifting safety from a global trade-off to state-dependent feasibility.The value function can be estimated model-free through temporal-difference learning, avoiding known-dynamics and grid-discretization requirements.Source: §1 · p. 2
  • The critic gates optimization: reward is optimized inside the estimated safe set, while recovery is used near unsafe regions, avoiding a persistent reward-cost trade-off.The framework uses model-free HJ reachability to estimate the feasible safe operating region.Source: §1 · p. 2
  • ShieldVLA reduces cumulative safety cost by 57% and improves task success rate by +0.13 over SafeVLA across five navigation and manipulation environments.The benchmarks span multiple VLA backbones and embodiments.Source: §1 · p. 2
  • Rubric-based VLM safety scores turn semantic feedback into structured critic targets without requiring manually annotated per-step safety labels.The rubric decomposes per-frame safety into interpretable axes such as collision risk, hazardous proximity, and manoeuvring room.Source: §1 · p. 2

2 Related Work

Prior safe-RL and VLA approaches rely on soft constraints, known dynamics, or runtime-only intervention. ShieldVLA instead uses HJ reachability to gate fine-tuning and rubric-based supervision to obtain visual cost targets.

2 Related Work

  • Lagrangian constrained optimization underlies SafeVLA, but its soft expected-cost constraints bring oscillatory dual-variable dynamics and hyperparameter sensitivity.Other approaches may retain soft constraints, require known dynamics, or act only at test time.Source: §2 · p. 2
  • Vision-based formal safety methods use runtime filters on fixed pretrained policies, whereas ShieldVLA gates training to optimize task reward alongside safety.The related work distinguishes safety filtering from fine-tuning foundation models for reward.Source: §2 · p. 3
  • ShieldVLA combines model-free HJ reachability for training-time objective gating with calibrated rubric-based VLM supervision for per-step costs without ground-truth labels.Structured weighted criteria are described as more calibrated than raw VLM scores [Kwon et al., 2023, Yu et al., 2023, Ma et al., 2024, Yang et al., 2022].Source: §2 · p. 3

3 Background

The background contrasts soft CMDP cost constraints with HJ reachability, which evaluates safety through a worst-case margin value. It also frames the practical challenge of learning margin labels from visual observations and using the critic in policy optimization.

3 Background

  • The standard CMDP objective maximizes reward subject to a soft constraint on expected cumulative violation, with cost defined as the non-negative magnitude of safety-margin violations.The signed margin ℓ(s) is safe when ℓ(s) ≥ 0 and indicates violation when ℓ(s) < 0.Source: §3 · p. 3
  • HJ reachability defines a safety value function tracking the discounted worst-case margin along a trajectory; a nonnegative value certifies that some policy can maintain safety indefinitely.A negative value means every policy eventually violates the constraint.Source: §3 · p. 3
  • The HJ safety Bellman operator can be estimated model-free by temporal-difference learning on transitions from a replay buffer.The operator is a γ-contraction in the infinity norm.Source: §3 · p. 3
  • The problem requires per-state visual safety margins and a way to use the trained critic in policy optimization without restoring Lagrangian coupling.Hand-crafted margins for high-dimensional visual observations are impractical, and the learned critic inherits estimation error from finite data and the VLM cost signal.Source: §3 · p. 3

4 Method

ShieldVLA decomposes safe VLA fine-tuning into visual safety-margin estimation, online Hamilton–Jacobi critic learning, and critic-gated policy updates. The gate separates task-reward optimization in feasible states from safety recovery in infeasible states.

4 Method

  • ShieldVLA separates fine-tuning into rubric-based safety-margin estimation, online HJ critic learning, and critic-gated policy updates.Source: Figure 1 · p. 4

4.1 Safety-Gated Optimization with an Online Safety Critic

  • The online critic learns the HJ safety value model-free from replay, with concurrent training helping it remain calibrated as the policy changes.Replay training also exposes the critic to rare unsafe transitions that on-policy rollouts may under-sample.Source: §4.1 · pp. 4–5
  • A thresholded feasibility gate preserves PPO reward updates in feasible states and switches infeasible states to deterministic safety-improvement gradients.Source: §4.1 · p. 5

4.2 VLM-Based Estimation of the Safety Margin

  • A frozen VLM scores frames with structured rubrics; weak episode labels calibrate raw severity into signed per-frame margins for HJ updates.Source: §4.2 · pp. 5–6
  • Refinement-through-Differentiation proposes rubrics from safe–unsafe disagreement pairs, surfacing analogous corrective criteria on TurtleBot-Nav and Safety-CHORES Nav.Source: §4.2 · p. 6

5 Experiments

Experiments compare ShieldVLA with unconstrained, SafeVLA, and oracle HJ baselines across five environments, then assess seed stability, component ablations, and robustness to visual shifts.

5.1 Experimental Setup

  • The five-task suite spans Dubins-VL, TurtleBot-Nav, Safety-CHORES Nav and Fetch, and Franka-Reach, with varied observation and action setups and pretrained backbones.The suite includes a Dubins car, TurtleBot 4 navigation, Stretch-based navigation and fetch, and Franka FR3 tabletop reaching.Source: §5.1 · pp. 6–7

5.2 Baselines and Metrics

  • Baselines are unconstrained VLA, SafeVLA, and Dubins-VL oracle HJ; SR counts safe goal-reaching episodes, while CSC is mean episode safety cost.A method is Pareto-superior when it has both higher SR and lower CSC.Source: §5.2 · p. 7

5.3 Main Results

  • ShieldVLA pairs the lowest collision rates with the highest success rates in nearly all benchmarks; on Dubins-VL, its success-rate exception is against HJ (GT cost), whose zero cost it matches.The HJ baseline uses privileged ground-truth cost and serves as an idealized upper bound.Source: §5.3 · pp. 7–8, Table 2 · p. 8
  • Across three training seeds, standard deviations were small relative to the SafeVLA point-estimate gap in every environment, indicating seed-robust safety-task trade-offs.Each seed used 200 evaluation episodes.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 Ablation Studies

  • Binary costs yield TurtleBot-Nav SR 0.31 vs 0.54 and Franka-Reach SR 0.25 vs 0.45; without off-collision gradient information, the gate fires too early.This ablation keeps the HJ critic and gate identical while replacing calibrated VLM rubric margins with a binary collision indicator.Source: Table 4 · p. 8, §5.4 · p. 8
  • On TurtleBot-Nav, a Lagrangian penalty trails feasibility gating on both axes when the HJ critic and rubric supervision are held fixed.The ablation changes the policy update, replacing the gated objective with a penalty using the same critic.Source: Table 5 · p. 8, §5.4 · p. 8

5.5 Out-of-Distribution Robustness

  • The OOD test freezes the policy and critic while applying held-out color, lighting, or combined renderer changes at every frame across evaluation episodes.The perturbations are absent from training rollouts; results cover TurtleBot-Nav, Safety-CHORES Nav, and Safety-CHORES Fetch.Source: §5.5 · p. 9
  • ShieldVLA has the lowest CSC in every tested environment-condition cell; on Safety-CHORES Nav, Avg. ∆CSC is −0.49 versus SafeVLA’s +1.19.Safety-CHORES Fetch retains positive residual degradation, which the authors relate to its tighter manipulation clearances.Source: Table 6 · p. 9, §5.5 · p. 9

6 Conclusion

ShieldVLA combines an online model-free HJ safety critic, feasibility-gated policy updates, and rubric-based VLM supervision. Across five benchmarks it improves safety and task success over SafeVLA, with limitations in scorer capacity and near-boundary exploration.

6 Conclusion

  • ShieldVLA uses an online model-free HJ safety critic, a feasibility gate that switches between reward and recovery gradients, and a rubric pipeline converting episode outcomes into calibrated per-step margins.Offline refinement addresses rubric tail failure modes.Source: §6 · p. 9
  • Across five benchmarks, ShieldVLA reduces cumulative safety cost by 57% on average and improves task success rate by +0.13 over SafeVLA, while degrading gracefully under test-time visual perturbations.Source: §6 · p. 9
  • Results are bounded by sub-8B scorer mode collapse on tail frames and the need for sufficient near-boundary exploration, which is difficult when unsafe events are rare.The authors identify physical-robot validation and scorer scaling as immediate next steps.Source: §6 · p. 9

A Proof of Safety Bellman Convergence

The safety Bellman operator is shown to contract at rate γ, yielding a unique fixed point to which repeated application converges.

A Proof of Safety Bellman Convergence

B Limitations

The paper’s safety guarantees are empirical and depend on rubric-derived supervision, visited states, and simulator evaluation.

Limitations

  • The classical HJ contraction does not transfer cleanly to function approximation with finite data, so safety claims are empirical rather than formal.Source: B Limitations · p. 13
  • The rubric-derived safe set can differ from the one induced by simulator dynamics, and the gap is bounded only on Dubins-VL by the HJ (GT cost) row in Table 2.Source: B Limitations · p. 13
  • Sub-8B VLM scorers exhibit mode collapse on tail frames, imposing a practical lower bound on supervisor capacity.Source: B Limitations · p. 13
  • The critic learns boundaries only in rollout-visited regions, making threshold δ effectively hand-tuned when unsafe events are rare.Source: B Limitations · p. 13
  • Evaluation is simulator-only; real-robot validation requires rubric recalibration for domain-shifted observations and threshold retuning.Source: B Limitations · p. 13

C Training Algorithm

Algorithm 1 trains a safety critic off-policy from VLM-scored rollouts, then uses the critic to gate policy optimization.

Training Algorithm

D.1 Dubins-VL

The appendix specifies four simulation settings and a rubric-based visual safety-supervision pipeline; its refinement experiments report improved pre-contact discrimination with a larger scorer and added criteria.

D.1 Dubins-VL

  • Dubins-VL uses a top-down RGB-observed Dubins car, with obstacle clearance as its safety margin and a VLM rubric supplying ShieldVLA supervision.Increasing gate threshold δ enforces a wider safety margin but is expected to reduce task progress; Figure 3 depicts feasible and infeasible regions across headings.Source: §D.1 · p. 14, Table 7 · p. 14, Figure 3 · p. 15

D.2 TurtleBot-Nav

  • TurtleBot-Nav evaluates colour-conditioned navigation in randomized MuJoCo scenes using egocentric RGB, continuous velocity actions, and time-in-obstacle cost.The setting uses OmniVLA and an independent safety critic with a frozen visual encoder.Source: §D.2 · p. 15

D.3 Safety-CHORES Nav and Safety-CHORES Fetch

  • Safety-CHORES Nav and Fetch use the same Stretch RE1 simulator, dual RGB cameras, and 20-action set, differing in whether the goal is navigation or object fetching.The safety critic uses the same SPOC-based architecture as SafeVLA for comparison.Source: §D.3 · pp. 15–16

D.4 Franka-Reach

  • Franka-Reach evaluates a 7-DoF arm reaching around a randomly placed cylinder, with simulator contact events reserved for evaluation rather than training supervision.The stochastic OpenVLA-OFT policy uses twin critics and a pessimistic Bellman target.Source: §D.4 · p. 16

E VLM Cost Pipeline

  • The VLM cost pipeline comprises rubric design, offline RTD refinement, validation on two platforms, and the experimental scorer prompts.Source: E VLM Cost Pipeline · p. 16

E.1 Rubric design rationale

  • Rubrics ask about one hazard axis at a time and assign anchored continuous severity, addressing opaque aggregation, scale drift, and difficult-to-localize scoring failures.The paper reports a 2B scorer’s near-constant per-frame scores under the multi-rubric prompt, which the 8B scorer breaks open.Source: §E.1 · p. 17

E.2 Refinement-through-Differentiation (RTD)

  • RTD calibrates rubric scores against episode outcomes, finds unsafe-safe disagreement pairs, and asks a proposer VLM to suggest criteria that are deduplicated and vote-ranked.The refinement loop runs offline before the main training loop.Source: §E.2 · pp. 17–18
  • 0.608 pre-contact AUROC was achieved on TurtleBot-Nav after rubric refinement with an 8B scorer, versus 0.500 initially, a +0.108 end-to-end change.On Safety-CHORES Nav, refinement proposed obstacle-related criteria similar to those found for TurtleBot-Nav.Source: §E.2 · p. 17

E.3 Empirical validation

The validation tests the VLM cost pipeline on TurtleBot-Nav and Safety-CHORES Nav, using different forms of safety supervision. Across the reported tests, the 8B scorer improves safety-cost results over 2B, while rubric refinement adds useful criteria.

E.3 Empirical validation

  • The two case studies use different platforms and supervision: TurtleBot-Nav has per-frame ground-truth distance, while Safety-CHORES Nav has episode-level outcome bits only.The shared pipeline is reported to produce shielding filters on both platforms without oracle distance.Source: §E.3 · pp. 18–19
  • The paper reports that increasing scorer capacity from 2B to 8B and refining the rubric from 5R to 7R composed multiplicatively, producing a +0.108 AUROC end-to-end gain.The 2B scorer emitted identical per-rubric scores on a given frame; the 8B scorer broke this collapse, allowing RTD criteria to contribute once it could evaluate them independently.Source: §E.3 · p. 18, Table 8 · p. 18
  • RTD proposed obstacle-overlap and obstacle-edge-proximity criteria for TurtleBot-Nav, addressing a camera-inside-obstacle sign-inversion failure.The 72B proposer generated these criteria from disagreement pairs; the section reports corresponding criteria on Safety-CHORES Nav as well.Source: §E.3 · pp. 18–19
  • On TurtleBot-Nav ID episodes, the 8B scorer matches the 2B scorer on SR while incurring lower CSC at no extra training cost.The evaluation used 50 in-distribution episodes with δ = 0.Source: §E.3 · p. 18, Table 9 · p. 19
  • Across every OOD family, the 8B scorer dominates the 2B scorer on both CSC and SR.The families aggregate color variants, lighting variants, or all five held-out variants.Source: Table 10 · p. 19, §E.3 · p. 19
  • On Safety-CHORES Nav, the 8B scorer’s Spearman correlation with episode safety cost was +0.272, versus +0.094 for 2B.The passage reports per-frame AUROC in the same regime as TurtleBot-Nav and says the pipeline still produced a useful end-to-end shield.Source: §E.3 · p. 19, Table 11 · p. 19

F Compute Budget

The VLM cost pipeline is run offline or cached during data collection, keeping it out of per-iteration training cost. Rubric refinement is a one-time preprocessing step, and the full pipeline takes under an hour per environment.

F Compute Budget

  • Under 1 hour per environment is required for the full VLM pipeline on 8 GPUs, while outcome-bit calibration takes less than 1 minute.VLM cost evaluation is performed offline before training or cached during data collection, so it adds no per-iteration training cost; rubric refinement is one-time preprocessing.Source: F Compute Budget · p. 23

G Hyperparameters

Related papers

Loading 2609.13231v1…