PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-Stem: A Lightweight Safety Mechanism for Vision-Language-Action Models via Reinforcement Learning

Bojun Zou, Wenhao Yu, Jiaohao Li, Jiaxuan Lin, Guanting Ye, Haohan Geng, Weihao Yu, Yanyong Zhang, Jianmin Ji

arXiv:2610.37378v1cs.RO

TL;DR

VLA policies often lack explicit responses to physical safety demands, while avoidance-data collection can be costly and policy fine-tuning may require additional training. VLA-Stem uses a frozen VLA’s motion reference with trajectory-level guidance and a separately trained Stem policy. It achieves the highest collision-free task success across the evaluated constrained settings, with reported StackCube results of 0.5141 and 0.5419 SUCC in static and dynamic settings.

  • Problem

    VLA policies often lack explicit regulation for physical safety demands, while collecting avoidance data is costly and adapting a pretrained VLA requires additional policy training.

    Source: §I · p. 1

  • Method

    VLA-Stem conditions a separately trained Stem policy on frozen VLA motion references and physical context, using trajectory-level guidance and teacher–student distillation.

    Source: §I · p. 2, C. Teacher–Student Distillation · p. 4

  • Results

    VLA-Stem achieves the highest collision-free task success across evaluated constrained settings; on StackCube, SUCC is 0.5141 static and 0.5419 dynamic.

    Source: 1) Main Results · p. 6

  • Takeaways & Limitations

    The evaluation supports VLA-Stem’s potential for safer execution while retaining base-policy performance in obstacle-free settings and direct deployment of the simulation-trained policy on a real robot.

    Source: §V · p. 8

    The current evaluation focuses on structured manipulation tasks and collision avoidance using low-dimensional obstacle representations.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models integrate high-level reasoning with low-level action execution for end-to-end robotic control. However, most VLA models target static manipulation in unconstrained environments, overlooking spatial constraints and dynamic obstacles. Fine-tuning VLAs on obstacle-avoidance data is a straightforward solution, but collecting such data is costly, especially for dynamic scenarios, and may degrade their original task capabilities. To address these limitations, we propose VLA-Stem, a lightweight, reinforcement-learning-based safety mechanism for VLA systems. Without fine-tuning the underlying VLA policy, VLA-Stem converts its action chunks into motion references and uses trajectory-level guidance to preserve their spatial intent while allowing flexible motion and timing adjustments. Conditioned on these references, robot proprioception, and local obstacle information, the Stem policy adaptively modifies execution at inference time, preserving the original task intent while enabling responsive adaptation to spatial constraints and dynamic obstacles. For robust and sample-efficient training, we combine teacher-student distillation with curriculum learning. We evaluate VLA-Stem on reference following, static spatial constraints, and dynamic obstacle avoidance. Across six constrained simulation settings, VLA-Stem achieves 52.8% average collision-free task success versus 23.7% for the strongest safety baseline, while preserving base-policy performance in obstacle-free settings; real-world experiments further show consistent collision reductions under static and dynamic obstacles. These results demonstrate that VLA-Stem couples high-level VLA task intent with rapid, reflex-like responses to immediate physical risks, providing a brainstem-inspired mechanism for robust robot execution.

I. INTRODUCTION

The introduction argues that VLA policies need rapid safety responses that remain coordinated with task intent, and presents VLA-Stem as an inference-time mechanism that adapts execution without fine-tuning the base policy. It situates this approach among VLA, safety, and test-time action-adjustment methods.

A. Vision-Language-Action Models

  • VLA models combine high-level capabilities with low-level action execution, but most lack explicit lower-level regulation responsive to physical safety demands.The introduction uses rapid sensorimotor responses, including brainstem functions, as its analogy for the missing regulation.Source: §I · p. 1

B. Safety for Manipulators

  • Existing safety approaches include avoidance-data training, conventional safety modules, trajectory planners, and standalone learned generators, each with stated data, tuning, or task-coordination limitations.The introduction notes that avoidance data can be costly to collect, while safety modules may fail to coordinate corrections with VLA task behavior.Source: §I · pp. 1–2, B. Safety for Manipulators · p. 2

I. INTRODUCTION

  • VLA-Stem couples task-directed motion references with runtime safety adaptation, using trajectory-level guidance and teacher–student training without fine-tuning the underlying policy.Its stated aim is to preserve task behavior while allowing local motion and timing adjustments and improving recovery from avoidance-induced deviations.Source: §I · p. 2

C. Test-Time VLA

  • Test-time VLA methods adjust actions during deployment, but candidate-sequence search can reduce real-time efficiency; VLA-Stem instead uses a compact policy without search or VLA fine-tuning.The cited examples are VLA-Reasoner and V-GPS.Source: C. Test-Time VLA · p. 2

III. METHOD

The method pairs a frozen VLA with a separately trained Stem policy: the VLA supplies a short-horizon motion reference, and the Stem uses that reference and current physical context to produce executable actions. The framework also includes trajectory-level guidance, distillation, and staged RL training.

III. METHOD

  • VLA-Stem uses a frozen VLA motion reference and current physical context to condition the Stem policy, which generates executable actions.The method overview adds trajectory-level guidance, teacher–student distillation, and curriculum-based RL training.Source: §III · pp. 2–3, A. VLA-Conditioned Execution · p. 3

A. VLA-Conditioned Execution

  • At each execution interval, the frozen VLA produces an action chunk that is converted into future end-effector poses relative to the interval-start pose.The Stem uses this motion reference rather than directly executing the VLA chunk, and the VLA is queried again after the interval.Source: A. VLA-Conditioned Execution · p. 3
  • During each control step, the Stem conditions on the active reference, robot proprioception, and local obstacle information to generate an executable action.The obstacle representation uses obstacle centers and bounding-box dimensions; the next VLA chunk follows the execution interval.Source: A. VLA-Conditioned Execution · p. 3

B. Trajectory-Level Guidance for Spatial Intent Preservation

Step-wise corrections can misalign the robot state with later time-indexed VLA actions and accumulate deviation from the base policy’s expected state distribution. Trajectory-level guidance instead tracks spatial proximity to the reference while allowing forward progression without strict temporal correspondence.

B. Trajectory-Level Guidance for Spatial Intent Preservation

  • Step-wise safety corrections can misalign execution with later VLA actions, producing discontinuous corrections and accumulated drift from the base policy’s expected state distribution.The issue also arises when step-wise action or waypoint alignment guides RL exploration.Source: B. Trajectory-Level Guidance for… · p. 3
  • Trajectory-level guidance measures deviation from local reference segments, preserving spatial consistency while relaxing strict time-step correspondence.The method projects the end-effector position onto trajectory segments and interpolates orientation along each segment.Source: B. Trajectory-Level Guidance for… · p. 3
  • A non-decreasing matched-segment index and forward corridor keep matching near the current reference segment during waiting or local detours.The corridor look-ahead range is K, and the segment index resets at the start of each execution interval.Source: B. Trajectory-Level Guidance for… · p. 3
  • The matched-segment deviation supplies a dense trajectory-following reward, with pose sensitivity set by k_pose and an early-stage progress bonus based on normalized index advancement.The progress bonus uses (m_t − m_{t−1})/(h − 1).Source: B. Trajectory-Level Guidance for… · p. 3

C. Teacher–Student Distillation

Teacher–student distillation addresses recovery after large deviations by using privileged task information during teacher training, then deploying a student without that information. RL training proceeds through reference following, static constraints, and dynamic obstacles of increasing difficulty.

C. Teacher–Student Distillation

  • Large deviations can alter later observations and VLA references, making recovery difficult when guidance is task-agnostic and deployable observations lack explicit task-state information.The teacher receives privileged task information during training, such as target-object positions, while the student uses deployable observations only.Source: C. Teacher–Student Distillation · pp. 3–4, Figure 2 · p. 4
  • The student combines the PPO objective with a KL distillation loss to imitate the privileged teacher, then serves as the deployable policy without privileged task inputs.The distillation weight is λ = 0.1; teacher and student share the reward design but differ in observation spaces.Source: C. Teacher–Student Distillation · p. 4

D. RL Optimization and Curriculum Learning

  • Curriculum training progresses from obstacle-free reference following to static constraints and then moving obstacles, where execution timing may include waiting.Reward terms cover task completion, collision avoidance, and consistency with the VLA motion reference; collision or spatial-constraint violations receive −1.Source: D. RL Optimization and Curriculum… · p. 4

IV. EXPERIMENT

The experiments evaluate a frozen-VLA execution mechanism across three manipulation tasks and obstacle-free, static, and dynamic scenarios. VLA-Stem preserves observed obstacle-free performance and achieves the highest collision-free task success across the evaluated constrained settings.

A. Experimental Setup

  • The simulation uses PickCube, StackCube, and Object Stowing, with each corresponding base VLA frozen during Stem training and shared checkpoints for the compared safety mechanisms.Each base VLA was fine-tuned on 100 demonstrations for 30,000 optimization steps.Source: A. Experimental Setup · pp. 4–5
  • Static scenarios place obstacles near target regions or along transfer routes, while dynamic scenarios send moving obstacles across the robot’s path during approach or transfer.Randomized crossing times require adapting both motion and execution timing.Source: A. Experimental Setup · p. 5

1) Main Results:

  • In normal settings, VLA-Stem’s success rates are 0.7325, 0.7061, and 0.7521 on PickCube, Object Stowing, and StackCube, versus 0.7166, 0.6138, and 0.7450 for the corresponding π0.5 policies.The reported results show no reduction in observed task success in these three obstacle-free settings.Source: 1) Main Results · pp. 5–6
  • On PickCube, π0.5 SUCC falls from 0.7166 normally to 0.0325 static and 0.0131 dynamic, with COLL of 0.8781 and 0.9328 in those constrained settings.VLA-finetune improves some constrained settings but remains inconsistent across tasks; the authors report that added obstacle-response behavior can interfere with original manipulation without consistently establishing safety adaptation.Source: 1) Main Results · p. 6
  • VLSA-ARM reaches COLL of 0.2253 on PickCube-Static, below VLA-Stem’s 0.2316, but its SUCC is 0.1516 versus VLA-Stem’s 0.5219.The authors attribute collision-free VLSA failures to corrections disrupting fine-grained manipulation or moving execution away from the nominal motion, complicating recovery.Source: 1) Main Results · p. 6
  • VLA-Stem achieves the highest collision-free task success across evaluated constrained settings; on StackCube, SUCC is 0.5141 static and 0.5419 dynamic, with COLL of 0.3353 and 0.0784, respectively.The simulation results assess task preservation in normal settings and adaptation to static and dynamic constraints.Source: 1) Main Results · pp. 5–6, Table I · p. 5

2) Analysis and Ablations:

The ablations test whether VLA references, trajectory-level guidance, and teacher–student distillation support task execution and recovery. Controlled reference perturbations further show that Stem execution follows spatial changes while subsequent VLA references can redirect execution toward task completion.

Reference-conditioned execution

  • Removing VLA references and reference-based guidance rewards yields zero SUCC for student policies in all three scenarios, despite low collision rates in obstacle settings.The framework uses references to convey task-directed motion and dense guidance to direct exploration under a sparse task-reward design.Source: 2) Analysis and Ablations · pp. 6–7

Reference intervention

  • Across left, right, and up reference perturbations, 88 of 90 rollouts (97.8%) were closer to the modified than original VLA reference; normalized mean distances were 0.230 versus 0.894.Perturbations used displacements of 15, 17.5, and 20 cm. Distances were normalized by the complete arc length of the corresponding original VLA reference chunk.Source: 2) Analysis and Ablations · p. 7
  • After local perturbation, subsequent VLA references generated from the updated robot state redirect execution toward task completion, alongside 60.0%–70.0% success under altered references.This links responsive Stem execution to continued VLA task progression.Source: 2) Analysis and Ablations · p. 7

Effectiveness of trajectory-level guidance

  • Trajectory-level guidance raises average success from 30% to 63% versus step-wise action alignment across all three scenarios.The authors attribute the stronger safety–task performance to allowing flexible local motion and timing while maintaining consistency with task-directed motion.Source: 2) Analysis and Ablations · p. 7, Table III · p. 7

Necessity of teacher–student distillation

  • VLA-Stem increases average success from 14% to 63% and reduces average collision rate from 37% to 15% in constrained scenarios versus the variant without distillation.The authors link the improvement to privileged task observations providing informative recovery-training signals transferred to the deployable student through distillation.Source: 2) Analysis and Ablations · pp. 7–8, Table IV · p. 7

C. Real-World Evaluation

In real-world object stowing and stacking, the simulation-trained Stem policy reduces collisions and increases task success under static and dynamic obstacles while retaining observed base-policy success in normal settings. The evaluation is limited to structured manipulation and low-dimensional obstacle representations; longer-horizon tasks, contact- or force-related safety, and robust visual perception are future work.

C. Real-World Evaluation

  • In normal settings, VLA-Stem retains the observed base-policy success rates on object stowing and stacking.The real-world evaluation used 15 trials per task and scenario on a UR7e robot.Source: C. Real-World Evaluation · p. 8
  • Under static and dynamic obstacles across both tasks, collision rates fall from 73–100% to 20–40% and task success rises from 0–20% to 47–60%.The Stem policy was trained only in simulation and deployed directly on the real robot without real-world training.Source: C. Real-World Evaluation · p. 8

V. CONCLUSION

  • The evaluation focuses on structured manipulation and collision avoidance with low-dimensional obstacle representations; future work targets longer-horizon tasks, contact- or force-related safety, and robust visual perception.The conclusion identifies noisy obstacle observations as a reliability concern for visual perception.Source: §V · p. 8

APPENDIX

The frozen VLA supplies a short-horizon trajectory reference, while the Stem policy uses it with robot state and obstacle information to generate executable actions. The deployable policy uses separate encoders for these low-dimensional inputs and an MLP actor–critic.

1) VLA-Conditioned Execution:

  • At each execution interval, the VLA action chunk is converted into future end-effector poses relative to the interval’s starting pose, serving as a reference rather than being executed directly.The Stem policy then generates executable actions at each control step using proprioception and local obstacle information alongside the reference.Source: 1) VLA-Conditioned Execution · p. 10
  • The deployable policy uses 16-dimensional proprioception, an 8 × 8 trajectory feature sequence, and a 33-dimensional obstacle vector for up to three obstacles, without raw visual observations.Its output is a seven-dimensional normalized action; after each interval, the frozen VLA generates a new reference from the updated observation.Source: 1) VLA-Conditioned Execution · p. 10

2) Policy Architecture:

  • The Stem policy uses an MLP actor–critic with three independent feature encoders; the encoded features form a 576-dimensional representation shared by the actor and critic.The reference encoder has three 256-unit hidden layers; proprioception and obstacle encoders have dimensions (256, 256) and (64, 64), respectively, while each head has hidden dimensions (1024, 1024, 512).Source: 2) Policy Architecture · p. 10
  • The deployed student has approximately 4.56 million trainable parameters and uses only 16-dimensional proprioception, whereas the privileged teacher receives task-related information in a 23-dimensional state input.The actor parameterizes a seven-dimensional diagonal Gaussian action distribution; deterministic inference uses its tanh-transformed mean, and the critic outputs a scalar state-value estimate.Source: 2) Policy Architecture · p. 10

1) Reward Components:

The reward combines trajectory guidance, sparse task rewards, and a collision penalty, with reward weights varying across three curriculum stages. Teacher and student policies are optimized with PPO using GPU-parallel simulation and Adam.

1) Reward Components:

  • Trajectory-level guidance is separated into trajectory-following and progress terms, alongside sparse grasping and terminal-success rewards and a collision penalty.Grasping and success each receive a reward of 1 when achieved, and each event is rewarded only once per episode; the progress reward encourages advancement along the reference during early curriculum stages.Source: 1) Reward Components · p. 10

2) PPO Optimization and Training Budget:

  • Teacher and student policies are optimized using PPO with GPU-parallel simulation and Adam, and reported environment transitions include both teacher and student training.The student PPO uses generalized advantage estimation with λGAE = 0.9, a clipping coefficient of 0.2, a value-loss coefficient of 0.5, an entropy coefficient of 0, gradients clipped to a maximum norm of 0.5, and Adam with ϵ = 10−5.Source: 2) PPO Optimization and Training Budget · p. 10

3) Teacher–Student Distillation:

Teacher–student distillation trains a deployable student to imitate a privileged teacher while using PPO. Reference-segment pose deviation measures the end-effector’s position and orientation relative to the trajectory segments.

3) Teacher–Student Distillation:

  • The student is jointly optimized with PPO and distillation from a frozen teacher’s action distributions, using deployable and corresponding privileged observations during training.The teacher is first trained with PPO using deployable observations and privileged task-level information.Source: 3) Teacher–Student Distillation · pp. 10–11
  • Privileged inputs supervise the teacher but do not change the reward; after distillation, deployment uses only the student without privileged task information.The training objective includes PPO components and a distillation coefficient λ = 0.1.Source: 3) Teacher–Student Distillation · p. 11

1) Pose Deviation to Reference Segments:

  • Pose deviation is computed against each reference segment by projecting the current end-effector position and interpolating waypoint orientations with the same projection coefficient.The segment-level deviation combines positional and rotational components; waypoint positions define piecewise-linear segments.Source: 1) Pose Deviation to Reference Segments · p. 11

Related papers

Loading 2610.37378v1…