PaperlayerView exact revision on arXiv

Source-linked AI summary

VLS: Steering Pretrained Robot Policies via Vision-Language Models

Shuo Liu, Ishneet Sukhvinder Singh, Yiqing Xu, Jiafei Duan, Ranjay Krishna

arXiv:2602.03973v1cs.ROcs.CV

TL;DR

Pretrained robot policies can degrade under observation, language, and environmental shifts because imitation learning couples actions to training-specific contexts. VLS addresses this with training-free inference-time steering of frozen policies, and experiments show improved performance across simulation and real-world tasks, with computational latency remaining a limitation.

  • Problem

    Imitation-trained robot policies struggle to produce constraint-satisfying behaviors when observation, language, or scene configurations depart from the training distribution.

  • Method

    VLS steers the sampling process of a frozen pretrained policy using vision-language-model-generated differentiable rewards without modifying policy parameters.

  • Results

    VLS outperforms evaluated steering and vision-language-action baselines across simulation and real-world tasks, including 94% success on MovableObjects and 69% average real-world success in in-distribution tasks.

  • Takeaways & Limitations

    Inference-time steering can adapt reusable pretrained motor primitives to out-of-distribution spatial and task requirements without retraining.

  • Takeaways & Limitations

    VLS incurs high inference overhead from batch sampling, MCMC runs, and forward-kinematics resampling.

Abstract

from arXiv · show

Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train-test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time. We propose Vision-Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation-language inputs without modifying policy parameters. By leveraging vision-language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements. Across simulation and real-world evaluations, VLS consistently outperforms prior steering methods, achieving a 31% improvement on CALVIN and a 13% gain on LIBERO-PRO. Real-world deployment on a Franka robot further demonstrates robust inference-time adaptation under test-time spatial and semantic shifts. Project page: https://vision-language-steering.github.io/webpage/

I. INTRODUCTION

Pretrained robot policies often fail under spatial and semantic shifts even when the needed motor behaviors were seen during training. VLS addresses this mismatch by steering frozen policy sampling at inference time using vision-language-derived guidance.

  • Motivation: Retraining or fine-tuning is costly and misaligned when the required behaviors already exist but must be recomposed under altered conditions.The paper frames this as an inference-time control problem rather than a need for broader data coverage.
  • VLS: VLS steers a frozen diffusion or flow-matching policy’s sampling process without modifying policy parameters.The base policy supplies reusable motor primitives while steering controls their composition under OOD inputs.
  • VLS: Vision-language models interpret OOD observation-language inputs, decompose tasks into stages, and synthesize differentiable rewards for action proposals.These rewards provide dense guidance during inference-time denoising.
  • Evaluation: VLS is evaluated on CALVIN and LIBERO-PRO under observation and language shifts, including unseen appearances, positions, and target substitutions.The evaluations cover simulation and practical deployment conditions.
  • Motivation: Imitation-trained policies can degrade sharply when scene geometry or task context departs from the training manifold.Examples include clutter, support-surface shifts, and changed object layouts.
  • Related work: Related steering methods rely on learned critics, dynamics models, discrete verification, or online learning, whereas VLS targets pure inference-time adaptation.The paper contrasts these alternatives with retaining the pretrained policy as the skill prior.

III. PROBLEM FORMULATION

Imitation learning fits action distributions to expert demonstrations conditioned on observations and language, but deployment inevitably encounters OOD inputs. Diffusion and flow-matching policies generate action trajectories through iterative denoising from noise.

  • A. The OOD Dilemma in Imitation Learning: Imitation learning trains a policy πθ from expert demonstrations to model the state-conditional action distribution p(a|o).The demonstrations contain observations, actions, and language instructions.
  • A. The OOD Dilemma in Imitation Learning: The training target maximizes the likelihood of an action chunk conditioned on the current observation and language instruction.The chunk spans horizon T from time t through t+T.
  • A. The OOD Dilemma in Imitation Learning: After training, freezing θ yields the base policy π⋆, which faces OOD observation-language pairs outside the expert dataset.Observation shifts include changed backgrounds or object layouts, while semantic shifts include unseen instructions.
  • A. The OOD Dilemma in Imitation Learning: Because π⋆ overfits spatial and semantic correlations in the training manifold, it becomes brittle when deployment conditions differ.The paper identifies this as a distribution-dependent limitation of the static training objective.
  • B. Diffusion and Flow Matching Policies: Diffusion policies transform Gaussian noise into complex action distributions through forward diffusion and reverse denoising.Forward diffusion adds Gaussian noise to clean action trajectories before reverse generation.
  • B. Diffusion and Flow Matching Policies: A denoising network predicts added noise at step k using the observation and instruction, producing a clean action trajectory approximation.The reverse process iteratively samples from the noisy trajectory toward the action distribution.
  • B. Diffusion and Flow Matching Policies: Flow matching replaces discrete diffusion steps with a continuous velocity field over time interval [0,1].Both frameworks are unified using k to denote denoising progression, with k=1 as noise and k=0 as the clean trajectory.

C. Problem Formulation

The paper formulates adaptation as classifier-guided steering of a frozen policy’s denoising process for OOD observation-language conditions. The central challenge is constructing guidance that captures structured spatial and semantic constraints while providing dense trajectory gradients.

  • C. Problem Formulation: VLS uses classifier guidance to steer a pretrained base policy π⋆ toward OOD conditions without fine-tuning.The guidance gradient changes the direction of denoising for proposed action trajectories.
  • C. Problem Formulation: The guidance function represents the score of the joint action-and-OOD-condition distribution.Its gradient with respect to the action proposal supplies the steering direction.
  • C. Problem Formulation: For diffusion policies, guidance modifies the predicted noise, with λ controlling guidance strength.The paper uses the analogous idea for flow matching by controlling the predicted velocity field.
  • C. Problem Formulation: Flow-matching policies can be steered for the condition y=(o,l)OOD by controlling their predicted velocity field.This extends the guidance formulation beyond discrete diffusion denoising.
  • C. Problem Formulation: Real-world OOD inputs are structured specifications rather than simple class labels, encoding spatial and semantic constraints.A useful guidance function must interpret the geometry and logical structure induced by the condition.
  • C. Problem Formulation: The guidance must provide dense, informative gradients with respect to the proposed action trajectory.This requirement complements accurate interpretation of the OOD condition’s geometry and logic.

IV. OUR APPROACH: VLS

VLS converts OOD observation-language inputs into differentiable reward guidance for a frozen policy’s denoising trajectory. Its pipeline grounds inputs geometrically, scores proposals stage by stage, and injects those gradients during inference.

  • Pipeline: VLS approximates the guidance signal needed for inference-time adaptation under OOD inputs.The signal is used to steer the denoising trajectory of the frozen base policy.
  • Pipeline: A differentiable surrogate score R evaluates how well an action proposal satisfies spatial and semantic constraints from the OOD input.R is differentiable with respect to the action trajectory.
  • 1) OOD input grounding and VLM-generated differentiable scoring: A vision-language model synthesizes stage-wise programmatic reward functions over the grounded geometric scaffold.These rewards provide differentiable scores for action proposals.
  • 2) Inference-time denoising guidance: VLS injects reward gradients into diffusion or flow-matching denoising updates while combining gradient refinement with particle-level diversity.The guidance is applied during inference without fine-tuning the frozen policy.
  • 1) OOD input grounding and VLM-generated differentiable scoring: VLS maps the high-dimensional OOD observation-language pair into a compact set of task-relevant spatial variables.This grounding makes it possible to define a differentiable function over the action space.

1) OOD Input Grounding:

VLS grounds out-of-distribution observation–language inputs into task-relevant 3D keypoints, then uses a VLM to generate differentiable, stage-aware rewards for action evaluation.

  • VLS identifies relevant objects and regions, segments them with SAM, and extracts masked DINOv2 features.
  • Masked pixels are reprojected with depth into object-centric 3D point clouds whose points combine visual features and spatial coordinates.
  • Clustering produces task-relevant keypoints P, with each keypoint anchoring a physically meaningful spatial constraint.
  • The resulting geometric scaffold deterministically exposes the spatial variables needed for downstream differentiable reward evaluation.
  • A VLM decomposes the task into sequential stages and synthesizes a differentiable reward function for each stage’s spatial constraints.
  • Rewards are implemented as PyTorch functions, allowing gradients through action proposals while the VLM remains non-differentiable and off-graph.

B. Action Denoising Process Guidance

VLS guides frozen-policy denoising toward trajectories satisfying OOD constraints by combining reward-gradient refinement with diversity preservation and resampling.

  • VLS injects reward gradients into diffusion or flow-matching denoising updates to steer action trajectories toward OOD constraints.
  • The pipeline uses grounded keypoints and VLM-generated rewards to guide action generation while preserving diversity under complex inputs.
  • A batch of B action proposals is initialized independently for each environment timestep.
  • A pairwise repulsive force prevents premature batch collapse and maintains broad coverage of the action manifold.
  • Multiple stochastic inner refinements per denoising step smooth exploration and reduce sensitivity to noisy or locally distorted gradients.

3) Gradient-Free Resampling via Feynman–Kac Steering:

VLS supplements continuous gradient guidance with Feynman–Kac particle resampling and closed-loop guidance adaptation for multimodal, uncertain execution.

  • Feynman–Kac steering treats action proposals as an interacting particle system and periodically resamples them using reward-based potentials.
  • Reward-weighted multinomial resampling replicates high-reward particles and prunes proposals violating OOD constraints.
  • Combining gradient refinement with gradient-free resampling helps the frozen policy navigate complex, multimodal constraint landscapes.
  • Closed-loop execution feedback adaptively regulates guidance strength and determines when to switch stage-specific rewards.
  • Guidance is increased when action chunks deviate from stage constraints and reduced as execution improves, allowing the base policy to dominate near completion.

2) Schmitt-Trigger-Based Stage Switching:

VLS uses hysteresis-based stage switching to coordinate multi-stage execution under uncertainty, and evaluates adaptation across controlled spatial and semantic shifts.

  • 2) Schmitt-Trigger-Based Stage Switching:: VLS uses two reward thresholds, R_high and R_low, to compute a switching signal and avoid oscillatory stage transitions.
  • 2) Schmitt-Trigger-Based Stage Switching:: Hysteresis prevents premature transitions and repeated oscillations during physically uncertain, complex OOD execution.
  • 2) Schmitt-Trigger-Based Stage Switching:: When switching occurs, a VLM selects the next-stage reward or continues the current stage with updated guidance strength.
  • V. EXPERIMENTS: Evaluation covers CALVIN and LIBERO-PRO simulations plus Franka Emika real-world deployment under spatial and semantic test-time shifts.
  • V. EXPERIMENTS: Test-time observation shifts add distractors, alter object attributes, and change task-object or support-surface positions and orientations.
  • V. EXPERIMENTS: Language shifts change target objects and goal behaviors, while experiments test steering necessity, comparative adaptation, component contributions, and real-world overhead.
  • V. EXPERIMENTS: 94% average success on CALVIN movable objects corresponds to a 7.4× improvement over the base policy.

A. Baselines

VLS is evaluated against pretrained VLA models, inference-time steering methods, and ablated variants that remove individual components.

  • The experiments compare VLS with seven baselines spanning VLA models and inference-time steering methods.
  • Four pretrained VLA models—OpenVLA, π0, π0.5, and π0.5 LeRobot—are evaluated without fine-tuning on OOD test scenarios.
  • DynaGuide uses DINO-feature distances for denoising guidance, whereas ITPS selects predefined guidance functions according to the detected OOD condition.
  • Three VLS ablations remove gradient guidance, Feynman–Kac resampling, or RBF-based diversity initialization, isolating their contributions.

B. Results

Across simulation and real-world evaluations, VLS improves adaptation to spatial, semantic, and task shifts, while ablations expose the roles of guidance, resampling, and diversity.

  • Inference-Time Steering Is Necessary: LIBERO-PRO evaluates OOD perturbations across object, position, semantic, task, and environment dimensions, with position and task shifts matching this paper’s target scenarios.
  • Inference-Time Steering Is Necessary: Pretrained VLAs struggle under joint observation-language perturbations despite strong in-distribution performance, while VLS consistently outperforms them.
  • VLS Outperforms Existing Steering Methods: On CALVIN, VLS reaches 94% average success on MovableObjects and 87% on ArticulatedParts, improving over the base policy by 7.4× and 9.6×, respectively.
  • Ablation Study of components: Removing gradient guidance causes success rates to approach failure across tasks and substantially increases episode lengths.
  • Ablation Study of components: Removing FK resampling or RBF diversity has smaller success-rate effects but degrades efficiency and stability by reducing sample efficiency and global coverage.
  • Scaling with sample batch size: Increasing sample batch size improves success rates and reduces episode length, but raises inference latency, creating a tunable compute–performance trade-off.
  • Ablation Study of components: Both gradient-free exploration and gradient-based refinement are necessary for robust inference-time control.
  • VLS Enables Efficient Real-World Deployment: On a Franka robot, VLS achieves 69% average in-distribution success, exceeds the frozen π-0.5 baseline by 19%, and succeeds in 40% of unseen-mug OOD trials.

VI. CONCLUSION & LIMITATION

The paper concludes that VLS provides training-free OOD adaptation for pretrained robotic policies, while identifying inference latency as a limitation.

  • VLS guides pretrained robotic policies with VLM-generated differentiable rewards to address OOD deployment without training.
  • VLS introduces high inference overhead because batch sampling, MCMC runs, and Feynman–Kac resampling increase computational latency.
Loading 2602.03973v1…