PaperlayerView exact revision on arXiv

Source-linked AI summary

Proxy Policy Steering

Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma, Kuan Fang

arXiv:2609.09148v3cs.RO

TL;DR

Generalist robot policies are hard to specialize from limited demonstrations without degrading their broad capabilities. PPS trains a reference proxy and a task proxy whose velocity difference steers a frozen base sampler. Across 12 manipulation tasks, PPS improves the π0.5 base policy’s average success rate by 55% absolute while preserving recovery behavior.

  • Problem

    Specializing generalist robot policies from limited demonstrations while preserving their broad capabilities remains a deployment challenge.

    Source: §1 · pp. 1–2

  • Method

    PPS steers a frozen flow-matching VLA with a velocity-space residual between a base-distilled reference proxy and a task-fine-tuned proxy.

    Source: §1 · p. 2

  • Results

    55% absolute average success-rate gain for the π0.5 base policy across 12 manipulation tasks, with zero-to-one gains on tasks it never solves.

    Source: §7 · p. 8

  • Takeaways & Limitations

    PPS preserves base recovery behaviors while outperforming LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.

    Source: §1 · p. 3, §7 · p. 8

    The study covers the single-task regime and validates PPS only on flow-matching VLAs; cross-task, environment, and embodiment transfer remains open.

    Source: §6 · p. 8

Abstract

from arXiv · show

Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.

1 Introduction

Generalist robot policies are difficult to specialize from limited demonstrations without sacrificing their broad capabilities. PPS uses two lightweight proxies to steer a frozen base policy toward task-specific behavior while retaining its priors.

Motivation

  • Limited demonstrations make it difficult to acquire task-specific behavior while preserving the broad capabilities of generalist robot policies.The deployment distribution may differ from pretraining data.Source: §1 · p. 1

Prior adaptation limits

  • Fine-tuning can compromise broad capabilities, while existing steering methods cannot reach task behaviors the base assigns negligible probability.The latter methods reweight behaviors already produced by the base.Source: §1 · pp. 1–2

PPS overview

  • PPS estimates the task-supervision-induced change in action behavior and applies it to the frozen sampler without modifying base weights.A reference proxy models base behavior on target-task observations; a task proxy is initialized from it and fine-tuned on demonstrations, and their velocity difference steers the base at each denoising step.Source: §1 · p. 2
  • PPS lifts the π0.5 base policy’s average success rate by 55% absolute across 12 tasks and outperforms LoRA, specialists, residual policies, and prior steering methods.It also reports zero-to-one gains on four real-world tasks the base never solves and preserves recovery behavior.Source: §1 · pp. 2–3

2 Related Work

Prior adaptation approaches either change the base policy’s parameters or intervene in its sampling process. PPS instead uses a velocity-space correction designed to steer the sampler toward task-relevant modes before it commits to an action.

Existing adaptation approaches

  • Fine-tuning [7, 8] can overwrite manipulation priors [9] [11] and is expensive at VLA scale, while sampling-time methods intervene at different stages of action generation.Sampling-time methods include noise optimization [16], residual policies [22] [23] [25], candidate selection [26] [14], and dynamics-guided trajectory steering [17] [28].Source: §2 · p. 3

Proxy Policy Steering

  • PPS applies an additive density-ratio correction in velocity space at every denoising step, steering toward task-relevant modes before the sampler commits.This contrasts with output-level residual policies and candidate selection, which act after the base has sampled an action mode.Source: §2 · p. 3
  • PPS’s guidance pair—a base-distilled reference proxy and a task-fine-tuned proxy—aims to isolate task-supervision changes rather than arbitrary network differences.The method builds on proxy tuning [29] [30] [31] [32] and additive score-space guidance [33] [34] [36] [37], but uses the proxy pair to construct its task-specific residual.Source: §2 · p. 3

3 Preliminaries

The paper focuses on VLA policies, which map observations, proprioceptive states, and language instructions to robot actions. For flow-matching action heads, actions are generated by integrating a velocity field from noise to a clean action.

Generalist policies

  • Vision-Language-Action (VLA) models map observations, proprioceptive states, and language instructions to robot actions.Their conditional action distribution is denoted πbase(a | o, l).Source: §3 · p. 3

Flow-matching action heads

  • Flow-matching action heads transform Gaussian noise into clean actions by predicting a velocity field over intermediate action states.The flow path is xk = kϵ + (1 − k)a, with k = 1 denoting noise and k = 0 a clean action; inference integrates the velocity field from noise to the final action.Source: §3 · p. 3

Adaptation setting

  • The adaptation goal is to change the base policy’s task action distribution while preserving its priors, without modifying base parameters.The base is treated as query-only: its parameters and gradients are unavailable, and it supplies forward velocity predictions.Source: §3 · p. 3

4 Proxy Policy Steering

PPS expresses adaptation as a relative correction to a frozen base policy, implements it as an additive velocity residual, and trains proxies so that residual reflects task supervision. The method is developed for flow-matching VLAs and evaluated across real-world and simulation tasks.

4.1 Adapting Generalist Policies by Proxy

  • PPS frames adaptation as a relative density correction that amplifies actions made more likely by task supervision while preserving actions already explained by the base.The reference proxy models base behavior on target-task observations, and the task proxy captures how that behavior changes under demonstrations.Source: §4.1 · p. 4

4.2 Flow-Matching Proxy Guidance

  • Under a shared flow-matching schedule, the relative correction becomes an additive task-minus-reference velocity residual on the frozen base field.The steering strength γ controls the correction; γ = 0 recovers the base sampler, and agreeing proxies leave the base in control.Source: §4.2 · pp. 4–5

4.3 Learning Proxy Policies

  • PPS first distills a reference proxy on base-generated trajectories, then initializes the task proxy from it and fine-tunes on demonstrations.Shared initialization and approximation errors help their difference isolate task-induced change; independent training or replacing the reference with the base leaves error in the residual.Source: §4.3 · p. 5
  • Figure 2 covers 8 real-world and 4 simulation manipulation tasks, including in-distribution and out-of-distribution settings for the base VLA.The figure reports success rates averaged over tasks separately for simulation and the real world.Source: Figure 2 · p. 6

5 Experiments

The experiments evaluate PPS on real-world and simulation tasks using 50 demonstrations per task, comparing it with a frozen base and adaptation baselines. They report overall gains, test recovery and efficiency, and analyze steering strength, proxy calibration, and failure modes.

5.1 Experimental Setup

  • The evaluation covers 8 real-world and 4 simulation tasks, uses 50 demonstrations per task, and compares PPS with Specialist, Residual, DSRL, and LoRA.Success rates use a 1200-action-step budget, with 10 real-world rollouts and 100 simulation rollouts per method and task.Source: §5.1 · p. 6

5.2 Comparative Results

  • 55% absolute success-rate gain averaged across all 12 tasks shows PPS substantially improves π0.5 on downstream tasks.PPS also enables the base to solve tasks it could not complete on its own.Source: §5.2 · p. 6
  • PPS beats Specialist and LoRA; DSRL improves only where the base has nonzero success, while residual learning underperforms Specialist.DSRL needs either a large offline dataset or nonzero base success to bootstrap.Source: §5.2 · pp. 6–7
  • Under mid-rollout disturbances, PPS loses 7.8 success-rate points, versus 15.8 for LoRA and 26.1 without the reference proxy.Figure 3 depicts PPS recovering from failures absent from demonstrations, while LoRA fails due to overfitting.Source: §5.2 · p. 7, Figure 3 · p. 7
  • PPS trains 5× faster than LoRA (2.9h vs. 16.3h) and adds 6 ms inference latency per action chunk; applying it to π0 raises real-world success from 5% to 55%.The same procedure gives zero-to-one gains on all six tasks π0 never solves.Source: §5.2 · p. 7

5.3 Analysis on PPS

  • The optimal steering strength γ in the 50-demonstration regime lies inside (0.4, 0.6); experiments fix γ = 0.4 without per-task tuning.γ = 0 recovers the unsteered base sampler, while larger values impose stronger task-specific steering.Source: §5.3 · p. 7
  • Removing the reference, velocity supervision, or reference initialization lowers success from 64% to 55%, 50%, and 48%, respectively, by breaking residual-calibration conditions.The ablations test matching base velocities on task observations and sharing proxy approximation errors and randomness.Source: §5.3 · pp. 7–8
  • PPS cuts incorrect-mode failures by 93% versus frozen π0.5 and OOD-state failures by 87% versus Specialist.The analysis groups failures into incorrect mode, imprecise execution, OOD states, and other causes.Source: §5.3 · p. 8

6 Limitations

The study is limited to single-task settings and flow-matching VLAs, and PPS's benefit depends on a sufficiently strong base policy.

6 Limitations

  • PPS is studied only in the single-task regime and on flow-matching VLAs; its benefit may be smaller when the base is substantially weaker than the proxy.In that case, training a specialist from scratch may offer more benefit; transfer of shared manipulation primitives across tasks, environments, and embodiments remains open.Source: §6 · p. 8

7 Conclusions

PPS specializes a frozen flow-matching VLA using a calibrated velocity-space residual between two lightweight proxies, and the conclusion reports gains across 12 tasks while retaining recovery behaviors. Appendix A describes the proxy and baseline architectures and the shared control and sampling setup.

7 Conclusions

  • Across 12 manipulation tasks, PPS lifts π0.5 by 55% absolute success rate on average and transfers to π0, while retaining recovery behaviors that LoRA degrades.The conclusion states that PPS requires only forward velocity predictions from the base, allowing use with closed-source or API-only policies.Source: §7 · p. 8

A Implementation Details

  • Appendix A covers proxy and baseline architectures, control and sampling configuration, and shared training hyperparameters.These implementation details are organized across Secs. A.1–A.3.Source: A Implementation Details · p. 12

A.1 Network Details

  • Each proxy has about 32M trainable parameters, roughly 1% of the frozen π0.5 base; both use a jointly trained DINOv3 visual encoder and a Gemma transformer action expert.Specialist uses the proxy architecture; Residual adds a base-action encoder, while LoRA inserts adapters into the base VLM and action expert.Source: §A.1 · p. 12

A.2 Control and Sampling Setup

  • Policies run at 15 Hz with 10 Euler denoising steps and a 15-step action horizon, executing the first 8 steps before replanning.PPS applies guided velocity at each denoising step and uses γ = 0.4 for every task without per-task tuning.Source: §A.2 · p. 12

A.3 Training Hyperparameters

The appendix specifies proxy and baseline training hyperparameters and points to the task descriptions and evaluation protocol.

A.3 Training Hyperparameters

  • Proxy policies and baselines train for 20,000 gradient steps with batch size 64, using a shifted Beta(1.5, 1) distribution for flow-matching noise level k.Reference distillation instead supervises noise levels visited along the base sampler's denoising trajectory.Source: §A.3 · p. 12
  • DSRL is trained for 1,000 steps because its reinforcement-learning objective on a small dataset becomes unstable with longer training.The reported checkpoint is the last one before its RL loss diverges.Source: §A.3 · p. 12

B Additional Details of Experimental Setup

  • Appendix B lists each task's instruction, success criterion, and demonstration collection, and describes the success-rate measurement protocol.The referenced task details and evaluation protocol are in Secs. B.1 and B.2.Source: B Additional Details of Experimental… · p. 12

B.1 Task Design

This appendix specifies task prompts, demonstrations, and evaluation protocols, then reports per-task outcomes and parameter- and data-scaling results.

B.1 Task Design

  • Table 1 lists each task’s language instruction and success criterion; real-world tasks use 50 teleoperated demonstrations, while simulation uses 50 generated from five teleoperated source demonstrations.The task set includes eight real-world and four simulation tasks.Source: §B.1 · pp. 12–13, Table 1 · p. 13

B.2 Evaluation Protocol

  • Real-world evaluation uses 10 rollouts per method per task with randomized object layouts; simulation uses 100 rollouts per task with identical initial seeds across methods.Source: §B.2 · p. 13

C.1 Per-Task Success Rates

C.2 Scaling up Parameters and Data

  • 64% success with 32M trainable parameters puts PPS within 10 points of either method’s best, while LoRA needs 8× more parameters and Specialist plateaus near 54% at 256M.Source: §C.2 · p. 14
  • PPS leads on Tea and Capsule at every demonstration budget from 50 to 200; Specialist improves steadily and catches up near 200 demonstrations.Source: §C.2 · p. 14

C.3 Training, Inference, and Parameter Efficiency

The supplemental evaluations show PPS transferring to π0 and supporting added sensing modalities, while retaining recovery under rollout perturbations. They also report efficiency gains over LoRA and a benchmark where both methods reach 100% success.

C.3 Training, Inference, and Parameter Efficiency

  • PPS takes 2.9 hours versus LoRA’s 16.3 under identical training, adds 6 ms per action chunk and 0.13 GB at inference, and LoRA needs roughly 8× more trainable parameters to match PPS at 32M.The training comparison uses a single A6000 GPU; the proxies run in parallel with the base policy on a single 4090 GPU.Source: §C.3 · p. 14

C.4 Other Base Policies

  • Across eight real-world tasks, PPS lifts π0’s average success rate from 5% to 55%, including zero-to-1 improvements on six tasks where unsteered π0 scores 0%.This supports compatibility with another base policy in the π0 family.Source: Figure 7 · p. 15, §C.4 · p. 15

C.5 Comparison on a Standard Benchmark

  • On the LIBERO benchmark, both PPS and DSRL lift π0 from 17% to 100% success, so the saturated benchmark does not distinguish them.The methods separate more sharply on tasks the base policy does not solve on its own.Source: §C.5 · p. 15

C.6 Cross-modality Steering

  • PPS outperforms all baselines on Marble point-cloud and Phone spectrogram tasks, including methods that also use the added modality.The comparisons include Specialist, DSRL, and LoRA-MM.Source: §C.6 · p. 15, Figure 8 · p. 16

C.7 Robustness Under Mid-Rollout Perturbations

  • Under disturbances, ∆SR is −7.8 points for PPS, −15.8 for LoRA, and −26.1 for PPS w/o ref; PPS also recovers from two Jeans-task interventions and completes.Values closer to zero indicate better retention of recovery ability.Source: §C.7 · pp. 15–16, Figure 9 · p. 16

C.8 Velocity Alignment Analysis

Velocity alignment comparisons calibrate the task-induced shift against both a fine-tuned base and the reference proxy. The steering visualization shows how the task–reference residual redirects denoising from the base’s incorrect action mode toward the task-success mode.

C.8 Velocity Alignment Analysis

  • 0.946 cosine agreement between vref and vbase on the steered path, and 0.935 on the base model’s own path, supports the reference proxy as a base stand-in.On-policy distillation aligns the reference with the base in the region where guidance is applied.Source: §C.8 · p. 16
  • 0.890 Tuned–Base similarity calibrates the task-induced shift, compared with 0.865 for Task–Ref and 0.767 for Task–Base.The additional Task–Base divergence reflects the architecture and capacity gap between the 32M proxy and 3B base, rather than task signal.Source: §C.8 · p. 16
  • Using vtask − vbase as the residual mixes task-induced change with proxy-specific approximation error, which the reference proxy cancels.The w/o ref ablation and disturbance results show this error is costly.Source: §C.8 · p. 16
  • Figure 10 compares mid-rollout disturbance effects on success rate, where changes nearer zero are better, and velocity-field cosine similarity calibrated against the tuned base.The cosine similarity is measured along the steered denoising trajectory.Source: §C.8 · p. 16, Figure 10 · p. 17

C.9 Steering Visualization

  • The task proxy points toward the success mode while the reference tracks the base’s incorrect mode; their residual steers the denoising trajectory toward success.The guided velocity is vbase + γ(vtask − vref) at each denoising step, unlike the base-only trajectory from the same initial sample, which ends in the incorrect mode.Source: §C.9 · pp. 17–18, §C.8 · p. 17

D Velocity Derivation

The derivation converts a product-of-experts score correction into a velocity-space steering rule for flow-matching policies. Its exactness depends on stated assumptions, including a per-noise-level correction that is generally not the noised marginal of the clean target.

D Velocity Derivation

  • The derivation is exact given A1–A3, but the per-level product-of-experts correction generally does not equal the noise-level marginals of the clean target except at k = 0 or γ = 0.A1 and A2 are satisfied by design; A3 is a modeling choice because noising does not commute with density products and ratios.Source: D Velocity Derivation · p. 18
  • For the linear path, velocity is v(x, k) = A(k)x + B(k)s_k(x), with coefficients determined by the noise schedule and the score-to-velocity relation invertible only for k in (0, 1).At k = 0, B(0) = 0; at k = 1, the coefficients diverge.Source: D Velocity Derivation · p. 18
  • Under assumptions A1–A3, the level-k target’s marginal velocity satisfies the steering identity, obtained by substituting the shared score-to-velocity relation.At k = 0 the identity holds trivially because every policy’s velocity is −x.Source: D Velocity Derivation · pp. 18–19
  • Eq. (11) steers the frozen base velocity by adding γ times the task-proxy minus reference-proxy velocity, and the residual vanishes when the proxies agree.The scalar γ controls how strongly the base velocity is tilted toward the task proxy.Source: D Velocity Derivation · pp. 18–19

Related papers

Loading 2609.09148v3…