PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, Sergey Levine

arXiv:2410.13816v2cs.ROcs.LG

TL;DR

Generalist robotic policies trained on diverse demonstrations can inherit mixed-quality data and struggle with precise manipulation. V-GPS uses an offline-RL value function to re-rank actions at deployment, improving performance across policies and tasks. The approach does not require fine-tuning or access to policy weights.

  • Problem

    Robotic datasets are often mixed in quality, and it remains challenging to obtain high-performing generalist policies that preserve their broad capabilities.

    Source: §1 · p. 2

  • Method

    V-GPS learns a robotic value function via offline RL and uses it at test time to re-rank action proposals from a generalist policy.

    Source: §1 · p. 2

  • Results

    V-GPS improved real-world manipulation by +82% and consistently enhanced five open-source policies across multiple embodiments and 12 tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same value function can improve policies with different architectures and training datasets without altering or accessing their weights.

    Source: §1 · p. 2, §7 · p. 8

    The setup assumes a language-conditioned policy that can sample multiple actions for a state and instruction.

    Source: §3 · p. 3

Abstract

from arXiv · show

Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS

1 Introduction

Generalist robotic policies can inherit mixed-quality demonstrations and struggle with precise manipulation, especially under environmental shifts. V-GPS steers them at deployment by re-ranking proposed actions with an offline-RL value function, improving performance across policies and tasks.

1 Introduction

  • Large robotic datasets are typically mixed in quality, making it challenging to recover high-performing policies that match the desired performance.Imitation learning can replicate the demonstration distribution, but that distribution may fall short of desired performance.Source: §1 · p. 2
  • Generalist policies can exhibit imprecise manipulation despite strong semantic generalization, with failures worsening under environmental shifts.Examples of shifts include changes in table texture, camera pose, and distractors.Source: §1 · p. 2
  • Fine-tuning is described as infeasible in the non-stationary real world because adaptation requires costly data collection and hyperparameter tuning to preserve generalist capabilities.The paper motivates test-time steering as an alternative to repeated adaptation.Source: §1 · p. 2
  • At deployment, an offline-RL value function re-ranks generalist-policy action proposals, without fine-tuning or requiring access to policy weights.The approach is modular and plug-and-play with different generalist policies.Source: §1 · p. 2
  • V-GPS improved real-world manipulation by +82% and consistently enhanced five open-source generalist policies across multiple embodiments and 12 tasks.Evaluations covered simulated and real-world environments and policies including Octo, RT1-X, and OpenVLA.Source: §1 · p. 2

2 Related Work

Prior work established large-scale robotic datasets, generalist policies, and value-based methods, but left a gap in using learned values to guide low-level actions for real-world generalist policies. V-GPS addresses that gap with test-time sampling-based action selection.

Large-scale Robotic Datasets and Policies

  • Large-scale datasets enabled generalist policies that generalize across manipulators, unseen tasks, and language instructions, yet those policies can fail under environmental shifts.The section describes V-GPS as integrating a value function at test time with off-the-shelf policies.Source: §2 · pp. 2–3

Value-based Offline RL for Robotics

  • Conventional model-free offline-RL methods use value functions to train and update policies, often limiting policy models to Gaussian distributions and complicating use of expressive pre-trained policies.V-GPS is presented as a more flexible way to use values with black-box pre-trained policies.Source: §2 · p. 3

Sampling-based Action Selection

  • Prior sampling-based value selection improved language-model tasks and simulation benchmarks, but had not been applied to diverse, high-dimensional real-world robotic tasks.Earlier robotic work scored high-level language commands or low-level actions in D4RL simulation tasks.Source: §2 · p. 3

3 Preliminaries and Problem Statement

The problem setting assumes a language-conditioned generalist policy that can sample action candidates without exposing its weights. The paper frames value-guided steering using an MDP and language-annotated robot trajectories.

3 Preliminaries and Problem Statement

  • The policy receives a language command and current state, samples multiple actions, and may be accessed as a black box without exposing its weights.The policy is denoted π(a | s_t, l), with candidate actions a_1 through a_K.Source: §3 · p. 3
  • The formal setup uses an MDP with state and action spaces, dynamics, reward, and discount factor, and defines Q as long-term discounted return.The displayed return expression continues in the following passage.Source: §3 · p. 3
  • The dataset consists of language-annotated robot trajectories, each pairing a sequence of states and actions with a natural-language command describing the task.The learned value function is intended to steer a pre-trained generalist policy.Source: §3 · p. 3

4 Analysis: Failure Modes of Generalist Policies

The paper analyzes Octo-small-1.5 failures on a real WidowX pepper-placement task and identifies grasp precision as a challenge. The illustrated failures also include premature drops and holding an object too long.

4 Analysis: Failure Modes of Generalist Policies

  • Octo-small-1.5 was examined on a real WidowX robot using the “put pepper in pot” task, where a slippery, uneven pepper surface makes grasp selection consequential.The analysis uses the open-source model trained on the OXE dataset.Source: §4 · pp. 3–4
  • Imprecise grasp points or gripper actions can cause the pepper to fall during task execution, even after an initial grasp.Reliable handling requires choosing both the grasp point and gripper-action magnitude appropriately.Source: §4 · p. 4
  • The illustrated Octo failures include imprecise grasping, dropping an object prematurely, and holding onto it too long.The figure presents these failure modes in its first, second, and third rows, respectively.Source: Figure 2 · p. 4

5 V-GPS: Value-Guided Policy Steering

V-GPS trains a language-conditioned value function on robotic demonstrations and uses it at deployment to re-rank actions proposed by generalist policies. Its training and selection design aims to estimate useful values across diverse data and steer multiple policies without replacing them.

5 V-GPS: Value-Guided Policy Steering

  • V-GPS combines a dataset-trained language-conditioned value function with generalist policies at test time to re-rank sampled actions and avoid policy failure cases.The paper presents this as test-time steering rather than iterative re-ranking.Source: §5 · p. 4

5.1 Training: Learning Value Function via Offline RL Pre-training

  • The language-conditioned Q function is trained from demonstrations by assigning +1 rewards to each trajectory’s last three transitions and 0 elsewhere.The experiments shift these reward values to 0 and −1.Source: §5.1 · pp. 4–5
  • Cal-QL is chosen to approximate the best-in-support optimal value while conservatively penalizing out-of-distribution actions and compensating for pessimism on dataset actions.The paper also reports that IQL is effective for V-GPS.Source: §5.1 · p. 5
  • The value function uses a ResNet-34 image encoder with FiLM language conditioning, with language instructions processed by a frozen MUSE encoder.FiLM conditioning passes language features into every ResNet block.Source: §5.1 · p. 5

5.2 Deployment: Test-Time Action Re-Ranking

  • V-GPS samples multiple actions from a generalist policy, scores them with a value function, and selects an action greedily or from a temperature-controlled re-ranked distribution.The temperature parameter controls the distribution’s sharpness and balances reliance on the policy and value function.Source: §5.2 · p. 5

6 Experimental Evaluation

The evaluation tests a shared value function across real and simulated tasks, multiple robot embodiments, and five generalist policies. V-GPS improves the policies’ success rates, while real-world examples illustrate gains on grasp precision and object-release timing.

6.1 Experimental Scenarios and Comparisons

  • Evaluation spans 12 tasks: six across three real-world WidowX scenes, plus four simulated WidowX and two simulated Google Robot tasks, using one cross-embodiment value function across policies.Source: §6 · p. 6, Figure 3 · p. 6

6.2 What Kind of Failure Modes of the Generalist Policy Does V-GPS Address?

  • Real-world success-rate improvements are +55% in Scene A, +92% in Scene B, and +100% in Scene C; pepper-in-pot success rises from 15% to 35%.V-GPS consistently improves Octo-small-1.5 across all six real-world tasks.Source: §6.2 · p. 7
  • On the mushroom task, V-GPS upweights the gripper-close action until the object is over the cloth; mushroom performance doubles, while sushi performance triples.The examples address premature release and late dropping, respectively.Source: §6.2 · p. 7, Figure 4 · p. 7

6.3 Can V-GPS improve various generalist policies across different embodiments?

  • V-GPS improves all five generalist policies across SIMPLER embodiments and achieves an 82.8% average improvement in real-world Octo success rates.The SIMPLER results use the same single value function across policies and embodiments.Source: §6.3 · p. 7, Table 2 · p. 8, Table 1 · p. 8
  • V-GPS produces its largest SIMPLER gains on the eggplant-in-basket task, which combines a height difference, an obstructing wall, and precise grasping of a slippery object.The benefit is reported across all five evaluated policies.Source: §6.3 · p. 8
  • In SIMPLER, V-GPS largely mitigates Octo-small-1.5’s performance degradation relative to Octo-small, suggesting robustness to checkpoint or training-recipe variation.The authors frame this as a possible implication of the result.Source: §6.3 · p. 8

7 Discussion and Future Work

V-GPS can improve policies without changing their weights, but deployment requires stochastic action sampling and adds computation. The section also reports that IQL works as an alternative value function.

7 Discussion and Future Work

  • V-GPS requires stochastic policy sampling, adds deployment computation, and has limited ability to handle completely unseen languages and objects.The value function was trained on data from only two robotic embodiments; added computation may limit applicability in high-frequency tasks.Source: §7 · p. 8

A V-GPS with IQL

B V-GPS Implementation Details

B.1 Value Function Training

The implementation trains a language-conditioned value function on Bridge and Fractal data, then uses it to re-rank sampled policy actions at deployment. The appendix also specifies model checkpoints and evaluation-code sources.

B.2 Test-Time Action Re-Ranking

  • At each time step, V-GPS samples K proposals from the policy and re-ranks them with Q; real-world Cal-QL uses K = 50 and β → 0.Simulation sweeps K = {10, 50} and β = {0, 0.1, 1.0}, reporting the best result for each policy.Source: §B.2 · p. 14

B.1 Value Function Training

  • The value function uses a ResNet-34 image encoder with FiLM language conditioning, and predicts a scalar Q value from image, language, and action inputs.Training uses Bridge and Fractal data, batch size 512, and either 1M Cal-QL steps or 200K IQL steps.Source: §B.1 · p. 14, Figure 5 · p. 15

C Baseline Implementation Details

  • The evaluations use publicly released checkpoints; OpenVLA requires K separate forward passes because it does not support batch inference.The real-world and simulated evaluations build on the Octo and SIMPLER evaluation code, respectively.Source: C Baseline Implementation Details · p. 15

D Experimental Setup

The evaluation spans real-world and simulated tasks, including scene and object-configuration changes. In comparisons with fine-tuning and single-dataset training, V-GPS is the only method reported to outperform the generalist policy.

D Experimental Setup

  • Real-world evaluation covers six tasks across three scenes with 20 trials per task; SIMPLER covers six tasks across WidowX and Google Robot.SIMPLER trials use three random seeds for RT1-X and Octo variants, and 50 trials per task for OpenVLA.Source: D Experimental Setup · p. 15, Table 5 · p. 16, Table 6 · p. 16
  • V-GPS improves over the pre-trained policy in SIMPLER and Scenes A, B, and C, including changes in table height and unseen tabletops or backgrounds.Source: D Experimental Setup · p. 16

E Additional Comparisons to Policy Fine-Tuning Approaches

  • The fine-tuning comparison addresses the computational cost of tuning large policies and the fact that OXE already contains Bridge and Fractal data.Fine-tuning OpenVLA requires 8 × A100s; additional training on those individual datasets does not necessarily improve performance.Source: E Additional Comparisons to Policy… · p. 16
  • V-GPS is the only compared method that improves over the generalist policy; neither fine-tuning nor single-dataset training improves over Octo.The comparison includes Octo-finetune, Octo-scratch, and Resnet-DP.Source: E Additional Comparisons to Policy… · p. 16

F Ablation Over the Size of Dataset

The dataset-size ablation evaluates IQL value functions trained on smaller Bridge subsets for SIMPLER’s eggplant task. A 50% subset retains the same improvement, while a 10% subset still improves over Octo-small.

F Ablation Over the Size of Dataset

G Analysis of the Overhead in Inference Time

The analyses examine V-GPS's inference overhead, sensitivity to the number of candidate actions, actor-only alternatives, component contributions, and model size.

G Analysis of the Overhead in Inference Time

  • 1.28 times slower at K=10 and 1.59 times slower at K=50; the authors report no significant practical slowdown, particularly with WidowX's 0.2-second blocking-control interval.The timing analysis used the Octo-small model on the real-world evaluation inference machine.Source: Table 9 · p. 17, G Analysis of the Overhead in… · p. 17

H Ablation Over the Number of Actions K

I Comparisons to the Actors of Cal-QL & IQL

  • Cal-QL and IQL actors achieve zero success in SIMPLER tasks; their common failure is consistently outputting the open-gripper action.The authors connect this issue to Gaussian actors' difficulty modeling extreme gripper open/close modes in manipulation datasets.Source: Table 11 · p. 18, I Comparisons to the Actors of Cal-QL… · p. 18

J Comparison to Using a Random Policy or Random Action Selection

  • Random action selection with Octo-small performs similarly to the naive model, while a random policy with V-GPS achieves zero success; both policy proposals and value-based ranking matter.The reported comparison concludes that combining the pretrained policy and value function gives the best performance.Source: Table 12 · p. 18, J Comparison to Using a Random Policy… · p. 18

K Details of the Network Size

Loading 2410.13816v2…