PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Language Models are In-Context Value Learners

Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, Fei Xia

arXiv:2411.04549v1cs.ROcs.AIcs.LG

TL;DR

Robots need temporal progress estimates that generalize across diverse tasks and domains, but existing approaches face challenges in task and scene generalization, partial observability, and long-horizon consistency. GVL asks frozen VLMs to autoregressively predict task completion percentages for shuffled video frames, optionally using multimodal in-context examples. GVL provides strong zero-shot value predictions across 51 datasets, 20 embodiments, and more than 300 tasks, including challenging bimanual manipulation, with performance improving through multimodal in-context examples.

  • Problem

    Robots need temporal progress estimates that generalize across diverse tasks and domains, but existing approaches face challenges in task and scene generalization, partial observability, and long-horizon consistency.

    Source: §1 · pp. 1–2

  • Method

    GVL asks frozen VLMs to autoregressively predict task completion percentages for shuffled video frames, optionally using multimodal in-context examples.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    GVL provides strong zero-shot value predictions across 51 datasets, 20 embodiments, and more than 300 tasks, including challenging bimanual manipulation, with performance improving through multimodal in-context examples.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    GVL supports dataset filtering, success detection, and advantage-weighted regression for real-world robot learning without model training or fine-tuning.

    Source: §1 · p. 3, §5 · p. 13

    Value-Order Correlation is most suitable for a-periodic tasks with a unique expert-frame ordering, so wiping or stirring may be difficult to assess.

    Source: §5 · p. 13

Abstract

from arXiv · show

Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (\GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression -- all without any model training or finetuning.

1. Introduction

GVL addresses the challenge of generalizable temporal progress estimation by using VLMs to predict values over shuffled video frames. It supports zero- and few-shot prediction across diverse real-world tasks and downstream robot-learning applications.

Motivation and approach

  • Shuffling frames counters VLMs’ tendency to produce uninformative monotonic values caused by strong temporal correlations in chronological video.The approach encourages the model to use semantic and temporal grounding to distinguish task progress across frames.Source: §1 · p. 2

Evaluation

  • GVL is evaluated across 51 datasets, 20 embodiments, and more than 300 real-world tasks, including challenging bimanual manipulation.The evaluation uses Value-Order Correlation and reports highly positive scores on most datasets, with further gains from multimodal in-context examples.Source: §1 · p. 3

Applications

  • GVL enables downstream dataset-quality estimation, success detection, and advantage-weighted regression for robot learning.These applications use value predictions at different data abstraction levels, from datasets and trajectories to transitions.Source: Figure 1 · p. 2, §1 · p. 3

Contributions

  • GVL introduces universal value prediction through VLM in-context autoregressive estimation on shuffled video frames.The framework is evaluated without robot- or task-specific training and supports zero-shot and few-shot use.Source: §1 · pp. 2–3

2. Related Work

Prior work has used foundation models for rewards, planning, imitation, and action generation, but generally not for dense, generalizable per-frame value estimation on real-world tasks. GVL differs by using pretrained VLMs for flexible multimodal in-context value learning without robot-specific fine-tuning.

Reward and value foundation models

  • Earlier reward and value methods used robot or human videos, language models, and vision-language models for reward signals, preferences, or success detection.These approaches commonly provide sparse success-oriented feedback rather than task-progress values.Source: §2 · p. 4
  • Existing VLM reward-modeling work generally does not exploit autoregression, long context, or in-context learning for real-world per-frame value estimation.Prior evaluations often focus on simple or simulated tasks.Source: §2 · p. 4

In-context learning for robotics

  • Earlier robotics in-context learning methods typically require explicit, often extensive task training and generalize narrowly.GVL instead obtains multimodal in-context value learning from pretrained VLMs without robot-specific fine-tuning.Source: §2 · p. 4

3. Generative Value Learning

GVL frames temporal value estimation as autoregressive prediction of task progress over a shuffled sequence of trajectory frames. Its three components—autoregressive prediction, observation shuffling, and in-context learning—promote consistent and more faithful values without task-specific model training.

Overview

  • GVL predicts task progress as an autoregressive next-token problem over a batch of shuffled trajectory frames.The VLM receives a natural-language task description and generates progress values for the frames.Source: §3 · p. 4

1. Autoregressive value prediction

  • Autoregressive generation lets the VLM condition each value on the full trajectory and previous predictions, supporting globally consistent long-horizon estimates.This supplies a consistency mechanism without training a classical feed-forward value function.Source: §3 · p. 5

2. Input observation shuffling

  • Chronological inputs encourage a shortcut in which the VLM outputs monotonic values, so GVL randomly shuffles observations to force attention to individual frames and trajectory quality.The first frame is retained as an anchor because fully shuffling frames can make the original temporal direction ambiguous.Source: §3 · p. 5

3. In-context value learning

  • GVL prepends shuffled videos with labeled task-progress examples to improve predictions through few-shot in-context learning without fine-tuning.Examples can come from unrelated tasks or human videos, and gains are especially useful for difficult bimanual dexterous tasks.Source: §3 · pp. 5–6

Practical implementation

  • The practical implementation asks the VLM for integer-valued progress percentages from 0 to 100 using 30-frame input sequences.Videos are subsampled to make findings comparable across datasets with different lengths and capture frequencies.Source: §3 · p. 6

4. Experiments

The experiments evaluate GVL’s zero-shot generalization, in-context scaling, downstream utility, and key design choices across diverse real-world robotics settings. Results show strong OXE performance, improvement from robot and human demonstrations, useful quality and policy-learning applications, and dependence on autoregressive prediction and frame shuffling.

4.1. Large-scale real-world evaluation

  • GVL’s zero-shot VOC scores outperform LIV across OXE datasets with both language and image goals.Across 1,000 trajectories, GVL recovers temporal structure from shuffled videos; LIV is especially weak on language goals, while GVL remains better on image goals.Source: §4.1 · p. 7, Figure 3 · p. 7
  • On 250 challenging ALOHA bimanual tasks, GVL achieves positive VOC on more than 60% of demonstrations with median VOC 0.12.The evaluation covers 500 human-teleoperated demonstrations and long-horizon skills, with lower performance than on OXE.Source: §4.1 · pp. 7–8, Figure 4 · p. 8

4.2. Multi-Modal In-Context Value Learning

  • One in-context trajectory raises ALOHA performance to 90% positive VOCs and median VOC 0.37, while additional examples steadily improve average VOC.GVL also improves from human demonstrations of the same tasks, demonstrating cross-embodiment in-context learning without fine-tuning.Source: §4.2 · p. 8, Figure 6 · p. 9

4.3. GVL Applications

  • GVL VOC supports dataset quality estimation, success detection, filtered imitation learning, and advantage-weighted regression for visuomotor control.VOC rankings distinguish higher-quality demonstrations from exploration data, while GVL-SD improves success filtering and high-VOC predictions correlate with policy-learning gains.Source: §4.3 · pp. 10–13

4.4. Ablations

  • Removing autoregressive batch prediction drops RT-1 VOC from 0.74 to −0.08, while removing frame shuffling produces degenerate monotonic predictions.These ablations support both autoregressive value generation and shuffled-frame ordering as necessary design choices for discriminative value estimates.Source: §4.4 · p. 13

5. Conclusion

GVL provides zero-shot and few-shot value prediction across diverse robotic tasks by autoregressively estimating progress from shuffled video frames. Its evaluation demonstrates downstream benefits, while VOC and multi-view and finetuning questions define important boundaries for future work.

5. Conclusion

  • GVL produces dense value predictions for diverse real-world robotic tasks across embodiments and task categories, with performance improving through few-shot examples from varied tasks or embodiments.The method uses autoregressive value prediction on shuffled video frames.Source: §5 · p. 13
  • GVL values support dataset, trajectory, and transition selection that improves downstream policy learning performance and generalization.Source: §5 · p. 13
  • VOC is most suitable for a-periodic tasks with a unique expert-frame ordering, making wiping or stirring difficult to evaluate with this metric.Source: §5 · p. 13
  • The study leaves open whether VLM finetuning or multi-view observations can improve value prediction quality.Source: §5 · p. 13

Acknowledment

The authors acknowledge Jie Tan, Pannag Sanketi, Oliver Groth, and the Google DeepMind Robotics team for discussions and feedback.

Acknowledment

  • The authors thank Jie Tan, Pannag Sanketi, Oliver Groth, and the Google DeepMind Robotics team for helpful discussions and feedback.Source: Acknowledment · p. 14

A. Prompt

The prompt asks a VLM to estimate task-completion percentages for randomly ordered robot frames using task descriptions and examples. It anchors completion at 0 for the initial scene and requires per-frame formatted outputs.

A. Prompt

  • The prompt asks the VLM to predict task-completion percentages between 0 and 100 for frames associated with a specified robot task.Source: A. Prompt · p. 20
  • Examples and queried frames are presented in random order, requiring the VLM to reason about individual frames rather than sequence position.Source: A. Prompt · p. 20
  • The initial robot scene is assigned 0% completion before the prompt requests percentages for the subsequent randomly ordered frames.Source: A. Prompt · p. 20
  • Each response must include a frame description and task-completion percentage in the specified per-frame format.Source: A. Prompt · p. 20

B. GVL OXE Dataset VOC Breakdown

This section reports average VOC scores for each OXE dataset and identifies where backbone-specific VOC results are provided.

B. GVL OXE Dataset VOC Breakdown

C. Simulation Tasks

The simulation study collects balanced success-and-failure rollouts for six dexterous manipulation tasks to support success detection and filtered imitation learning.

C. Simulation Tasks

  • Six simulation tasks use VR teleoperation to collect 500 trajectories for initial policy training, followed by 1,000 rollouts from the converged policy.The resulting datasets contain approximately equal numbers of successful and failed trajectories.Source: C. Simulation Tasks · p. 20

D. Additional Results

Additional analyses show that shuffled-frame prompting is central to GVL’s informative value predictions, while the method remains effective across backbones, tasks, viewpoints, and language specifications. These results also support downstream success-filtered imitation learning and flexible cross-task in-context use.

GVL and No-Shuffling Ablation

  • GVL produces temporally varied value predictions, whereas removing frame shuffling collapses predictions into a few monotonic patterns.The qualitative comparison identifies shuffling as important for recovering informative temporal structure.Source: D. Additional Results · p. 20, Figure 11 · p. 23

Zero-Shot ALOHA Simulation

  • GVL-SD consistently outperforms SuccessVQA for zero-shot success-filtered imitation learning, and gains persist across VOC thresholds.The result is reported on ALOHA tasks and is qualitatively similar to the one-shot setting.Source: D. Additional Results · p. 23, Figure 12 · p. 23

Different VLM Backbone

  • Across backbone VLMs, GVL produces VOC distributions skewed toward coherent temporal ordering, with backbone differences mainly reflected in refusal and response-template conformity.The evaluation aggregates 1000 VOC scores across trajectories, while the figure highlights the tall bar at −1.0 associated with these response behaviors.Source: D. Additional Results · pp. 23–24, Figure 13 · p. 24

Cross-Task In-Context Learning

  • Cross-task demonstrations improve GVL’s VOC relative to no cross-task example, although same-task examples improve more because they provide target-specific progress cues.Cross-task in-context learning nevertheless provides foundation-model guidance without a task-specific prior.Source: D. Additional Results · p. 24, Figure 14 · p. 24

Camera Viewpoints

  • GVL works best zero-shot on the Table viewpoint on average, but in-context examples consistently improve performance across all camera viewpoints.Few-shot prompting can compensate when a viewpoint is later found to be sub-optimal, including gripper views that are more out-of-distribution for VLM training.Source: D. Additional Results · pp. 24–25, Figure 15 · p. 25

Task Specification

  • GVL achieves the highest VOC for matching video and language descriptions on 9 of 13 tasks, while no shuffling sharply reduces task-sensitive predictions.For many unmatched pairs, GVL refuses to predict because the frames and description are unrelated; the shuffled and no-shuffling comparisons appear in Figures 16 and 17.Source: D. Additional Results · p. 25, Figure 16 · p. 26, Figure 17 · p. 27

Related papers

Loading 2411.04549v1…