PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Yanru Wu, Weiduo Yuan, Ang Qi, Vitor Guizilini, Jiageng Mao, Yue Wang

arXiv:2603.16065v2cs.ROcs.AI

TL;DR

Robot RL refinement is bottlenecked by the difficulty of designing generalizable, dense reward functions. The paper adapts foundation VLMs into frame-level reward generators and reports improved downstream RL performance on ManiSkill3, outperforming state-of-the-art video-based generative reward models.

  • Problem

    RL refinement is bottlenecked by the difficulty of obtaining accurate, dense, informative, and generalizable rewards without labor-intensive human labeling or brittle task-specific objectives.

    Source: §I · p. 1

  • Method

    Large Reward Models adapt foundation VLMs into dense, frame-level reward generators trained on diverse data and used to refine an initial imitation-learning policy.

    Source: §I · pp. 1–2

  • Results

    LRM-generated rewards improve downstream RL performance on ManiSkill3 and targeted reward supervision outperforms state-of-the-art video-based generative reward models.

    Source: §V · p. 8

  • Takeaways & Limitations

    In real-world experiments, Task Completion Reward serves as an autonomous supervisor for iterative policy refinement through automated trajectory filtering.

    Source: §V · p. 8

Abstract

from arXiv · show

Reinforcement Learning (RL) has shown great potential in refining robotic manipulation policies, yet its efficacy remains strongly bottlenecked by the difficulty of designing generalizable reward functions. In this paper, we propose a framework for online policy refinement by adapting foundation VLMs into online reward generators. We develop a robust, scalable reward model based on a state-of-the-art VLM, trained on a large-scale, multi-source dataset encompassing real-world robot trajectories, human-object interactions, and diverse simulated environments. Unlike prior approaches that evaluate entire trajectories post-hoc, our method leverages the VLM to formulate a multifaceted reward signal comprising process, completion, and temporal contrastive rewards based on current visual observations. Initializing with a base policy trained via Imitation Learning (IL), we employ these VLM rewards to guide the model to correct sub-optimal behaviors in a closed-loop manner. We evaluate our framework on challenging long-horizon manipulation benchmarks requiring sequential execution and precise control. Crucially, our reward model operates in a purely zero-shot manner within these test environments. Experimental results demonstrate that our method significantly improves the success rate of the initial IL policy within just 30 RL iterations, demonstrating remarkable sample efficiency. This empirical evidence highlights that VLM-generated signals can provide reliable feedback to resolve execution errors, effectively eliminating the need for manual reward engineering and facilitating efficient online refinement for robot learning.

I. INTRODUCTION

The paper addresses the bottleneck of generalizable, dense rewards for refining imitation-learned robot policies. It proposes frame-level VLM reward generation and reports improved performance on zero-shot long-horizon tasks.

I. INTRODUCTION

  • Generalizable, accurate, dense rewards remain a bottleneck for RL refinement because human labeling and hand-coded objectives are labor-intensive or task-specific.IL policies can plateau on tasks requiring high precision or long-horizon coordination.Source: §I · p. 1
  • Prior VLM rewarders score entire episodes post hoc, use expanding visual histories, or learn proxy rewards from interactive observation preferences [1] [2] [3].The paper identifies sparse delayed feedback and growing inference latency as constraints of the first two approaches.Source: §I · p. 1
  • LRMs adapt foundation VLMs into dense, frame-level reward generators trained on data from 24 sources, including robot, human-object interaction, and simulation videos.The named sources include Open X-Embodiment, HOI4D, EgoDex, LIBERO, and RoboCasa.Source: §I · p. 1
  • The reward formulation combines temporal contrastive relative progress, continuous absolute progress, and terminal task-completion signals.The modalities respectively compare temporal frames, estimate completion from a frame, and verify semantic task requirements.Source: §I · pp. 1–2
  • The framework is evaluated on long-horizon ManiSkill tasks that are entirely zero-shot to its reward models.The authors present this setting as a test of physical generalization rather than environment-specific overfitting.Source: §I · p. 2
  • LRM-generated rewards resolve execution errors and significantly improve policy performance within a highly efficient number of RL iterations.The evaluation covers complex, zero-shot long-horizon tasks.Source: §I · p. 2

II. RELATED WORK

Related work spans real-robot RL, learned reward models, process-level dense supervision, and autonomous closed-loop improvement. These approaches address policy refinement through simulation transfer, learned or generated rewards, and failure-driven adaptation.

Real-robot reinforcement learning

  • Real-robot RL research has progressed from simulation transfer using domain or dynamics randomization to locomotion and manipulation, with recent work refining generalist policies through multi-stage pipelines [19] [20] [25].Earlier direct real-world RL had limited success [11] [12].Source: §II · p. 2

Learned reward models for robotics

  • Learned reward approaches range from video- and trajectory-based rewards to universal representations, LLM-optimized reward code, temporal progress estimation, token-probability scoring, and trajectory comparison [29] [31] [33] [2].The passage also describes RoboReward as a systematization of VLMs as general reward predictors [1].Source: §II · p. 3

Process-level supervision and dense feedback

  • Process reward models provide step-level supervision, including robotics adaptations for manipulation, while Progressor uses self-supervised temporal learning on expert videos [35] [36].Source: §II · p. 3

Autonomous self-improvement and closed-loop control

  • Autonomous improvement methods analyze failures to synthesize rewards, coevolve policies with video world models, or constrain exploration to valid action manifolds [38] [39] [40].Source: §II · p. 3

III. METHODOLOGY

The methodology adapts foundation VLMs into three frame-level reward modalities for online policy refinement. It constructs training supervision from temporally sampled videos drawn from diverse domains and sources.

III. METHODOLOGY

  • Large Reward Models map visual observations and task descriptions to dense, frame-level rewards in three modalities for closed-loop policy refinement.The modalities are Temporal Contrastive, Absolute Progress, and Task Completion Rewards.Source: §III · p. 3

A. Extracting Reward Signals from Multi-domain Unlabeled Videos

  • The training dataset is built by extracting keyframes and progress labels from unlabeled trajectories aggregated from 24 sources, using temporal monotonicity as supervision.The dataset is represented as task descriptions, frames, and progress values.Source: A. Extracting Reward Signals from… · p. 3
  • The dataset combines robot corpora, human-object interaction videos, and simulated benchmarks to support broad zero-shot generalization.The sources include Open X-Embodiment, HOI4D, EgoDex, LIBERO, and RoboCasa.Source: A. Extracting Reward Signals from… · p. 3
  • Each episode is sampled at 11 normalized temporal progress points from 0.0 to 1.0 and paired with its original task description.The sampling is intended to provide a consistent, monotonic mapping between visual states and task completion levels.Source: A. Extracting Reward Signals from… · p. 3

B. Tri-faceted Reward Formulation

The tri-faceted reward formulation evaluates task states through relative progress, continuous completion estimates, and binary goal verification. Together, the modalities provide complementary supervision derived from temporal progress in multi-domain videos.

B. Tri-faceted Reward Formulation

  • The three reward formats are derived from temporal progress in multi-domain videos and are designed to provide complementary supervision for policy refinement.Source: B. Tri-faceted Reward Formulation · p. 4
  • Temporal Contrastive Reward compares current and previous frames to assign +1.0, -1.0, or 0.0 according to relative closeness to the goal.The comparison is intended to provide a dense directional signal while mitigating absolute-score calibration issues.Source: B. Tri-faceted Reward Formulation · p. 4
  • Absolute Progress Reward regresses a normalized completion score from the current observation, initial state, and task description.It maps execution onto eleven discrete completion stages for granular progress estimates.Source: B. Tri-faceted Reward Formulation · p. 4
  • Task Completion Reward classifies whether the current state satisfies the task’s semantic requirements, returning 1 for success and 0 otherwise.The binary signal anchors goal attainment within long-horizon sequences.Source: B. Tri-faceted Reward Formulation · p. 4

C. Training LRMs for Reward Generation

The paper specializes a vision-language model to generate reward signals from visual observations and task descriptions. It trains contrastive, progress, and completion modalities with distinct objectives.

C. Training LRMs for Reward Generation

  • The Qwen3-VL-8B-Instruct model is adapted with LoRA to map visual observations and task descriptions to reward intensities.Progress and completion models use the current observation, optionally anchored by the initial observation; contrastive inputs use temporal image pairs.Source: C. Training LRMs for Reward Generation · p. 4
  • The contrastive model uses Direct Preference Optimization to judge which of two temporal frames is closer to task completion.A reasoning step precedes its +1, −1, or 0 reward, grounding judgments in physical interactions such as object displacement.Source: C. Training LRMs for Reward Generation · p. 4
  • Progress estimation and completion judgment are trained with supervised fine-tuning to predict continuous progress and binary success, respectively.The progress model jointly predicts reasoning and its label, while the completion model directly predicts the binary signal.Source: C. Training LRMs for Reward Generation · p. 4
  • The completion model’s goal-satisfaction judgment supplies a definitive terminal signal for online policy refinement.The paper describes this signal as reinforcement for precise refinement.Source: C. Training LRMs for Reward Generation · p. 4

D. Online Policy Refinement via LRM-Generated Rewards

The method refines an imitation-initialized policy through closed-loop interaction, using frozen LRM rewards to guide policy updates. Rewards are held between queries, and PPO with GAE performs the optimization.

D. Online Policy Refinement via LRM-Generated Rewards

  • A policy initialized from imitation learning interacts with the environment in a closed loop while frozen LRMs provide online rewards.The process shifts the policy from passive imitation to reward-driven control.Source: D. Online Policy Refinement via… · p. 4
  • LRM rewards are cached and held for K steps, creating a continuous supervisory stream for recognizing and correcting execution errors.The models are queried at intervals to generate a reward for the active modality.Source: D. Online Policy Refinement via… · pp. 4–5
  • Proximal Policy Optimization updates the policy parameters to maximize expected return.Generalized Advantage Estimation computes the advantage used with dense LRM feedback.Source: D. Online Policy Refinement via… · p. 5
  • The advantage uses temporal-difference error calculated from LRM rewards, anchoring policy gradients in semantically grounded feedback.The paper describes this mechanism as helping resolve sub-optimal behaviors through physical priors encoded in LRM feedback.Source: D. Online Policy Refinement via… · p. 5

IV. EXPERIMENTAL RESULTS

The evaluation examines both reward-signal quality and the usefulness of LRM rewards for policy refinement. It includes zero-shot simulation benchmarks and real-world trajectory filtering for iterative policy fine-tuning.

IV. EXPERIMENTAL RESULTS

  • The evaluation has two stages: testing semantic grounding across three reward modalities, then using LRM rewards for online policy refinement.The modalities are Temporal Contrastive, Absolute Progress, and Task Completion.Source: §IV · p. 5
  • ManiSkill3 simulation experiments assess zero-shot reward-guided policy refinement across diverse object identities.The reward models are evaluated on a benchmark described as zero-shot to them.Source: §IV · p. 5
  • In real-world deployment, the completion reward automatically filters successful hardware-rollout trajectories for iterative policy fine-tuning.Source: §IV · p. 5

A. Evaluating Large Reward Models (LRMs)

The specialized LRM improves contrastive ranking and progress-estimation performance over the zero-shot baseline, while completion accuracy changes only marginally. The reported comparisons cover held-out reward evaluations and cumulative accuracy across tolerance thresholds.

A. Evaluating Large Reward Models (LRMs)

  • The evaluation compares specialized reward models with the zero-shot Qwen3-VL-8B-Instruct baseline on held-out sets across three modalities.The assessment aims to measure specialization gains and the stability and semantic grounding of rewards for downstream reinforcement learning.Source: A. Evaluating Large Reward Models (LRMs) · p. 5
  • 15.3% gains in both Kendall’s τ and Spearman’s ρ show improved temporal ranking correlation over the zero-shot baseline.These metrics assess consistent temporal orderings for a directional gradient.Source: A. Evaluating Large Reward Models (LRMs) · p. 5, Table I · p. 5
  • 8.6 percentage points higher accuracy at ±0.2, with MAE down 20.0% and RMSE down 19.3%, shows improved progress estimation.The LRM outperforms the baseline across tolerance thresholds, with its largest gains in the high-precision regime; Table II summarizes progress-estimation metrics.Source: A. Evaluating Large Reward Models (LRMs) · p. 5, Table II · p. 5, Figure 2 · p. 6
  • 69.38% accuracy for the fine-tuned LRM versus 69.23% for the zero-shot baseline indicates only a marginal completion-model gain.The paper suggests the base model already has robust zero-shot semantic goal-recognition capability.Source: A. Evaluating Large Reward Models (LRMs) · pp. 5–6

B. Policy Refinement on the ManiSkill3 Benchmark

On ManiSkill3, all three LRM reward modalities improve closed-loop success over the πSFT baseline, and the progress reward outperforms the compared video-based progress models. Open-loop reward-quality measures also improve after RL training.

B. Policy Refinement on the ManiSkill3 Benchmark

C. Real-World Policy Self-Improvement

The real-world evaluation adapts an SFT pick-and-place policy using LRM task-completion rewards to filter trajectories. The refined policy improves task success over the SFT baseline.

C. Real-World Policy Self-Improvement

  • The pick-and-place evaluation starts from an SFT policy fine-tuned on 20 teleoperated demonstrations and uses 60 real-world rollouts capped at 30 seconds.The task is to grasp a toy giraffe and place it inside a target bowl.Source: C. Real-World Policy Self-Improvement · p. 7
  • Task Completion Reward classifies terminal goal satisfaction, and successful transitions are isolated for iterative policy refinement.A terminal reward of rcomp = 1 marks goal satisfaction and provides an automated labeling signal.Source: C. Real-World Policy Self-Improvement · p. 7
  • 51.7% success after refinement, up from 38.3% for the SFT baseline.The rollout comparison shows the refined policy placing the giraffe in the bowl where the baseline failed.Source: C. Real-World Policy Self-Improvement · p. 7

V. CONCLUSIONS

The conclusion presents LRMs as frame-level reward models with three modalities for online policy refinement. The paper reports improved ManiSkill3 and real-world policy performance, and describes reward quality becoming more aligned with policy behavior during RL.

V. CONCLUSIONS

  • LRMs provide Temporal Contrastive, Absolute Progress, and Task Completion reward modalities as frame-level supervision for policy refinement.The conclusion describes this targeted reward supervision as outperforming state-of-the-art video-based generative reward models.Source: §V · p. 7
  • On ManiSkill3, LRM-generated rewards improve downstream RL performance and narrow the gap to privileged ground-truth environment rewards without manual reward engineering.The Env Reward uses privileged simulator states unavailable to vision-only models.Source: §V · p. 8
  • In real-world experiments, Task Completion Reward serves as an autonomous supervisor for iterative policy refinement through trajectory filtering.The conclusion reports higher real-world task success than the SFT baseline.Source: §V · p. 8, Table VII · p. 8
  • As policies internalize LRM physical priors, their trajectories develop clearer semantic markers and physical transitions that are more interpretable to the model.The conclusion describes this as synchronization between RL-driven behavior and LRM-perceived value.Source: §V · p. 8
Loading 2603.16065v2…