PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboReward: General-Purpose Vision-Language Reward Models for Robotics

Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, Chelsea Finn

arXiv:2601.00675v2cs.RO

TL;DR

RoboReward addresses the limited availability of accurate, informative rewards for real-world robotic reinforcement learning, where existing approaches rely on human labeling or brittle handcrafted objectives. It constructs a benchmark and augmented dataset for vision-language reward models, trains RoboReward 4B/8B, and finds that RoboReward 8B performs best overall while improving real-robot policy learning over Gemini Robotics-ER 1.5 and narrowing the gap to human-provided rewards.

  • Problem

    Real-world robotic reinforcement learning lacks accurate, informative reward models and typically depends on labor-intensive human labels or brittle handcrafted reward functions.

  • Method

    The paper builds RoboReward and RoboRewardBench from diverse real-robot episodes, augments success-heavy data with counterfactual and temporally clipped negatives, and trains 4B/8B vision-language reward models.

  • Results

    RoboReward 8B achieves the best overall accuracy across 22 models on RoboRewardBench and improves real-robot task success over Gemini Robotics-ER 1.5 while narrowing the gap to human-provided rewards.

  • Takeaways & Limitations

    Targeted reward supervision can outperform substantially larger general-purpose VLMs, and offline reward accuracy can serve as a meaningful proxy for downstream real-world RL usefulness.

  • Takeaways & Limitations

    Performance is not uniform across embodiments and scenes, and future work must extend reward modeling to longer-horizon, multi-stage tasks with harder credit assignment and progress estimation.

Abstract

from arXiv · show

A well-designed reward is critical for effective reinforcement learning-based policy improvement. In real-world robotics, obtaining such rewards typically requires either labor-intensive human labeling or brittle, handcrafted objectives. Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood. In this work, we aim to close this gap by introducing (1) RoboReward, a robotics reward dataset and benchmark built on large-scale real-robot corpora from Open X-Embodiment (OXE) and RoboArena, and (2) vision-language reward models trained on this dataset (RoboReward 4B/8B). Because OXE is success-heavy and lacks failure examples, we propose a negative examples data augmentation pipeline that generates calibrated negative and near-misses via counterfactual relabeling of successful episodes and temporal clipping to create partial-progress outcomes from the same videos. Using this framework, we build a large training and evaluation dataset spanning diverse tasks and embodiments to test whether state-of-the-art VLMs can reliably provide rewards for robot learning. Our evaluation of open and proprietary VLMs finds that no model excels across tasks, highlighting substantial room for improvement. We then train general-purpose 4B- and 8B-parameter models that outperform much larger VLMs in assigning rewards for short-horizon robotic tasks. Finally, we deploy the 8B model in real-robot reinforcement learning and find that it improves policy learning over Gemini Robotics-ER 1.5 while narrowing the gap to RL training with human-provided rewards. We release the full dataset, trained reward models, and evaluation suite on our website to advance the development of general-purpose reward models in robotics: https://crfm.stanford.edu/helm/robo-reward-bench (project website).

1 INTRODUCTION

RoboReward addresses the lack of accurate, informative reward models for real-world robotic reinforcement learning by introducing a diverse benchmark, augmented data, and trained VLM reward models. Its models outperform larger VLMs, while evaluations show current models remain unreliable across robotic settings.

  • Real-world robotic RL remains limited by labor-intensive human reward labeling and brittle handcrafted objectives.
  • RoboReward combines a real-robot reward dataset, benchmark, negative-example augmentation, and general-purpose RoboReward 4B/8B models.The augmentation pipeline uses counterfactual relabeling and truncated successful rollouts to produce wrong, near-miss, and partial-progress examples.
  • 54,135 automatically generated examples augment success-heavy demonstration data with wrong and near-miss outcomes, with human verification for the test set.
  • RoboRewardBench evaluates 22 VLMs on 2,831 robot episodes spanning diverse tasks, 14 embodiment types, and egocentric and exocentric views.
  • The released dataset, benchmark, checkpoints, and evaluation suite are intended to support general-purpose reward-model development in robotics.
  • RoboReward 4B and 8B rank first and fourth among 22 models, while RoboReward 8B improves real-robot RL over Gemini Robotics-ER 1.5 and narrows the gap to human rewards.

2 RELATED WORK

Prior robotics reward work spans learned trajectory rewards, VLM preferences, video-derived rewards, and specialized settings. RoboReward differs by evaluating general-purpose end-of-episode reward prediction across diverse real robots and both successful and unsuccessful trajectories.

  • Real-robot reinforcement learning: Real-robot RL research has progressed across locomotion, manipulation, and generalist-policy fine-tuning, but reward design continues to limit broad application.
  • Learned reward models for robotics: Learned robotics rewards include methods based on human videos, robot trajectories, VLM preferences, individual videos, and in-context value learning.
  • Learned reward models for robotics: Other learned reward methods target narrower domains such as legged locomotion, autonomous driving, text-to-video rewards, and real-to-sim keypoints.
  • Learned reward models for robotics: RoboReward targets general-purpose end-of-episode reward prediction across broad real-robot tasks and embodiments, with a unified progress rubric.
  • Learned reward models for robotics: Compared with OpenGVL, RoboReward evaluates more VLMs on successful and unsuccessful trajectories across more tasks and embodiments, and releases prompts and raw predictions.OpenGVL evaluates 14 VLMs on six tasks using successful demonstration videos and a Value-Order Correlation metric.
  • Non-robot reward models: Language-model reward benchmarks such as RewardBench and RewardBench 2 provide related evaluation precedents outside robotics.

3 THE ROLE OF REWARD IN REINFORCEMENT LEARNING

The paper studies which episodic reward formulations best support robotic RL and whether offline reward accuracy predicts downstream policy performance. Simulation results favor progress-based rewards, and reward accuracy strongly correlates with RL outcomes.

  • RL seeks a policy π that maximizes expected discounted reward V^π, so reward specification must align learned behavior with the desired robotic objective.
  • Discrete progress is selected for RoboReward because it combines fast convergence with more consistent human annotation than fully continuous rewards.
  • The study compares binary success, continuous progress, and five-bin discrete progress rewards learned from full-episode videos.
  • Progress-based rewards converge faster than learned binary rewards, while continuous and discrete progress rewards converge comparably quickly across three Robomimic tasks.
  • r = 0.83: higher reward accuracy predicts better downstream RL performance across reward-model checkpoints and all three Robomimic tasks.Accuracy is measured using held-out mean absolute error, while the plotted x-axis is maximum possible MAE minus model MAE.
  • Held-out offline reward accuracy is an effective signal for estimating downstream RL performance when using the corresponding reward model.

4 THE ROBOREWARD DATASET AND BENCHMARK

RoboReward combines diverse real-robot videos from OXE and RoboArena with counterfactual and temporal augmentation to create calibrated progress-labeled examples. It provides a benchmark and trained reward models for evaluating and improving VLM-based robotic rewards on unseen tasks.

  • Data Sources: OXE contributes approximately 1M demonstrations across 22 embodiments, while RoboArena adds diverse DROID evaluations containing successful and naturally failed rollouts.
  • Negative Examples Data Augmentation: Counterfactual relabeling generates alternative instructions for the same successful video, producing partial-success or failure instruction-video pairs without fabricating videos.
  • Negative Examples Data Augmentation: Negative clipping creates partial-progress outcomes by ending successful videos at early, middle, or late rollout fractions while preserving the original task instruction.
  • Rubric and Validation: A fixed five-level rubric ranges from no success to perfect completion, and VLM validation retains only coherent, video-grounded labels whose scores match the rubric.
  • Data Cleaning and Augmentation: The offline label-generation pipeline is too slow and operationally complex for large-scale RL loops, so it serves as a teacher for distilling rewards into a single-pass model.
  • Dataset Splits: The augmented corpus contains 54,135 automatically generated examples, while task descriptions remain disjoint across train, validation, and test splits.
  • Benchmark: RoboRewardBench is formed by human-verifying test examples and discarding labels that are not justified by the rollout video and task description.

5 EXPERIMENTS

The experiments benchmark VLM reward models, test RoboReward in real-world policy improvement, and isolate the effects of negative-example augmentation. RoboReward models achieve strong benchmark accuracy and improve real-robot RL, while broad generalization still varies across subsets.

  • 5.1 BENCHMARKING FRONTIER VLMS WITH ROBOREWARDBENCH: RoboReward 8B achieves the lowest overall RoboRewardBench MAE, 0.665, outperforming evaluated frontier VLMs including GPT-5 mini at 0.691.On the RoboArena subset, RoboReward 8B and 4B also outperform the evaluated frontier VLMs.
  • 5.1 BENCHMARKING FRONTIER VLMS WITH ROBOREWARDBENCH: Targeted supervision improves MAE over same-size Qwen3-VL Instruct by 0.227 for 8B models and 0.187 for 4B models.The 8B comparison is 0.892 to 0.665, while the 4B comparison is 1.032 to 0.845.
  • 5.1 BENCHMARKING FRONTIER VLMS WITH ROBOREWARDBENCH: Benchmark accuracy varies substantially across embodiments, scenes, and viewpoints, so no model is uniformly strong across RoboRewardBench subsets.GPT-5 and Gemini 3 Pro each show large differences between their strongest and weakest subsets.
  • 5.2 TRAINING REAL-ROBOT POLICIES WITH VLM REWARD MODELS: RoboReward 8B improves success from 5% to 50% on pick-and-place monkey and from 10% to 80% on open drawer during real-world RL.Human rewards reach 75% and 90%, while RoboReward 8B substantially outperforms Gemini Robotics-ER 1.5.
  • 5.2 TRAINING REAL-ROBOT POLICIES WITH VLM REWARD MODELS: RoboReward 8B has lower RoboRewardBench MAE than Gemini Robotics-ER 1.5, 0.665 versus 0.906, and induces greater downstream RL gains.The ordering of models by benchmark performance is consistent with the real-world RL performance they induce.
  • 5.3 DATA MIXTURE ABLATIONS: Removing clipped negatives worsens overall MAE from 0.845 to 1.075 while changing RoboArena MAE only from 0.806 to 0.813.The ablations indicate that clipping broadens failure-mode coverage without harming organic performance.

6 DISCUSSION

RoboReward introduces a dataset, benchmark, evaluation suite, and reward models for generalist VLM-based robotic reward prediction. Results show targeted supervision improves benchmark accuracy and downstream RL, while generalization remains uneven across embodiments and scenes.

  • RoboReward provides a training dataset, RoboRewardBench, a human-verified evaluation suite, and finetuned 4B/8B reward VLMs.
  • Across 22 frontier and open-weight models, RoboReward 8B achieves the best overall RoboRewardBench accuracy.
  • Improved offline reward accuracy translates into improved downstream RL performance in simulation and real-robot settings.
  • Current frontier VLMs exhibit large, non-uniform generalization gaps across embodiments and scenes, while small reward mistakes can meaningfully affect policy improvement.
  • The authors identify longer-horizon, multi-stage tasks as an important direction because credit assignment and progress estimation become more challenging.

A.1 DATASET SOURCES

The RoboReward corpus combines diverse real-robot datasets spanning many embodiments, viewpoints, environments, and manipulation tasks. The resulting corpus contains 54,135 examples across training, validation, and test splits.

  • 54,135 examples comprise the corpus: 45,072 training, 6,232 validation, and 2,831 test examples.
  • The sources cover exocentric and egocentric recordings from robot arms including Franka, xArm, WidowX, PR2, Jaco, Sawyer, UR5, and others.
  • Tasks span tabletop and household manipulation, including pick-and-place, stacking, pouring, opening drawers, kitchen activities, folding, sweeping, and object insertion.
  • Several datasets include multi-step or complex interactions, such as preparing food, setting tables, manipulating appliances, and combining objects with tools.
  • The dataset sources include varied objects, scenes, camera perspectives, and manipulation requirements, supporting broad task and embodiment coverage.

A.2 DATA CLEANING AND AUGMENTATION DETAILS

The cleaning and augmentation pipeline converts success-heavy robot demonstrations into calibrated progress examples and validates reward labels before constructing the benchmark. It uses video analysis, command generation, rubric-based checking, and human verification.

  • Negative example generation: Successful episodes with reward 5 yield counterfactual commands scored 1–4 and clipped videos scored 1–4 for the original command.
  • Negative example generation: The generation process uses GPT-5 mini for video analysis and planning, and Qwen3-4B-Instruct-2507 for command generation and text correction.
  • Video analysis: Video analysis samples rollout frames at 1 FPS, includes the final frame, and describes scenes, robot actions, and final states relative to the task.
  • Negative example generation: The planning step enforces strictly ordered scores 1 < 2 < 3 < 4 < 5 using distinct, visible failure modes and progressively closer final states.
  • Automated validation: Each example is checked for coherent grounding and rubric-consistent scores, with rejection criteria covering mismatched objects, invalid tasks, and inconsistent relations.
  • Human verification: The test split is human-verified, retaining only examples whose end-of-episode reward labels are justified by the video and task description.

B.1 ROBOMIMIC EXPERIMENTS

The Robomimic experiments use DSRL-NA with environment-specific hyperparameters reported in the accompanying tables. These tables provide the common and environment-specific settings for the experiments.

  • DSRL-NA is used for all Robomimic experiments.
  • Environment-specific hyperparameters follow the settings specified by Wagenmaker et al. (2025).
  • Tables 5 and 6 report the common and environment-specific DSRL hyperparameters.

B.2 BENCHMARKING WITH ROBOREWARDBENCH

RoboRewardBench reports model performance by dataset subset, ranking models with mean absolute error. Qualitative failures include both false-positive and false-negative progress scores from Gemini Robotics-ER 1.5.

  • Models are ranked by overall group-wise mean absolute error, with lower values indicating better performance.The benchmark table also reports absolute error for each dataset subset.
  • Gemini Robotics-ER 1.5 assigns 5/5 to a failed drawer pull and 2/5 to a successful drawer pull.The figure presents these as false-positive and false-negative reward predictions, respectively.

B.3 THE PITFALLS OF STATE-OF-THE-ART VLMS AS REWARD MODELS

Frontier VLMs can understand scenes and broad actions while still making consequential reward errors on fine-grained grasping, opening, and placement outcomes. Gemini Robotics-ER 1.5 produces both false positives and false negatives on real robot rollouts, including during RL training.

  • Representative reward mistakes: Gemini Robotics-ER 1.5 gives 5/5 when a drawer remains closed, 2/5 when another drawer opens, and 5/5 when placement is unclear.These errors respectively over-reward a failure, under-reward a success, and hallucinate a completed placement.
  • Representative reward mistakes: The model correctly recognizes objects and broad intent but incorrectly asserts a precise grasp and successful pull not clearly shown in the rollout.
  • Representative reward mistakes: For the drawer-opening success, the description accurately tracks approach and alignment but stops before narrating the decisive task outcome.
  • Representative reward mistakes: For the monkey task, the model claims placement on the towel even though the monkey may remain held above it in the final frame.The unsupported release or placement event leads to an incorrect 5/5 score.
  • Implications: These examples indicate that reward modeling depends on fine-grained spatial and temporal details that frontier VLMs can miss despite high-level scene understanding.The relevant distinctions include secure grasping, drawer opening, and true placement versus hovering.

B.4 REAL-WORLD RL EXPERIMENT

The real-world RL experiment uses a diffusion-policy base and DSRL-SAC training, with hyperparameters documented for the policy and RL procedure. Benchmark tables report model rankings and subset errors.

  • Method: The base diffusion policy uses a diffusion transformer pretrained on BridgeData V2 with goal-image conditioning.DSRL modifies the input noise to the policy’s denoising process.
  • Method: DSRL-SAC warm-starts replay with 20 base-policy rollouts before reinforcement-learning training begins.All reward models use the same DSRL hyperparameters.
  • Experimental configuration: The experiment documents separate hyperparameter tables for the base diffusion policy and DSRL WidowX training.
  • Benchmark reference: RoboRewardBench reports full benchmarking results with models ranked by mean absolute error and columns for each dataset subset.
Loading 2601.00675v2…