PaperlayerView exact revision on arXiv

Source-linked AI summary

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, Jesse Zhang

arXiv:2603.02115v2cs.ROcs.AIcs.LG

TL;DR

Absolute progress supervision is difficult to scale to failed and suboptimal robot trajectories because dense labels are ambiguous. Robometer combines frame-level progress with trajectory preferences and trains on RBM-1M, achieving stronger reward generalization and downstream robot-learning performance, while remaining limited by unobserved physical states and incomplete failure coverage.

  • Problem

    Absolute progress labels are ambiguous and costly for failed trajectories, limiting the use of abundant suboptimal data and reward-model scalability.

  • Method

    Robometer combines an expert frame-level progress loss with a trajectory-comparison preference loss and trains on RBM-1M with real and augmented failure data.

  • Results

    Across diverse real-world robot-learning applications, ROBOMETER achieves 2.4 −4.5× higher success rates than the best baseline in each category.

  • Takeaways & Limitations

    Pairwise preferences, dense progress targets, and failure data together produce more generalizable reward functions and improve robot learning across multiple applications.

  • Takeaways & Limitations

    ROBOMETER may miss long-horizon events, rare or task-specific failures, latent physical states, and safety-critical states that are not visually observable or represented in training.

Abstract

from arXiv · show

General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.

I. INTRODUCTION

Robometer addresses the limits of absolute, trajectory-local progress labels by adding global trajectory preferences, enabling scalable use of suboptimal and failed data. The resulting model and dataset improve reward generalization and downstream robot learning.

  • Motivation: Absolute progress labels are easy for experts but ambiguous and costly for failed attempts, limiting scalability and generalization.Failed trajectories may fluctuate in progress, making dense annotation difficult and preventing effective use of abundant suboptimal data.
  • Motivation: Trajectory preferences complement frame-level progress labels by imposing global ordering constraints across diverse trajectories.Relative comparisons can be curated without additional human annotation and provide supervision beyond individual rollouts.
  • Method: Robometer combines expert frame-level progress loss with trajectory-comparison preference loss in a dual reward-prediction objective.Preference supervision improves discrimination between successful and suboptimal trajectories, while additional unlabeled suboptimal data further improves performance.
  • Dataset: RBM-1M contains over one million trajectories from 21 robot platforms, including substantial suboptimal and failed data.The dataset spans single-arm, bimanual, and mobile manipulators, human demonstrations, and augmented preference pairs.
  • Results: 2.4 −4.5× higher success rates than the best baseline were achieved across diverse real-world robot learning applications.The applications include online RL, offline RL, imitation-learning dataset filtering, and zero-shot failure detection.

II. RELATED WORK

Prior reward-learning methods rely on task-specific demonstrations, preference supervision, or foundation-model representations, but often remain limited in task or domain generalization. Robometer combines these directions into a scalable, general-purpose reward-modeling framework.

  • Reward Learning from Demonstrations: Inverse reinforcement learning commonly infers rewards from human or expert demonstrations but often requires new task-specific demonstrations when tasks or reward specifications change.Robometer instead targets reward models that generalize to new tasks.
  • Preference-Based Reward Learning: Preference-based reward learning uses relative comparisons rather than absolute numerical scales, with rewards identifiable up to monotone transformations.Such supervision can nevertheless support effective policy optimization.
  • Rewards from Foundation Models: Foundation-model approaches generate or shape rewards from images, videos, language, or language-conditioned state representations.These methods broaden reward construction beyond conventional task-specific formulations.
  • Progress-Based Reward Models: Zero-shot VLM reward prediction can be noisy, while smaller per-task models may overfit domain-specific visual and semantic features.These limitations constrain generalization across tasks and environments.
  • Progress-Based Reward Models: Recent VLM and VLA methods improve progress prediction through fine-tuning, relative progress targets, distance-to-goal values, or discretized progress labels.Robometer differs by introducing a more scalable framework combining dense progress rewards with trajectory preferences.

III. SCALABLE REWARD MODEL TRAINING

The proposed recipe instantiates reward modeling with ROBOMETER by combining diverse data, a pretrained VLM backbone, and complementary frame-level and trajectory-level supervision.

  • Core Recipe: ROBOMETER provides dense rewards for robot manipulation through a scalable training recipe.The recipe is designed as a general reward-modeling framework rather than a task-specific demonstration method.
  • Core Recipe: The recipe uses unlabeled failure trajectories, a pretrained VLM backbone, and an objective combining per-frame rewards with trajectory preferences.These three pillars jointly support diversity, generalization, and comparative supervision.
  • Core Recipe: Trajectory preferences supplement dense progress supervision with comparisons across behaviors of differing quality.This enables the framework to use failure trajectories that lack absolute progress labels.

A. RBM-1M Dataset

RBM-1M aggregates progress-labeled and unlabeled trajectories while emphasizing diversity across viewpoints, scenes, and robot embodiments. It contains over one million trajectories from 21 robot embodiments.

  • Dataset Representation: RBM-1M represents trajectories as image observations, a language instruction, and an optional scalar progress label.Expert demonstrations use progress 1.0, partially labeled datasets retain provided scores, and unlabeled failures use no progress label.
  • Data Composition: The dataset aggregates expert robot data, human videos, and additional sources to emphasize viewpoint, scene, and embodiment diversity.Its composition prioritizes diversity rather than simply maximizing trajectory quantity.
  • Dataset Scale: Over one million trajectories span 21 robot embodiments, with separate in-distribution and out-of-distribution evaluation datasets.RBM-EVAL-ID and RBM-EVAL-OOD are used to evaluate reward-model generalization.

B. ROBOMETER Architecture and Tokenization

ROBOMETER uses a causally masked VLM with learned tokens that separately support single-trajectory progress inference and paired-trajectory preference prediction.

  • ROBOMETER builds on QWEN3-VL-4B-INSTRUCT to process one video for reward inference or two videos for preference training.
  • Learned progress tokens are interleaved within the first video, while a preference token is appended after a separator between trajectories.The separator demarcates the two video trajectories, and the preference token aggregates information from the instruction and both videos.
  • Causal masking restricts progress tokens to current and previous frames of the first trajectory, while the preference token attends to both trajectories.This produces dense frame-level progress estimates for single-trajectory inference and relative judgments for paired trajectories.
  • Both trajectories are fixed to length T = 8 to prevent trajectory length from serving as a proxy for quality.

C. Training Objectives

ROBOMETER combines preference, progress, and success objectives so rewards retain absolute task-progress calibration while capturing trajectory quality differences.

  • The composite objective is L = Lpref + Lprog + Lsucc, combining preference, progress, and success losses.This jointly trains relative trajectory comparison with frame-level progress and success prediction.
  • A binary preference classifier predicts which trajectory better satisfies the language instruction from the preference-token representation.The target y identifies the preferred trajectory.
  • For expert demonstrations, ROBOMETER predicts continuous frame-level progress using ten categorical bins over [0, 1] and cross-entropy supervision.Ground-truth progress is defined as pt = t/T, projected onto neighboring bin centers by linear interpolation; inference recovers a continuous estimate from the bin expectation.
  • The success head uses frame-level binary targets, with success set to one at the final task frame and zero earlier.Training uses balanced binary cross-entropy to address negative-sample imbalance, while endpoint sampling corrects recordings that continue after completion.

D. Data Sampling and Augmentation

ROBOMETER constructs trajectory comparisons across RBM-1M to exploit failures, distinguish task instructions, and model regressions through synthetic rewinding.

  • Progress-Based Comparisons (Different Expertise): Progress-based comparisons pair trajectories sharing an instruction but differing in expertise, outcome, or progress, preferring the higher-progress or expert trajectory.Unlabeled failures can therefore be contrasted with successful demonstrations without absolute failure-progress labels.
  • Instruction Negatives (Different Tasks): Instruction negatives pair trajectories from different tasks, rewarding the trajectory matching the selected instruction and assigning zero progress to the other.
  • Video Rewind (Augmented Failures): Video rewind reverses a segment of an expert trajectory to generate synthetic preferences that model undoing progress.The augmentation targets a common failure mode in reinforcement-learning rollouts.
  • ROBOMETER remains a dense per-frame reward model, querying prefixes o1:t to produce T rewards for a trajectory of length T.Subsampling always includes the first and current frame.
  • RBM-1M contains progress-labeled and progress-unlabeled data, and comparisons are sampled from failures, different tasks, and rewound videos.At inference, ROBOMETER handles one trajectory for progress and success predictions or two trajectories for preference prediction.

IV. EXPERIMENTS

The experiments evaluate reward alignment, trajectory ranking, and downstream policy learning against video-language and zero-shot-capable baselines on in-distribution and out-of-distribution settings.

  • The evaluation asks whether rewards reflect progress across unseen tasks and embodiments, how components contribute, and how rewards affect downstream policy learning.
  • RoboDopamine-8B and RoboReward-4B/8B provide additional comparison points through goal-image-conditioned reward prediction and discrete progress prediction, respectively.
  • RBM-EVAL-OOD contains 976 trajectories from three academic institutions across six embodiments, including three absent from baseline training data.The dataset is collected specifically to avoid overlap with baseline training data.
  • Figure 4 compares progress predictions for failed, suboptimal, and successful trajectories of the same task.ROBOMETER reduces progress after a marker drop, whereas RoboReward retains high progress; ROBOMETER alone correctly predicts success for the successful trajectory.

Q1: Reward Evaluation

ROBOMETER generalizes across unseen tasks, embodiments, and domains while distinguishing successful, suboptimal, and failed trajectories. It also provides a strong initialization for domain-specific fine-tuning.

  • ROBOMETER accurately aligns rewards with task semantics across unseen embodiments, camera viewpoints, and scenes.
  • ROBOMETER performs best across reward-alignment tests on RBM-EVAL-OOD and RBM-EVAL-ID.Reward alignment is measured with Value Order Correlation between predicted frame rewards and ground-truth timestep values.
  • ROBOMETER achieves Kendall-τa 0.66 on RBM-EVAL-OOD, versus 0.50 for RoboReward-4B and 0.47 for RoboReward-8B.The metric evaluates ordering among failed, suboptimal, and successful trajectories.
  • ROBOMETER fine-tuned on RoboFAC performs best, while fine-tuned Qwen3 performs worse than zero-shot ROBOMETER on Kendall τ and success-fail difference.LoRA and full fine-tuning of ROBOMETER both outperform the compared alternatives.
  • ROBOMETER outperforms baselines in generalization and successful-versus-failed trajectory discrimination, while also serving as a strong fine-tuning initialization.

Q2: Ablations: Why does ROBOMETER Perform so Well?

Ablations test preference supervision, additional failed data, and pretrained multimodal backbones. Preference supervision and failed trajectories improve ranking and policy learning, while replacing the pretrained VLM causes severe degradation.

  • Adding preference supervision improves policy ranking even without paired failure trajectories.
  • Adding 1,929 generated failed LIBERO trajectories produces the largest gains across ranking-based metrics.
  • Replacing the pretrained VLM with a scaled ReWiND architecture severely degrades all metrics, confirming the importance of a large-scale multimodal pretrained backbone.
  • On LIBERO-90, failed-data training raises Kendall-τ to 0.92 and increases success-failure reward separation more than 4× over progress-only training.
  • ROBOMETER achieves 2−4× better sample efficiency than sparse reward on unseen LIBERO-90 tasks.The comparison uses online RL policies trained from scratch on two tasks.
  • Offline IQL with ROBOMETER rewards achieves a 2.4× success-rate improvement over the best baseline for each task.

Q3: Accelerating Robot Learning with Generalizable Rewards

ROBOMETER supports downstream learning across automatic online RL, offline RL with mixed-expertise data, retrieval, and zero-shot failure detection. Its dense rewards and success predictions improve learning and evaluation across these settings.

  • Automatic Online RL: DSRL with ROBOMETER improves π0 from 20% to 85% on a single-stage task within 10k timesteps.Success detection is automated, with human intervention only for physical scene resets.
  • Automatic Online RL: DSRL with ROBOMETER improves π0 from 20% to 70% on a two-stage task, versus RoboReward’s 20%.ROBOMETER success predictions automatically trigger progression between stages.
  • Offline RL: ROBOMETER rewards achieve a 2.4× success-rate improvement over the best baseline in offline IQL with noisy and expert trajectories.The best performance occurs with discount factor γ = 0.9.
  • Data Filtering & Retrieval: ROBOMETER retrieval attains an average 4.5× success-rate improvement over other retrieval methods.It retrieves task-relevant subtrajectories using either preference comparisons or progress-value correlation.
  • Failure Detection: ROBOMETER achieves the highest average F1 score for zero-shot failure detection across manipulation trajectories.It detects irreversible, insufficient-progress, and non-terminal failures across tasks and environments.

V. LIMITATIONS AND FUTURE WORK

ROBOMETER’s subsampled per-frame video modeling limits long-horizon and partial-observation memory, while incomplete failure coverage and missing latent physical-state access constrain failure recognition.

  • Subsampled per-frame video inputs limit ROBOMETER’s ability to memorize key events in very long-horizon or partially observed tasks.
  • Rare, subtle, or task-specific failure modes may fall outside the dominant patterns represented in current training data.
  • As a vision-language model, ROBOMETER lacks direct access to contact forces, grasp stability, and compliance until these factors become visually observable.
  • Safety-critical states are not among the states ROBOMETER is pretrained to predict.

APPENDIX A DATASET DETAILS

RBM-1M combines diverse robot, human, simulation, and failed-trajectory sources, with dataset-specific filtering and task-end thresholding to support reward learning.

  • Preference-Only Data: Preference comparisons allow noisier datasets with unreliable termination timing to contribute training signal without requiring dense progress labels.This enables use of sources that would otherwise be unsuitable for progress and end-state success prediction.
  • Dataset Sources: Dataset curation includes expert demonstrations, preference-only sources, and evaluation datasets spanning in-distribution and out-of-distribution settings.Some sources are restricted to preference prediction because trajectory termination times vary relative to task completion.
  • Diversity: The dataset includes broad embodiment and viewpoint diversity, including single-arm, bimanual, mobile, humanoid, and human demonstrations.Viewpoint selection is dataset-dependent; DROID uses both wrist and external cameras, while many other sources use external views only.
  • Task End-State Adjustment: Task-finished thresholds are manually selected per data source using ten sampled trajectories and the 90th percentile of completion frames.Thresholds are typically around 80–95% of trajectory length because teleoperators often reset after completing tasks.

APPENDIX C ADDITIONAL REWARD EVALUATION RESULTS

Additional evaluations show that ROBOMETER improves preference prediction, reward generalization, failure detection, and downstream policy learning across diverse settings.

  • Preference Prediction: 27.0% and 32.4% average preference-accuracy improvements over RL-VLM-F occur on different-quality and different-task trajectory pairs, respectively.ROBOMETER compares full video sequences, whereas RL-VLM-F uses only final frames.
  • Long-Horizon Tasks: ROBOMETER’s instruction-conditioned progress can increase while the robot temporarily moves away from intermediate objects during multi-step tasks.With brief or underspecified instructions, predicted progress may stagnate or decrease.
  • Objective Ablations: Removing any training component reduces performance, with trajectory rewinding producing the largest degradation in success–failure separation and trajectory ranking.Different-task negatives mainly affect instruction grounding, while suboptimal pairs help calibrate mixed-quality behavior.
  • Training Loss Weights: Uniform loss weighting, (λpref, λprog) = (1, 1), performs best on LIBERO-90 across alignment, ranking, and success–failure separation.Upweighting either preference or progress performs worse in this sweep.
  • Automatic Online RL: 55% improvement over RoboReward after 10k online RL steps is reported for ROBOMETER in single-stage experiments.RoboReward often assigns maximum reward to wrong-task behavior in cluttered scenes, producing false positives.
  • Failure Detection: ROBOMETER identifies failures through regressions or stagnation in predicted progress, covering irreversible, semantic, and insufficient-progress failures.Irreversible failures show sharp post-event drops, while semantic failures remain low and insufficient-progress failures plateau or oscillate.
  • Reward Generalization: ROBOMETER reaches VOC r=0.884 and Kendall τ=0.802 after adaptation on RoboFAC, outperforming the strongest from-scratch baseline by +21.6% in VOC.LoRA achieves essentially the same performance, indicating the reported gains primarily come from the pretrained initialization.
  • Downstream Policy Learning: ROBOMETER produces a 3.5× success-rate improvement over DreamZero on a cluttered ice-cream placement task evaluated over 10 trials.Integrating ROBOMETER corrected placements into the intended plate.
Loading 2603.02115v2…