TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Shirui Chen; Cole Harrison; Ying-Chun Lee; Angela Jin Yang; Zhongzheng Ren; Lillian J. Ratliff; Jiafei Duan; Dieter Fox; Ranjay Krishna

Source-linked AI summary

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

arXiv:2602.19313v2Feb 22, 2026cs.ROcs.AIcs.LG

TL;DR

Robot learning needs dense, instruction-conditioned rewards, but existing approaches often depend on manual annotations, demonstrations, or trained reward models. TOPReward instead extracts task-completion token probabilities from pretrained VLMs to produce zero-shot progress rewards. Across ManiRewardBench and Open X-Embodiment, it outperforms prior training-free methods on open-source models and remains competitive with a trained baseline without reward-model training.

Problem
Scalable dense progress feedback remains difficult because existing robot reward methods often rely on manual annotations, task-specific demonstrations, or curated training data.
Method
TOPReward uses pretrained VLM token probabilities for instruction completion as a continuous reward instead of generating numerical progress values.
Results
TOPReward substantially outperforms GVL on open-source models across Open X-Embodiment and ManiRewardBench, while remaining competitive with trained reward models without reward-model training.
Takeaways & Limitations
The reward is instruction-sensitive and supports success detection and reward-weighted behavior cloning across real-world manipulation tasks.
Takeaways & Limitations
TOPReward may produce noisy estimates for tasks requiring fine-grained spatial reasoning, precise alignment, or small-object manipulation, and depends on prompt and tokenizer choices.

Abstract

from arXiv · show

General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.

1 Introduction

TOPReward addresses the difficulty of obtaining scalable, instruction-conditioned progress feedback by using pretrained VLMs for zero-shot reward estimation. It evaluates this approach across diverse manipulation tasks and downstream settings without task-specific training.

Existing robot reward methods often rely on curated datasets, task-specific demonstrations, or fine-tuning, limiting scalability and generalization.
TOPReward provides instruction-conditioned progress signals across diverse manipulation tasks and multiple VLM backbones without task-specific training.
TOPReward supports success detection, policy improvement, and reward-weighted offline behavior cloning as downstream applications.The introduction reports reward-weighted fine-tuning improvements on six real-world SO-100 manipulation tasks.
TOPReward extracts token-probability shifts toward task-completion answers instead of generating numerical progress values.The method uses pretrained VLM representations without additional training or fine-tuning.
ManiRewardBench contains 130 unique real-world manipulation tasks spanning multiple robot platforms with temporal progress annotations.The benchmark supports evaluation of progress tracking across varied embodiments and tasks.

2 Related Work

Prior reward approaches range from task-specific visual representations and coarse success classifiers to trained generalist models and training-free VLM estimators. TOPReward extends internal-representation methods by using completion-token probabilities as a zero-shot reward across datasets and robot platforms.

Embedding-based reward methods require task-specific fine-tuning and provide limited language grounding, while VQA-style methods produce signals too coarse for dense shaping.
Generalist reward models broaden coverage but still require domain-specific training data and may struggle across embodiments and environments.
Training-free GVL estimates progress by prompting VLMs to assign scores to shuffled trajectory frames.
Internal logits, hidden states, and embeddings can track certainty more reliably than generated text, motivating representation-based rewards.
TOPReward asks a binary completion question and uses the affirmative token probability as a continuous, zero-shot reward without fine-tuning or domain-specific data.The approach scales to ManiRewardBench and Open X-Embodiment across multiple robot platforms.

3 TOPReward

TOPReward converts a pretrained VLM’s token probabilities for task completion into causal prefix rewards, then samples and optionally normalizes them into temporal progress curves. Its design avoids numerical text generation while supporting downstream positive weighting.

Given an instruction and video trajectory, TOPReward produces a scalar score for each prefix reflecting accumulated evidence of task completion.
3.1 Token probability as the reward: The method scores the probability of an affirmative completion token, such as “True,” as the reward.Boolean tokens were selected because they showed the clearest success–failure separation in the authors’ comparison.
3.1 Token probability as the reward: TOPReward computes rt = log pθ(a | c(τ1:t, u)), producing a causal completion score from the video prefix and instruction.The raw score can increase as visual evidence accumulates and can compare complete successful and failed trajectories.
Direct prompt scoring is preferred for open-source VLMs because chat templates can substantially reduce performance.
3.1 Token probability as the reward: Evaluating probabilities over the entire multi-token instruction was less effective than the selected token-based formulation.
3.2 Progress estimation from trajectory prefixes: Uniformly spaced prefix sampling produces rewards that summarize how completion evidence accumulates over time.
3.2 Progress estimation from trajectory prefixes: Per-episode min-max normalization maps raw log-probabilities to bounded progress scores for within-trajectory visualization and evaluation.This normalized score is non-causal and is not used as the raw online reward.
For reward-weighted offline behavior cloning, TOPReward increments create positive weights that downweight decreasing-reward steps rather than treating them as negative supervision.

4 Experiments

TOPReward is evaluated as a training-free progress estimator across large, diverse robot datasets, with complementary tests of metric validity, instruction sensitivity, success detection, and downstream behavior cloning. It substantially outperforms GVL on open-source models, remains competitive with trained reward baselines, and improves reward-weighted behavior cloning over BC.

Evaluation setup: TOPReward is evaluated on 33 filtered Open X-Embodiment datasets and ManiRewardBench, which covers 113 successful-task evaluation tasks and 497 episodes across four robot platforms.The benchmark suite includes diverse tasks, camera configurations, embodiments, and stage-aware progress annotations.
Evaluation setup: VOC measures temporal rank alignment, but complementary Kendall tau-b, Pearson correlation, and MAE are needed because VOC alone does not establish instruction grounding.Higher values are better except MAE.
Progress estimation: TOPReward substantially outperforms GVL on open-source models across OXE and ManiRewardBench and is competitive with trained Robometer-4B progress estimates on ManiRewardBench.On OXE, TOPReward is slightly better than Robometer-4B on the same filtered split, while Gemini-2.5-Pro is a weaker case for TOPReward because of chat-template effects.
Progress estimation: TOPReward produces smooth, generally increasing progress traces that track stage-aware completion, with plateaus at intermediate subtask completions and accelerations during active manipulation.Gemini-GVL shows noisier, more frequently nonmonotonic predictions in the qualitative comparison.
Instruction sensitivity: After Sinkhorn normalization, mean diagonal mass is 0.420 for TOPReward Qwen3-VL-8B, 0.376 for TOPReward Qwen3-VL-32B, and 0.149 for Robometer-4B, indicating stronger video–instruction matching for TOPReward.The random baseline on the 20-task split is 1/20 = 0.05.
Downstream applications: Qwen3-VL-32B reaches 0.965 ROC-AUC and Qwen3-VL-8B reaches 0.939 ROC-AUC for success detection, exceeding the trained Robometer-4B success head without reward-model training or success-label supervision.The evaluation uses 150 successful and 129 failed cleaned LeRobot episodes.
Downstream applications: TOP-RWBC improves over behavior cloning on all six real-world single-arm SO-100 tasks when TOPReward-derived positive weights are used.The partial-success score is computed as the fraction of predefined subtasks completed per trial, summed over 10 trials.

5 Limitations

TOPReward’s estimates inherit the underlying VLM’s visual perception limits and also depend on causal versus non-causal score processing and prompt configuration.

Fine-grained spatial reasoning, precise alignment, and small-object manipulation may produce noisy progress estimates when intermediate states are visually indistinguishable.The raw prefix score is causal, whereas per-episode min-max normalization is non-causal and intended for within-trajectory evaluation and visualization.
Deployments should validate chat-template formatting and answer-token choices for the selected VLM backbone.

6 Conclusion

TOPReward uses pretrained video VLM token probabilities to estimate instruction-conditioned progress without reward-model training. It outperforms GVL on open-source models, remains competitive with trained robotics reward models, and supports success detection and reward-weighted behavior cloning.

6 Conclusion: TOPReward repurposes pretrained video VLM token probabilities as instruction-conditioned progress signals for robotic manipulation.It queries the model’s belief about instruction completion rather than requiring calibrated numerical outputs.
6 Conclusion: TOPReward substantially outperforms GVL on open-source models and remains competitive with robotics reward models trained on large-scale data across progress-estimation metrics.These results require no reward-model training.
6 Conclusion: TOPReward supports success detection and reward-weighted behavior cloning, which yields consistent improvements over standard BC across six real-world SO-100 manipulation tasks.
A Alternative Reward Formulation: The alternative formulation scoring the probability of generating the entire instruction is less effective than the main completion-query formulation.The authors hypothesize that visible entities can receive high probability even when the instructed action is incomplete, distracting progress estimation.

B Why the True Token?

TOPReward selects the affirmative token True because it provides the clearest separation between successful and failed trajectories. Its success-detection score combines prefix-based completion evidence with a direct full-video completion query using standardized scores.

Token selection: True is selected because it is a single vocabulary token and shows the largest, most consistent successful–failed separation at the final step.Figure 3 compares the top tokens by absolute difference in mean final-step probability.
Score construction: TOPReward’s trajectory-level success score uses completion-margin changes across sampled prefixes and a direct binary completion query on the full video.The prefix component compares early and late sampled prefixes, while the direct query asks whether the trajectory completed the instruction.
Score construction: The two success-detection components are combined with equal weights after standardization over evaluated trajectories.The standardization puts the scores on a comparable scale and uses no learned classifier or fitted task-specific reward model.

D Dataset-level breakdown

The dataset-level analysis compares TOPReward with zero-shot GVL across filtered Open X-Embodiment and ManiRewardBench evaluation sets. It reports aggregate and per-episode VOC distributions, backbone-specific differences, and qualitative progress traces.

Dataset-level VOC: Figure 4 compares mean dataset-level VOC for GVL (0-shot) and TOPReward across filtered OXE and ManiRewardBench evaluation sets.The OXE set contains 33 datasets with 20 episodes each; the ManiRewardBench subset contains 4 datasets, 113 tasks, and 497 episodes.
Per-episode VOC: Figure 5 breaks down per-episode VOC distributions by evaluation set and model backbone.The evaluation sets are ManiRewardBench and Open X-Embodiment.
Backbone comparison: Figure 6 shows the distribution of dataset-level ΔVOC = VOC(TOPReward) − VOC(GVL) separately for each model backbone.Positive values indicate TOPReward outperforms GVL, and dashed lines mark per-model means.
Dataset-level VOC: Table 5 compares GVL (0-shot) and TOPReward per filtered dataset and model backbone, highlighting the method with higher VOC for each pairing.The table uses the same 33 high-quality OXE datasets as the main table.

E Additional qualitative results and ablations

The reward-weighted behavior cloning objective uses fixed hyperparameters for its weighting and maximum normalization settings.

Behavior cloning: Reward-weighted behavior cloning uses β = 0.2 and δmax = 2.0.These values are applied in Equations (5) and (3), respectively.

E.2 Chat-template ablation

The chat-template ablation tests whether prompt formatting affects TOPReward. Chat formatting substantially degrades VOC for the evaluated open-source backbones, while the qualitative evaluation uses six SO-100 tasks and includes a “Place doll in box” example.

Chat-template ablation: Chat formatting substantially reduces TOPReward VOC for both Qwen3-VL-8B and Molmo2-8B.The ablation evaluates the probability of the answer being True after wrapping the prompt in a chat template.
Qualitative evaluation: The qualitative reward-weighted behavior cloning evaluation uses six real-world single-arm SO-100 manipulation tasks.The task set is summarized in Figure 9.
Qualitative evaluation: On “Place doll in box,” the pretrained policy and BC fail, whereas TOP-RWBC succeeds consistently.TOP-RWBC is fine-tuned using reward weights from TOPReward, with frames uniformly sampled from evaluation rollouts.
Chat-template ablation: Chat templates degrade Qwen3-VL-8B performance by nearly 50% and Molmo2-8B performance by 20%.The result indicates sensitivity of the logit-based formulation to prompt formatting.

F Additional details of ManiRewardBench

ManiRewardBench contains 130 unique manipulation tasks across four robot platforms, spanning diverse fine-control, deformable-object, symbolic, and multi-step activities. Episodes are organized into ordered, manually annotated subtasks, with dataset-specific task and episode compositions summarized across the benchmark.

Benchmark scope: ManiRewardBench contains 130 unique tasks spanning Franka, SO-100/101, bimanual YAM, and single-arm YAM platforms.The benchmark covers everyday activities including stacking, sorting, and container interaction.
Task diversity: Multi-step tasks combine sequential subtasks such as puzzle spelling, pyramid construction, color grouping, obstacle removal, packing, and box closing.The examples require spatial reasoning, categorical reorganization, precise positioning, or multiple task phases.
Task diversity: The benchmark includes fine-manipulation tasks involving spatial alignment, precise rotation, orientation, and controlled pouring.Examples include horizontally aligning cubes, rotating objects by specified angles, orienting a screw toward glue, and pouring tea.
Task diversity: Deformable-object tasks require careful manipulation of towels and cloths whose non-rigid geometry complicates folding or stacking.Examples include folding towels and stacking one cloth on another.
Annotation and composition: The benchmark combines successful and failed LeRobot demonstrations with Franka, bimanual YAM, and single-arm YAM datasets using task-specific episode counts.The listed datasets include approximately 7 episodes per task for the failure split, 3 for most Franka tasks, and 5 for YAM tasks.
Annotation and composition: Episodes are manually segmented into strictly ordered, non-overlapping predefined subtasks with start and end times recorded for each subtask.The subtasks represent execution stages such as reaching, grasping, and placing.