PaperlayerView exact revision on arXiv

Source-linked AI summary

D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

Marius Memmel, Shawn Yin, Urvi Rutia, Angad Wadhwa, Eldon Li, Gavin Angeloff, Ankit Goyal, Jesse Zhang, Dieter Fox, Fabio Ramos, Anqi Li, Abhishek Gupta

arXiv:2611.03622v1cs.ROcs.AIcs.CLcs.LGcs.SYeess.SY

TL;DR

Zero-shot VLM reward estimation can hallucinate progress, while post-training requires robotics data and model-specific retraining. D&A separates trajectory description from assessment against an explicit success criterion. The paper reports generalization across tasks, embodiments, and benchmarks, with reward models that match or surpass post-trained models and support real-world reinforcement learning.

  • Problem

    Zero-shot VLM reward methods can hallucinate progress on failures, while post-trained approaches require hard-to-obtain robotics data and repeated training for each base model.

    Source: §I · p. 1

  • Method

    D&A uses a prompt-based pipeline that describes robot behavior and assesses progress against an explicit success criterion.

    Source: §I · p. 1

  • Results

    D&A generalizes across tasks, embodiments, and benchmarks, matching or outperforming post-trained models; its rewards improve real-world reinforcement learning outcomes.

    Source: §I · p. 2

  • Takeaways & Limitations

    The prompt-based method requires no robotics data or fine-tuning and can be applied across tasks, embodiments, and language.

    Source: §I · p. 2

    D&A inherits frontier-model constraints, including high inference latency and token costs, limited context windows, API stochasticity, and occasional semantic hallucinations.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) hold promise for zero-shot video task-progress estimation in robot learning, yet existing frameworks frequently suffer from confirmation bias, hallucinating progress in failed executions due to implicit success assumptions or relative temporal ranking designs. Naive prompting of frontier models collapses into a binary success detector that fails to capture nuanced, non-linear progress and failures. While post-training approaches can mitigate this through explicit training, they demand expensive real-world robotics data and require computationally intensive post-training for every individual base model. To resolve these limitations, we introduce D&A (Describe & Assess): our approach decouples progress estimation into an objective, task-grounded textual description of scene dynamics followed by an assessment phase using explicit rubrics. By compelling the VLM to synthesize an explicit, text-based definition of absolute success before scoring, the framework anchors evaluations to an external, global criterion rather than the video's internal timeline. Empirical results demonstrate that our method calibrates zero-shot estimates and mitigates confirmation bias, providing fine-grained, partial progress metrics that accurately penalize failed executions while correctly interpolating successful runs. Being purely prompt-based, D&A generalizes across tasks, embodiments, and benchmarks---matching or surpassing post-trained reward models---and its reward estimates enable efficient downstream reinforcement learning.

I. INTRODUCTION

Naïve zero-shot VLM reward estimates can hallucinate progress on failed tasks, while post-trained models require hard-to-obtain robotics data and repeated model-specific training. D&A instead separates trajectory description from success assessment, and the paper reports generalization across tasks, embodiments, and benchmarks.

I. INTRODUCTION

  • Naïve zero-shot reward methods can hallucinate progress on failures, and their misleading evaluations can impede downstream policy learning.Post-trained alternatives require diverse real-world robotics data, are limited to smaller open-source models, and must be repeated for each base VLM.Source: §I · p. 1
  • D&A separates objective textual description of robot behavior from assessment against an explicit semantic success criterion, without robotics-specific training.It also generates dense rewards and natural-language feedback for policy improvement.Source: §I · pp. 1–2
  • D&A is prompt-based and model-agnostic, generalizes across tasks and embodiments, and matches or outperforms post-trained models on reward benchmarks.The paper also reports improved real-world online and offline reinforcement learning and scaling across four model families and 28 models.Source: §I · p. 2

II. RELATED WORK

  • Prior work includes pixel-based zero-shot reward estimators and robotics-data-trained models; D&A routes reward estimation through description and assessment instead.The cited related-work passage contrasts direct scalar extraction with D&A’s two-phase approach.Source: §II · p. 2

III. DEMONSTRATING BIAS IN ZERO-SHOT VLM PROMPTING

  • In the shirt example, GPT-5.3 interpreted a folding video as unfolding and rewarded frame-to-frame motion rather than task completion.The passage links this shortcut to confirmation bias associated with fine-tuning on human feedback [3, 4, 5, 6, 7, 8].Source: §III · p. 2, Figure 2 · p. 2
  • The paper reports that existing zero-shot reward methods perform poorly across benchmarks, real-world tasks, and embodiments despite prompt engineering.The evaluation includes both closed- and open-source VLMs with custom prompting and reward-extraction schemes.Source: §III · p. 2

IV. D&A: DECOUPLING DESCRIPTION AND ASSESSMENT

D&A maps a task video to dense rewards through separate description and assessment stages. The describer processes the full trajectory without the task instruction, while the assessor scores the resulting description against an explicit success criterion.

IV. D&A: DECOUPLING DESCRIPTION AND ASSESSMENT

  • D&A composes a Describer and Assessor to convert a video and instruction into dense per-frame rewards and language feedback.The Describer narrates without the instruction; the Assessor scores each timestep against an explicit success criterion.Source: Figure 3 · p. 3, §IV · p. 3
  • The pipeline downsamples videos to H frames, such as H = 8, then interpolates rewards back to T frames to reduce inference cost and time.The figure describes outputs as dense per-frame rewards in [0, 1].Source: §IV · p. 3, Figure 3 · p. 3

A. Describing the Robot Trajectory Video

  • The describer processes the full trajectory in one pass, producing temporally consistent descriptions informed by events before and after each timestep.To retain task-relevant entities without exposing the task instruction to the grounded describer, an entity extractor lists scene entities before description.Source: A. Describing the Robot Trajectory Video · p. 3

B. Success Criteria Generation and Reward Assessment

  • Scoring descriptions directly can collapse to all-zero rewards or binary success detection, so the assessor first defines an observable fully completed state.It then scores each timestep against that criterion, allowing partial progress and non-monotonic rewards while resisting both collapse modes.Source: B. Success Criteria Generation and… · pp. 3–4

C. From Progress to Rich Language Feedback

D&A turns progress assessment into a diagnostic signal that can support both dense-reward learning and corrective feedback. Across the reported evaluations, it distinguishes subtle failures, improves downstream policy results, and performs strongly against reward-model baselines.

C. From Progress to Rich Language Feedback

  • D&A’s assessor tracks criteria fulfillment over time to localize execution deviations, while its feedback generator produces failure analyses and corrective instructions.This extends scalar progress scores with targeted language feedback for policy improvement.Source: C. From Progress to Rich Language… · p. 4

D. Using D&A for Downstream Policy Improvement

  • In online RL, D&A’s absolute progress scores serve as dense rewards; in offline RL, its bounded scores become potential-based rewards and advantages for filtering or conditioning training data.For code-as-policies, D&A provides between-attempt failure analyses and proposed corrections.Source: D. Using D&A for Downstream Policy… · p. 4

V. EXPERIMENTAL EVALUATION

  • D&A (Gemini 3 Flash) is the strongest zero-shot method across three benchmarks, matching or surpassing post-trained models; D&A* (GPT-6 Astra) tops every benchmark.The comparison covers D&A-Bench, RBM-EVAL-OOD, and RoboRewardBenchmark.Source: Figure 5 · p. 5
  • 100%: With D&A rewards, online RL raises task success from the policy’s initial 10% in about 90 minutes; TOPReward leaves success at 5%.D&A assigns zero reward to ice-cream-cone trajectories and high, dense reward for manipulating the banana.Source: §V · pp. 5–6
  • For “hit the enter key,” D&A distinguishes success from a visually similar failure by detecting the gripper near, but not pressing, the correct key.Robometer scores the two alike, RoboReward and SOLE-R1 score the failure as success, and GVL and TOPReward score both high.Source: Figure 7 · p. 6
  • D&A scales with model capability across four open- and closed-source model families, while GVL scales less and TOPReward and RoboReward show inverse performance trends.Source: Figure 8 · p. 6
  • +80%: D&A improves behavior cloning by filtering suboptimal demonstrations in the offline book-moving task, outperforming the other models.The dataset contained optimal, suboptimal, and failure demonstrations.Source: §V · p. 6
  • 18.8 ± 1.8 seconds: D&A-filtered behavior cloning completes the task faster than the baseline’s 28.5 ± 9.2 seconds, with lower rollout variance.The comparison is for successful runs on the long-horizon book task.Source: §V · pp. 6–7

(Q2) How well does D&A generalize to different tasks and embodiments?

D&A is evaluated for generalization on a manually collected, out-of-distribution benchmark and two external benchmarks. It scores highest on D&A-Bench and assigns zero progress to mismatched video–instruction pairs while scoring correct matches accurately.

D&A-Bench

  • D&A-Bench contains 54 success and failure trajectories across 14 YAM tasks, with ten VLM-generated instruction paraphrases per trajectory to test language robustness.The tasks probe nuanced scene understanding, counting, and semantic reasoning.Source: (Q2) How well does D&A generalize to… · p. 7
  • 0.67: D&A scores highest on D&A-Bench, ahead of Robometer (0.56) and RoboReward (0.43); zero-shot TOPReward and GVL score 0.24 and 0.27.D&A improves +0.23 over Naive VLM on this benchmark.Source: (Q2) How well does D&A generalize to… · p. 7

Semantic understanding

  • D&A assigns zero progress to mismatched video–instruction pairs while accurately scoring correct matches; post-trained models also separate the pairs, whereas zero-shot models do not.The confusion matrices pair unseen task videos with unseen task instructions.Source: (Q2) How well does D&A generalize to… · p. 7, Figure 9 · p. 7

(Q3) How does D&A scale with the underlying VLM?

D&A’s trajectory-ranking performance rises with base VLM capability, reaching state of the art with two models; its gains are largest on smaller models. Ablations identify object-grounded descriptions as an effective design choice, while rich D&A feedback improves CaP task success over history alone.

(Q3) How does D&A scale with the underlying VLM?

  • 0.734 with Gemini 4 Argon and 0.714 with GPT-6 Astra achieve SOTA on RBM-EVAL-OOD as D&A performance rises with base-model capability.GVL also scales, but at a much lower rate; TOPReward faces closed-source compatibility barriers and follows a negative scaling trend reported in its original publication [12].Source: (Q3) How does D&A scale with the… · p. 7
  • D&A yields the biggest gains on smaller models, outperforming RoboReward on both benchmarks and Robometer on D&A-Bench.GPT-6 Astra brings smaller but still substantial gains; qualitative examples are referenced in Sec. X.Source: (Q3) How does D&A scale with the… · pp. 7–8

(Q4) What components contribute to D&A’s performance?

  • +0.26 over Naive VLM results from grounding descriptions in objects present in the video, outperforming post-trained models in some cases.Adding an explicit success criterion alone improves performance by +0.18; combining it with decoupling hurts when relevant objects are missing or unidentifiable in the description.Source: (Q4) What components contribute to… · p. 8, Table I · p. 8

(Q5) Can rich feedback generated by D&A improve over scalar rewards?

VI. LIMITATIONS AND DISCUSSION

  • D&A inherits frontier-model latency, token-cost, context-window, API-stochasticity, and semantic-hallucination constraints, while language descriptions limit highly nuanced or continuous rewards.The paper gives precise spatial relationships and absolute geometric distances as examples of hard-to-verbalize reward structures.Source: §VI · p. 8

VII. OFFLINE RL VIA ADVANTAGE CONDITIONING: FULL DERIVATION

  • Step-wise rewards are temporal differences of progress scores, capturing the instantaneous rate of task improvement.The derivation treats each trajectory as observations and actions and sets the initial progress score to zero.Source: §VII · p. 10
  • Offline returns are discounted Monte Carlo sums, and the global value baseline is their empirical mean across all timesteps and trajectories.The centered return defines each state-action advantage; the resulting advantages condition the downstream policy optimization.Source: §VII · p. 10
  • CFGRL conditions the policy on advantages, with π0.5 optimized using a supervised log-likelihood objective.The conditional policy takes state, action, language instruction, and advantage as inputs.Source: §VII · p. 10

VIII. ONLINE RL VIA LATENT SPACE NOISE STEERING: FULL DERIVATION

The online DSRL pipeline uses D&A progress scores as dense environmental rewards and learns a policy over latent steering variables rather than directly over robot actions. The described updates use a critic’s temporal-difference error and an entropy-regularized policy objective; the latent-space approach is presented as mitigating forgetting and collapse.

VIII. ONLINE RL VIA LATENT SPACE NOISE STEERING: FULL DERIVATION

  • Online DSRL uses D&A’s absolute progress scores as dense rewards while optimizing a policy over latent steering variables mapped through a frozen base model to robot actions.The approach learns a lower-dimensional steering vector or noise variable instead of directly optimizing in the high-dimensional action space.Source: §VIII · p. 10
  • The policy objective maximizes expected critic return with entropy regularization during online optimization.The policy samples latent variables from πθ; the critic uses a soft action-value function and target state-value network.Source: §VIII · p. 10
  • Optimizing in a frozen expert’s latent noise space mitigates catastrophic forgetting and policy collapse, enabling sample-efficient online behavior optimization powered by D&A rewards.Source: §VIII · p. 10

IX. ADDITIONAL RESULTS

X. GAINS OF D&A OVER BASE MODEL CAPABILITIES - FRONTIER MODELS

D&A improves frontier-model reward estimates and retains gains as Gemini backbones strengthen. The supplied experiments also describe downstream evaluation setups and qualitative benchmark examples.

X. GAINS OF D&A OVER BASE MODEL CAPABILITIES - FRONTIER MODELS

  • D&A improves reward modeling on recent VLMs to SOTA on both benchmarks, except Gemini 4 Argon is on par with Naive VLM on RBM-EVAL-OOD.The authors hypothesize Gemini 4 Argon may have been trained on RBM-EVAL-OOD, potentially diluting the benchmark results.Source: Figure 11 · p. 11, §X · p. 11

XI. GAINS OF D&A OVER BASE MODEL CAPABILITIES - GEMINI MODEL FAMILY

  • As Gemini models strengthen, D&A gains persist; reducing test-time compute to low thinking matches results and can even improve performance.The authors report that competing baselines either have closed-model technical barriers or cannot be applied to proprietary models.Source: §XI · p. 11, Table V · p. 12

XIII. CAP EXPERIMENT DETAILS

  • The LIBERO evaluation uses an open-loop code-as-policies planner across 10 tasks, 500 initial conditions, and 25 attempts per condition.D&A scores each attempt's video, and its output is appended to the planner's history for the next attempt; Success@k counts unique conditions solved cumulatively.Source: §XIII · pp. 12–13

XIV. D&A-BENCH BENCHMARK AND QUALITATIVE RESULTS

  • D&A-Bench includes 14 real-world bimanual YAM manipulation tasks, with 560 episode-instruction pairs scored per reward model.The accompanying figures show per-trajectory predictions for tasks including car assembly, block towers, and measuring coffee.Source: §XIV · p. 13, Figure 15 · p. 13, Figure 23 · p. 17

XV. D&A PROMPTS

The naive VLM baseline prompt specifies object-grounding rules, frame-by-frame progress scoring, and a JSON response format.

A. Naive VLM Baseline

  • The task wording fixes the required objects' category, color, shape, and wording; video frames alone provide evidence about the scene.Source: A. Naive VLM Baseline · p. 21
  • A required object may match only a visible referent of the same entity, and a conflicting or unclear referent is treated as absent.Source: A. Naive VLM Baseline · p. 21
  • Objects may be tracked across frames only when they are plainly the same physical entity already visible in the video.Source: A. Naive VLM Baseline · p. 21
  • For each timestep, the baseline estimates progress toward the fully completed state on a 0–100% scale.Source: A. Naive VLM Baseline · p. 21
  • The response is a list of progress values ordered by video frame and returned in the specified JSON format.Source: A. Naive VLM Baseline · p. 21

1) Grounding:

The prompt grounds task objects in the scene, describes video dynamics objectively, and assesses progress against an explicit fully completed state.

1) Grounding:

  • Grounding lists all visible objects and identifies which, if any, are relevant to the task.Source: 1) Grounding · p. 21

2) Grounded Video Description:

  • Grounded video descriptions detail the robot and task objects at each timestep, including motion and distances, while avoiding judgments about intent.The prompt requests descriptions of the first and last frames as scene anchors and exactly one description per frame.Source: 2) Grounded Video Description · p. 21

3) Success Criteria and Assessment:

  • Assessment matches task-required objects only to non-contradictory scene referents, defines full completion, and scores progress toward it at each timestep.If all required objects are absent, the task is treated as impossible in that scene and receives 0% progress; progress may be non-monotonic.Source: 3) Success Criteria and Assessment · p. 22

4) Success Criteria and Assessment - Modified for RoboRewardBenchmark:

The modified assessment prompt scores only the final state against a five-level completion rubric, with explicit rules separating major from minor misses.

4) Success Criteria and Assessment - Modified for RoboRewardBenchmark:

5) Success Criteria and Assessment - Modified for Failure Analysis and Feedback Prediction:

The assessment grounds task requirements in explicitly described scene objects, defines a fully completed state, and scores progress toward it. It also reports failure and corrective feedback when completion is not achieved.

5) Success Criteria and Assessment - Modified for Failure Analysis and Feedback Prediction:

  • Task requirements may match only clearly identified scene entities with compatible attributes; absent or contradictory objects cannot be substituted with plausible alternatives.Short paraphrases may be aligned across timesteps only when they plainly refer to the same physical object.Source: 5) Success Criteria and Assessment -… · p. 22
  • When all required objects are absent, the task is assigned 0% progress; otherwise, the assessment defines the fully completed scene before scoring partial progress.Progress is measured toward that completed state and need not be monotonic.Source: 5) Success Criteria and Assessment -… · p. 23
  • For incomplete tasks, the assessment describes the failure and gives feedback on what the robot should have done differently.Before responding, the prompt asks the model to reconsider whether the task is actually complete and adjust progress values if needed.Source: 5) Success Criteria and Assessment -… · p. 23
  • The response format returns progress values ordered by frame alongside failure, feedback, and completion-state fields.The prompt supplies scene descriptions as the assessment input.Source: 5) Success Criteria and Assessment -… · pp. 22–23

XVI. FAILURE ANALYSIS AND FEEDBACK PREDICTION EXAMPLES

The examples pair scene descriptions with failure analysis, completion criteria, and progress assessment. They show partial task execution being distinguished from full completion across object placement, lighting, and measured pouring tasks.

XVI. FAILURE ANALYSIS AND FEEDBACK PREDICTION EXAMPLES

  • The candy-block example identifies that only the yellow and pink blocks reached the bowl, while the red block was dropped elsewhere, leaving the three-block task incomplete.The completion state requires all three blocks in the blue bowl.Source: §XVI · p. 24
  • The lighting example records contact with the switch and plug area but no illumination; completion requires the cord connection and switch to be on.The feedback calls for actuating the switch and ensuring the plug is fully inserted; touching the bulb was unnecessary.Source: §XVI · p. 25
  • The coffee-pouring example reports a final measurement of 14.8g rather than the required 16.0g, attributing the shortfall to stopping early.The feedback recommends continuing to pour while monitoring the scale until 16.0g is reached.Source: §XVI · p. 26
Loading 2611.03622v1…