PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models

Shengbang Liu, Zhengye Du, Zhilong Wan, Honghao Su, Chenxiang Xia, Chang Ge, Jinyang Xiao, Nan Wang, Zhipeng Hu, Huining Yuan, Shu'ang Yu, Yue Chen, Tianxing Chen, Chuankang Li, Maoqing Yao, Wenbo Ding, Yu Wang, Guanghui Ren, Chao Yu

arXiv:2610.26663v1cs.RO

TL;DR

Existing evaluations cover selected aspects of robotic execution understanding, leaving the reliability of value feedback insufficiently tested. RoboValue evaluates four complementary capabilities using diagnostic trajectories and instruction variations across simulation and real-world manipulation. Its results expose persistent weaknesses in instruction grounding, progress tracking, recovery assessment, and consistent subtask credit.

  • Problem

    Existing evaluations cover selected aspects of execution understanding, leaving the reliability of robotic value feedback insufficiently tested.

    Source: §1 · p. 3

  • Method

    RoboValue uses shared interfaces and diagnostic trajectories to evaluate task-state understanding, temporal progress, failure and recovery reasoning, and value consistency.

    Source: §1 · p. 3

  • Results

    Models retain errors in instruction grounding, history-dependent progress tracking, recovery assessment, and subtask credit across valid execution orders, despite strong aggregate outcome or progress scores.

    Source: §1 · p. 4, §8 · p. 21

  • Takeaways & Limitations

    The benchmark, dataset, and public leaderboard provide a common basis for diagnosing model limitations and assessing future advances.

    Source: §8 · p. 21

    Differences in training data and model configurations limit attribution of observed capability gaps to individual components.

    Source: §7 · p. 20

Abstract

from arXiv · show

Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations cover only selected aspects of execution understanding, leaving the reliability of value feedback insufficiently tested. We introduce RoboValue, a fine-grained sim-and-real benchmark for unified evaluation of robotic value models. Its framework evaluates four complementary capability dimensions through shared interfaces: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency. Diagnostic trajectories and instruction variations test whether value judgments accurately reflect task requirements and changes throughout execution. RoboValue-Dataset covers 15 simulation and 20 real-world manipulation tasks, providing expert demonstrations for training and annotated trajectories for fine-grained evaluation. We evaluate representative robotic value models in zero-shot and one-shot settings under standard conditions and separate shifts in robot embodiment and environment. Our results reveal errors in fine-grained instruction grounding, history-dependent progress tracking, and recovery assessment, even in models with strong outcome and forward-progress scores. The same semantic subtask can also receive substantially different value gains across valid execution orders. These findings identify concrete targets for value supervision, while the dataset and public leaderboard support continued assessment of feedback reliability. The website is available at https://robovalue-benchmark.github.io/.

1 Introduction

Robotic value feedback shapes policy learning and execution, yet existing evaluations can miss unreliable judgments about intermediate execution. RoboValue tests four complementary capabilities with diagnostic trajectories and instruction variations, and its evaluation reveals weaknesses that motivate targeted supervision.

1 Introduction

  • Value-estimation errors can propagate into policy learning and execution because predictions influence which behaviors are selected, reinforced, or corrected.Imperfect feedback can increase predicted reward while degrading actual task performance.Source: §1 · p. 3
  • RoboValue evaluates task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency.These capabilities express the requirement that values reflect task fulfillment throughout execution, including relevant history, unresolved errors, and valid solutions.Source: §1 · p. 3
  • Diagnostic trajectories and instruction variations test goal grounding, history-dependent progress, recovery outcomes, and consistency across valid subtask orders.Counterfactual instructions change the requested goal while holding execution fixed, and recurring visual states require history to disambiguate progress.Source: §1 · p. 4
  • Models can score outcomes or forward progress strongly while still misgrounding instructions, mishandling recurring states, or misjudging failure and recovery.The same semantic subtask can also receive inconsistent value gains across valid execution orders.Source: §1 · p. 4
  • RoboValue-Dataset combines expert demonstrations and annotated test trajectories, while its benchmark and leaderboard support shared evaluation and capability profiles.The dataset covers simulation and real-world manipulation, and the framework jointly assesses limitations that outcome discrimination and forward-progress correlation alone can overlook.Source: §1 · p. 4

2 Related Work

Robotic value models use varied prediction meanings and evidence, while evaluations test distinct aspects of feedback quality. RoboValue connects these concerns through a shared sim-and-real protocol that jointly diagnoses four capabilities.

2.1 Value Models for Robotic Manipulation

  • Value models estimate completion ratings, task progress, stage-aware progress, temporal distance, or relative progress and local advantages.Some frameworks expose multiple interfaces for policy refinement, while recent methods incorporate temporal context, recovery, or synthesized failures.Source: §2.1 · pp. 4–5

2.2 Evaluation of Robotic Value Models

  • Prior evaluations cover completion labels, alignment, progress correlation, ranking, temporal order, subtask progress, demonstration-conditioned estimates, and recurring-state context.Process-level benchmarks also probe temporal questions, micro-progress, regression, hesitation, and retry behavior.Source: §2.2 · p. 5
  • Failure-oriented systems diagnose failures or guide correction, while robustness benchmarks test data-source variation and instruction sensitivity.RoboValue connects these concerns by testing recovery outcomes and sensitivity to instructions that change task requirements.Source: §2.2 · p. 5
  • RoboValue jointly tests task-state understanding, temporal progress, failure and recovery reasoning, and value consistency under diagnostic trajectories.Its shared sim-and-real protocol examines recurring visual states, recovery outcomes, and subtask gains across valid execution orders.Source: §2.2 · p. 5

3 RoboValue Dataset

RoboValue-Dataset provides simulation and real-world manipulation tasks with expert demonstrations and annotated test trajectories. Its trajectory designs, collection settings, and annotations support evaluation of execution quality, history, recovery, and valid solution variation.

3.1 Dataset Composition

  • The dataset covers 35 dual-arm manipulation tasks—15 in simulation and 20 in the real world—with 3,500 training demonstrations and 2,792 disjoint test trajectories.Test trajectories span standard, cross-embodiment, and cross-environment settings; each out-of-domain setting varies either embodiment or environment relative to standard conditions.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 Trajectory Design

  • Six trajectory types probe execution quality, history, recovery outcomes, and subtask order, including demonstrations, error continuations, recovery attempts, repeated states, and valid alternatives.Alternative valid solutions can vary operating arm or division of work while preserving task requirements.Source: §3.2 · p. 6

3.3 Data Collection

  • VR teleoperation trajectories include synchronized RGB-D observations from a head camera and two wrist cameras, plus robot states and action targets.Test collection covers standard, cross-embodiment, and cross-environment settings in simulation and the real world; training demonstrations use only the standard setting.Source: §3.3 · p. 7, Figure 4 · p. 7

3.4 Trajectory Annotation

  • Manual test annotations mark successful subtask intervals and failure-stage boundaries, including when recovery outcomes become observable.These timestamps support failure localization and separate evaluation of value trends during failure, continuation, recovery attempts, and outcomes.Source: §3.4 · p. 7

4 RoboValue Benchmark

RoboValue defines shared interfaces and evaluation protocols for assessing robotic value models across task understanding, progress monitoring, failure and recovery reasoning, and value consistency. Its metrics probe whether scores reflect requested goals, execution history, error outcomes, and equivalent progress across valid solutions.

4.1 Preliminaries; 4.2 Evaluation Protocol

  • A robotic value model maps visual observations and task instructions to a scalar assessment of goal satisfaction or progress, while score semantics and scales remain model-dependent.Shared adapters support scalar, pairwise, and subtask outputs; evaluation covers zero-shot and one-shot tracks, with embodiment and environment shifts assessed separately.Source: §4.1 · p. 8, §4.2 · p. 8

4.3 Capability Dimensions and Metrics

  • The benchmark organizes metrics around task outcomes, evolving progress, error and recovery effects, and the stability and consistency of value feedback.The four dimensions are task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency.Source: §4.3 · p. 8, Table 1 · p. 9

4.3.1 Task-State Understanding

  • Task-state metrics test success discrimination, sensitivity to instruction changes, and identification of the active subtask during execution.Success accuracy compares terminal scores of successful and unsuccessful episodes; task grounding compares gains under correct and negative instructions with the visual execution fixed, while subtask descriptions are judged against candidate subtasks.Source: §4.3.1 · pp. 9–10

4.3.2 Temporal Progress Monitoring

  • Temporal metrics assess progress ordering, response to reversed playback, and use of history when visually similar states recur.Cycle-VOC reverses recorded states while playback time advances; Memory-VOC uses recurring visual states, but a high Memory-VOC score alone does not establish effective history use.Source: §4.3.2 · p. 10

4.3.3 Failure and Recovery Reasoning

  • Failure and recovery metrics test whether value declines align with error onset and whether values track initial failure, corrective attempts, and recovery outcomes.Trajectory Recovery Reasoning distinguishes continued error, successful recovery, and failed recovery; expected value trends differ across their annotated stages.Source: §4.3.3 · p. 11

4.3.4 Value Consistency

  • Value Stability combines trend consistency with non-flat time coverage, penalizing reversals and prolonged intervals without value updates.It is interpreted alongside progress-oriented metrics because a stable trend can still move in the wrong direction.Source: §4.3.4 · p. 12
  • Cross-Solution Value Consistency compares gains for the same semantic subtask across valid solutions, allowing their overall value curves to differ.Differences in gains require interpretation in light of model value semantics and execution context; agreement alone does not establish that gains reflect task progress.Source: §4.3.4 · p. 12

5 RoboValue Leaderboard

The leaderboard compares representative value models across four capability dimensions in zero-shot and one-shot settings. Its leaders vary by capability, while failure/recovery reasoning and value consistency remain challenging.

5.1 Evaluated Value Models

  • The evaluation covers completion, progress, temporal-distance, and relative-progress models, plus LIV and RoboFAC.TOPReward uses 8B and 32B Qwen3-VL backbones; ProcVLM and VLAC are evaluated zero-shot and one-shot.Source: §5.1 · p. 13

5.2 Leaderboard

  • The leaderboard reports four capability scores and their overall mean, with separate zero-shot and one-shot rankings.It aggregates normalized in-domain and out-of-domain scores where available before averaging capability scores.Source: §5.2 · p. 13, Table 2 · p. 13
  • RoboMeter-4B leads zero-shot overall, while Robo-Dopamine 2.0-8B Preview leads one-shot and three capability dimensions; failure/recovery reasoning and value consistency remain challenging.RynnValue-4B is strongest in task-state understanding.Source: §5.2 · p. 13, Table 2 · p. 13
  • Zero-shot and one-shot evaluations do not train on the full RoboValue dataset; a Full-Shot track will permit training or fine-tuning on 3,500 demonstrations.All three tracks share the held-out test set and metrics, with separate rankings.Source: §5.2 · p. 14

6 Experiments

Across the evaluated models, strong outcome or forward-progress performance does not ensure reliable fine-grained instruction grounding, history-dependent progress tracking, failure and recovery assessment, or consistent subtask credit. The experiments also find broader degradation under environment shifts than embodiment shifts.

6.1 Main Results

  • Strong outcome and forward-progress performance can coexist with errors in fine-grained grounding, history-dependent tracking, failure and recovery assessment, and consistent subtask credit.The experiments also report that visually similar recurring states challenge models’ memory of completed subgoals.Source: §6.1 · pp. 14–15

6.2 Generalization Results

  • Models largely retain in-domain strengths across robot morphologies, but tested environment shifts cause broader degradation across models.Robo-Dopamine 2.0 better preserves progress tracking and retains the highest value stability under environmental shifts.Source: §6.2 · p. 16

6.3.1 Counterfactual Instruction Grounding

  • Counterfactual tests reveal both instruction insensitivity and responsive but semantically incorrect grounding; ProcVLM-2B adaptation raises overall TGA-CF from 14.36% to 51.16%.Object changes are especially challenging, and adaptation does not fully resolve fine-grained grounding.Source: §6.3.1 · p. 17

6.3.2 Subtask Identification

  • Zero-shot ProcVLM outperforms RoboFAC and FailSafe across all three SIA settings, but generally low scores show active-subtask identification remains difficult.One-shot ProcVLM slightly improves in-domain SIA while reducing performance under both out-of-domain shifts.Source: §6.3.2 · p. 17

6.3.3 Failure and Recovery Reasoning

  • RoboFAC more consistently predicts declines during failures and on continued-error and failed-recovery branches, while RoboMeter more consistently predicts increases after successful recovery.Across models, corrective activity may raise values even when recovery fails; unrelated value declines can also obscure failure onset, and rebounds can occur while errors remain unresolved.Source: §6.3.3 · pp. 17–18

6.3.4 Value Consistency Across Valid Solutions

  • Valid changes in semantic subtask order can substantially alter credit: the tea-set full-task CSVC scores are −40.98 for Robo-Dopamine 2.0-8B Preview and 2.30 for RoboMeter.This order dependence can treat semantically matched training segments differently solely because of execution context and position.Source: §6.3.4 · pp. 18, 20

7 Analysis

The analysis relates value reliability to model inputs, supervision targets, model size, and backbone choice. It identifies specific gaps in history-dependent progress, instruction grounding, recovery, and semantic consistency, while noting that comparisons do not isolate individual components.

7.1 How Do Input Interfaces Relate to Value Reliability?

  • Despite strong VOC and Cycle-VOC, history-using models show near-zero or negative Memory-VOC, indicating difficulty tracking accumulated progress.Sampled context can omit brief interactions or fail to integrate visible events; the analysis motivates context and memory mechanisms that preserve completed-subtask evidence.Source: §7.1 · p. 20
  • Broader camera coverage alone does not establish which task requirements are fulfilled, as Robo-Dopamine’s strong progress scores coexist with weak grounding and limited history tracking.The analysis emphasizes connecting visible interactions to their task-specific meaning.Source: §7.1 · p. 20
  • Progress gains from one-shot demonstrations do not ensure better subtask identification, making scalar progress and subtask recognition complementary checks.ProcVLM uses task-specific parameter adaptation, while VLAC conditions predictions on reference observations.Source: §7.1 · p. 20

7.2 Which Distinctions Should Supervision Capture?

  • RoboMeter and RynnValue distinguish broad tasks more reliably than fine-grained counterfactual instructions, motivating supervision that varies one requirement while holding execution and reference evidence fixed.Such examples can reward the instruction matching observed behavior rather than visual similarity to a demonstration.Source: §7.2 · p. 20
  • Temporal reversal supervision yields strong Cycle-VOC, but low Memory-VOC and recovery-stage errors show that reversal alone does not capture accumulated progress or successful correction.The analysis proposes pairing similar states with different completed subgoals and tying recovery examples to unresolved task conditions.Source: §7.2 · p. 20
  • Semantic-preserving visual perturbations are associated with better progress tracking under environment shifts in Robo-Dopamine 2.0 Preview, while valid execution orders still alter subtask gains.The analysis proposes cross-solution supervision to align gains for equivalent subtask completions across admissible orders.Source: §7.2 · p. 21

7.3 What Do Model Size and Backbone Choice Explain?

  • Model size yields mixed capability changes: larger checkpoints improve some measures while reducing others, so size alone incompletely predicts execution judgments.VLAC-8B improves grounding and progress tracking over VLAC-2B but has lower TRR; TOPReward-32B improves Cycle-VOC and TRR yet lowers SA, VS, and CSVC versus its 8B counterpart.Source: §7.3 · p. 21
  • RoboMeter outperforms RoboReward on SA and Cycle-VOC despite sharing its backbone, but differing objectives, data, temporal inputs, and score construction limit attribution to backbone identity.The analysis therefore points to supervision coverage and prediction granularity alongside model capacity.Source: §7 · p. 20, §7.3 · p. 21

8 Conclusion

RoboValue provides a unified sim-and-real evaluation of robotic value models. The conclusion reports persistent reliability gaps across instruction grounding, history, recovery, and valid execution orders, with broader degradation under environment than embodiment shifts.

8 Conclusion

  • Strong outcome discrimination and forward-progress correlation do not ensure accurate grounding, history-dependent progress tracking, failure localization, recovery assessment, or consistent subtask gains across valid orders.Models generally retain in-domain strengths under embodiment shifts, whereas environment shifts produce broader performance degradation.Source: §8 · p. 21

A Trajectory Types for Metric Evaluation

The trajectory categories supply distinct evidence for task understanding, progress, failure and recovery, temporal context, and consistency across valid solutions. Table 7 links these categories to the evaluation metrics.

A Trajectory Types for Metric Evaluation

  • Trajectory categories divide metric evidence across complete demonstrations, failure and recovery contrasts, long-horizon temporal cases, and multi-solution executions.Failure and recovery trajectories support SA and TRR; Error Continuation and Effective Recovery support FPL; multi-solution trajectories support SA, SIA, and CSVC comparisons.Source: A Trajectory Types for Metric Evaluation · p. 26, Table 7 · p. 26

B Comparison with Existing Benchmarks

RoboValue compares benchmark coverage and generalization protocols while using controlled execution contrasts to diagnose value feedback. Its task suite spans manipulation skills and includes decomposed tasks, instruction counterfactuals, and separate embodiment and environment shifts.

B Comparison with Existing Benchmarks

C.1 Manipulation Skill Coverage

  • The simulation suite covers 15 representative manipulation skills, from transport and placement to insertion, folding, tool use, handovers, and coordinated holding.Composing skills into multi-stage tasks introduces varied spatial relations, contact conditions, and ordering constraints.Source: §C.1 · p. 28, Figure 8 · p. 28

C.2 Task Descriptions and Subtask Decomposition

  • Task cards pair instructions and initial scenes with procedures and semantic-subtask decompositions; illustrated procedures represent one admissible ordering when multiple solutions exist.For example, mug hanging permits any order, while the toaster task requires both slices to be inserted before pressing the lever.Source: §C.2 · pp. 28, 30, 31

C.3 Original and Counterfactual Instructions

  • Counterfactual instruction tests evaluate five instructions against the same successful execution, changing objects, actions, placement, or constraints to test semantic sensitivity.The table covers 13 simulation tasks; two tasks dedicated to Memory-VOC are excluded.Source: §C.3 · p. 36, Table 10 · p. 37

C.4 Out-of-Domain Scenario Construction

  • Out-of-domain settings change embodiment or environment separately while preserving task goals and using the shared protocol without adaptation.The environment setting varies surface appearance, clutter, and lighting while retaining the standard embodiment.Source: §C.4 · p. 36

C.5 Task Execution Visualizations

This section documents task execution visualizations, real-world skill coverage, task and counterfactual instructions, subtask decompositions, and out-of-domain conditions, including embodiment and environment shifts used to examine value judgments of task progress and outcomes under visual and execution changes.

C.5 Task Execution Visualizations

  • Figure 9 presents all 15 simulation tasks as seven chronologically ordered frames tracing each execution from its initial scene through manipulation to successful completion.The frames show intermediate stages as well as the final outcome.Source: §C.5 · p. 38, Figure 9 · p. 38

D.1 Manipulation Skill Coverage

  • Figure 10 covers 15 representative real-world skills, from picking and placing to alignment, container manipulation, pouring, pressing, shaking, and striking.The skill range spans object position, configuration, and accumulated progress under physical sensing and contact conditions.Source: §D.1 · p. 39, Figure 10 · p. 39

D.2 Task Descriptions and Subtask Decomposition

  • Task cards document all 20 real-world tasks with original instructions, initial scenes, procedures, and semantic subtask decompositions, including constraints and alternative valid orders.For example, drawer operations specify an ordered sequence, while some placement tasks allow flexible ordering; the listed procedure gives one admissible execution when alternatives exist.Source: §D.2 · pp. 39, 43

D.3 Original and Counterfactual Instructions

  • Table 11 lists original instructions and four counterfactual variants for 14 TGA-CF real-world tasks, excluding six Memory-VOC tasks, evaluated on the same successful execution.Evaluation uses each complete modified instruction.Source: §D.3 · p. 50, Table 11 · p. 50

D.4 Out-of-Domain Scenario Construction

  • Real-world out-of-domain evaluation changes embodiment and environment separately while retaining task goals and using the shared protocol without additional adaptation.Source: §D.4 · p. 51
  • The cross-embodiment test compares the standard AgiBot Genie02 with a dual-arm ARX setup performing the same tasks with different visible morphologies and execution patterns.It examines transfer of value judgments about task progress and outcomes to a different physical robot.Source: §D.4 · p. 51
  • The cross-environment test varies tablecloth color, lighting, and task-irrelevant clutter while retaining the standard embodiment to assess attention to task-relevant interactions.The two configurations use different tablecloth colors and lighting setups; trajectories include two to four irrelevant objects with varied positions and orientations.Source: §D.4 · p. 51

D.5 Task Execution Visualizations

Figures 11 and 12 visualize successful executions of all 20 real-world tasks through seven chronologically ordered frames per task.

D.5 Task Execution Visualizations

  • Seven chronological frames per row trace successful executions across the first ten and remaining ten real-world tasks.Together, the figures cover all 20 real-world tasks, from initial scene through intermediate interactions to final configuration.Source: §D.5 · p. 51, Figure 11 · p. 51, Figure 12 · p. 52

E Data Collection Protocol

The collection protocol defines valid task outcomes, alternative execution orders, and controlled error and recovery branches in simulation and real-world examples. It also profiles the evaluated baseline models and their value-prediction approaches.

E.1 Simulation Collection Example

  • The simulation Fill Pen Holder task requires four pens inside an upright holder; insertion order is unrestricted, yielding 24 valid orders with varied arm assignments.The collection samples 20 orders, with two trajectories per order under the standard ARX setting.Source: §E.1 · p. 53, Table 12 · p. 53
  • Simulation collection includes 75 successful and 65 failure/recovery trajectories, with EC and IR unsuccessful and ER resolving the seeded error.Each of five error scenarios contributes seven error continuations, three effective recoveries, and three ineffective recoveries; 100 trajectories are in-domain and 20 in each OOD setting.Source: §E.1 · pp. 53–54

E.2 Real-World Collection Example

  • The real-world Family Tea Set task requires five placement conditions; four placement units admit 24 valid orders while teapot and lid placement stay consecutive.Teapot and lid retain separate annotated subtask boundaries despite forming one unit for order enumeration.Source: §E.2 · p. 54, Table 13 · p. 54
  • The real-world collection contains 58 successful and 52 failure/recovery trajectories; EC and IR end unsuccessfully, whereas ER satisfies all five success conditions.Four seeded scenarios cover dropped lids, misoriented tongs, misplaced teapots, and misplaced canisters.Source: §E.2 · pp. 54–55

F.1 Baseline Model Profiles

  • ProcVLM structures progress around procedures and subtasks, Robo-Dopamine predicts signed progress changes, and RoboFAC generates video-based task and failure assessments.RoboMeter combines frame-level progress supervision with trajectory-pair preference learning; the benchmark evaluates its progress head.Source: §F.1 · pp. 55–56
  • RoboReward assigns ordinal completion ratings, RynnValue estimates remaining time, and TOPReward uses affirmative-token log probability as its completion signal.The evaluated models include RoboReward 4B and 8B, RynnValue 4B and 8B, and Qwen3-VL-8B-Instruct and 32B-Instruct for TOPReward.Source: §F.1 · p. 56
  • VLAC combines signed pairwise progress and completion judgments; FailSafe describes subtasks and recovery actions, while LIV scores image–instruction similarity.RoboValue uses FailSafe subtask descriptions for SIA, not its failure verdicts or recovery suggestions as scalar values.Source: §F.1 · pp. 55–56

F.2 Input–Output Interfaces and Adapter Implementations

Model adapters map predictions to shared scalar-value, directional-comparison, and subtask-description interfaces. The scoring protocol normalizes metrics and aggregates them into four capability dimensions and an overall score.

F.2 Input–Output Interfaces and Adapter Implementations

  • Adapters map model outputs to shared scalar-value, directional-comparison, and subtask-description interfaces while retaining native value meaning and aligning larger values with greater advancement.For scalar-only models, pairwise comparison is the difference between the two scalar values; Robo-Dopamine and VLAC provide native pairwise predictions.Source: §F.2 · p. 56, Table 14 · p. 57
  • SIA evaluates generated current-subtask descriptions by judging them against candidate subtasks, with label probabilities aggregated geometrically within each task.FailSafe, ProcVLM, and RoboFAC descriptions are generated without candidate labels.Source: §F.2 · p. 56
  • ProcVLM’s one-shot adapter uses demonstration-derived subtask and progress examples, then reads generated progress percentages as scalar values and extracts the current unfinished action.Its adaptation targets are constructed independently of the benchmark’s test annotations.Source: §F.2 · pp. 56–57
  • Robo-Dopamine accumulates pairwise progress changes and fuses them with initial- and goal-anchored predictions; VLAC accumulates signed pair outputs for scalar values.Both models retain native directional predictions for pairwise comparisons.Source: §F.2 · pp. 57–58
  • RoboFAC converts five-level A–E rubric logits into scalar values, while its generated subtask description is passed to the SIA judge.The rubric and equal level spacing define this adapter readout rather than a calibrated native progress scale.Source: §F.2 · pp. 57–58
  • Other scalar adapters use model-specific readouts, including progress-head outputs, discrete completion levels, sign-reversed remaining time, affirmative-token log probability, and image–instruction similarity.The adapters use the models’ scalar outputs for shared-protocol comparisons and outcome metrics.Source: §F.2 · pp. 56, 58

F.3 Leaderboard Scoring

  • Leaderboard normalization uses theoretical metric ranges; OOD scores average cross-embodiment and cross-environment results before combining with ID scores at weights of one-half, one-quarter, and one-quarter.Missing metrics contribute zero after normalization without redistributing their weights.Source: §F.3 · p. 58
  • The overall score equally averages task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency.Task-state scores average SA and TGA, with SIA reported separately; the other three dimensions combine their specified component metrics.Source: §F.3 · p. 58

G Sensitivity to Evaluation Settings

Across the tested parameter settings, most pairwise model rankings remain stable, though some closely ranked models reverse under specific settings. Absolute scores can still vary, so comparisons should use matched evaluation settings.

Overall Findings

  • 126 of 132 simulation pairwise comparisons (95.45%) and 126 of 130 real-world comparisons (96.92%) preserve their reference ordering across the tested parameter values.Figure 13 shows largely stable rankings despite changes in absolute scores.Source: G Sensitivity to Evaluation Settings · p. 59, Figure 13 · p. 61

Endpoint Windows: SA and TGA

  • At the 4% endpoint window, SA reverses adjacent RoboMeter–ProcVLM rankings in simulation and RynnValue–ProcVLM rankings in the real world, with gaps of 3.82 and 1.60 percentage points.For TGA-CT, rankings are preserved across 1%, 2%, and 4% windows in simulation; only RoboMeter and RynnValue reverse at 1% in the real world, while TGA-CF orderings remain unchanged.Source: G Sensitivity to Evaluation Settings · pp. 59–60

Temporal Settings: FPL and TRR

  • FPL reverses the close Robo-Dopamine–RynnValue ordering in simulation at 10 Hz and α = 0.1, while the real-world ranking remains unchanged.TRR rankings match at 10 and 15 Hz; at 5 Hz, RoboMeter and Robo-Dopamine reverse only in the real world, and varying the minimum sample count causes no strict reversals.Source: G Sensitivity to Evaluation Settings · p. 60

Value Stability and Consistency: VS and CSVC

  • VS preserves model order across tested sampling rates, but absolute scores decrease with denser sampling; its rankings are stable while absolute scores require matched sampling settings.The sweep does not isolate the contributions of efficiency-ratio and non-flat-time terms to the score change.Source: G Sensitivity to Evaluation Settings · p. 60
  • In simulation, CSVC reversals occur at two boundary pairs for RynnValue/ProcVLM and six for RoboMeter/RynnValue, with respective score gaps of 0.47 and 0.75 points; other simulation orderings and the full real-world ranking remain unchanged.Cycle-VOC preserves model ordering across all tested sampling densities in both environments, with largest absolute deviations from the reference of 0.22 points in simulation and 0.42 points in the real world.Source: G Sensitivity to Evaluation Settings · p. 60

G Sensitivity to Evaluation Settings

H.1 Main Results

The simulation results are organized as metric-specific tables for standard and shifted settings, while the supplied passages give the clearest model comparison in the cross-task score matrices. Those matrices show differences in instruction selectivity across model configurations.

H.1 Main Results

  • The in-domain simulation tables report metric-specific results on trajectories collected with the standard ARX setup, but the supplied passages do not state their comparative scores.Source: §H.1 · p. 62, Table 18 · p. 62, Table 19 · p. 62

H.2.1 Cross-Embodiment Generalization

  • Cross-embodiment simulation evaluation replaces ARX with a dual-arm UR5e while preserving task goals; the supplied passages describe the test but not its results.Source: §H.2.1 · p. 67, Table 29 · p. 67

H.2.2 Cross-Environment Generalization

  • Cross-environment simulation evaluation retains ARX while varying surface appearance, lighting, and clutter, with task goals held fixed; no outcome is stated in the supplied passages.Source: §H.2.2 · p. 71, Table 38 · p. 71

H.3 Cross-Task Score Matrix Visualizations

  • In the simulation cross-task matrices, RoboMeter and RynnValue-4B show strong zero-shot diagonal selectivity, while one-shot ProcVLM is more selective than Robo-Dopamine variants.Robo-Dopamine variants show broad off-diagonal responses, indicating gains assigned to unrelated instructions.Source: §H.3 · p. 74, Figure 14 · p. 75, Figure 15 · p. 75

I.1 Main Results

The real-world results are organized into metric-specific tables for standard and shifted settings, while the supplied passages report model comparisons in cross-task score matrices. These matrices show differences in instruction selectivity across model configurations.

I.1 Main Results

  • The in-domain real-world tables report metric-specific results on trajectories collected with the standard AgiBot Genie02 setup, but the supplied passages do not state comparative scores.Source: §I.1 · p. 76, Table 47 · p. 76, Table 48 · p. 76

I.2.1 Cross-Embodiment Generalization

  • Cross-embodiment real-world evaluation replaces AgiBot Genie02 with a dual-arm ARX setup while preserving task goals; the supplied passages describe the test but not its results.Source: §I.2.1 · p. 81, Table 58 · p. 81

I.2.2 Cross-Environment Generalization

  • Cross-environment real-world evaluation retains AgiBot Genie02 while varying tablecloth appearance, lighting, and tabletop clutter; no outcome is stated in the supplied passages.Source: §I.2.2 · p. 85, Table 67 · p. 85

I.3 Cross-Task Score Matrix Visualizations

  • In real-world cross-task matrices, RoboMeter and RynnValue variants retain strong zero-shot diagonal selectivity, while one-shot ProcVLM is more selective than other demonstration-based configurations.Robo-Dopamine often assigns large gains to both matching and mismatched instructions.Source: §I.3 · p. 88, Figure 16 · p. 89, Figure 17 · p. 89

Related papers

Loading 2610.26663v1…