Source-linked AI summary
R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models
Yidi Wang, Feixiang Ruan, Ruoqu Chen, Jie Yin, Yang Yu, Mengdi Xu, Kaifeng Zhang
TL;DR
Robot policy evaluation is labor-intensive, unstable, and limited by binary success metrics that do not fully describe execution quality. R2S-Eval combines real-to-sim calibration with VLM pairwise preference evaluation to rank rollout behaviors, and experiments show reliable, stable, human-aligned, effort-reducing, quality-aware conclusions.
Problem
Conventional real-world evaluation requires repeated hardware operation and mainly uses success rates, limiting information about execution quality and producing unstable rankings.
Method
R2S-Eval generates rollout videos in a real-world-calibrated simulator and aggregates VLM judgments of pairwise video preferences into policy rankings.
Results
Experiments show reliable and stable policy conclusions that agree with human judgments, reduce manual effort, and capture execution-quality differences beyond success counts.
Takeaways & Limitations
R2S-Eval shifts robot evaluation toward automated, statistically stable, and behavior-aware assessment based on preferences over rollout behaviors.
Abstract
from arXiv · showhide
Evaluating robot manipulation policies is becoming increasingly important as generalist models, particularly vision-language-action (VLA) models, are deployed on physical robots. However, conventional real-world evaluation remains labor-intensive, unstable, and insufficiently informative. It requires repeated hardware trials, manual scene resets, and continuous operator monitoring, may produce different policy rankings across repeated evaluations, and primarily relies on success-rate metrics that provide limited information about execution quality. In contrast, humans assess robot performance by observing and comparing complete behaviors rather than relying solely on binary success outcomes. To this end, we propose R2S-Eval, an evaluation pipeline that combines real-to-sim calibration with vision-language model (VLM) preference evaluation. The real-to-sim component efficiently generates rollout videos in a simulator calibrated to the real-world evaluation setting, thereby reducing the need for repeated hardware trials. The VLM evaluator assesses the execution quality of rollout videos and produces pairwise preferences, which are subsequently aggregated into policy rankings. We further introduce a protocol to assess whether the proposed evaluation pipeline yields validated policy conclusions while mitigating the key challenges of conventional real-world evaluation. Experiments in both simulation and real-world settings demonstrate that R2S-Eval produces reliable and stable policy conclusions, achieves agreement with human preferences, substantially reduces repeated hardware-operation effort, and reveals behavior-quality differences that are not captured by binary success labels. In general, R2S-Eval advances robot evaluation from manual success counting toward automated, statistically stable, and quality-aware evaluation of robot behavior. Project page: https://r2s-eval.github.io.
I. INTRODUCTION
R2S-Eval addresses the labor, instability, and limited informativeness of conventional hardware evaluation by combining calibrated real-to-sim rollouts with VLM preference-based policy ranking. It evaluates complete behaviors rather than only binary success outcomes and validates conclusions across reliability, stability, human agreement, effort, and behavioral informativeness.
- Motivation: Conventional evaluation deploys fine-tuned policies on hardware, repeatedly records task successes, and ranks policies by empirical success rates.Each trial involves scene setup, execution, object reset, and hardware monitoring.
- Motivation: Hardware evaluation is labor-intensive and unstable because small changes in placement, contact, perception, or robot state can alter outcomes and rankings.Repeated trials may therefore produce inconsistent policy conclusions.
- Motivation: Success rates are too coarse to distinguish motion quality, corrective behavior, or progress among rollouts sharing the same success label.Humans instead compare complete executions and can differentiate both successful and failed behaviors.
- R2S-Eval: R2S-Eval calibrates a simulator to the real world, deploys candidate policies in closed loop to generate rollout videos, and uses VLM comparisons to produce preferences and policy rankings.The pipeline is designed to obtain behavior videos efficiently while judging execution quality automatically.
- R2S-Eval: R2S-Eval formulates manipulation evaluation as preference estimation over rollout behaviors, producing rankings that reflect execution quality rather than binary success counts.Observed pairwise preferences are summarized in a comparison matrix and used to infer latent policy quality scores.
- Validation: The validation protocol examines ranking reliability and stability, agreement with human preferences, reduced hardware-operation effort, and behavioral information beyond success counts.These axes are intended to assess whether evaluation conclusions remain meaningful and stable across settings and accumulating evidence.
B. Real-to-Sim Calibration
R2S-Eval uses real-to-sim calibration to obtain behavior videos efficiently while preserving factors that determine policy behavior. Policies are adapted for simulation, executed in closed loop, and evaluated through their simulated rollouts as proxies for hardware behavior.
- Calibration goal: Real-to-sim calibration provides an efficient alternative to directly collecting rollout videos on hardware.It addresses the labor and instability inherited from conventional real-world video collection.
- Calibration goal: The calibrated simulation matches robot geometry, kinematics, joint limits, control interfaces, tasks, object placements, and camera viewpoints.The objective is behavioral calibration rather than a perfectly photorealistic digital twin.
- Adaptive policy training: Residual differences in friction, compliance, and sensing noise can make direct simulation deployment unstable in contact-rich manipulation.Without mitigation, rollouts may reflect domain mismatch instead of policy differences of interest.
- Evaluation rollout: R2S-Eval evaluates simulation-adapted counterparts of candidate policies in closed-loop calibrated simulation and collects videos from their own executions.The simulated policies serve as proxies for the corresponding hardware policies, whose ranking consistency is assessed experimentally.
- Evaluation rollout: Calibrated simulation reduces the need for repeated hardware trials during policy evaluation.The resulting rollout videos are passed to VLM preference evaluation.
C. VLM Preference Evaluation
R2S-Eval compares rollout videos with a VLM using structured execution criteria, then aggregates pairwise judgments into policy rankings with a Bradley–Terry model.
- VLM judgment: The VLM acts as an automated video-based preference judge intended to match human judgments of overall execution quality.It is used as a general visual judge rather than a hand-designed task scorer.
- VLM judgment: Each comparison uses videos from the same task and initial configuration, with balanced task contributions to avoid task-frequency and difficulty confounds.Repeated comparisons estimate preferences between policy rollout distributions.
- VLM judgment: The VLM describes motion smoothness, temporal continuity, task progress, and visually observable contact and effort control before comparing anonymized video pairs.The pairwise output is binary and uses the same criteria for both descriptions.
- Preference aggregation: Pairwise task-level judgments are pooled into a preference matrix and converted into policy-level rankings with the Bradley–Terry model.The model represents the probability that one policy is preferred over another from their latent quality scores.
- Preference aggregation: The preference model estimates policy scores by fitting Bradley–Terry strength parameters and ranking policies by their estimated scores.The parameters are fit with minorization-maximization under a sum-to-one identifiability constraint.
D. Validation Protocol
The validation protocol evaluates whether R2S-Eval produces reliable, stable, practical, and behaviorally informative policy conclusions. It emphasizes agreement with target-setting performance, convergence, reduced intervention, and execution details beyond success.
- Validation axes: Reliability requires policy rankings to agree with a reference ranking that reflects the actual capabilities of the evaluated policies.Stability additionally requires scores or rankings to converge with low variance across trials and configurations.
- Validation axes: Behavioral informativeness requires preserving differences in progress, recovery, and control quality that binary success records discard.These differences are especially relevant when rollouts share the same success or failure outcome.
IV. EXPERIMENTS
Experiments test whether calibrated real-to-sim evaluation preserves hardware-consistent policy conclusions and whether VLM preferences support stable, informative rankings. Across real-world, simulation, and real-to-sim settings, the evaluation compares six VLA policies using rollout behavior and preference scores.
- Experimental Questions: The experiments address real–sim consistency, policy-performance reflection, evaluation stability and cost, human agreement, and execution-level distinctions beyond success labels.These questions are evaluated across simulation and calibrated real-to-sim settings.
- Experimental Setup: Six candidate VLA policies are evaluated, including π0, π0.5, OpenVLA, NORA-Long, SmolVLA, and X-VLA.The real-world and real-to-sim experiments use these candidate policies.
- Real–Sim Consistency: 2.13 percentage points is the mean absolute real–sim success-rate difference across policy–task pairs.The largest gap is 7.0 percentage points, so calibration does not eliminate every discrepancy.
- Real–Sim Consistency: The calibrated simulation preserves comparative policy conclusions despite not implying behavioral equivalence between real and simulated policies.Real and simulated rollouts also show qualitatively similar failure modes, including oscillations without progress.
B. VLM Preference Evaluation Results
VLM preference evaluation ranks policies from rollout-video comparisons and is assessed for agreement, stability, effort, and sensitivity to execution quality. Results show strong ranking consistency, substantial human agreement, reduced hardware effort, and distinctions among rollouts sharing binary outcomes.
- Evaluation Protocol: The evaluation measures ranking agreement, score correlation, rank violations, confidence-interval width, and agreement with human preferences.Reference performance is mean success rate across tasks, and bootstrap intervals use 1,000 replicates.
- Simulation Results: Across all VLMs, LIBERO rankings achieve average Spearman’s ρ of 0.823, Pearson r of 0.924, and MMRV of 0.018.The narrow BT confidence intervals indicate robust ranking estimation with low uncertainty.
- Simulation Results: 82.9% human agreement is achieved, while all evaluated VLMs exhibit similar performance despite architecture and scale differences.The VLM ranking generally follows success-rate ranking, with minor inversions among nearly tied policies.
- Order-Swap Consistency: 91.5% of real–sim video-pair preferences remain unchanged when A/B order is reversed.Random assignment of anonymized videos to A and B is used to average out residual positional bias.
- Repeated Hardware Effort: Repeated hardware-operation time increases approximately linearly with tasks or trials, whereas R2S-Eval requires no additional real-world hardware effort.The measured conventional effort includes execution, initialization, reset, monitoring, and success recording.
- Behavior Case Study: VLM preferences distinguish execution quality when rollout pairs share identical success labels.The comparisons include both two-successful-rollout and two-failed-rollout pairs.
- Behavior Case Study: The evaluator uses task progress, motion continuity, and control quality to capture behavior differences invisible to binary success labels.In the examples, it prefers smooth or progressing executions over retries or oscillatory motions.
- Summary: R2S-Eval combines calibrated simulation for efficient video collection with VLM preference evaluation for reliable, stable, quality-aware policy conclusions.The reported scope includes simulation and real-world experiments, human agreement, reduced manual effort, and behavior-quality differences beyond success counts.
Appendix
The appendix documents the real-world platform, unified action interface, calibrated simulation assets, shared environment configuration, and matching observation and control interfaces. These implementation choices align simulated evaluation with the real system while preserving common task and workspace structure.
- Real-World Platform: Real-world experiments use SharpaNorth 1, a dual-arm humanoid with two 7-DoF arms, 22-DoF hands, and multiple body joints and cameras.The observation system includes two head-mounted RGB cameras and two wrist-mounted fish-eye cameras operating at 30 Hz.
- Action Interface: The unified action interface contains 65 joint targets, with 63 actively controlled DoFs because two waist dimensions remain fixed.The retained waist dimensions provide interface compatibility.
- Evaluation Fairness: All evaluated policies receive the same camera streams and use identical control frequency, initialization, termination criteria, and unified action conversion.Each model consumes the observation subset supported by its architecture.
- Simulation Environment: Simulation reconstructs task objects as corresponding assets and shares the real-world robot, workspace, cameras, lighting, rendering, and task-specific initial states.The common scene configuration differs across tasks only in manipulation objects and initial states, with tabletop height adjustable when needed.
- Simulation Interface: The simulator exposes the same observation modalities and control interface as the real-world system for data collection and closed-loop evaluation.Simulation parameters are summarized in Table VI.
B.3. Task Implementation
Task implementation aligns simulated and real-world task specifications while estimating object poses and generating replay trajectories in the calibrated simulator. The appendix also records automatic action replay and policy-training procedures used for evaluation.
- Task Configuration: Each simulated task matches its real-world counterpart in language instruction, initialization protocol, episode configuration, and success evaluation.Movable objects are randomly initialized within predefined regions while avoiding initial collisions unless otherwise specified.
- Object Pose Estimation: Initial object poses are estimated using open-vocabulary detection and segmentation with Grounding DINO and SAM.The first RGB frame is segmented to obtain object masks, centroids, and principal orientations.
- Automatic Action Replay: Recorded joint-angle action sequences are replayed in the calibrated simulator, with automatic searches over nearby object-pose initializations to address contact-sensitive discrepancies.Small initialization errors can change contact timing and cause real-world successes to fail in simulation.
- Policy Training: Real-world and simulation-side policy counterparts use the same pretrained checkpoint, task definitions, interfaces, training recipe, and update budgets within each VLA family.New action dimensions are randomly initialized when a pretrained policy does not natively support the target dimensionality.
- Automatic Action Replay: Automatic action replay resets the simulator, initializes the robot from the first recorded joint state, executes the action sequence, and returns a trajectory when the success evaluator passes.The algorithm includes a retry limit and samples perturbations from a specified distribution after the initial pose.
C.2. Detailed Training Parameters
The appendix specifies training, execution, video-description, and preference-comparison settings for policy evaluation. VLM descriptions emphasize observable motion quality, action continuity, task completion, and force control, while comparisons produce binary choices.
- Each VLA predicts a 25-step action chunk, executes the first five actions, and is then queried again under receding-horizon control.
- VLA policies are fine-tuned until the training objective stabilizes, with training on NVIDIA H20 GPUs and inference and simulation on RTX 4090 GPUs.
- Video descriptions assess motion smoothness, action continuity, task success, and force or effort control using only visibly executed behavior.
- The description rules require explicit treatment of incomplete subtasks as failures and restrict final outcomes to successful, partially successful, or failed.
- Preference comparisons use the same four criteria, treat smooth continuous actions as positive evidence and jerky or unnecessarily adjusted actions as negative evidence.
- The comparison procedure internally extracts evidence and scores both videos, then outputs only one uppercase choice: A or B.
D.2. Detailed VLM Parameters
The VLM configuration appendix points to the inference settings used for video description and preference evaluation. The supplied passages identify the configuration table but do not provide its parameter values.
- VLM inference configurations for video descriptions and preference evaluation are summarized in Table VII.
D.3. Additional Results on Preference Matrix
Additional results document preference-matrix construction and Bradley–Terry aggregation for LIBERO and real-to-sim evaluations. Human annotations provide reference preferences for assessing VLM judgments.
- Win-rate matrices at the final 2,000-comparison checkpoint are reported for LIBERO and real-to-sim settings across different VLMs.
- Bradley–Terry scores are reported in Table IX for LIBERO and Table X for real-to-sim, with fitting details specified in Table VIII.
- Human references come from three independent annotators evaluating 200 task-stratified video pairs per setting, with majority vote defining each preference.
- The annotation sets include 66 success–success, 67 failure–failure, and 67 success–failure pairs in both LIBERO and real-to-sim settings.
- Annotators compare videos using motion smoothness, continuity or fluidity, task completion, and force or effort control, while following rules against hallucinating unstated details.
D.8. Order-Swap Evaluation
The order-swap evaluation audits whether VLM preferences depend on which video is presented as A or B. It reverses presentation order while preserving the underlying video identities and reports consistency across evaluators.
- Each VLM repeats preference evaluation after reversing the A/B order for every task-stratified real-to-sim video pair.
- A comparison is consistent when both queries select the same underlying video after outputs are mapped back to video identity.
- 91.5% mean order-swap consistency is obtained across the eight VLM evaluators.
- The swap test serves only as a position-bias audit and does not alter the main preference matrix.
- Evaluation materials include seven real-world tasks, seven simulated tasks, LIBERO and real-to-sim win-rate matrices, and convergence curves for VLA policies.