PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Real-World Robot Manipulation Policies in Simulation

Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, Ted Xiao

arXiv:2405.05941v1cs.ROcs.CVcs.LG

TL;DR

Generalist manipulation policies are difficult to evaluate scalably and reproducibly in the real world. The paper introduces SIMPLER, which mitigates real–sim control and visual gaps without requiring full digital twins. Paired evaluations show strong correspondence with real performance and behavior modes.

  • Problem

    Real-world evaluation of increasingly broad generalist manipulation policies is expensive, inefficient, and difficult to reproduce.

  • Method

    The paper builds SIMPLER environments using approaches that mitigate control and visual disparities, including system identification, green-screening, and texture baking.

  • Results

    Paired sim-and-real evaluations show strong correlation in policy performance and accurately reflect real-world behavior modes such as sensitivity to distribution shifts.

  • Takeaways & Limitations

    SIMPLER supports simulated manipulation-policy evaluation as a reliable, scalable, and reproducible complement to costly real-world evaluations.

  • Takeaways & Limitations

    The current environments focus on rigid-object tasks; green-screening is limited to fixed cameras and manual asset curation remains necessary.

Abstract

from arXiv · show

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

I. INTRODUCTION

SIMPLER addresses the scalability and reproducibility challenges of evaluating generalist manipulation policies by using purpose-built simulation rather than full digital twins. It mitigates control and visual gaps and shows strong real–sim performance and behavior-mode correspondence.

  • Real-world evaluation becomes increasingly expensive and difficult to reproduce as generalist policies span more tasks, environments, and robot embodiments.
  • SIMPLER evaluates policies trained on real data in purpose-built simulated environments as a scalable, reproducible complement to real-world evaluation.
  • Exact digital twins are unnecessary; environments only need sufficient realism to correlate simulated policy performance with real-world performance.
  • The proposed pipeline mitigates control and visual disparities through offline system identification, real-background green-screening, and texture baking from real-world images.
  • Paired evaluations across RT-1 and Bridge-V2 setups show strong correlation between SIMPLER and real policy performance, including sensitivity to distribution shifts.
  • SIMPLER provides open-source environments, policy-inference code, and workflows for evaluating policies and creating new environments.

III. USING PHYSICS SIMULATORS FOR EVALUATION OF ROBOT MANIPULATION POLICIES

The paper frames simulated evaluation as a scalable, reproducible complement to real-world testing, targeting agreement in relative policy performance rather than exact behavioral reproduction. It motivates MMRV alongside Pearson r because Pearson alone can misjudge ranking consistency and is sensitive to narrow performance ranges.

  • Problem formulation: Simulated evaluation aims to reduce costly real-world testing while providing broader, controlled sweeps of policy performance.It is intended as a complement rather than a replacement for real-world evaluation.
  • Problem formulation: The target is agreement in relative policy performance: if one policy outperforms another in reality, simulation should preserve that comparison.The simulator need not reproduce real-world behavior one-to-one.
  • Metrics: Pearson r measures linear consistency between real and simulated performance, with values near 1 indicating a well-functioning pipeline.It reflects whether real-world success-rate improvements correspond to linear simulated improvements.
  • Metrics: Pearson r can penalize correct rankings without linear recovery and can vary sharply when real-world policy performances occupy a narrow range.Small differences in that range may reflect evaluation noise rather than meaningful pipeline quality.
  • Metrics: MMRV measures ranking consistency while weighting incorrect pairwise rankings by their real-world performance margins.It averages each policy’s worst-case rank violation across policies; lower values indicate better agreement.

IV. BUILDING A REAL-TO-SIM EVALUATION SYSTEM BY ADDRESSING CONTROL AND VISUAL GAPS

The evaluation pipeline addresses real-to-sim control disparities through offline system identification, aligning simulated robot responses with real open-loop trajectories. The resulting controller improves end-effector tracking and can recover a missed coke-can grasp.

  • Control gap: The control gap is defined by differences between simulated and real robot responses to the same policy actions.The desired outcome is comparable 6D end-effector and joint-position trajectories.
  • System identification: System identification compares real and simulated open-loop trajectories under shared action sequences and controller parameters.The losses include translation and rotation errors, with simulated stiffness and damping parameters optimized against real motion.
  • System identification: A small sample of existing offline demonstration trajectories supplies the data for computing system-identification losses without collecting new data.The datasets come from existing open-source real-robot demonstrations.
  • System identification: Simulated annealing searches normalized stiffness and damping ranges, repeating optimization across three progressively reduced ranges.The parameters with the lowest Lsysid initialize the next search round.
  • Results: After system identification, the simulated arm tracks real motion more accurately and successfully reproduces the coke-can pick-up.Naive controller parameters produced inaccurate tracking and a missed grasp.

B. Mitigating the Real-to-Sim Visual Gap

Visual Matching reduces the real-to-sim appearance gap by combining real-background compositing with texture adaptation for objects and robot links. The paper also evaluates Variant Aggregation, which estimates performance by aggregating results across randomized visual variants.

  • Motivation: Visual discrepancies can shift learned policy behavior, while manually recreating realistic assets and lighting is labor-intensive.The approach therefore targets a modest-effort visual alignment rather than full scene reconstruction.
  • Visual Matching: Background green-screening alone can be insufficient because policies may respond to foreground object and robot textures.Simulation assets can differ from real objects through texture, material, and lighting.
  • Visual Matching: Visual Matching combines green-screening of simulated interactive assets onto real backgrounds with texture matching for simulation assets.These are the two core components shown in Fig. 5.
  • Texture matching: Texture matching projects segmented real-object textures onto simulation meshes or selectively transfers real colors to existing robot texture maps.The choice depends on whether the simulated asset’s texture map already resembles the real counterpart.
  • Texture matching: Multiple tuned robot-arm colors are averaged across evaluations to reduce confounding from color changes during manipulation.The colors are selected to match real-world textures at different task phases.
  • Variant Aggregation: Variant Aggregation randomizes visual scene aspects and aggregates evaluation results across variants as an alternative to minimizing the visual gap.The paper empirically compares it with Visual Matching.

V. SIMPLER: SIMULATED MANIPULATION POLICY EVALUATION FOR REAL ROBOT SETUPS

SIMPLER provides simulated environments for common Google Robot and WidowX BridgeV2 evaluation setups, covering varied manipulation conditions. Its construction uses standard assets and alignment procedures, while experiments test policy correlation, physical-property sensitivity, and simulator independence.

  • SIMPLER environments: SIMPLER targets the Google Robot from RT-series work and the WidowX BridgeV2 setup.It is designed for commonly used real-robot evaluation configurations.
  • SIMPLER environments: The suite spans tasks, skills, objects, object poses, backgrounds, and lighting conditions representative of corresponding training datasets.Tasks emphasize largely rigid-body objects whose dynamics modern simulators can approximate.
  • Implementation: SIMPLER is built on SAPIEN, with experiments showing that the contributions can also be reproduced in Isaac Sim.This tests whether the approach depends on a particular physics simulator.
  • Implementation: Environment construction combines public or ROS-exported robot assets, repository or reconstructed object models, dimension matching, and texture-tuned assets.Articulated objects such as cabinets require manual model construction and represent the most human effort.
  • Implementation: Object densities and friction coefficients are assigned from material properties or mass-volume estimates, while simulated robot and camera poses are aligned to real observations.Users can install the environments through pip and access them through the Gym API.
  • Evaluation scope: Experiments evaluate whether simulated rankings correlate with real rankings, reproduce policy behavior modes, tolerate physical-property gaps, and transfer across simulators.These questions define the suite’s empirical evaluation scope.

A. Experimental Setup

The paired evaluations compare open-source manipulation policies across real and SIMPLER environments on Google Robot and BridgeData V2 tasks. SIMPLER, especially with Visual Matching, more closely reflects real-world policy rankings than validation MSE.

  • Main evaluation results: SIMPLER shows strong correlation between simulated and real-world policy performance across most evaluated checkpoints and tasks.High-performing policies in reality also perform highly in SIMPLER, while low-performing policies show correspondingly low simulated performance.
  • Main evaluation results: For BridgeData V2, SIMPLER ranks all policies correctly on all but one task, with MMRV and Pearson r computed per task and averaged.The figure reports real-versus-simulation success rates for BridgeData V2 tasks.
  • Evaluation protocol comparison: SIMPLER produces much stronger real-evaluation correlation than validation MSE for ranking six open-source policy checkpoints.Visual Matching outperforms Variant Aggregation in the comparison reported for Google Robot tasks.
  • Evaluation protocol comparison: SIMPLER evaluations reflect relative policy performance more accurately than validation MSE, which yields high MMRV and low Pearson r.This comparison concerns imitation-learning policy ranking on Google Robot tasks.

C. SIMPLER Evaluations Accurately Model Policy Robustness to Distribution Shifts

SIMPLER captures policy sensitivity to distribution shifts and remains effective across plausible physical-property settings and alternative physics simulators. The ablations show that visual-gap mitigation works best when scene components are matched jointly.

  • Distribution-shift robustness: SIMPLER accurately reflects robustness to shifts in background, lighting, distractors, table texture, and robot camera pose.Camera-pose and table-texture changes significantly affect performance, whereas lighting and distractor changes have relatively minor effects.
  • Distribution-shift robustness: 4% versus 25% real-world performance decreases show greater robustness to unseen solid table colors than unseen patterned textures.SIMPLER reproduces this pattern with 2% versus 24% average simulated performance decreases.
  • Novel distribution shifts: 0% versus 29.3% success for Octo-Base under untuned versus tuned simulated arm textures identifies strong texture sensitivity that SIMPLER predicts in real evaluations.Real-world tests using gift-wrapping papers support the finding that Octo-Base is more arm-texture-sensitive than RT-1-X.
  • Visual-gap ablations: Jointly matching backgrounds, object textures, and robot textures significantly improves real-to-sim correlation, whereas partial visual matching does not.Tuning only the drawer or only foreground textures yields no correlation improvement over the unmatched baseline.
  • Physical-property sensitivity: Across plausible physical-property parameters, SIMPLER maintains low MMRV and high Pearson correlation despite success-rate changes of up to 15%.The tested variations include can mass, gripper-finger friction, and articulated-cabinet joint friction.
  • Physics-simulator sensitivity: SAPIEN and Isaac Sim both produce strong real-to-sim correlation, with MMRV = 0.082 and r = 0.923 for SAPIEN versus MMRV = 0.058 and r = 0.919 for Isaac Sim.The results suggest simulator choice is not critical for the tested rigid-body manipulation tasks.

VII. CONCLUSION

SIMPLER provides scalable simulated evaluation for generalist manipulation policies, correlating strongly with real-world performance while capturing sensitivity to distribution shifts. Its current scope is limited to rigid-object tasks, fixed-camera green-screening, and partly manual environment construction.

  • SIMPLER evaluates policies in simulated environments designed for common real robot setups and complements costly real-world evaluations.
  • Paired sim-and-real experiments across multiple open-source policies show strong correlations between SIMPLER results and real evaluations.
  • SIMPLER captures fine-grained policy behavior, including robustness to various distribution shifts, beyond average performance.
  • Current environments focus on rigid-object manipulation because its physics are relatively straightforward to simulate with modern physics simulators.
  • The green-screening approach is limited to fixed cameras and does not accurately capture object shadows and other visual details, while scene generation still requires manual effort.

APPENDIX A CONTRIBUTIONS

The appendix documents contributor roles, evaluation details, language-conditioned Google Robot tasks, and task-specific trial configurations for SIMPLER experiments.

  • The appendix attributes project leadership and methodology to Xuanlin Li, Jiayuan Gu, and Kyle Hsu, with separate teams handling implementations and robot experiments.
  • SIMPLER environment and evaluation protocols are described in a dedicated details section.
  • Google Robot tasks: Google Robot tasks include picking a coke can, moving one object near another, manipulating drawers, and completing a drawer-placement sequence.
  • Google Robot tasks: The drawer task uses 9 robot positions, 3 drawers, and 2 actions, yielding 54 trials.
  • Tabletop tasks: The spoon-on-towel and carrot-on-plate tasks use tabletop placement configurations, with spoon trials requiring gripper reorientation.
  • Tabletop tasks: The block-stacking task varies block placement and square size, creating 2 × 12 = 24 trials.
  • Tabletop tasks: The eggplant task places an eggplant in a sink and evaluates placing it into a basket across 24 trials.

EVALUATION SYSTEM

The evaluation system combines robot-specific simulated controllers, reconstructed assets, visual matching, and environment variations to approximate real robot evaluation setups.

  • Controllers: The Google Robot controller converts model translation, rotation, and gripper outputs into simulator actions using a dedicated algorithm.
  • Controllers: The controller distinguishes simulation frequency from control frequency and uses Ruckig for time-optimal joint motion planning under velocity, acceleration, and jerk limits.
  • Controllers: The WidowX controller is specified separately for translating policy actions into simulated robot commands.
  • Robot and object assets: Robot assets use converted or exported URDF descriptions, while cameras match the Google Robot egocentric and WidowX third-view setups.
  • Simulation setup: The simulation enables the Temporal Gauss-Seidel solver to avoid mesh penetration during Google Robot grasping.
  • Robot and object assets: Object assets are sourced from public repositories, scans, single-view generation, or manual modeling, then dimensioned and simplified for simulation.
  • Environment variations: Variant Aggregation varies lighting, backgrounds, and table textures across ReplicaCAD-derived scenes for evaluation.
  • Robot and object assets: Visual matching crops real objects, aligns simulated and real segmentation masks, and optimizes asset pose through differential rendering before texture unprojection.

C. SIMPLER-Variant Aggregation

Variant Aggregation averages policy performance across varied simulator appearances to produce a more faithful evaluation signal, with full results showing strong correlation with real-world evaluations.

  • Variant Aggregation varies visual simulator characteristics, including textures and lighting, and averages policy performance across all environment variants.
  • Full evaluation results in Tables IV and V demonstrate strong performance relationship correlations between SIMPLER environments and real-world evaluations.

CORRELATION EXPERIMENTS UNDER ENVIRONMENT

SIMPLER evaluations correlate policy performance with real-world evaluations across distribution shifts and simulation settings. Additional comparisons show stronger alignment than simulation-free action-prediction evaluation, while evaluation can be interrupted when a policy risks damaging the robot.

  • SIMPLER evaluations show strong policy behavior correlations with real-world evaluations under different environment distribution shifts.
  • Real evaluation of a policy on Drawer tasks was terminated after two trials because operators judged that it could damage the robot.
  • Distribution shifts: SIMPLER tracks policies’ robustness to distribution shifts with low MMRV and high Pearson correlation coefficient with real-world evaluations.
  • Simulation-free comparison: SIMPLER produces significantly better real-and-sim performance correlations than validation-trajectory action-prediction MSE, with substantially lower MMRV and higher Pearson correlation.
  • Single-task policy: SIMPLER preserves performance rankings after adding a policy trained solely on Pick Coke Can demonstrations, maintaining low MMRV and high Pearson correlation.

B. Other Metrics: Kruskal Wallis

The Kruskal-Wallis analysis complements relative performance alignment metrics by testing whether simulation and real-world success distributions differ for individual policies. Visual Matching generally avoids significant distribution shifts, while the visual-gap ablation identifies asset and background choices that improve correlation.

  • Metric: The Kruskal-Wallis test compares each policy’s real and simulated trial-success distributions and counts significant shifts using p<0.05.
  • Results: With Visual Matching, simulation trial-success distributions are not significantly different from real results across many tasks and policies.
  • Metric complementarity: Kruskal-Wallis metrics complement MMRV by assessing absolute distribution shifts, whereas MMRV measures real-to-sim relative performance alignment.
  • Visual matching: Combining green-screened backgrounds with curated foreground object and robot assets yields the best real-and-sim performance correlation among tested visual-gap settings.
Loading 2405.05941v1…