PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Jenai Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, Jonathan Tremblay

arXiv:2604.09860v4cs.ROcs.AI

TL;DR

Existing simulation benchmarks provide limited evidence of true generalization because they can saturate and overlap with training environments. RoboLab introduces scalable, high-fidelity evaluation of real-world-trained policies through generated scenes, tasks, and granular analyses, finding low and instruction-sensitive performance while preserving policy rankings with real-world evaluation.

  • Problem

    Existing robotics benchmarks lack high-fidelity real-world-policy evaluation, saturate on static tasks, and offer limited analysis of failure modes.

  • Method

    RoboLab combines scalable human- and LLM-enabled scene/task generation with robot- and policy-agnostic environments and metrics beyond binary success rates.

  • Results

    Policies show low out-of-domain performance, partial task understanding, asymmetric competency generalization, and degradation under more abstract instructions.

  • Takeaways & Limitations

    RoboLab-120 provides a meaningful proxy for real-world evaluation while exposing policy behavior and sensitivity that raw success rates can obscure.

  • Takeaways & Limitations

    RoboLab focuses on rigid-body tabletop scenes and underrepresents deformable objects, contact-rich force-controlled skills, and ambiguous long-horizon tasks.

Abstract

from arXiv · show

The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.

I. INTRODUCTION

RoboLab addresses limitations in simulation benchmarks by combining high-fidelity, scalable task generation with evaluation and analysis of real-world-trained generalist policies.

  • Current benchmarks lack high-fidelity evaluation, saturate rapidly on static tasks, and provide limited analysis of policy failure modes.
  • Training and evaluation on nearly identical simulated environments can trivialize assessment and obscure true task generalization.
  • Compared with costly real-to-sim approaches requiring approximately ∼1hr per scene, RoboLab targets high realism with low overhead.
  • RoboLab enables human-authored and AI-enabled generation of robot- and policy-agnostic scenes and tasks, supporting scalable benchmark expansion.
  • RoboLab-120 contains 120 diverse pick-and-place tasks spanning visual, procedural, and relational competencies across three difficulty levels.
  • RoboLab evaluates real-world-trained policies and provides subtask scoring, event tracking, and sensitivity analysis beyond binary success rates.

III. ROBOLAB

RoboLab separates scene, task, and environment construction, then automates scene and task generation through language models, geometric validation, and simulation-based refinement.

  • RoboLab generates diverse, real-world-representative tasks that are policy- and robot-agnostic for multifaceted generalization analysis.
  • Users create scenes, define language-instruction tasks, and instantiate environments by selecting robots, policies, and scene variations.
  • Runtime robot binding allows identical tasks to be evaluated across robot embodiments and policy variants without embodiment-specific training.
  • The formal environment combines a task, robot embodiment, policy parameters, and camera, lighting, background, and pose variations.
  • Scaling Scene Generation: Automated scene generation prompts an LLM for object layouts, validates placements geometrically and physically, and refines invalid plans using simulation feedback.
  • Scaling Task Generation: Automated task generation produces task code from scene and competency information, validates syntax and asset selections, and refines invalid tasks.

B. Benchmark Design

RoboLab evaluates task-generalist policies across visual, procedural, and relational competencies using structured tasks, graded subtasks, language variants, and trajectory-quality metrics.

  • Competency axes: RoboLab labels tasks by visual, procedural, and relational competencies, including perception, action-oriented reasoning, language conjunctions, counting, and spatial relationships.
  • Task structure: Tasks decompose into sequential subtasks containing parallel events, enabling evaluation of intermediate execution beyond final task completion.For example, multi-object placement can sequence object-specific PickPlace subtasks, each containing grasp, hover, drop, and completion events.
  • Task structure: DifficultyScore combines the number of subtasks with the highest required competency weight, assigning greater weight to procedural, reorientation, and dynamic tasks.Weights range from 0 for visual identification to 3 for reorientation and dynamic tasks.
  • Evaluation metrics: RoboLab supplements binary success with discrete and continuous metrics to characterize policy performance and failure modes.The framework includes normalized graded scores, language-variation analysis, trajectory metrics, and event tracking.
  • Evaluation metrics: SPARC evaluates motion smoothness from the normalized Fourier magnitude spectrum of the end-effector velocity profile; smoother motions approach zero.The adaptive cutoff uses the smaller of 10 Hz and the highest frequency exceeding threshold α = 0.05.
  • Evaluation metrics: Trajectory optimality also uses end-effector speed and path length, with shorter paths generally indicating more direct, higher-quality motion.

D. Sensitivity Analysis

RoboLab combines controlled policy evaluations with Bayesian robustness analysis and event tracking to examine performance, environmental sensitivity, and behaviors hidden by final success.

  • Robustness analysis: Simulation-Based Inference learns approximate posterior distributions over environmental parameters conditioned on observed outcomes such as task success.The parameters can include continuous variables such as object distance and camera displacement, as well as discrete variables.
  • Results: Overall performance of current foundation models remains limited despite emerging capabilities across diverse task dimensions.The benchmark caption directs readers to more granular trajectory-quality results in Tables IV and V.
  • Behavioral analysis: RoboLab records wrong-object grasps, dropped objects, and gripper collisions to expose undesirable rollout behaviors that final task success may miss.
  • Sensitivity analysis: RoboLab evaluates off-the-shelf generalist models under controlled ablations and environmental perturbations targeting language, scene complexity, and task difficulty.
  • Sensitivity analysis: Policies are evaluated on 120 tasks spanning simple, moderate, and complex difficulty levels, using models fine-tuned on the real-world DROID dataset.

B. Task Results

RoboLab-120 reveals substantial weaknesses in task-generalist policies across relational, visual, and procedural competencies, while controlled perturbations identify specific behavioral sensitivities. Partial-credit scoring further exposes meaningful progress that strict success rates miss.

  • Overall results: 13.5% success but 0.44 score on complex tasks shows π0.5 often reaches partial milestones without completing the full task.The score/success gap indicates partial task understanding followed by failure in final execution stages.
  • Competency axes: Procedural understanding was most challenging, while conjunctions and counting outperformed spatial relations and visual grounding remained low across attributes.These asymmetric results indicate brittle language-to-object binding beyond familiar descriptions.
  • Instruction specificity: π0.5 success fell from 28.0% with default instructions to 15.3% with vague instructions, revealing sensitivity to instruction abstraction.The same underlying tasks and scenes were evaluated across vague, default, and specific language conditions.
  • Scene complexity: Success generally degraded as visual clutter increased, although π0.5 completed some tasks despite high numbers of distractor objects.Additional distractors can overwhelm target identification and manipulation for some policies.
  • Interpretation: RoboLab isolates where generalization fails, supporting analysis that can inform data-collection and model-improvement priorities.The framework addresses both performance measurement and identification of factors associated with failure.
  • Sensitivity analysis: Successful execution required wrist-camera displacement near zero, and object placements around 0.5m from the robot had the highest success probability.The sensitivity analysis associates policy outcomes with camera calibration and object reachability.

D. Real-World Verification

RoboLab compares simulation performance with real-world policy evaluation and reports preserved policy rankings. Its current scope remains bounded by tabletop rigid-body tasks, underrepresented contact-rich skills, and residual visual distribution shift.

  • Real-world verification: Spearman ρ = 1.00 preserved policy rankings between RoboLab-120 success rates and RoboArena Elo scores.The comparison uses four policies with both measurements available; Pearson r = 0.68 also indicates positive correlation.
  • Real-world verification: RoboLab evaluates real-world-trained policies in high-fidelity simulation to reduce dependence on simulation-specific training overlap.The framework is intended to assess whether simulation behavior can serve as a proxy for real-world policy quality.
  • Behavioral analysis: Sensitivity analysis identifies wrist-camera calibration and object distance near 0.5m as factors associated with successful execution.These findings connect simulation perturbations to concrete policy-behavior dependencies.
  • Limitations: RoboLab currently focuses on rigid-body tabletop scenes and does not fully capture deformable-object manipulation.Cloth, cables, and bags are explicitly outside full coverage.
  • Limitations: Contact-rich skills requiring precise force control, compliant interaction, or complex frictional dynamics remain underrepresented and depend on simulation fidelity.Open-ended ambiguous long-horizon tasks can also exceed the subtask evaluation system and require human judgment.
  • Limitations: A residual visual distribution shift remains between high-fidelity simulation and real-world deployment, requiring further characterization and validation.The paper specifically calls for analysis of visual-perception robustness and extensive real-world deployment validation.

APPENDIX A DETAILS ON THE ROBOLAB BENCHMARK

The appendix details RoboLab’s benchmark assets, evaluation protocol, scoring, trajectory metrics, and MNPE-based sensitivity analysis. It also documents statistical limits of ten-episode per-task evaluations and caveats in interpreting partial progress and long-horizon results.

  • Benchmark construction: RoboLab uses approximately 300 object assets with visual and collision meshes, mass and friction properties, language descriptions, and object labels.Assets draw from YCB, HOT3D, HOPE, HANDAL, and VoMP benchmarks.
  • Benchmark construction: RoboLab-120 contains 120 manually generated tasks, while users can compose scenes and generate tasks through manual or AI-assisted pipelines.The appendix contrasts existing high-overhead pipelines with RoboLab’s drag-and-drop and batch AI generation workflow.
  • Evaluation protocol: Each policy-task evaluation uses N=10 episodes, with results reported as aggregate benchmark outcomes rather than per-task policy comparisons.The protocol accounts for simulator and policy stochasticity but leaves individual task estimates coarse.
  • Statistical significance: A single per-task success rate with ten trials has a 95% confidence interval of approximately ±30% near p=0.5 and ±19% near p=0.9.The paper recommends at least 100 episodes per task for per-task conclusions or differences smaller than 10%.
  • Evaluation metrics: π0.5 reaches 13.5% success and 0.44 score on complex tasks, retaining 0.35 of milestones within failed episodes.The same partial-progress pattern appears in procedural categories where full completion is rare.
  • Long-horizon caveat: The subtask=7 spike reaches approximately 20% success and 0.68 score for π0.5, but is dominated by repetitive CubesAndBlocksInBinTask sequences.This task composition does not provide a clean test of causal long-horizon reasoning.
  • MNPE sensitivity analysis: MNPE infers how scene parameters relate to policy outcomes and interprets concentrated posteriors near reference values as high sensitivity.Broad posteriors indicate robustness to variation; camera experiments use external and wrist displacement parameters.
  • Evaluation metrics: Score measures normalized partial subtask completion, while Score (fail) isolates progress within episodes that did not complete.Succ% measures full completion; SPARC and Speed separately characterize trajectory quality.

APPENDIX C DETAILS ON SCALING SCENE GENERATION

RoboLab includes scene-generation infrastructure that supports additional implementation detail and high-fidelity mixed Gaussian-splat and mesh scenes. These scenes can host vision-language-action policies during task execution.

  • Scene generation: RoboLab provides additional implementation details for scene generation in Section III-A1.The cited appendix material points readers to the main scene-generation section rather than describing a separate generation method.
  • Scene generation: A demonstrated scene combines a Gaussian-splat background, a collision mesh estimated with 3DGRUT, and a mesh foreground.Objects have spatially varying density, with mass estimated using VoMP, and a VLA runs a task in the resulting scene.

A. Stage I: Predicates for Semantic Planning

Stage I represents scene layouts with structured spatial predicates, allowing semantic relationships to be specified independently of raw coordinates.

  • PlaceIn requires object x to be contained within object y, such as fruit inside a bowl.
  • PlaceOn requires object x to be supported by object y, such as a mug on a coaster.
  • ClusterAround makes object x the anchor for a group of target objects.
  • PlaceAnywhere places an object freely on the global table surface and downgrades invalid anchor references to preserve the object.

B. Stage II: Geometric Constraint Solving

Stage II converts semantic placement constraints into collision-free geometric layouts through rejection sampling, packing heuristics, and adaptive recovery strategies.

  • Global placement samples positions within table bounds and checks collisions against placed objects using SAT on oriented bounding boxes.
  • High-density scenes use progressively relaxed collision margins and random position perturbations when the solver reaches a local minimum.
  • Stacking samples support-surface positions for up to K = 20 attempts while avoiding overlap with objects already on that support.
  • Containment discretizes a container floor into grid cells, accepts unoccupied cells within scaled bounds γ = 0.7, and assigns object heights from the container geometry.
  • The baseline performs one-pass grid assignment with uniform within-cell jitter, providing basic separation but preventing iterative feedback.

D. Experiments

Experiments evaluate scene-generation quality against a domain-randomization-style baseline and report consistently stronger performance for RoboLab across diverse scenes and metrics.

  • The comparison uses VQA, GPT preference, visual realism, functionality, layout correctness, quality, and scene completeness metrics.
  • Quantitative comparisons cover 100 generated scenes and split results by scenes containing [5], [6] [15], and [20] objects.
  • Algorithm 1 takes objects, predicates, and table bounds as input and outputs 2D coordinates for base objects.
  • The spatial solver initializes placements, repeatedly applies relative constraints, resolves collisions, and perturbs positions when progress stalls.
  • RoboLab consistently outperforms the baseline across all reported scene-generation metrics, with particularly large gains in visual realism and semantic functionality.

APPENDIX D DETAILS ON TASK GENERATION EVALUATION

The appendix evaluates automatically generated tasks and scene-generation components, finding strong instruction alignment and broad task quality while identifying conservative predicate coverage as a trade-off.

  • Task generation evaluation: Tasks are generated from scene descriptions and category templates, then assessed for instruction-code alignment across six semantic, linguistic, and feasibility dimensions.
  • Task generation evaluation: 812 tasks are generated across 59 scenes and 7 categories, with one task per category for 2 simple scenes and two tasks per category for 57 remaining scenes.
  • Scene-generation evaluation: Scene generation is evaluated with realism, functionality, layout, quality, VQA, GPT preference, and difficulty-split comparisons against a baseline.
  • Task generation evaluation: Alignment aggregates the six sub-dimensions, while clarity, feasibility, and match separately measure language quality, physical realizability, and semantic accuracy.
  • Task generation evaluation: 0.91 mean alignment and 76% full alignment verdicts indicate strong overall task-generation quality.
  • Task generation evaluation: Conjunction and recognition reach alignment scores of 0.97 and 0.96, whereas color-based tasks score 0.81 and spatial feasibility reaches 0.89.
  • Scene-generation setup: The method uses realistic clusters, containers, supports, varied orientations, runtime scene fields, size constraints, and feedback for failed validation checks.
  • Geometric placement: The placement solver separately handles stacking and containment using support-surface sampling, capacity filtering, grid dimensions, jitter, and computed object heights.
Loading 2604.09860v4…