PaperlayerView exact revision on arXiv

Source-linked AI summary

SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes

Ninad Khargonkar, Sai Haneesh Allu, Yangxiao Lu, Jishnu Jaykumar P, Balakrishnan Prabhakaran, Yu Xiang

arXiv:2306.15620v3cs.ROcs.CVcs.LG

TL;DR

Real-world robot manipulation lacks common test scenes that are easy to reproduce, making comparisons across methods difficult. SceneReplica constructs reproducible YCB-object scenes and evaluates model-based and model-free grasping pipelines; the reported comparisons favor GDRNPP among model-based frameworks and MSMFormer combined with Contact-GraspNet among model-free pipelines.

  • Problem

    Real-world robot tasks lack common test datasets, and reproducing scenarios across research labs is difficult.

    Source: §I · p. 1

  • Method

    SceneReplica creates 20 reproducible pick-and-place scenes using 16 YCB objects and benchmarks model-based and model-free 6D grasping pipelines.

    Source: §I · p. 1

  • Results

    GDRNPP achieves higher grasping and pick-and-place success than the other model-based framework, while MSMFormer with Contact-GraspNet performs best among model-free pipelines.

    Source: §V · pp. 5–6

  • Takeaways & Limitations

    The benchmark combines pipeline effects and can serve as a reference for robot manipulation research.

    Source: §I · p. 2

    Evaluation is limited to 16 YCB objects and the Fetch mobile manipulator; extending it to other robots or external cameras requires a third-person view and hand-eye calibration.

    Source: §VI · p. 6

Abstract

from arXiv · show

We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide our experimental results and analyzes for model-based and model-free 6D robotic grasping on the benchmark, where representative algorithms are evaluated for object perception, grasping planning, and motion planning. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods.

I. INTRODUCTION

Robotics lacks widely adopted benchmarks that make real-world manipulation scenarios reproducible, complicating fair comparisons across methods. SceneReplica provides reproducible pick-and-place scenes and evaluates model-based and model-free grasping pipelines on the same real-world tests.

I. INTRODUCTION

  • Widely adopted benchmarks enable fair, standardized algorithm comparisons, but most robotics benchmarks have not achieved widespread use.The introduction points to established benchmarks including ImageNet, KITTI, GLUE, and SuperGLUE.Source: §I · p. 1
  • Existing real-world protocols can have inaccurate object placement or require AR tags that create unnatural test scenes.The cited protocols include and GRASPA.Source: §I · p. 1
  • The benchmark provides 20 pick-and-place scenes with five YCB objects each, generated procedurally and designed around robot reachability and pose diversity.A tool guides scene recreation without AR tags or external equipment.Source: §I · p. 1
  • SceneReplica evaluates model-based and model-free 6D grasping pipelines systematically and fairly on the same real-world scenes.The benchmark covers perception, grasp planning, motion planning, and control.Source: §I · pp. 1–2
  • SceneReplica is intended as an easy-to-setup, reproducible real-world benchmark whose scene-generation approach can transfer across robotic platforms.The authors plan to maintain it and evaluate additional perception, planning, and control methods.Source: §I · p. 2

II. RELATED WORK

Simulation benchmarks offer reproducible evaluation at scale, but their performance may not transfer directly to real-world conditions. SceneReplica targets reproducible tabletop pick-and-place scenes and distinguishes itself among grasping benchmarks by enabling local recreation without AR markers.

II. RELATED WORK

  • Simulation benchmarks provide reproducible evaluation and many tasks, but sim-to-real gaps limit direct transfer to real-world performance.Lighting, friction, and external perturbations are among the real-world nuances described as difficult to model.Source: §II · p. 2
  • For grasping and pick-and-place, SceneReplica is described as the only benchmark enabling locally reproducible scenes without AR markers.Other reproducible benchmark scenes cited are NIST Assembly, FurnitureBench [15], and GRASPA.Source: §II · p. 2

III. THE SCENEREPLICA BENCHMARK

  • SceneReplica focuses on tabletop cluttered pick-and-place scenes with separated objects, generated in Gazebo using ground-truth object data for constraint checking.The target is not piled clutter, which the passage describes as harder to replicate exactly.Source: §III · p. 2

A. Creating Scenes in Simulation

SceneReplica builds simulated pick-and-place scenes from accessible YCB objects, using stable poses and robot reachability to support reproducible real-world testing. It selects 20 scenes for object-pose coverage and balanced counts.

Object Set

  • Sixteen readily available YCB objects provide the benchmark’s object set, with selection also motivated by pose-estimation use and parallel-jaw gripper fit.The objects are a subset of the YCB Video dataset.Source: A. Creating Scenes in Simulation · pp. 2–3

Stable Poses of Objects

  • Stable object poses replace randomly piled clutter, which the authors describe as difficult to tune and reproduce; these poses are also less likely to fall and disturb scenes.Stable poses are orientations in which objects rest statically on a planar surface.Source: A. Creating Scenes in Simulation · p. 3

Determining the Reachable Space of a Robot

  • Reachable tabletop locations are found by testing feasible motion plans to stand-off poses on a 16 × 16 grid, removing locations without a plan.The table is 1m × 1m × 0.745m, and each grid cell is tested with a 3cm × 3cm block.Source: A. Creating Scenes in Simulation · p. 3

Scene Generation

  • Scenes are generated by adding up to five objects at nearby reachable, collision-free locations with randomly sampled stable poses, producing clutter and occlusion.The generation process repeats until five objects have been added.Source: A. Creating Scenes in Simulation · p. 3

Scene Selection

  • The selected 20 scenes balance object counts, cover stable poses, and keep objects reachable; the set with the highest pose-coverage score is chosen.For a different robot, users re-render reference images using its camera parameters and recreate the scenes from those images.Source: A. Creating Scenes in Simulation · p. 3

B. Reproducing Scenes in the Real World

SceneReplica reproduces simulated scenes by matching the real camera pose and table height, then guiding object placement with an overlay of the reference and live images. For another robot, only the reference images need to be regenerated.

B. Reproducing Scenes in the Real World

  • Scene replication aligns the camera extrinsic between simulation and reality and keeps table height at 0.745m before overlaying reference and live RGB images for placement guidance.Users interactively check image overlap to align the objects.Source: B. Reproducing Scenes in the Real World · pp. 3–4
  • The reference set comprises 20 scenes with five YCB objects each.These are the selected SceneReplica scenes.Source: Figure 3 · p. 4
  • For a different robot platform, users regenerate reference images with its camera; matching camera extrinsics makes object poses in the robot base frame replicate those in the camera frame.The described procedure assumes a height-adjustable table to match the 0.745m table height.Source: B. Reproducing Scenes in the Real World · pp. 3–4

IV. MODEL-BASED AND MODEL-FREE 6D GRASPING

SceneReplica evaluates tabletop pick-and-place through grasping, lifting, gripper rotation, and drop-off, using both model-based and model-free 6D grasping pipelines. The two approaches differ in object representation and perception, grasp planning, and obstacle handling.

IV. MODEL-BASED AND MODEL-FREE 6D GRASPING

  • The benchmark evaluates grasping, lifting, gripper rotation, and object drop-off in sequence, distinguishing grasping success from completion of the full pick-and-place pipeline.The task requires 6D grasping, controlling both gripper orientation and translation rather than only top-down grasping.Source: §IV · p. 4

A. Model-based 6D Grasping

  • SceneReplica integrates perception with planning and control so pose-estimation accuracy can be examined in relation to grasping success.The text notes that pose accuracy can be assessed on datasets, while its impact on grasping remains unclear.Source: A. Model-based 6D Grasping · p. 4, §IV · p. 4
  • Model-based grasping estimates 6D object pose, plans grasps from 3D object models, and uses motion planning to execute a feasible trajectory.The benchmark uses GraspIt! for grasp planning and OMPL algorithms for motion planning; object models also serve as obstacles in the planning scene.Source: A. Model-based 6D Grasping · p. 4

B. Model-free 6D Grasping

  • Without 3D object models, model-free grasping segments unseen objects, synthesizes grasps from point clouds, and plans around oriented 3D bounding boxes.The grasp-synthesis networks learn from large datasets of object-grasp pairs and can plan for new objects from partial point clouds.Source: B. Model-free 6D Grasping · pp. 4–5

V. EXPERIMENTS

The experiments compare real-world pick-and-place pipelines across 100 trials, measuring task success, perception, and failure types. GDRNPP leads among the reported model-based methods, while MSMFormer with Contact-GraspNet is the strongest model-free combination.

Experimental Evaluation

  • In pick-and-place, the evaluation assessed success rate for transporting objects between locations, perception, and failure analysis over 100 trials (20 scenes × 5 objects per scene) across diverse grasping frameworks.Source: §V · p. 5

Failure Analysis

  • Errors are classified as perception, planning, or execution failures; perception errors occur when pose estimation or segmentation inaccuracies cause grasp failure.Pose estimation failure uses ADD-S > 0.1 * d, while segmentation failure uses an overlap F-measure below 75%.Source: §V · p. 5

Grasping Orders

  • The experiments compare near-to-far grasping order with a fixed order defined for each scene when the order can be programmed.Near-to-far order clears closer objects first, which may occlude other objects.Source: §V · p. 5

Model-based Grasping Frameworks

  • The GDRNPP-based framework achieves higher success in both grasping and pick-and-place than the other evaluated model-based methods.The comparison uses SceneReplica with a Fetch mobile manipulator.Source: §V · p. 5, Table II · p. 6

Model-free Grasping Frameworks

  • Among model-free pipelines, MSMFormer with Contact-GraspNet achieves the best performance, while MSMFormer with top-down grasping is a strong baseline.Contact-GraspNet-based methods outperform 6D GraspNet-based methods, and MSMFormer has higher segmentation accuracy than UCN.Source: §V · pp. 5–6, Table III · p. 6

Failure Analysis

  • Perception and planning errors are dominant, perception errors occur more often, and small, thin objects such as scissors and markers are harder to grasp.The paper reports detailed success and failure analyses by object and method in supplementary material.Source: §V · p. 6, Figure 8 · p. 6

VI. DISCUSSION AND FUTURE WORK

The discussion identifies limits in the benchmark’s object and robot coverage and outlines planned extensions to methods and tasks.

Limitations

  • Testing is limited to 16 YCB objects, and a more diverse object set would improve the benchmark.Source: §VI · p. 6
  • Reported results use only the Fetch mobile manipulator; extending to other robots or external cameras requires a third-person view setup and hand-eye calibration.With a new camera pose, scene reference images can be re-rendered for scene recreation.Source: §VI · p. 6

Future Work

  • Planned work includes evaluating reinforcement-learning and end-to-end grasping approaches, with possible extension to object rearrangement.The authors also plan to continuously evaluate additional methods and release results.Source: §VI · p. 6

Supplementary Material

The supplied supplementary-material passages provide author and publication metadata.

Author Information

Publication Information

A Grasp and Motion Planning

The benchmark generates candidate scenes in simulation and uses grasp-planning procedures to support reproducible real-world manipulation tests.

A Grasp and Motion Planning

  • GraspIt! grasps are sampled offline, then farthest-point sampling spreads grasp translations across each object’s surface; top-down grasps are tried if motion planning fails.The offline computation avoids running GraspIt! at every scene iteration because of time constraints.Source: A Grasp and Motion Planning · p. 8, Figure 2 · p. 9
  • Top-down grasping uses the smaller X-Y point-cloud principal component for gripper width and alignment; model-free planning uses an oriented bounding box because partial-cloud bounding boxes are too large in clutter.The figure shows a top-down grasp from a point cloud.Source: A Grasp and Motion Planning · p. 8, Figure 1 · p. 8

B Scene Generation

  • Scene generation restricts object placements to reachable table locations, samples stable poses and Z-axis rotations, and uses nearby placements to create packed, collision-free scenes.A candidate scene is rejected if any of its five objects lacks a feasible motion plan using the offline grasp dataset.Source: B Scene Generation · p. 9
  • The selected 20-scene set is chosen from hundreds of candidates to balance object counts and favor diversity in object poses.Pose diversity is scored using entropy derived from the count distribution of unique poses for each object.Source: B Scene Generation · p. 9

C Additional Results

Failure analysis finds perception and planning failures dominate pick-and-place errors, while pose validation reports GDRNPP ahead of other methods. The section also describes how reachable space is checked when adapting scenes to a different robot platform.

C.1 Pick-and-place Failure Analysis

  • Perception and planning failures dominate pick-and-place errors, with small, thin objects especially difficult because of depth uncertainty and limited graspable area.Tables 3 and 4 report per-object grasping statistics for Methods 1–8 and Method 9, respectively.Source: §C.1 · p. 10, Table 3 · p. 11, Table 4 · p. 12

C.2 Pose Estimation Validation

  • GDRNPP outperforms other methods in pose estimation, and this translates to better pick-and-place grasping.Pose validation uses an ADD threshold-accuracy curve and a histogram of pose rotation-angle error.Source: §C.2 · p. 10, Figure 3 · p. 12

D.1 Robot Reachable Space Verification

  • For a Franka-Panda arm, reachable space is checked by testing motion plans to cubes spawned in a dense table grid and removing unreachable locations.The resulting reachable space can guide scene placement, with reference images generated using chosen camera parameters.Source: §D.1 · p. 10

D.2 Reference Image Generation with Different Camera

Reference images can be regenerated in simulation for a different robot platform and camera setup. Accurate replication requires matching the real-world camera parameters to those used in simulation.

D.2 Reference Image Generation with Different Camera

  • For a different robot platform, reference scene images can be regenerated in simulation using the new camera’s parameters, even when the camera is separate from the robot.The example table presents two camera settings with the same ground-truth object poses.Source: §D.2 · pp. 12–13, Table 5 · p. 13
  • The procedure places objects within the new robot’s reachable space, adjusts the camera until all objects are visible, then renders reference images for real-world setup.The rendered simulation images are used as references when setting up the scene in the real world.Source: §D.2 · p. 13
  • Accurate scene replication requires the real-world camera parameters to match those used in simulation.Table 5 shows reference images for two camera settings.Source: §D.2 · p. 13, Table 5 · p. 13

Related papers

Loading 2306.15620v3…