PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, Hao Su

arXiv:2302.04659v1cs.ROcs.AI

TL;DR

Existing manipulation benchmarks inadequately cover object-level variation, fully dynamic simulation, and diverse manipulation settings needed for generalizable skills. ManiSkill2 addresses these gaps with a unified, heterogeneous benchmark combining large-scale assets and demonstrations, flexible interfaces, dynamic rigid- and soft-body simulation, and shared rendering resources. It provides 20 task families, over 2000 objects, over 4M demonstration frames, and 2000+ FPS visual sample collection under the reported setup, while evaluation comparisons remain challenging across frameworks and action conversion can accumulate execution errors.

  • Problem

    Existing benchmarks lack sufficient object-level topological and geometric variation, fully dynamic simulation, or native support for multiple manipulation task types.

  • Method

    ManiSkill2 provides a unified benchmark with heterogeneous task families, multiple controllers, large-scale demonstrations, coupled rigid- and soft-body simulation, and shared rendering resources.

  • Results

    ManiSkill2 covers 20 task families, over 2000 objects, and over 4M demonstration frames, with 2000+ FPS achieved using a render server and 16 parallel environments on one GPU.

  • Takeaways & Limitations

    ManiSkill2 offers a unified and accessible basis for benchmarking sense-plan-act, imitation-learning, and reinforcement-learning methods across diverse manipulation settings.

  • Takeaways & Limitations

    Framework comparisons remain challenging because simulation and rendering fidelity differ, and action-space conversion can suffer from accumulated execution errors.

Abstract

from arXiv · show

Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI. However, existing benchmarks, mostly composed of a suite of simulatable environments, are insufficient to push cutting-edge research works because they lack object-level topological and geometric variations, are not based on fully dynamic simulation, or are short of native support for multiple types of manipulation tasks. To this end, we present ManiSkill2, the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. ManiSkill2 includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D-input data simulated by fully dynamic engines. It defines a unified interface and evaluation protocol to support a wide range of algorithms (e.g., classic sense-plan-act, RL, IL), visual observations (point cloud, RGBD), and controllers (e.g., action type and parameterization). Moreover, it empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS with 1 GPU and 16 processes on a regular workstation. It implements a render server infrastructure to allow sharing rendering resources across all environments, thereby significantly reducing memory usage. We open-source all codes of our benchmark (simulator, environments, and baselines) and host an online challenge open to interdisciplinary researchers.

1 INTRODUCTION

ManiSkill2 addresses the need for a versatile, public benchmark for generalizable manipulation skills by targeting object-level variation and practical challenges. It extends prior benchmarks with fully simulated dynamics, diverse articulated objects, and large-scale demonstrations while supporting multiple controllers.

  • Motivation: Generalizable manipulation skills involve handling unseen objects and configurations within learned object categories.These skills are positioned as a cornerstone of Embodied AI and can support long-horizon daily chores.
  • Motivation: A versatile public benchmark is needed to support interdisciplinary and reproducible research on object-level topological and geometric variation.The benchmark should also address practical manipulation challenges.
  • Benchmark gap: Most prior benchmarks insufficiently support and evaluate progress in generalizable manipulation skills.The paper attributes this gap to limited benchmark capabilities and diversity.
  • Contribution: ManiSkill2 extends the SAPIEN ManiSkill benchmark with fully simulated dynamics, diverse articulated objects, and large-scale demonstrations.It is presented as the next generation of the earlier benchmark.
  • Contribution: ManiSkill2 supports multiple controllers, including joint-space and task-space controllers, for different manipulation components.The paper notes that controller choice affects exploration, robustness, and sim2real transferability.

2 BUILDING ENVIRONMENTS FOR GENERALIZABLE MANIPULATION SKILLS

ManiSkill2 builds a heterogeneous suite of manipulation tasks and supports flexible controller and demonstration-action interfaces. Its environment design emphasizes diverse task types, precise contact-rich assembly, and reproducible action-space conversion, while acknowledging accumulated conversion errors.

  • Heterogeneous task families: ManiSkill2 contains 20 task families spanning rigid and soft bodies, stationary and mobile bases, and single- and dual-arm manipulation.Each family shares an objective while varying assets and initial states.
  • Heterogeneous task families: Six soft-body tasks require agents to move or deform soft bodies into specified goal states through interaction.Examples include filling, hanging, excavation, pouring, pinching, and writing, with observations including RGBD images, point clouds, or depth maps.
  • Heterogeneous task families: Assembly tasks emphasize millimeter-level precision and contact-rich insertion rather than only position or rotation errors.PegInsertionSide uses 3mm clearance, while PlugCharger models realistic holes with 0.5mm clearance.
  • Multi-controller support: The benchmark supports multiple controller action spaces, including joint position, delta joint position, and delta end-effector pose.Controllers translate desired configurations into joint torques driving the corresponding motors.
  • Action-space conversion: Demonstration action conversion maps source controller trajectories into target action spaces while reproducing the demonstrations’ kinematic and dynamic processes.The described example converts joint-position demonstrations for use with a delta end-effector pose controller.
  • Action-space conversion: The conversion method is closed-loop but suffers from accumulated execution errors.A target environment is instantiated to acquire the required current pose information during conversion.

3 REAL-TIME SOFT BODY SIMULATION AND RENDERING

ManiSkill2 introduces a real-time soft-body simulator coupled dynamically with an existing rigid-body simulator. Its GPU MPM solver and bidirectional coupling support real-time soft-body manipulation environments, with implementation optimizations targeting efficient execution.

  • Simulator: ManiSkill2 introduces a highly efficient GPU MPM simulator for soft bodies.The simulator is designed for real-time soft-body manipulation environments.
  • Rigid-soft coupling: The simulator supports two-way dynamic coupling between soft bodies and the SAPIEN rigid-body simulator.This coupling enables interactions between the new soft-body simulator and an external rigid-body simulator.
  • Rigid-soft coupling: The coupling transfers rigid-body poses to the soft-body simulator and soft-body forces to the rigid-body simulator.External collision shapes are copied into the soft-body simulator at the beginning of simulation.
  • Performance optimization: The soft-body simulator uses Warp to translate Python code into native C++ and CUDA for performance comparable to C and CUDA.The implementation also optimizes data transfer between CPU and GPU, including pose and force transfers for coupling.

4 PARALLELIZING PHYSICAL SIMULATION AND RENDERING

ManiSkill2 parallelizes simulation and rendering with asynchronous rendering and a render server to improve visual-RL throughput while reducing data transfer and GPU memory use. In comparison experiments, it achieves 2000+ FPS with 16 parallel environments on one GPU, while shared rendering resources keep additional environments’ memory requirements low.

  • Asynchronous Rendering: Asynchronous rendering starts reward computation before GPU rendering finishes, increasing CPU utilization because reward computation often does not depend on rendered images.The sequential pipeline leaves the CPU idle while GPU rendering completes.
  • Render Server: The render server eliminates repeated GPU-to-CPU-to-GPU image copies and shares GPU rendering resources across environments, reducing transfer time and memory usage.Rendering requests are handled by a thread pool in the main Python process.
  • Comparison: The comparison evaluates PickCube sample-collection speed across Habitat 2.0, RoboSuite 1.3, Isaac Gym, and ManiSkill2 under matched computational resources.Experiments use 16 CPU cores, one RTX Titan GPU, 128×128 images, and up to 512 parallel environments.
  • Comparison: Fair cross-framework comparison remains difficult because the simulators and renderers differ in fidelity, despite efforts to match simulation parameters.This is an author-identified limitation of the benchmark comparison.
  • Throughput: 2000+ FPS is achieved with 16 parallel environments on a single GPU when ManiSkill2 uses asynchronous rendering and a render server.Asynchronous rendering without the render server already outperforms the other compared frameworks.
  • Memory Sharing: With 74 YCB objects per scene, ManiSkill2’s shared meshes and textures require very little additional memory as environments increase, unlike Habitat 2.0’s linear growth.The comparison uses the extended PickClutterYCB environment and a matched Habitat 2.0 setting.

5 APPLICATIONS

ManiSkill2 supports sense-plan-act, imitation learning, reinforcement learning, and sim-to-real evaluation across diverse manipulation settings. Experiments expose strengths and remaining challenges in generalization, precision, controller choice, and transfer.

  • 5.1 SENSE-PLAN-ACT: Contact-GraspNet achieved a 43.24% success rate across 74 objects and five initial states per object in PickSingleYCB.Failures included low-confidence predictions in 27.03% of trials and low-quality or unreachable high-confidence grasps in 29.73%.
  • 5.2 IMITATION & REINFORCEMENT LEARNING WITH DEMONSTRATIONS: Behavior cloning performed poorly on rigid-body tasks and struggled with precise soft-body control, especially for Excavate, Pour, Pinch, and Write.Performance was higher on Fill and Hang, which impose less stringent precision requirements.
  • 5.2 IMITATION & REINFORCEMENT LEARNING WITH DEMONSTRATIONS: Point cloud-based DAPG+PPO agents outperformed RGBD-based agents on pick-and-place tasks, while assembly-task success rates remained near zero.Increasing assembly clearance and reducing task difficulty produced much higher performance, indicating that highly precise control remains challenging.
  • 5.2 IMITATION & REINFORCEMENT LEARNING WITH DEMONSTRATIONS: Replacing the delta end-effector pose controller with a delta joint position controller reduced PickSingleYCB success to 0.22±0.18.The result demonstrates the importance of controller selection for learned manipulation performance.
  • 5.3 SIM2REAL: A real-world PickCube evaluation achieved a 60.0% success rate over 50 trials with different initial states.The authors conjecture that the performance drop arose from a depth-map domain gap, given the limited simulation-time augmentation.
  • 5.3 SIM2REAL: Matched simulation and real-world Pinch executions produced similar plasticine deformation, supporting the fidelity of the 2-way coupled rigid-MPM simulation.The comparison used the same motion-planned action sequence in both settings.

6 CONCLUSION

ManiSkill2 is a unified benchmark for generic and generalizable manipulation skills, combining broad task and asset coverage with efficient simulation, rendering, and controller support. Its applications span sense-plan-act, imitation learning, and reinforcement learning, with learned policies showing potential for real-world deployment.

  • 6 CONCLUSION: ManiSkill2 provides 20 manipulation task families, over 2000 objects, and over 4M demonstration frames.The benchmark covers diverse manipulation settings and supports broad algorithmic evaluation.
  • 6 CONCLUSION: The benchmark combines efficient rigid-body simulation, rendering, real-time MPM-based soft-body environments, and multi-controller conversion support.These capabilities support varied manipulation algorithms and physical interaction settings.
  • 6 CONCLUSION: Applications demonstrate ManiSkill2 with sense-plan-act, imitation learning, and reinforcement learning algorithms, while learned policies show potential for direct real-world deployment.

REPRODUCIBLITY STATEMENT

ManiSkill2 uses a verification-driven, iterative workflow to build scalable manipulation environments. The workflow organizes task creation, reward design, and observation configuration, then uses complementary planning and learning methods for demonstrations and verification.

  • A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT: The workflow is designed for scalable and affordable continuous integration of assets and tasks.Verification occurs iteratively as demonstrations are generated for environments.
  • A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT: Task creation defines assets, robots, initial states, and success metrics before reward and observation design.Asset preparation can include convex decomposition and texture baking.
  • A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT: Reward design prototypes shaped rewards required by methods such as model-predictive control and reinforcement learning.The workflow treats reward development as a distinct stage from environment construction.
  • A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT: Observation configuration tailors camera parameters and placements so task observations contain adequate information.This stage is part of the three-stage environment-building workflow.
  • A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT: TAMP, MPC, and RL can be used individually or together to collect demonstrations and verify environments according to task characteristics and difficulty.TAMP avoids reward crafting for many stationary tasks, whereas MPC depends on well-designed shaped rewards.

A.2 CLOUD BASED EVALUATION SYSTEM

The cloud evaluation system runs user-submitted solutions in flexible Docker environments and supports configurable observations, controllers, and evaluation protocols. Its controller suite maps policy actions to robot control signals across joint- and end-effector representations.

  • Cloud evaluation: Users submit Docker images containing the code and models needed to solve benchmark tasks.The submission interface requires a function that accepts observations and returns actions.
  • Configuration: Solutions can configure RGB-D or point-cloud observations and controllers such as delta end-effector pose or joint position.
  • Cloud evaluation: The evaluation server pulls each submission, runs the evaluation code, stores results in a database, and displays them publicly.
  • Controllers: Controllers translate policy actions into joint control signals, spanning joint position, velocity, delta-position, and end-effector pose parameterizations.The suite also includes target and non-target variants, normalized action spaces for most controllers, and joint-position gripper control.
  • Demonstration conversion: Demonstration conversion enables evaluation or learning with a target action space different from the controller used to generate demonstrations.The conversion experiments use PickSingleYCB, AssemblingKits, TurnFaucet, and Write demonstrations, with final success averaged across two evaluation stages.
  • Observations: ManiSkill2 supports observation modes including state, RGBD, point clouds, and optional robot-joint segmentation, with proprioception and task-specific goals where applicable.Default RGBD and point-cloud modes exclude ground-truth segmentation, while separate segmented modes provide it.

C.4 MISCELLANEOUS TASKS

The miscellaneous task suite covers navigation, soft-body interaction, and articulated-object manipulation under varied goals, assets, initial states, and physical conditions. Soft-body tasks use coupled rigid–soft simulation and include precise success criteria for material state and robot behavior.

  • Navigation: Dense obstacle navigation randomizes obstacle shape and color while requiring the end-effector to reach a goal pose within 2.5 cm and 15 degrees.The task provides 1976 trajectories and evaluates 100 episodes per stage across different obstacle layouts.
  • Soft-body manipulation: Soft-body success conditions combine material outcomes with physical constraints, including target amounts or shapes, spill limits, height, static velocity, and object placement.Examples include more than 90% of clay in the beaker, liquid within 4 mm of a target line, and Chamfer distance below 0.3t.
  • Data and evaluation: The soft-body suite supplies 200 successful motion-planned trajectories for several tasks, while Pinch provides 1556 heuristic-planned trajectories across target shapes.Evaluation varies bucket poses, heightmaps, bottle and beaker positions, target shapes, and gripper or rod poses.
  • Articulated and mobile manipulation: Miscellaneous articulated tasks include opening randomized-friction drawers and doors, pushing a chair, and lifting buckets while preserving an enclosed ball.The tasks use single- or dual-arm mobile robots and include held-out assets for some evaluations.
  • Simulation: Rigid-soft coupling transfers rigid-body poses to the soft simulator and soft-body forces to the rigid simulator during fully coupled simulation.The algorithm updates MPM particles and rigid bodies through penalty forces, accumulated forces, and torques.
  • Rendering: Rendering uses screen-space particle splatting with bilateral depth filtering, while a single GPU-GPU copy transfers particle positions to the renderer.Scaling the filter by camera pixel distance makes results visually consistent most of the time, and the copy reduces rendering latency.
  • Performance: Soft-body environments run at around 17–18 FPS individually and 80–84 FPS with 16 parallel environments on one RTX Titan GPU.The reported parallel performance is approximately 4x real time.

E.3 MORE DETAILS ON SAMPLE COLLECTION SPEED COMPARISON

This section details sample-collection comparisons, manipulation baselines, demonstration processing, and visual policy training. The experiments evaluate both speed and task success under specified controller, observation, and asset settings.

  • Sample collection speed: The PickCube speed comparison uses 25 physical simulation steps per control step, GPU rendering, and a 16-logical-core CPU budget.
  • Manipulation baselines: Contact-GraspNet demonstrations include successful can-grasp trajectories and failure modes involving rotation, distance, friction, geometry, and kinematic constraints.
  • Manipulation baselines: Transporter Network success is much lower under ManiSkill2’s criterion requiring pieces to fully fit in holes, with failures mainly caused by imprecise rotation or position prediction.The comparison is reported for AssemblingKits on training and test assets.
  • Demonstration processing: Demonstrations are translated from joint-position control to delta end-effector pose for rigid tasks and delta joint position for soft-body tasks, then filtered to retain successful trajectories.
  • Visual policies: RGBD policies use IMPALA and point-cloud policies use PointNet after camera fusion and preprocessing.The visual inputs combine camera observations with robot state and task information according to the configured observation mode.
  • Training setup: Demonstration-based training uses 1000 trajectories for most rigid environments and 200 for most soft-body environments, with larger counts for multi-asset tasks and Pinch.
  • Training setup: DAPG+PPO combines online rollout transitions with demonstration transitions in its policy objective, with the demonstration weight set to ω = 0.1 · 0.995^N.N denotes the PPO epoch count.

F.5 FURTHER ANALYSIS OF IMITATION LEARNING ON SOFT-BODY TASKS

The analysis finds that imitation learning struggles with fine-grained soft-body control and precise shape deformation, while point-cloud performance depends on representation choices. Relaxing assembly clearances improves some results but leaves additional difficulties.

  • Soft-body control: Behavior cloning struggles to estimate how actions affect fine-grained soft-body displacement and deformation.Fill succeeds more often than Pour because Pour requires precise bottle tilt to match the final liquid level.
  • Soft-body control: Tasks requiring accurate measurements are harder for behavior cloning: Pour needs liquid-level precision and Excavate needs depth control, whereas Hang is easier.
  • Shape deformation: Behavior cloning performs poorly on Pinch and Write because it cannot effectively use target shapes to guide precise soft-body deformation.The learned motions make progress toward pinching or drawing but do not reach the required goals.
  • Point-cloud learning: Point-cloud learning is strongly affected by controller choice, coordinate-frame selection, and the addition of appropriate visual cues.The ablations use PickSingleYCB with DAPG+PPO and compare against a delta end-effector pose baseline.
  • Assembly analysis: Increasing assembly clearance raises performance for some imitation-learning and demonstration-based RL agents, but many agents still perform poorly.The analysis points to additional challenges such as occluded target slots during insertion.

G COMPARISON WITH OTHER BENCHMARKS FOR ROBOTIC MANIPULATION

ManiSkill2 is compared with robotic-manipulation benchmarks through task coverage, physical grasping, demonstrations, object variety, and controller support. Its scope includes fully physical manipulation, while photorealism, domain randomization, and scene-level variation remain active limitations.

  • Benchmark comparison: ManiSkill2 provides large-scale demonstrations, varied objects, multi-controller support, demonstration action-space conversion, and fully physical grasping.These comparison dimensions are explicitly identified as ManiSkill2 features.
  • Simulation scope: ManiSkill2 uses an MPM-based soft-body simulator based on Warp, but deformable objects such as cloth are not yet supported.The limitation applies specifically to the current soft-body simulation scope.
  • Benchmark difficulty: Low success rates on many tasks indicate that the benchmark poses challenging manipulation problems for the research community.The authors present these success rates as evidence of task difficulty.
  • Limitations: The authors identify photorealism, domain randomization, and scene-level variation as current limitations under active development.The paper focuses on physically realistic short-horizon, low-level visuomotor manipulation while expanding these areas.

H CONTRIBUTIONS

The contributions cover benchmark infrastructure, rigid- and soft-body environments, rendering, simulation, evaluation experiments, and project management. They include both system construction and experiments spanning sense-plan-act, RL, IL, and real-world settings.

  • Infrastructure: The team designed and implemented ManiSkill2 infrastructure.This contribution is attributed to Jiayuan Gu, Fanbo Xiang, and Rui Chen.
  • Environments and demonstrations: Rigid-body environments were designed and demonstrations collected.This work is attributed to Jiayuan Gu, Xuanlin Li, Xiqiang Liu, and Tongzhou Mu.
  • Environments and demonstrations: Soft-body environments were designed and demonstrations collected.This work is attributed to Fanbo Xiang and Xinyue Wei.
  • Simulation and rendering: The project included server-based asynchronous rendering and a soft-body simulator.These systems were designed and implemented by separate contributors.
  • Experiments: Experiments covered sense-plan-act, RL and IL on rigid-body environments, IL on soft-body environments, and real-world evaluation.The listed experiments span simulated rigid-body, simulated soft-body, and real-world settings.
  • Evaluation and management: The project also implemented a cloud-based evaluation system and included project management or advisory work.These contributions are separately listed in the author-contribution statements.
Loading 2302.04659v1…