ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

1) A Unified Benchmark for Generic and Generalizable Manipulation Skills

• ManiSkill2 provides 20 task families, 2000+ objects, 4M demonstration frames, and a unified interface and protocol for evaluating diverse algorithms on seen and unseen assets and configurations.Tasks span stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation, with fully simulated dynamics, multiple observation modes and controllers, and a cloud-based evaluation system.Source: §1 · p. 2

2) Real-time Soft-body Environments

• ManiSkill2 introduces a custom GPU MPM simulator with two-way rigid-body coupling for real-time simulation and rendering of soft-body materials.The passage describes it as the first embodied AI environment to support two-way coupled rigid-MPM simulation and real-time MPM simulation and rendering.Source: §1 · p. 2

3) Multi-controller Support and Conversion of Demonstration Action Spaces

• ManiSkill2 supports varied and composable controllers across robot components and converts demonstration action spaces to a desired space, regardless of the demonstrations’ original controllers.Examples include combining a base velocity controller, arm task-space position controller, and gripper joint-space position controller.Source: §1 · p. 3

4) Fast Visual RL Experiment Support

• An RGBD-input PPO policy collects samples at about 2000 FPS using 1 GPU and 16 CPU processors on a regular workstation.Environments are scripted in Python, vectorized across multiple processes, and use an asynchronous RPC-based render server-client system to improve throughput and reduce GPU memory use.Source: §1 · p. 3

2 BUILDING ENVIRONMENTS FOR GENERALIZABLE MANIPULATION SKILLS

• Environment development uses verification-driven iteration, with TAMP, MPC, or RL generating demonstrations that also verify environments.Source: §2 · p. 3

2.1 HETEROGENEOUS TASK FAMILIES

• ManiSkill2 covers 20 task families spanning rigid and soft bodies, stationary and mobile bases, and single- and dual-arm manipulation.The section also describes mobile tasks such as PushChair, MoveBucket, OpenCabinetDoor, and OpenCabinetDrawer, plus stationary TurnFaucet and AvoidObstacles tasks.Source: §2.1 · pp. 3–4
• Six soft-body tasks require agents to move or deform bodies into goal states, testing how actions affect quantities such as displacement and deformation.Tasks include filling, hanging, excavating, pouring, pinching, and writing; targets can be provided as RGBD images, point clouds, or depth maps.Source: §2.1 · p. 3
• Peg-in-hole tasks emphasize millimeter-scale clearances and contact-rich insertion, with PegInsertionSide, PlugCharger, and AssemblingKits modeled as physical insertion tasks.The examples use 3 mm and 0.5 mm clearances, and programmatically carved slots that can use specified clearances such as 0.8 mm; TAMP generates demonstrations for these tasks.Source: §2.1 · p. 4
• A 6-DoF pick-and-place curriculum spans cube stacking, YCB and EGAD objects, and clutter scenes, challenging generalizable grasp prediction across object variations.Goals may be randomly placed in a 30×50×50 cm3 workspace, and some high-quality grasp poses may be unreachable when robot kinematic constraints are ignored.Source: §2.1 · p. 4

2.2 MULTI-CONTROLLER SUPPORT AND CONVERSION OF DEMONSTRATION ACTION SPACES

• Controllers map desired configurations to joint torques, and a closed-loop method converts demonstrations between action spaces while reproducing their kinematic and dynamic processes.Conversion can use forward kinematics to match desired end-effector poses, but the method suffers from accumulated execution errors.Source: §2.2 · pp. 4–5

• The GPU MPM simulator and two-way rigid-body coupling enable the first real-time MPM-based soft-body manipulation environment.Source: §3 · p. 5

Rigid-soft Coupling

• The coupling transfers rigid-body poses to the soft-body simulator and soft-body forces and torques back, using particle penalty forces that reduce artifacts such as penetration.Collision shapes are copied into the soft-body simulator as analytical SDFs for primitives or SDF volumes for meshes.Source: §3 · p. 5

Performance Optimization

• Performance optimizations use Warp, faster CPU-GPU data transfer, compiler caching, and a fast surface-rendering algorithm for MPM particles.These measures target simulation performance, compilation and startup time, and rendering for visual-learning environments.Source: §3 · p. 5

4 PARALLELIZING PHYSICAL SIMULATION AND RENDERING

• Asynchronous rendering starts reward computation after draw calls are submitted, overlapping CPU work with GPU rendering that would otherwise leave the CPU idle.Reward computation often does not depend on rendering results.Source: §4 · p. 6
• The render server avoids GPU-to-CPU-to-GPU image copies and shares GPU resources across environments, reducing transfer time and memory use.It uses a thread pool in the main Python process to serve rendering requests from simulation workers, requires minimal API changes, and supports network communication for multi-machine simulation and rendering.Source: §4 · p. 6, Figure 2 · p. 6
• The PickCube framework comparison matched simulation parameters as closely as possible, but differing simulation and rendering fidelity makes a fair comparison challenging.The experiment used 500 Hz simulation and 20 Hz control frequencies.Source: §4 · p. 6
• 2000+ FPS with 16 parallel environments on one GPU; asynchronous rendering alone also outpaces the other tested frameworks.The comparison tests PickCube with random actions and a randomly initialized Nature CNN policy; frameworks include Habitat 2.0, RoboSuite 1.3, and Isaac Gym.Source: §4 · pp. 6–7, Figure 3 · p. 7
• With 74 YCB objects per scene, Habitat 2.0 memory usage grows linearly as environments increase, while ManiSkill2 needs very little memory for additional environments.Habitat used GPU texture compression while ManiSkill2 used regular textures.Source: §4 · p. 7, Table 1 · p. 7

5.1 SENSE-PLAN-ACT

• The SPA experiments use Contact-GraspNet and Transporter Networks, in a framework where perception and planning are independently optimized and methods are usually open-loop.For PickSingleYCB, Contact-GraspNet ranks grasp poses by confidence, and motion planning tries them in order until it finds a valid plan.Source: §5.1 · p. 8, Contact-GraspNet for PickSingleYCB · p. 8

Contact-GraspNet for PickSingleYCB:

• 43.24% success across 74 PickSingleYCB objects reflects Contact-GraspNet’s grasping limits, including low-confidence predictions and high-confidence poses with poor grasp quality or unreachable targets.Each object received five trials; success required the object’s center of mass to be within 2.5 cm of the goal.Source: Contact-GraspNet for PickSingleYCB · p. 8

Transporter Network for AssemblingKits:

• 18% of AssemblingKits trials succeeded under the stricter fit-into-hole metric, compared with 99% under the original pose-accuracy metric.Transporter Networks used two camera views and 144 rotation bins, then motion planning to pick and place the peg.Source: Transporter Network for AssemblingKits · p. 8

5.2 IMITATION & REINFORCEMENT LEARNING WITH DEMONSTRATIONS

• Behavior cloning has low or zero success on rigid-body tasks and struggles with precise soft-body actions and target-guided deformation.It performs poorly on precision-sensitive Excavate and Pour and never succeeds on Pinch or Write, while Fill and Hang achieve much higher success rates.Source: §5.2 · pp. 8–9, Table 2 · p. 9
• Point cloud-based DAPG+PPO agents outperform RGBD-based agents on pick-and-place, but achieve near-zero success on other tasks, especially precise assembly.On PickSingleYCB, the success rate exceeds Contact-GraspNet with motion planning; increasing assembly clearance and reducing difficulty yields much higher performance.Source: §5.2 · p. 9, Table 3 · p. 9
• Replacing the delta end-effector pose controller with a delta joint position controller reduces PickSingleYCB success to 0.22±0.18.This ablation used point cloud-based agents and demonstrates the impact of controller choice on agent learning.Source: §5.2 · p. 9

5.3 SIM2REAL

• 60.0% of 50 real-world PickCube trials succeeded, versus 91.0% in simulation; the authors conjecture depth-map domain gap caused the drop.The policy was trained for 10M time steps with Gaussian noise and random pixel dropout as simulation data augmentation.Source: §5.3 · pp. 9–10
• Executing the same motion-planned action sequence in simulation and reality shows that 2-way coupled rigid-MPM simulation can reasonably reproduce plasticine deformation from multiple grasps.The comparison evaluates final shapes for the Pinch task.Source: §5.3 · p. 10

• ManiSkill2 provides 20 manipulation task families, over 2000 objects, and over 4M demonstration frames.The benchmark targets generic and generalizable manipulation skills.Source: §6 · p. 10
• Its infrastructure combines efficient rigid-body simulation and rendering for RL sample collection, real-time MPM-based soft-body environments, and versatile multi-controller conversion.Source: §6 · p. 10
• ManiSkill2 demonstrates applications in benchmarking sense-plan-act, imitation-learning, and reinforcement-learning algorithms, while learned policies show potential for real-world transfer.Source: §6 · p. 10

REPRODUCIBLITY STATEMENT

• The ManiSkill benchmark, simulators, renderers, demonstration-generation and learning code, and training assets are open source, while asset-generation and cloud-evaluation code are withheld for challenge fairness.The authors state that all reported results are reproducible and invite researchers to join the public challenge.Source: REPRODUCIBLITY STATEMENT · p. 11

A.1 VERIFICATION-DRIVEN ITERATIVE DEVELOPMENT

• Following agile software development, the workflow uses demonstration-generation approaches to verify environments and is designed for scalable, affordable continuous integration of assets and tasks.Figure 5 illustrates the environment-building workflow.Source: §A.1 · p. 15, Figure 5 · p. 15
• Task creation establishes essentials including assets, robot configurations, initial states, and success metrics.Asset work can include convex decomposition (Wei et al., 2022) and texture baking.Source: §A.1 · p. 15
• Reward design prototypes shaped rewards, which are requisite for methods such as Model-Predictive Control (MPC) and RL.Source: §A.1 · p. 15
• Observation configuration tailors camera parameters and placements so task observations contain adequate information.Source: §A.1 · p. 15
• TAMP, MPC, and RL can each collect demonstrations and verify environments, with their suitability depending on task demands.TAMP avoids reward crafting and suits many stationary tasks but struggles with underactuated systems; MPC searches difficult tasks without training or observations but needs carefully designed rewards; RL requires training and tuning but scales better than MPC at inference.Source: §A.1 · p. 15

A.2 CLOUD BASED EVALUATION SYSTEM

• The cloud benchmark accepts user solutions as Docker images that provide an observation-to-action function, lets users configure observation modes and controllers, and publishes evaluation results.The server pulls each image, runs the evaluation, and submits results to a database for display on a public benchmark.Source: §A.2 · pp. 15–16

B DETAILS OF OUR COMPREHENSIVE CONTROLLER SUITE; B.1 TERMINOLOGY; B.2 TARGET VS. NON-TARGET CONTROLLERS; B.3 NORMALIZED ACTION SPACE

• Controllers connect policy actions to robot control signals; target deltas refer to the previous desired pose, and most action spaces are normalized to [−1, 1].The Tool Center Point is a user-defined frame, often fixed relative to the end effector; for a two-finger gripper, it is defined between the fingers.Source: B DETAILS OF OUR COMPREHENSIVE… · p. 16, §B.2 · p. 16, §B.3 · p. 16

B.4.1 ARM CONTROLLERS

• The arm controller suite includes 7-dimensional unnormalized joint-position control and end-effector delta-position and delta-pose controllers, with the latter adding rotation input.The delta-position controller controls position but not rotation; the target delta-pose variant specifies the target transformation relative to the previous desired transformation.Source: §B.4.1 · p. 17

B.4.2 GRIPPER CONTROLLER (1-DIM)

• The gripper controller uses joint-position control and forces both fingers to share the same target position.Source: §B.4.2 · p. 17

B.5 EFFECTIVENESS OF CONVERSION OF DEMONSTRATION ACTION SPACES

• Conversion success was 99% for PickSingleYCB, 98% for AssemblingKits, 100% for Write, and 80% for TurnFaucet when converting joint-position demonstrations to end-effector delta-pose actions.The study sampled 100 demonstrations per task; the TurnFaucet policy’s inconsistent pushing contact can make execution sensitive to accumulated errors.Source: §B.5 · p. 17

C DETAILS OF OBSERVATIONS, TASK FAMILIES, DEMONSTRATIONS AND EVALUATION PROTOCOLS; C.1 SUPPORTED OBSERVATION MODES

• The evaluation averages results from two stages, and supported observations include state, RGB-D, and fused point clouds, optionally with robot-joint segmentation.Default RGB-D and point-cloud modes omit ground-truth segmentation; robot-link segmentation is separately supported and can be obtained from robot proprioception in real-world applications.Source: C DETAILS OF OBSERVATIONS, TASK… · p. 17, §C.1 · p. 18

C.2 PICK-AND-PLACE

• Pick-and-place coverage ranges from cube placement and stacking to 74 YCB objects, 1,600 EGAD training objects, and picking from cluttered YCB objects.Evaluation includes 40 additional test objects for YCB and a held-out EGAD set of 75 objects per stage; success generally requires the object within 2.5 cm of its goal and the robot static.Source: §C.2 · pp. 18–19

C.3 ASSEMBLY

• Assembly tasks cover inserting objects into kit slots, pegs into box holes, and chargers into wall receptacles; kit evaluation uses held-out shapes and a three-part fit criterion.For kit insertion, success requires center height within 3 mm of the plate, rotation error within 4 degrees, and position error within 2 cm; Figure 6 shows sample plate test assets.Source: §C.3 · pp. 19–20, Figure 6 · p. 20

C.4 MISCELLANEOUS TASKS

• Miscellaneous tasks cover navigating dense randomized obstacles to a goal end-effector pose and turning faucet handles past target angular distances.The navigation task supplies the goal pose and uses a 2.5 cm and 15-degree success tolerance; faucet observations include remaining rotation, target handle position, and rotation axis direction.Source: §C.4 · p. 20

C.5 SOFT-BODY MANIPULATION

• Soft-body tasks include transferring clay into a beaker, hanging a noodle on a rod, and lifting a specified amount of clay to a target height.Success requires over 90% of the clay in the beaker and soft-body velocity below 0.05; noodle and lifting tasks also impose placement, spill, height, and velocity conditions.Source: §C.5 · p. 21
• Other soft-body tasks pour liquid into a beaker, deform plasticine to a target shape, and write a character on clay.Pouring requires a liquid level within 4 mm of the red line and fewer than 100 spilled particles; shaping uses a Chamfer-distance threshold of 0.3t, while writing samples from over 50 characters and requires IoU above 0.8.Source: §C.5 · pp. 21–22

C.6 MOBILE MANIPULATION

• Mobile tasks include opening designated cabinet drawers or doors, pushing a swivel chair to a target, and lifting a bucket with a ball onto a platform.The drawer and door tasks use 10 unseen test cabinets; the chair task uses 10 unseen chairs, and the bucket task uses 10 unseen buckets. Success conditions include static open joints, an upright chair within 15 cm of its target, or an upright static bucket retaining the ball.Source: §C.6 · pp. 22–23

D.1 SOFT-BODY SIMULATION AND 2-WAY COUPLING ALGORITHM

• The soft-body coupling algorithm advances rigid and soft scenes together, transfers rigid poses and accumulated forces and torques, and steps MPM particles through particle-to-grid and grid-to-particle updates.Rigid-body states and MPM particles are copied to the SAPIEN renderer after simulation.Source: §D.1 · p. 23

D.2 SOFT-BODY RENDERING

• Soft-body rendering uses screen-space splatting of particle spheres, bilateral depth smoothing, and normals and lighting computed from the smoothed depth.A single GPU-to-GPU copy transfers particle positions to the renderer; filter scaling by camera distance makes results visually consistent most of the time despite possible view inconsistency.Source: §D.2 · pp. 23–24

D.3 SOFT BODY DETAILS

• 16 parallel soft-body environments run at 80–84 FPS (4× real time) on one RTX Titan GPU, while one environment runs at 17–18 FPS.The authors attribute the gain mainly to sequential CPU rigid-body and GPU soft-body simulation, with CPU kernel submission also limiting performance.Source: §D.3 · p. 24

E.1 RENDER SERVER IMPLEMENTATION; E.2 ADDITIONAL BENEFITS OF RENDER SERVER

• The render server handles concurrent requests through a thread pool, returns promptly to workers after scheduling draw calls, and shares GPU resources through a central manager; keeping rendering in one process also aids development and profiling.The authors report that Nsight System could not profile Vulkan across multiple processes on Linux in their experiments, making a single main rendering process useful for optimizing hardware utilization.Source: §E.1 · p. 24, §E.2 · p. 24

E.3 MORE DETAILS ON SAMPLE COLLECTION SPEED COMPARISON

• The PickCube speed comparison uses 25 physical simulation steps per control step, GPU rendering, 16 CPU cores, 128G memory, and one RTX Titan GPU.Collision-mesh vertex counts are 1009 for ManiSkill2 and Habitat-2.0 and 1210 for IsaacGym and Robosuite; visual meshes have 69747 vertices for all simulators.Source: §E.3 · p. 25

F.1 CONTACT-GRASPNET FOR PickSingleYCB

• Contact-GraspNet uses the pretrained model from its original authors (Sundermeyer et al., 2021), with examples showing both successful can grasping and common grasp failures.Failure examples include incorrect rotation, insufficient proximity, slipping, and motion-planning constraints.Source: §F.1 · p. 25, Figure 8 · p. 25, Figure 9 · p. 25

F.2 TRANSPORTER NETWORK FOR AssemblingKits

• The Transporter Network setup increases rotation-prediction bins from 36 to 144, tests suction and Franka Panda two-finger grippers, and trains on fused hand- and base-camera point clouds.Pick positions and placement goal poses come directly from environment state information.Source: §F.2 · p. 25
• Transporter Network success is much lower under ManiSkill2’s criterion requiring pieces to fully fit in holes, with failures mainly from imprecise rotation or position predictions.The comparison is against the original paper’s metric; supplied passages do not include the table’s numerical values.Source: §F.2 · pp. 25–26

F.3 DETAILED SETUP FOR IMITATION LEARNING & RL FROM DEMONSTRATIONS

• Demonstration-based agents translate joint-position demonstrations to delta end-effector pose control for rigid bodies or delta joint-position control for soft bodies, then train only on successful trajectories.Training uses 1000 trajectories for most rigid-body environments, with specified multi-asset exceptions, and 200 for most soft-body environments, except Pinch with 1550.Source: §F.3 · p. 26
• RGBD agents use IMPALA with concatenated 128x128 base- and hand-camera images, while point-cloud agents use PointNet on fused, ground-filtered, 1200-point clouds.Rigid-body point clouds are transformed into the end-effector frame, and PickCube and PickSingleYCB inputs include 50 sampled green points around the goal.Source: §F.3 · p. 26
• The demonstration-based online RL method modifies PPO and DAPG using an objective adapted from Jia et al. (2022), with ω = 0.1 · 0.995^N across PPO epochs.ρD denotes demonstration-transition distributions and ρπ online rollout-transition distributions; each PPO epoch samples online steps before policy and value updates.Source: §F.3 · p. 26

F.4 RESULTS FOR DAPG+PPO ON HELD-OUT OBJECT SETS

• Held-out evaluation reports mean and standard deviation of DAPG+PPO success rates on rigid-body test objects after a 25M-time-step training budget.The section contrasts held-out object-set evaluation with earlier results on training object sets; supplied passages provide no table values.Source: §F.4 · p. 26, Table 6 · p. 27

F.5 FURTHER ANALYSIS OF IMITATION LEARNING ON SOFT-BODY TASKS

• BC struggles when soft-body tasks require precise quantity or deformation control: Pour and Excavate demand fine judgments, while Fill and Hang are easier.Pour requires controlling bottle tilt to match a target liquid level, and Excavate requires digging to collect a specified amount of clay.Source: §F.5 · p. 27
• BC models make progress on Pinch and Write but use target shapes poorly to guide deformation, leaving pinches and drawn patterns short of their goals.The robot learns pinching motions and draws patterns, but they are not close enough to the targets.Source: §F.5 · p. 27

F.6 MORE RESULTS ON POINT CLOUD-BASED MANIPULATION LEARNING

• Point-cloud learning benefits from suitable coordinate frames and visual cues, alongside controller choices that significantly affect performance.The analysis reports results in Table 7 for point-cloud agents trained with DAPG+PPO on PickSingleYCB.Source: Table 7 · p. 27, §F.6 · p. 28

F.7 MORE ANALYSIS ON ASSEMBLY TASKS

• Increasing assembly-task clearance raises performance for some IL and RL agents, but many still perform poorly, potentially because target slots are occluded during insertion.The analysis varies clearance to make PegInsertionSide and PlugCharger easier; Table 8 reports the clearance settings and results scope.Source: §F.7 · p. 28, Table 8 · p. 28

F.8 NETWORK ARCHITECTURES AND HYPERPARAMETERS FOR IL & RL

• The paper defines the state vector as proprioceptive robot-state information concatenated with task-specific goal information, when provided.Source: §F.8 · p. 28
• IMPALA uses a modified ResNet10-like visual backbone, while PointNet uses hidden sizes [64, 128, 512] and layer normalization on intermediate outputs.IMPALA uses 64-unit layers, a first convolution with kernel size 4 and stride 4, and projects features to 384 dimensions; PointNet omits spatial normalization.Source: §F.8 · p. 28
• BC and DAPG+PPO use Adam with learning rate 3e-4; DAPG+PPO shares actor–critic visual backbones, and zero-initialized output-adjacent linear layers stabilize point-cloud policy learning.Source: §F.8 · p. 28

G COMPARISON WITH OTHER BENCHMARKS FOR ROBOTIC MANIPULATION

• ManiSkill2 emphasizes large-scale demonstrations for every task, object variety, multi-controller support, demonstration action-space conversion, and fully physically implemented grasping.The comparison also describes substantial work selecting, fixing, and remodeling objects, integrating them into tasks, and verifying those tasks.Source: G COMPARISON WITH OTHER BENCHMARKS FOR… · pp. 29–30, Table 10 · p. 29
• ManiSkill2 includes an MPM-based soft-body simulator built on Warp, but does not yet support deformable objects such as cloth.The simulator can support ray-tracing rendering, though the paper notes that photorealism can come at the cost of speed.Source: G COMPARISON WITH OTHER BENCHMARKS FOR… · pp. 29–30
• Low success rates across many tasks indicate that ManiSkill2 poses challenging problems for the community and supports large-scale visual manipulation benchmarking.The authors report generating demonstrations with fully physical grasping and carefully verifying tasks.Source: G COMPARISON WITH OTHER BENCHMARKS FOR… · p. 30
• The benchmark focuses on physically realistic, short-horizon, low-level visuomotor manipulation, while photorealism, broader domain randomization, and scene-level variation remain limitations.The authors report domain randomization for physical parameters and visual appearance in some tasks, and are introducing mobile manipulation with physically implemented base movement and grasp; planned tasks add scene variation and navigation demands.Source: G COMPARISON WITH OTHER BENCHMARKS FOR… · pp. 29–30

• Jiayuan Gu, Fanbo Xiang, and Rui Chen designed and implemented ManiSkill2 infrastructure; Xiang, Zhan Ling, and Gu implemented server-based asynchronous rendering.Source: H CONTRIBUTIONS · p. 30
• Jiayuan Gu, Xuanlin Li, Xiqiang Liu, and Tongzhou Mu designed rigid-body environments and collected demonstrations.Source: H CONTRIBUTIONS · p. 30
• Fanbo Xiang and Xinyue Wei designed soft-body environments and collected demonstrations; Xiang, Wei, and Xiaodi Yuan implemented the soft-body simulator.Source: H CONTRIBUTIONS · p. 30
• The team conducted sense-plan-act, rigid-body RL/IL, soft-body IL, and real-world experiments, implemented cloud-based evaluation, and managed or advised on the project.Source: H CONTRIBUTIONS · p. 30