ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Tongzhou Mu; Zhan Ling; Fanbo Xiang; Derek Yang; Xuanlin Li; Stone Tao; Zhiao Huang; Zhiwei Jia; Hao Su

Source-linked AI summary

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, Hao Su

arXiv:2107.14483v5Jul 30, 2021cs.LGcs.AIcs.CVcs.RO

TL;DR

Generalizable manipulation requires policies that handle unseen objects with substantial intra-class variation, but existing benchmarks provide limited support for this setting. ManiSkill addresses the gap with a full-physics benchmark combining diverse articulated objects, egocentric 3D observations, multiple learning tracks, and large-scale demonstrations. The benchmark establishes an open platform for studying object-level generalization, although its current assets, tasks, and sim-to-real validation remain limited.

Problem
Manipulation policies must generalize to unseen objects within a category despite topological and geometric variation, while existing benchmarks provide insufficient support for this setting.
Method
ManiSkill benchmarks physical manipulation with diverse articulated objects, full-physics environments, egocentric point-cloud or RGB-D inputs, demonstrations, and multiple policy-learning tracks.
Results
ManiSkill provides approximately 36,000 successful trajectories and 1.5 million point-cloud/RGB-D frames, while object-level generalization results remain low across diverse geometries.
Takeaways & Limitations
The benchmark supports reproducible research on object-level generalization by connecting 3D vision, reinforcement learning, robotics, and learning from demonstrations.
Takeaways & Limitations
The current benchmark has 162 articulated objects, four tasks that do not comprehensively cover household manipulation, and no sim-to-real experiments.

Abstract

from arXiv · show

Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the diversity of 3D shapes that align with real-world intra-class complexity in topology and geometry. Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator. 3D assets in ManiSkill include large intra-class topological and geometric variations. Tasks are carefully chosen to cover distinct types of manipulation challenges. Latest progress in 3D vision also makes us believe that we should customize the benchmark so that the challenge is inviting to researchers working on 3D deep learning. To this end, we simulate a moving panoramic camera that returns ego-centric point clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad set of researchers interested in manipulation research. Besides supporting the learning of policies from interactions, we also support learning-from-demonstrations (LfD) methods, by providing a large number of high-quality demonstrations (~36,000 successful trajectories, ~1.5M point cloud/RGB-D frames in total). We provide baselines using 3D deep learning and LfD algorithms. All code of our benchmark (simulator, environment, SDK, and baselines) is open-sourced, and a challenge facing interdisciplinary researchers will be held based on the benchmark.

1 Introduction and Related Works

ManiSkill is designed to benchmark generalizable manipulation over diverse articulated objects using full-physics simulation, 3D visual inputs, and multiple learning settings. It combines varied tasks, large-scale demonstrations, open-source infrastructure, and baselines to support reproducible interdisciplinary research.

Motivation: Human-like manipulation requires generalizing skills to unseen objects within a category despite substantial topological and geometric variation.The paper uses swivel chairs with differing armrests, headrests, wheel counts, and backrest shapes as an example.
Benchmark Design Challenges: Existing benchmarks remain insufficient because they often abstract away low-level physics or lack realistic intra-class object diversity and varied motion types.The paper identifies limitations in abstract action spaces, procedural modeling, and single-task benchmarks.
3D Visual Inputs: The benchmark uses robot-mounted egocentric panoramic cameras that provide point-cloud or RGB-D inputs and supports 3D neural policy baselines.This design targets 3D perception models using sensor data commonly accessible in real environments.
Learning Tracks: ManiSkill supports multiple policy-learning tracks for computer vision, reinforcement learning, and robotics researchers, including interaction-based and demonstration-based methods.The benchmark separates training settings because imitation and offline reinforcement learning use demonstrations, whereas online reinforcement learning requires environment interaction.
Benchmark Design: ManiSkill provides 162 manually processed objects across 3 categories and 4 object-centric tasks with distinct manipulation motions and skill requirements.The tasks target challenges including revolute and prismatic joint constraints and dual-arm collaboration.
Open Research Infrastructure: The benchmark and its simulator, environments, SDK, and baselines are fully open source to support reproducible comparisons.The paper contrasts this stack with common physical manipulation environments that depend on commercial software.
Demonstrations: Its demonstration dataset contains approximately 36,000 successful trajectories and 1.5 million point-cloud or RGB-D frames generated through scalable reinforcement learning.Dense rewards from a shared task-specific template help automate demonstration collection for difficult tasks.

2 ManiSkill Benchmark

ManiSkill is a full-physics benchmark for learning short-horizon manipulation skills from egocentric 3D observations and successful demonstrations. It spans diverse articulated objects, motion constraints, and evaluation settings while verifying environment solvability.

Basic Terminologies and Setup: ManiSkill defines manipulation skills as finite-horizon POMDPs over same-category objects and object-specific environment parameters.Observations support state, pointcloud, and RGB-D modes.
Basic Terminologies and Setup: Object-level generalization is evaluated by training on some objects and testing on unseen objects from the same category.Successful demonstrations are provided for each training environment.
Tasks with Diverse Motions and Skills: The benchmark covers revolute, prismatic, planar, and unconstrained motions through cabinet opening, chair pushing, and bucket-moving tasks.Tasks additionally vary in dual-arm coordination and suitability for motion planning.
Tasks with Diverse Motions and Skills: ManiSkill verifies every environment is solvable by generating successful trajectories while maintaining task difficulty.This avoids leaving solvability assessment to benchmark users.
Observations and Setup: RGB-D and point-cloud observations combine three robot-mounted cameras into an ego-centric panoramic view, with 400×160 resolution per camera.The view can contain self-occlusion and object occlusion because the cameras are mounted on the robot.
Demonstrations and Evaluation: The benchmark provides ~36,000 successful trajectories and ~1.5M frames, supports sparse and dense rewards, and evaluates mean success rate on predetermined test instances.Users can render visual observations from stored environment-state demonstrations.

3 Baseline Architectures, Algorithms, and Experiments

ManiSkill supplies point-cloud-based policy baselines and learning-from-demonstrations algorithms, then evaluates them in single-environment and object-level generalization settings. More demonstrations improve single-environment performance, but generalization across diverse objects remains difficult.

Baseline Architectures: The baselines use point-cloud features containing position, RGB, segmentation masks, and robot state for 3D policy learning.The benchmark includes a global PointNet architecture and a segmentation-aware PointNet-plus-Transformer architecture.
Algorithms: The evaluated learning-from-demonstrations methods include behavior cloning, BCQ, and TD3+BC.Behavior cloning matches predicted and ground-truth actions by minimizing L2 distance.
Single Environment Results: Increasing demonstration trajectories gradually raises success rates on a fixed OpenCabinetDrawer environment.PointNet + Transformer is more sample efficient than PointNet, while offline RL did not outperform BC in this experiment.
Object-Level Generalization Results: Object-level generalization produces low success rates on both training and test environments, even for the best PointNet + Transformer agent.Training across object variations is harder than training on one fixed environment.
Object-Level Generalization Results: The results identify physical manipulation across diverse geometries and generalization to novel objects as open challenges for current 3D learning and demonstration-based methods.The authors present the benchmark and baselines as a basis for further research.

4 Conclusion and Limitations

ManiSkill is proposed as a benchmark for generalizable physical manipulation from 3D visual inputs, using diverse object geometries and large-scale demonstrations. Its current scope is limited by asset, task, and sim-to-real coverage.

ManiSkill targets object-level generalizability by combining diverse object geometries with 3D computer vision, reinforcement learning, and robotics.The benchmark is intended to encourage research on manipulating unseen objects within a category.
162 articulated objects are currently provided, with additional PartNet-Mobility assets planned.
Four current tasks exemplify distinct manipulation challenges but do not comprehensively cover household manipulation skills.Planned additions include tasks such as pouring between buckets through two-arm coordination.
Sim-to-real experiments have not yet been conducted and are identified as future work.

Supplementary Material

The supplementary material documents ManiSkill’s architecture, system and task design, demonstration collection, baseline implementations, and environment evaluation details.

Figure 5 depicts the pipeline from re-modeled PartNet-Mobility objects and train/test splits to SAPIEN environments and evaluation.Successful demonstrations are generated on training environments, while policies are evaluated by mean success rate on test environments.
The supplementary material provides additional details on the system, tasks, and demonstration collection.
Baseline documentation covers point-cloud subsampling, PointNet-plus-Transformer architectures, imitation learning, and offline reinforcement learning.

B Further Details of Tasks and System

ManiSkill’s supplementary task and system details describe visual and physical simulation settings, randomized environments, segmentation masks, task success criteria, and runtime considerations.

ManiSkill reports environment FPS separately for state, pointcloud, and RGB-D modes, with physical simulation dominating state mode and rendering dominating visual modes.Measurements use one Intel i9-9960X CPU and one NVIDIA RTX TITAN GPU.
One environment step corresponds to five control steps through frame-skipping, shortening task horizons while making the environment FPS five times larger than without skipping.
Articulated objects and robots make physical simulation inherently slow, while visual observations add substantial rendering cost, especially with three cameras.Chairs can contain up to 20 joints and tens of thousands of collision meshes.
Environment parameters randomize robot initialization, target-object physical properties, and selected object poses across tasks.Randomized quantities include robot in-plane position and orientation, joint frictions, and bucket or chair placement and orientation.
Task-relevant binary segmentation masks identify robots and target parts or objects, including handles for cabinet-door and cabinet-drawer tasks.OpenCabinetDoor and OpenCabinetDrawer each provide three masks; PushChair and MoveBucket each provide one robot mask.
Success criteria require task-specific states to persist, including opening cabinet joints to 90% of their limits, placing chairs within 15 centimeters, and retaining balls in buckets.All episodes have a time limit of 200 steps.

B.5 Evaluation Kit

The evaluation kit accepts policies with configurable task and observation settings, evaluates them on test environments, and reports success-related metrics; supplementary material also documents controllers and environment testing.

The evaluation script takes a task name, observation mode, and solution file containing a policy function that maps observations to actions.
The evaluation kit runs the policy on test environments and reports average success and satisfactory rates for individual success conditions.MoveBucket examples include whether the ball is inside the bucket and whether the bucket is on or above the platform.
A YAML configuration system specifies reusable physical, rendering, scene, camera, randomization, and controller properties across environments.
Robot joints use velocity or position controllers, with inverse dynamics, PhysX PD control, filtering, and PID-based position-to-velocity conversion.
Environment testing checks modeling realism and solvability through visual inspection, physical-property inspection, and manual mouse-and-keyboard task execution.

B.9 Manually Processed Collision Shapes

ManiSkill manually processes Bucket collision meshes because standard VHACD can miss details and create artifacts. Its task environments use automatically generated, smooth multi-stage reward templates to guide reinforcement learning.

Manually Processed Collision Shapes: Bucket collision shapes are manually decomposed into convex shapes because standard VHACD misses details and produces bumps and seams.The processed mesh has 1,445 faces, compared with 2,340 faces for the standard VHACD result.
Reward Templates: Each task uses a manually designed reward template that is automatically translated into a smooth reward for every environment.This exploits the shared category structure across different objects.
Reward Templates: Multi-stage rewards make later-stage rewards strictly larger and smooth rewards at stage transitions.This is intended to prevent agents from remaining indefinitely in intermediate stages.
Reward Templates: OpenCabinetDoor and OpenCabinetDrawer use stages for approaching the handle, contacting the target link, and increasing the opening angle.The first stage penalizes Euclidean distance between the handle and gripper until a distance threshold is reached.
Reward Templates: PushChair rewards approach to the chair first, then distance to the target location after contact.The approach reward uses the logarithm of end-effector distance to the chair.
Reward Templates: MoveBucket uses four stages, adding rewards for stable two-gripper placement before lifting and subsequent positioning.The template uses gripper-to-bucket distance, gripper arrangement around the center of mass, and gripper height difference.

C.2 Agent Training and Demonstration Collection

Training one reinforcement-learning agent across many object environments becomes infeasible as object count grows, so ManiSkill collects demonstrations by training separate agents per environment. The resulting trajectories are generated from successful SAC policies, with quality checks and varied behaviors.

Agent Training and Demonstration Collection: Jointly training one SAC agent from scratch across many cabinets is infeasible, with success rates dropping sharply as the number of objects increases.The agents use state observations, dense rewards, and 106 time-steps in the reported OpenCabinetDrawer experiment.
Agent Training and Demonstration Collection: ManiSkill trains a population of SAC agents so each agent solves one specific environment before generating demonstrations there.This divide-and-conquer procedure targets the difficulty of multi-environment training.
Agent Training and Demonstration Collection: 300 successful trajectories are uniformly collected per environment after 2.0 × 10^6 training steps.Agents with success rates below 0.3 are retrained in a few cases.
Agent Training and Demonstration Collection: Removing early-done signals during training produces more robust near-end policies, while collection stops at the first success signal.Four parallel processes accelerate sample collection.
Agent Training and Demonstration Collection: The benchmark prioritizes solving manipulation tasks rather than comparing RL-generated demonstrations with human demonstrations.The authors therefore do not treat human-versus-RL demonstration diversity as the benchmark focus.

D Implementation Details of Baseline Architectures, Algorithms, and Experiments

ManiSkill downsamples dense camera point clouds before baseline training and publishes the resulting demonstration dataset. This reduces the need for users to render demonstrations locally.

Point Cloud Subsampling: Raw point clouds contain 192,000 points per frame, so ManiSkill downsamples them to 1,200 points.The procedure first samples up to 50 points per segmentation mask, then samples remaining non-ground points.
Point Cloud Subsampling: The downsampled point cloud demonstrations dataset is publicly available for researchers and benchmark users.Users can use the released data without rendering demonstrations locally.

D.2 Network Architectures

The baseline architectures combine point-cloud features with robot state, while the PointNet + Transformer variant processes segmentation-defined point groups separately. Centering point positions significantly improves performance.

Network Architectures: PointNet policies concatenate each point’s position, RGB, and segmentation-mask features with the robot state.The resulting features are used as the policy network input.
Network Architectures: Centering each point cloud around its mean coordinates significantly improves performance in PointNet policy models.The normalized position is concatenated with the original position after mean subtraction.
Network Architectures: The vanilla PointNet processes all point features jointly and maps its global feature through an MLP to output actions.Its hidden dimensions are [256, 512], followed by [512, 256, action_dim].
Network Architectures: The PointNet + Transformer model uses separate PointNets for each segmentation mask, unmasked points, and the entire point cloud.With k segmentation masks, it extracts k + 2 global features using PointNets with hidden dimension 256.

D.3 Implementation Details of Learning-from-Demonstration Algorithms

The benchmark evaluates learning from demonstrations with Behavior Cloning and two offline-RL algorithms, BCQ and TD3+BC. It also examines TD3+BC’s sensitivity to α and reports that non-zero α performed worse than BC in the tested setting.

Algorithms: Behavior Cloning, BCQ, and TD3+BC are used as learning-from-demonstration baselines.BC directly clones demonstration actions, whereas BCQ models demonstration-action distributions and selects actions using learned Q-values.
Implementation: BCQ and TD3+BC use specified hyperparameter configurations summarized in Tables 7 and 8.The supplied implementation details note that BCQ’s VAE latent vector has dimension 2 times the action-space dimension.
TD3+BC: α = 0 makes TD3+BC equivalent to BC.The original TD3+BC paper used α = 2.5, while the benchmark discussion distinguishes the α = 0 case from non-zero settings.
TD3+BC: Non-zero α made TD3+BC consistently worse than BC, even after α was reduced 100 times from the original value.The earlier comparison used α = 0.02 because α = 0 does not distinguish TD3+BC from BC.