PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI

Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, Hao Su

arXiv:2410.00425v2cs.ROcs.AI

TL;DR

Existing robotics simulators often cover narrow task and scene ranges and lack capabilities needed for scalable, generalizable manipulation and sim2real. ManiSkill3 addresses this with GPU-parallelized simulation and rendering, heterogeneous environments, diverse tasks and robots, and scalable demonstration generation. It reports up to 30,000+ FPS with lower GPU memory use while supporting broad manipulation research workflows.

  • Problem

    Existing simulation frameworks often support narrow ranges of scenes and tasks and lack heterogeneous simulation and fast parallel rendering needed for scalable visual manipulation learning.

  • Method

    ManiSkill3 combines GPU-parallelized simulation and rendering, heterogeneous environments, a task-building API, diverse robots and environments, and demonstration generation from limited demonstrations.

  • Results

    ManiSkill3 supports 12 environment categories and 20+ robots, with simulation-and-rendering speeds up to 30,000+ FPS and typically 2-3x lower GPU memory usage than other simulators.

  • Takeaways & Limitations

    The framework provides a broad, extensible platform for GPU-scaled robotics simulation, visual reinforcement learning, and demonstration-based learning.

  • Takeaways & Limitations

    Complex environments with many geometries can only be rendered and simulated with fewer parallel environments on one GPU, and the sim2real demonstration uses static cameras.

Abstract

from arXiv · show

Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features critical for scaling generalizable robotics and sim2real. We introduce and open source ManiSkill3, the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation. ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds/voxels visual input, and more. Simulation with rendering on ManiSkill3 can run 10-1000x faster with 2-3x less GPU memory usage than other platforms, achieving up to 30,000+ FPS in benchmarked environments due to minimal python/pytorch overhead in the system, simulation on the GPU, and the use of the SAPIEN parallel rendering system. Tasks that used to take hours to train can now take minutes. We further provide the most comprehensive range of GPU parallelized environments/tasks spanning 12 distinct domains including but not limited to mobile manipulation for tasks such as drawing, humanoids, and dextrous manipulation in realistic scenes designed by artists or real-world digital twins. In addition, millions of demonstration frames are provided from motion planning, RL, and teleoperation. ManiSkill3 also provides a comprehensive set of baselines that span popular RL and learning-from-demonstrations algorithms.

I. INTRODUCTION

ManiSkill3 targets generalized manipulation by addressing the limited task diversity, visual-rendering support, and heterogeneous simulation capabilities of existing GPU simulators. It combines fast GPU-parallelized simulation and rendering with diverse environments, extensible task-building tools, heterogeneous scenes, and demonstration generation.

  • Motivation: Existing GPU simulators support some robotics problems but remain limited for manipulation, visual inputs, heterogeneous environments, and fast parallel rendering.These limitations make visual-input reinforcement learning slower and constrain task generalization.
  • Core contributions: Up to 30,000+ simulation-and-rendering FPS and 2-3x lower GPU memory usage accelerate visual reinforcement learning in ManiSkill3.The system attributes these gains to fast parallel rendering and low system overhead, enabling on-device visual RL and larger training networks.
  • Core contributions: ManiSkill3 provides 12 environment categories and 20+ robot embodiments spanning mobile manipulation, room-scale scenes, drawing, humanoids, quadrupeds, and dextrous hands.The repository emphasizes templates and examples that users can extend for their own use cases.
  • Core contributions: Heterogeneous simulation allows different objects, articulations, or complete room-scale scenes in each parallel environment.A data-oriented design and API manage GPU memory despite differing object degrees of freedom.
  • Core contributions: A unified object-oriented API simplifies building diverse GPU-simulated tasks and supports domain randomization, trajectory replay, and controller action conversion.The API avoids complex tensor indexing and is accompanied by tutorials for customization.
  • Core contributions: A demonstration pipeline learns generalized policies from a few teleoperated or hardcoded demonstrations and rolls them out to create larger datasets.This pipeline targets tasks where reward design is difficult.

II. RELATED WORK

ManiSkill3 addresses limitations in existing robotics simulators by combining broad task coverage with GPU-parallelized visual simulation, rendering, and data generation. It supports scalable visual learning, domain randomization, digital-twin evaluation, and multiple demonstration-generation methods.

  • Related frameworks and datasets: Existing robotics frameworks differ in simulation backends, supported environments, and demonstration availability, while real-world datasets require substantial human labor to collect.CPU-only frameworks run slower than GPU-parallelized alternatives, and Open-X and DROID remain difficult to scale to vision/language-sized datasets.
  • Demonstration generation: ManiSkill3 generates demonstrations through motion planning, RL rewards, and online learning-from-demonstrations methods selected according to task complexity.RLPD and RFCL are used for complex tasks where motion-planning scripts or reward functions are difficult to define.
  • Task and robot coverage: ManiSkill3 provides 12 task categories spanning tabletop, mobile, room-scale, locomotion, humanoid, multi-agent, drawing, dextrous, tactile, digital-twin, and soft-body manipulation.The framework also supports 20+ robot embodiments and provides templates, documentation, and sim2real/real2sim setups.
  • GPU-parallelized learning: ManiSkill3 combines GPU-parallelized simulation and rendering with low system overhead, enabling visual reinforcement-learning tasks to train in minutes rather than hours.The platform reports up to 30,000+ simulation-plus-rendering FPS and typically uses 2–3x less GPU memory than other simulators.
  • Rendering and randomization: ManiSkill3 supports randomized camera poses, object textures, and parallel voxel and point-cloud rendering for visually diverse and 3D robot-learning inputs.The system can render more than 1,000 cameras with different extrinsics and intrinsics and supports 1,024 parallel environments in illustrated examples.
  • Digital twins: Its digital twins evaluate real-world-trained policies such as Octo at 60x to 100x real-world speed without human supervision.The framework implements four SIMPLER environments for evaluating generalist robotic policies.

C. Heterogeneous GPU Simulation

ManiSkill3 enables heterogeneous GPU simulation by allowing parallel environments to contain different objects, articulations, degrees of freedom, and scene configurations. This supports simultaneous training across varied manipulation instances.

  • Capability: Heterogeneous GPU simulation allows each parallel environment to contain different objects, articulations, degrees of freedom, or entire room-scale scenes.A data-oriented system design and GPU-memory API manage these varying configurations.
  • Examples: OpenCabinetDrawer builds a different cabinet with varying degrees of freedom and samples a random drawer link in each environment.This creates heterogeneous cabinet instances within one parallel simulation batch.
  • Examples: Pick Clutter YCB varies the number of objects per environment and samples a random goal object from each clutter scene.The differing clutter configurations are simulated and rendered in parallel.
  • Learning impact: Heterogeneous simulation lets PPO train simultaneously across every object in datasets such as YCB and PartNetMobility.The passage connects this capability to more generalizable manipulation learning.

D. Teleoperation

ManiSkill3 integrates immersive VR teleoperation, trajectory replay, digital twins, and task-building tools to support manipulation workflows across simulation and real-world settings.

  • D. Teleoperation: VR teleoperation translates real-time hand poses into robot actions while streaming 4K stereo video at 60 Hz for immersive scene exploration.The system supports wired connections and long-horizon or precise tasks.
  • D. Teleoperation: The VR system captures hand poses with a Meta Quest 3 headset, displays a 360-degree scene, and supports trajectory replay.
  • D. Teleoperation: ManiSkill3 supports digital twins for domain-randomized cube picking, vision-tactile key insertion, and real2sim spoon placement.The examples pair real-world setups with corresponding simulated environments.
  • D. Teleoperation: Its task-building API supports customization without requiring users to manage complex GPU memory details or design robot controllers.Tutorials accompany the API for customizing tasks.
  • D. Teleoperation: The framework provides object-oriented access from articulations and actors down to individual links, joints, and meshes.A cabinet drawer task illustrates access to a handle mesh pose and drawer joint angle for reward design.

2) Robots and Controllers:

ManiSkill3 provides native robot-description support, GPU-parallelized controllers, scalable demonstration generation, replay utilities, and reproducible learning baselines.

  • 2) Robots and Controllers:: ManiSkill3 natively supports URDF and Mujoco MJCF robot definitions with configurable GPU-parallelized joint-position and inverse-kinematic controllers.Its IK controllers build on PyTorch Kinematics.
  • 2) Robots and Controllers:: Demonstrations are generated through motion planning, converged RL policies, or teleoperation, with difficult tasks typically starting from about 10 demonstrations.RFCL or RLPD can generate additional data through online imitation learning.
  • 2) Robots and Controllers:: Trajectory replay works across CPU and GPU simulation, supports changing observations, rendering shaders, and stored rewards, and can convert controller actions.
  • 2) Robots and Controllers:: ManiSkill3 provides reproducible vision-based sim2real setups and baselines spanning PPO, SAC, TD-MPC2, offline imitation learning, and online imitation learning.

B. Learning From Demonstrations (LfD) / Imitation Learning

ManiSkill3 combines offline and online imitation-learning baselines with scalable demonstrations and a vision-based sim2real setup that transfers a trained policy to a real robot.

  • B. Learning From Demonstrations (LfD) / Imitation Learning: Offline LfD baselines include Behavior Cloning, Diffusion Policy, Action Chunking Transformer, and PerACT, while several VLA models are evaluation-only.Training VLA models on simulation data is left to future work.
  • B. Learning From Demonstrations (LfD) / Imitation Learning: Online imitation learning combines demonstrations with newly collected environment transitions through RLPD and RFCL.RFCL leverages simulation state resets.
  • B. Learning From Demonstrations (LfD) / Imitation Learning: LfD evaluations track demonstration count, source type, and provenance because algorithm performance depends heavily on how demonstrations are collected.
  • B. Learning From Demonstrations (LfD) / Imitation Learning: A low-cost Koch-arm setup trains a vision-based manipulation policy from RGB observations and joint positions without demonstrations or privileged state information.The setup uses a phone camera and supports end-to-end sim2real training.
  • B. Learning From Demonstrations (LfD) / Imitation Learning: 91.6% real-world success was achieved by a zero-shot policy evaluated 24 times across 3 training runs on varied cube sizes, colors, and start poses.The policy used the same controller during simulation training and real deployment.
  • B. Learning From Demonstrations (LfD) / Imitation Learning: Simulation and real-world success rates showed good correlation across intermediate checkpoints, suggesting some ManiSkill3 digital twins can reflect real-world performance.

V. LIMITATIONS

ManiSkill3’s parallel rendering and simulation remain constrained by scene complexity and task-specific requirements. Its current sim2real setup also depends on static cameras, reward engineering, and non-batched soft-body simulation.

  • Complex environments with many geometries support fewer parallel environments per GPU despite GPU-parallelized rendering and simulation.Room-scale scenes are the stated example.
  • The sim2real demonstration required reward engineering to encourage safer and more robust grasp behaviors.
  • The demonstrated sim2real setup is limited to static cameras, leaving mounted-camera rendering as future work.
  • The authors characterize ManiSkill3’s sim2real results as opening an avenue for visual RL rather than solving sim2real completely.
  • Not all environments are batched for GPU parallelization: soft-body environments use substantial GPU resources to simulate a single environment quickly.Vision-tactile simulation also uses algorithms different from most rigid-body environments.

Appendix Table of Contents

The appendix documents ManiSkill3’s broad robot and task coverage, emphasizing reusable open-source examples across manipulation, locomotion, humanoids, and realistic scenes. It also describes contact-rich scene support and GPU-parallelized environments.

  • Environments and Robots VII: ManiSkill3 provides reusable open-source task examples across a diverse array of possible robotics tasks rather than maximizing the number of distinct tasks.
  • Mobile Manipulation: Mobile manipulation combines a robot arm with a mobile base for tasks such as placing objects, opening cabinets, and picking objects from the ground.
  • Room-Scale Scenes: ManiSkill3 provides out-of-the-box code for ReplicaCAD, AI2-THOR, and RoboCasa scenes, with optional ray-tracing shaders for photorealism.
  • Room-Scale Scenes: Compared with CPU-based alternatives, ManiSkill3 targets contact-rich, photorealistic room-scale manipulation with GPU-parallelized simulation and rendering.RoboCasa is described as contact-rich and photorealistic but runs around 25 FPS without GPU parallelization.
  • Locomotion: Locomotion tasks control robot joints to move quadrupeds or humanoids between locations, with collision meshes and joint limits simplified for faster or more stable learning.
  • Humanoid/Bi-manual Manipulation: Humanoid and bi-manual tasks support manipulation with embodiments such as Unitree H1, while some legged embodiments can fix their legs to focus learning on manipulation.

F. Multi-Agent Robots

ManiSkill3 spans multi-agent robotics, drawing and cleaning, dextrous and vision-tactile manipulation, classic control, digital twins, and soft-body manipulation. These categories combine task-specific interfaces with GPU-parallelized or sim2real-oriented environments where supported.

  • F. Multi-Agent Robots: Multi-agent environments control multiple robots in one simulation, commonly for object handover, with dictionary action spaces following the PettingZoo API.A wrapper can flatten actions into a single vector for users who do not need multi-agent RL.
  • G. Drawing/Cleaning: ManiSkill3 supports drawing and cleaning tasks with GPU parallelization and rendering, including dynamically adding or removing represented ink or dirt.The implementation prebuilds thousands of thin cylinders and changes their visibility or poses during interaction.
  • H. Dextrous Manipulation: Dextrous manipulation targets multifingered hands and dense contacts, and ManiSkill3 additionally provides tactile sensing through touch sensors.
  • I. Vision-Tactile Manipulation: Vision-tactile tasks use tactile images for manipulation problems such as key insertion when visual occlusions prevent solving from vision alone.These environments were ported from the ManiSkill vision-based tactile manipulation challenge.
  • J. Classic Control: Classic-control environments provide GPU-parallelized simulation and rendering variants of tasks such as cart-pole balancing, hopping, and ant locomotion.
  • Digital Twins: ManiSkill3 ports and parallelizes SIMPLER environments for efficient evaluation of real-world-trained policies, while sim2real tools support training in simulation and deployment in the real world.The GPU-parallelized SIMPLER port is designed to preserve similar results and behaviors to the original CPU environments.

VIII. VISION-BASED SIM2REAL

ManiSkill3 demonstrates end-to-end vision-based sim2real by training a PPO RGB policy in simulation and deploying it zero-shot on a real robot. The setup combines matched control, visual-proprioceptive observations, grasp feedback, and randomized simulation conditions.

  • Vision-based sim2real: PPO trains an RGB-based manipulation policy in simulation for zero-shot deployment on a real robot.The authors describe this as an end-to-end vision-based sim2real setup.
  • Hardware and control: The setup uses a low-cost $300 Koch v1.1 arm and a third-view iPhone camera controlled through the LeRobot library.The hardware is intentionally accessible, though its less precise motors make deployment more difficult than on expensive arms.
  • Hardware and control: A target delta joint-position controller runs at 30Hz in both simulation and the real world, updating target positions by the policy’s action.The same controller formulation is used across domains to reduce dynamics mismatch for quasi-static tasks.
  • Hardware and control: The normalized action is clipped to [-1,1] and unnormalized to joint-position deltas of up to 0.05 radians per joint for the Koch arm.The controller bounds are l = -0.05 and h = 0.05.
  • Scope: The controller alignment is intended for quasi-static tasks; dynamic tasks such as catching thrown objects remain future work because they require higher control frequencies.Alignment assumes both simulated and real robots reach the target joint position within one control step.
  • Observation space: Observations combine robot proprioception with 128 × 128 RGB images and a binary grasping feature that compensates for camera occlusions.The grasping label checks whether the gripper joint is within 0.02 radians of its target, indicating that an object may be blocking motion.
  • Domain randomization: Simulation randomizes camera pose, robot initialization, cube pose, geometry, friction, and colors to cover real-world variation.Camera positions vary by up to 2.5 cm, while cube positions are sampled within a 10cm2 square and cube sizes range from 1.5 to 2.25 cm.

F. Training

The training section provides reinforcement-learning, imitation-learning, and vision-language-action baselines across state, RGB, voxelized, and multi-view inputs. It also reports that augmentation and camera configuration affect convergence, success, and memory usage.

  • RL baselines: PPO and TD-MPC2 are the primary RL baselines, representing wall-time-efficient and sample-efficient reinforcement learning, with SAC also provided.The baselines are implemented with PyTorch and use task-specific hyperparameter tuning.
  • RL baselines: State-based policies use three-layer 256-unit MLP actor-critic networks, while vision policies use a shared NatureCNN backbone.The vision architecture follows the default setup of Stable Baselines 3.
  • Training configuration: Training typically uses up to 4096 parallel environments for state observations and 256 to 1024 for RGB observations.The discount factor is usually 0.8, with further settings tuned per environment.
  • Imitation learning: Offline imitation-learning evaluation compares BC, Diffusion Policy, and ACT on four tasks using state and RGB observations.Motion planning supplies the demonstrations, and Diffusion Policy performs best overall, especially when demonstrations are limited.
  • VLA and voxel baselines: ManiSkill3 supports Octo and RT-x evaluation through SIMPLER and RDT-1B fine-tuning on ManiSkill3 demonstrations.PerAct is additionally provided as a voxelized-input sense-plan-act baseline.
  • VLA and voxel baselines: For PerAct, 50 demonstration trajectories are used on PushCube-v1 and StackCube-v1, with evaluation over 100 episodes.Results are reported over 80k training steps in Table IV and Figure 27.
  • Ablations: SE(3) augmentation improves success rate but slows convergence, while multi-view observations can accelerate convergence and improve success rates.Voxelization keeps GPU memory consumption consistent as additional cameras increase point-cloud size.

XI. SIMULATION AND RENDERING BENCHMARKING

ManiSkill3 is benchmarked against Isaac Lab for GPU simulation and visual rendering, showing lower memory usage and strong speed across camera configurations. The authors caution that visual and simulator differences limit strict apples-to-apples interpretation.

  • Simulation and rendering: Visual data collection uses 2-4x less GPU memory in ManiSkill3, with camera-enabled environments requiring 1.7GB versus Isaac Lab’s 4.8GB minimum.The benchmark records FPS while executing 1000 random actions and fetching 1000 visual observations.
  • Simulation and rendering: For 640x480 single-camera and 3x320x180-camera setups, ManiSkill3 uses about 2-4x less GPU memory and runs about 2x faster.These configurations reflect camera setups from the Open-X and Droid datasets.
  • Camera-size ablation: At larger square camera resolutions, ManiSkill3 reaches up to 2x higher speed and 4x lower GPU memory usage than Isaac Lab.At small resolutions and many environments, Isaac Lab can be at most about 1.25x faster.
  • Camera-size ablation: With fewer parallel environments, ManiSkill3 consistently achieves about 2-4x higher speed and 2-3x lower GPU memory usage.The authors note that small resolutions may be impractical for some manipulation tasks and that memory efficiency matters for GPU replay buffers.
  • Benchmark caveat: The rendering comparison cannot make all visual conditions identical because ManiSkill3 and Isaac Lab use fundamentally different parallel rendering systems.The authors therefore frame the results as approximate comparisons rather than perfectly matched measurements.

XII. VR TELEOPERATION

ManiSkill3 provides VR teleoperation for simulation and real robots through high-resolution stereo feedback, pose retargeting, inverse kinematics, and spatially aligned digital twins. The system supports diverse arms, hands, grippers, and coordinated dual-arm control.

  • VR system: ManiSkill3 supports mainstream VR devices through OpenVR, receiving headset and hand poses while sending 4K stereo video to the headset.The interface uses SteamVR and ALVR to connect hardware and software components.
  • VR system: The VR feedback loop runs at 60Hz, while asynchronous pose-to-action translation runs at 20Hz.This separates smooth visual feedback from computationally heavier action translation.
  • Motion retargeting: The teleoperation pipeline includes arm control, hand control, and gripper control modules for converting human motion into robot actions.The system provides robot-specific configurations, calibration tools, and controller-based gripper input.
  • Arm control: Modified closed-loop inverse kinematics maps human wrist poses to arm joint angles while addressing human-robot frame mismatches.The system also supports soft joint masks and SE(3) filtering for constrained, smooth coordinated motion.
  • Dual-arm control: A concatenated Jacobian solves inverse kinematics for two coupled end-effectors simultaneously in dual-arm robots.This avoids the large errors of independent solutions and the slow convergence of iterative alternatives.
  • Hand control: Hand retargeting scales human keypoint errors by finger-specific factors and regularizes consecutive robot joint positions for temporal consistency.The optimization is implemented with the NLopt solver; dexterous hands and grippers use different vector mappings.
  • Digital twin alignment: Spatially aligned point clouds allow simulation and real-world environments to function as digital twins without requiring matching visual textures.Robot position, kinematics, and control interfaces must align so identical human control signals produce identical robot actions.
Loading 2410.00425v2…