Source-linked AI summary
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, Animesh Garg
TL;DR
Robot-learning simulators often lack a unified combination of fast, accurate physics, realistic sensing, diverse assets, and reusable task interfaces. ORBIT addresses this gap with a modular Isaac Sim framework supporting varied worlds, agents, tasks, and learning workflows. It reports higher rigid- and deformable-body throughput than existing frameworks while providing a broad, extensible platform for robotics research.
Problem
Existing simulation platforms often trade off fast physics, high-fidelity sensing, diverse asset handling, and easy integration of new robotic tasks.
Method
ORBIT uses a unified, modular framework built on Isaac Sim, separating Worlds, Agents, and task specifications while supporting varied robots, sensors, actuators, motion generators, and authoring interfaces.
Results
ORBIT achieves approximately 10x higher rigid-body and 3x higher deformable-body throughput than CPU-based alternatives, with 50,000–75,000 FPS reported for GPU-optimized RL frameworks using 2048 environments.
Takeaways & Limitations
ORBIT provides a common, open-source platform for reinforcement learning, learning from demonstrations, motion planning, benchmarking, and sim-to-real research.
Abstract
from arXiv · showhide
We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes and high-fidelity rigid and deformable body simulation. With Orbit, we provide a suite of benchmark tasks of varying difficulty -- from single-stage cabinet opening and cloth folding to multi-stage tasks such as room reorganization. To support working with diverse observations and action spaces, we include fixed-arm and mobile manipulators with different physically-based sensors and motion generators. Orbit allows training reinforcement learning policies and collecting large demonstration datasets from hand-crafted or expert solutions in a matter of minutes by leveraging GPU-based parallelization. In summary, we offer an open-sourced framework that readily comes with 16 robotic platforms, 4 sensor modalities, 10 motion generators, more than 20 benchmark tasks, and wrappers to 4 learning libraries. With this framework, we aim to support various research areas, including representation learning, reinforcement learning, imitation learning, and task and motion planning. We hope it helps establish interdisciplinary collaborations in these communities, and its modularity makes it easily extensible for more tasks and applications in the future.
I. INTRODUCTION
ORBIT addresses the trade-off between fast physics, high-fidelity sensing, diverse assets, and accessible task development in existing simulators. It unifies these capabilities in an open-source framework with standardized tasks and GPU-based performance advantages.
- Motivation: Existing simulators often trade off rendering throughput, low-level interaction fidelity, physics accuracy, deformable-body simulation, and integration support.Vision-focused platforms simplify interactions, while robotics physics simulators commonly lack physically based rendering, deformable objects, or ROS support.
- Framework: ORBIT provides a unified, modular framework for photo-realistic scenes, high-fidelity rigid and deformable-body physics, and flexible task development.The framework is built on Isaac Sim and supports programmatic and interactive task design.
- Benchmarks: ORBIT includes eleven rigid-manipulation, thirteen deformable-manipulation, and two locomotion environments, with robots, objects, and sensors switchable within tasks.These standardized environments serve both benchmark purposes and examples for designing new tasks.
- Scope: ORBIT integrates rigid and deformable simulation, physically based rendering, multiple scene-authoring methods, and tooling for robotic learning and planning.Its stated application scope includes reinforcement learning, learning from demonstrations, and motion planning.
III. ORBIT: ABSTRACTIONS AND INTERFACES DESIGN
ORBIT organizes simulation around a World and an Agent, separating shared scene state from modular computation and task logic. Its interfaces support heterogeneous robots, sensors, actuators, frequencies, and action spaces while enabling reusable task specifications.
- Abstractions: ORBIT separates the simulated World from the Agent, whose computation graph receives observations and produces actions for the robot.Agent nodes can perform perception-based or action-based processing, and graph cuts define learning over selected computation nodes.
- Timing and control: ORBIT models asynchronous real-world operation by supporting differing sensor and action frequencies, actuator characteristics, delays, and noise models.For cube lifting, an inverse-kinematics controller may run at 50 Hz while a joint controller runs at 1000 Hz; motion generators can be switched within the task design.
- World: Worlds combine robots, sensors, objects, and visualization markers and can be authored procedurally, from scanned meshes, through Isaac Sim’s GUI, or by combining these methods.The combination supports architectural layouts from reconstructed meshes and interactive verification of scene physics.
- Robots and sensors: Common interfaces support robot categories, actuator models, physics-based and rendering-based sensors, and runtime configuration of task-relevant sensing.Actuator models include direct-control motors and series elastic actuators, while sensors use internal timers for differing operating frequencies.
- Assets: USD asset tooling converts URDF, OBJ, and STL files and automatically adds physics properties such as colliders and friction materials.USD stores assets, materials, physics, semantic, and rendering attributes in a hierarchical format used by Omniverse.
- Task and agent: A task module specifies goals, rewards, metrics, and resets, allowing multiple tasks to reuse the same World definition.This modularization supports task-specific learning without duplicating the underlying scene definition.
IV. ORBIT: FEATURES
ORBIT provides a configurable collection of robots, devices, motion generators, and manipulation tasks for developing diverse robotic learning environments. Its task suite spans rigid and deformable-object manipulation, legged control, and in-hand manipulation.
- Robots: ORBIT supports four mobile platforms, seven robotic arms, and six end-effectors, with tools for composing complex robotic systems.The supported components can be combined into platforms such as legged mobile manipulators and switched in the World.
- Environment Configuration: The framework supports switching robots, objects, observations, and action spaces through configuration files.The designed environments use hand-crafted state machines and task-space controllers with high-fidelity rigid and deformable-body simulation.
- I/O Devices: Keyboard, Xbox gamepad, and Spacemouse interfaces support real-time teleoperation, demonstration collection, and task-design debugging.I/O commands are parsed into control commands for subsequent nodes.
- Motion Generators: Motion generators transform high-level reference actions into lower-level commands through controllers such as inverse kinematics.ORBIT includes GPU implementations of differential IK, operational-space control, and joint-level control, plus CPU planners including RMPFlow and OCS2.
- Rigid- and Deformable-body Tasks: ORBIT provides tasks for rigid and deformable manipulation, legged control, and in-hand manipulation across skills including grasping, screwing, stacking, folding, and walking.The interfaces simplify switching among robots, objects, motion generators, observations, and domain randomization, and extend fixed-arm tasks to mobile manipulators.
V. EXEMPLAR WORKFLOWS WITH ORBIT
ORBIT combines pre-built environments with easy-to-use interfaces for extending and customizing simulation workflows. Its physics, sensor simulation, and rendering support perception and decision-making applications.
- Infrastructure: ORBIT combines pre-built environments with easy-to-use interfaces that support extensibility and customization.The framework is presented as useful for multiple robotics challenges in perception and decision-making.
A. Reinforcement Learning
ORBIT integrates with multiple reinforcement-learning frameworks while allowing action-space changes for different controllers. GPU-optimized frameworks achieve substantially higher training throughput than Stable-Baselines3 in the reported examples.
- Framework Integration: ORBIT provides wrappers for rl-games, RSL-rl, and stable-baselines-3, making environments usable across reinforcement-learning libraries.The wrappers format tensor-based data for different learning frameworks and provide access to more algorithms.
- Training Throughput: 50,000-75,000 frames per second (FPS) are achieved with RSL-rl and rl-games using 2048 environments, compared with 6,000-18,000 FPS using stable-baselines3.The comparison uses PPO on Franka-Reach and Franka-Cabinet-Opening with different action spaces; implementation differences also affect performance and training time.
B. Teleoperation and Imitation Learning
ORBIT supports demonstration collection through teleoperation and hand-crafted policies, then formats the data for imitation-learning workflows. Its motion-planning tools combine grasp generation, task-space control, motion previewing, and user teleoperation.
- Teleoperation and Demonstration Collection: ORBIT provides an I/O-based data-collection interface and stores demonstrations in robomimic-compatible formats for imitation learning.The interface is intended for interacting with environments and collecting data similar to RoboTurk.
- Imitation Learning: For Franka-LiftCube, demonstrations are collected under four combinations of fixed or random initial and desired object positions.The example collects 2000 trajectories for each setting and trains BC and BC-RNN policies.
- Imitation-Learning Evaluation: Table II reports success rates and trajectory lengths over 100 trials while varying initial states, goal states, or both.The evaluated conditions are No Change, I, G, and Both.
- Hand-crafted Policies: Hand-crafted state machines provide sequential goals for reaching, grasping, object interaction, and maneuvering to subsequent target positions.These policies can also collect expert demonstrations for challenging tasks such as cloth manipulation.
- Interactive Motion Planning: Interactive motion planning links grasp generation, teleoperation, task-space control, and motion previewing through a graphical interface.Users select grasp poses, preview generated robot motions, confirm execution, and then teleoperate the robot.
D. Deployment on real robot
ORBIT supports deploying simulated policies and commands on real robots through ZMQ and ROS connections, with demonstrations spanning manipulation and locomotion.
- Real-robot interfaces: ORBIT connects simulation to real robots through ZMQ and ROS transport layers for policy deployment and command execution.ZMQ is used for the Franka platform, while ROS supports deployment on ANYmal-D.
- Manipulation demonstrations: ORBIT demonstrates interactive grasp selection, motion preview, and execution for tabletop manipulation through a GUI-triggered image-based grasp generator.
- Manipulation demonstrations: A digital-twin demonstration applies the same commands to simulated and real Franka Panda systems, including simultaneous lifting with an Allegro hand.
- Franka deployment: A quintic interpolator upsamples 60 Hz simulator joint commands to 1000 Hz for real-time Franka execution.
- Franka deployment: Franka experiments cover teleoperation, state-machine deployment, and waypoint tracking with obstacle avoidance across two hand configurations.The configurations use the Franka Emika hand and Allegro hand, while sharing the same real-robot interface.
- ANYmal-D deployment: A locomotion policy trained entirely in simulation transfers to ANYmal-D using randomized base mass, simulated pushes, actuator modeling, and contact-force rewards.The demonstrated transfer indicates the viability of ORBIT’s simulated contact dynamics for contact-rich tasks.
E. Simulation Evaluation and Comparison
ORBIT’s evaluation examines deformable-body accuracy and simulation throughput against other frameworks. GPU parallelization scales better than CPU vectorization, while finer cloth meshes trade throughput for accuracy.
- Simulation accuracy: The deformable-body accuracy evaluation compares a simulated clamped silicone beam with motion-marker measurements from a physical beam.The simulation matches material properties and observes whether damped oscillations follow the real data.
- Throughput comparison: Throughput comparisons equalize action space, simulation frequency, and control decimation across ORBIT and competing rigid- and deformable-body frameworks.The evaluation uses a workstation with a 16-core Ryzen 5950X, 64 GB RAM, and RTX 3090.
- Throughput comparison: 3× higher throughput than DEDO is obtained for the cloth-hanging task using ORBIT’s PBD solver.Increasing cloth nodes or points adversely affects performance, and the dotted line marks memory exhaustion.
- Throughput comparison: ∼10x faster rigid-body and ∼3x faster deformable-body throughput is achieved with GPU-based parallelization compared with CPU-vectorized frameworks.CPU-based programs crash around 200–300 environments because of insufficient memory.
- Throughput comparison: Higher cloth-mesh resolution improves simulation accuracy but requires more computation time.
VI. DISCUSSION
ORBIT is presented as an interactive, modular, open-source framework intended to simplify robot-learning environment design and broaden supported research workflows. The discussion reports throughput gains and sim-to-real potential while identifying rendering and whole-simulator fidelity as remaining boundaries.
- Discussion: ORBIT simplifies environment design and task specification through an interactive, modular framework with diverse platforms, sensors, motion generators, and benchmark tasks.
- Discussion: Experiments show significant throughput improvement for ORBIT tasks relative to tasks in other frameworks and indicate potential for sim-to-real transfer.
- Discussion: Open-sourcing ORBIT aims to reduce application-development overhead and provide a unified platform for robot-learning research.
- Discussion: Physics simulation reaches up to 125,000 FPS, while ten-camera rendering of 640 × 480 images totals 270 FPS on an RTX 3090.The paper identifies camera rendering as a current bottleneck and reports ongoing GPU-based acceleration work.
- Discussion: Quantitative fidelity evaluation of the entire simulator, including rendering, sensors, and physics, remains future work.
- Future work: Planned enhancements include tactile and 6-axis force-torque sensors and loading assets directly in native formats such as URDF and OBJ.
APPENDIX A: AUTHOR CONTRIBUTIONS
The appendix attributes ORBIT’s development across infrastructure, robot integration, motion generators, environments, evaluations, sim-to-real experiments, Isaac Sim support, management, and writing.
- Infrastructure: M. Mittal, D. Hoeller, and N. Rudin designed and built ORBIT’s core infrastructure.
- Simulator integration: M. Mittal and J. Liu integrated robots into the simulator and implemented motion generators.
- Environments: Locomotion environments were added by M. Mittal, D. Hoeller, and N. Rudin.
- Environments: Rigid and deformable object-manipulation environments were added by M. Mittal, J. Liu, A. Yuan, C. Yu, and Q. Yu.
- Evaluations: Workflow evaluations and throughput comparisons were conducted by J. Liu, C. Yu, Q. Yu, and M. Mittal.
- Experiments and support: Sim-to-real experiments covered the Franka arm and ANYmal robot, while Y. Guo, H. Mazhar, B. Babich, and G. State provided Isaac Sim support.
- Project leadership: A. Mandlekar, M. Hutter, and A. Garg managed or advised the project, and M. Mittal and A. Garg wrote the paper.
APPENDIX B: MOTIVATION BEHIND ORBIT
NVIDIA’s simulation tools accelerate robotics research, but earlier offerings lacked several capabilities needed for general-purpose robot learning. Isaac Sim addresses these gaps, while its associated environments remain limited in learning-library integration.
- Isaac Gym: Isaac Gym provides high-performance GPU-based rigid-body physics simulation with direct Python access to simulation data.Its end-to-end GPU pipeline enables high frame rates compared with CPU-based physics engines.
- Isaac Gym: Isaac Gym is not a general-purpose robotics simulator because it lacks rigid–deformable interaction, high-fidelity rendering, and ROS support.It was primarily released as a preview showcasing the underlying physics engine.
- Isaac Sim: Isaac Sim extends robotics simulation through Omniverse, combining high-fidelity graphics and physics with ROS/ROS2, sensor simulation, domain randomization, and synthetic-data tools.It is built on Omniverse, a general-purpose platform for complex 3D workflows.
- Learning environments: IsaacGymEnvs and OmniIsaacGymEnvs provide starting points for robot learning but have limited integration with reinforcement-learning libraries.These repositories are designed to demonstrate simulator capabilities and help users understand available functionality.