Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning

Motivation and background

• GPU-native simulation makes large-scale, high-fidelity robotics training more accessible by avoiding frequent CPU-GPU transfers and reducing reliance on specialized computing resources.Isaac Gym established this paradigm by exposing physics results as PyTorch tensors and reducing training times from days to hours.Source: §1 · p. 2

Core platform

• Isaac Lab combines RTX photorealistic rendering, PhysX high-fidelity physics, USD scene authoring, and multi-GPU or multi-node scalability.These capabilities support large-scale RGB, depth, and segmentation data generation for policy training and sim-to-real transfer.Source: §1 · p. 2

Framework architecture

• Isaac Lab unifies actuator models, multi-frequency and custom sensors, low-level controllers, procedural environments, domain randomization, demonstrations, and reinforcement-learning libraries in one extensible framework.Its manager-based API organizes reusable environment components, while single-script setups remain available.Source: §1 · p. 3

Sensors and data

• The framework provides advanced sensor simulation, including tiled RTX rendering, Warp-based custom sensors, physics-based data, raycast LiDAR, height scans, and visuo-tactile sensing.These capabilities support rich observations for perception, policy training, and sim-to-real research.Source: §1 · p. 3

Learning workflows and applications

• Isaac Lab supports teleoperation and large-scale demonstration capture through spacemice, VR headsets, and other devices, alongside ready-to-use environments for reinforcement learning, imitation learning, and sim-to-real research.The technical report covers applications spanning locomotion, navigation, contact-rich manipulation, and other robotic tasks.Source: §1 · p. 3

2.1. USD for Robotics

• OpenUSD represents robotics scenes as hierarchical, schema-based graphs that unify geometry, physics, semantics, sensing, and appearance.Layering, references, and instancing support scalable, collaborative scene construction, while Isaac Lab adds robotics conventions, converters, and high-level wrappers.Source: §2.1 · pp. 4–5

2.2. Physics Simulation

• OmniPhysics parses USD stages into PhysX objects and exposes batched GPU simulation state through Tensor API views rather than manually indexed raw buffers.Views can select collections of objects through USD prim-path patterns and support read/write access through NumPy, PyTorch, or NVIDIA Warp.Source: Figure 3 · p. 5, §2.2 · p. 6
• PhysX supports rigid, articulated, deformable, fluid, and soft-body simulation with coupled solvers and robotics-specific features such as actuator, friction, load-cell, and SDF collision models.These capabilities support mixed-physics interactions and high-precision handling of non-convex geometries.Source: §2.2 · p. 5
• GPU PhysX execution enables direct state and control access in GPU memory, supporting downstream GPU kernels and avoiding traditional CPU–GPU transfer bottlenecks.Simulation parameters such as friction, masses, and joint properties still require PhysX CPU APIs.Source: §2.2 · pp. 5–6

2.3. Rendering

• RTX rendering provides physically based ray-traced RGB and ground-truth sensors, while MDL materials support realistic reflections, refractions, textures, and semantic labels.DLSS affects RGB outputs only; depth and segmentation remain at native resolution.Source: §2.3 · p. 7, Figure 4 · p. 7
• Tiled rendering batches thousands of cameras into one GPU framebuffer, preserving camera poses and enabling deterministic per-environment reconstruction without costly host–device transfers.The pipeline supports vision-in-the-loop learning and scales sensor throughput with GPU resources; active LiDAR and radar integration with tiled rendering is forthcoming.Source: §2.3 · pp. 7–8, Figure 5 · p. 8
• RTX also integrates 3D Gaussian primitives with ray-traced geometry, enabling photorealistic reconstructed environments that can contain robots and synthetic objects.The approach is intended to improve visual realism and facilitate policy transfer without hand-crafted assets.Source: §2.3 · p. 8, Figure 6 · p. 8

3. Isaac Lab Design and Features

• Isaac Lab lowers the learning barrier of general-purpose simulation through modular robot-learning interfaces that combine realistic actuators, asynchronous sensing, noise, uncertainty, and task abstractions.The framework supports reinforcement and imitation learning across robotic platforms and integrates peripherals for data collection.Source: §3 · pp. 8–9, Figure 7 · p. 9

3.1. Assets

• Asset interfaces wrap USD and OmniPhysics views to provide unified spawning, structured physical-state access, configuration properties, and derived robotics quantities.They support rigid, articulated, and deformable objects while hiding lower-level API differences.Source: §3.1 · pp. 9–10
• Articulation interfaces apply user-specified actuator models at each simulation step and allow different models for distinct joint groups, supporting heterogeneous actuation.The same actuator abstraction extends to multirotor thrust and torque, from direct RPM commands to higher-level motion setpoints.Source: §3.1 · p. 10

3.2. Actuators

• Isaac Lab actuators model friction, motor inertia, velocity limits, and effort limits, and separate into implicit and explicit classes.Implicit actuators use PhysX joint PD controllers and are typically more stable and more accurate at low sampling rates than explicit actuators.Source: §3.2 · pp. 10–11, §3.2.1 · p. 11

3.2.2. Explicit Actuators

• Explicit actuators better approximate discrete hardware control at high physics sampling rates but introduce numerical challenges from discrete-time control.Available models include ideal PD, velocity-dependent DC motor limits, communication delays, position-dependent effort limits, and neural-network actuator dynamics.Source: §3.2.2 · p. 11

3.3. Sensors

• Isaac Lab unifies physics-based sensors, cameras, Warp raycasters, and visuo-tactile sensing under a common interface for robot-learning environments.The sensor suite includes IMUs, frame transformers, contact sensors, parallelized cameras, geometric raycasting, and tactile images or force fields.Source: §3.3 · p. 11, Figure 9 · p. 12, §3.3.4 · p. 14
• Different sensor update frequencies model asynchronous sensing and communication delays instead of assuming synchronized sensing and control.This better approximates real-world sensor operation in parallelized environments.Source: §3.3 · p. 12

3.3.1. Physics-based Sensors

• Physics-based sensors provide frame transformations, inertial state, and contact forces, with options for offsets, noise, drift, filtered contacts, and contact histories.Frame Transformers batch relative-pose computations, IMUs expose pose, velocity, and acceleration, and Contact Sensors report forces and temporal contact information.Source: §3.3.1 · pp. 12–13

3.3.2. Rendering-based Camera

• Rendering-based cameras support pinhole and fisheye models with RGB, depth, semantic, instance, normal, and motion-vector outputs.USD-Cameras prioritize photorealistic rendering, whereas Tiled-Cameras aggregate environments for faster image generation with some quality trade-off.Source: §3.3.2 · p. 13

3.3.3. Warp-based Ray-Caster

• Warp-based RayCasters provide GPU-parallel geometric sensing with configurable patterns for height scanners and LiDAR, while raycast cameras prioritize efficiency over photorealism.Raycasts can target dynamic meshes and compute all rays at one time instance, avoiding rotating-sensor temporal distortion.Source: §3.3.3 · p. 13

3.3.4. Visuo-Tactile Sensor

• Visuo-tactile sensing approximates compliant contact and converts interactions into RGB images and distributed normal and shear force fields.GPU-parallel phases combine stiffness-and-damping contact modeling, tiled depth rendering, and penalty-based force computation for improved simulation performance.Source: §3.3.4 · p. 14

3.4. Controllers

• Isaac Lab organizes controllers into inverse kinematics, force control, and motion planning tools that generate joint-level commands from higher-level motion inputs.Controllers integrate with robot-learning task actions formulated as MDPs.Source: §3.4 · p. 14

3.4.1. Inverse Kinematics

• Inverse kinematics converts task-space motion into joint-space commands using differential Jacobian methods or Pink’s quadratic-programming formulation.Differential IK supports several singularity treatments, while Pink supports weighted simultaneous tasks and extensible objectives such as collision avoidance and joint limits.Source: §3.4.1 · pp. 14–15

3.4.2. Force Control

• Force control regulates environmental interaction through joint impedance and operational-space controllers.The implementations support feedforward compensation, variable impedance, hybrid force-motion control, gravity compensation, and null-space control.Source: §3.4.2 · p. 15

3.4.3. Motion Planning

• cuRobo provides GPU-parallelized, collision-aware motion planning by combining inverse kinematics, collision checking, trajectory optimization, and optional graph-based global planning.Isaac Lab initializes the planner from robot and world descriptions and manages tensor consistency on a dedicated CUDA device.Source: §3.4.3 · p. 15

3.5. Teleoperation Support

• Teleoperation in Isaac Lab supports direct robot control, controller and policy evaluation, and demonstrations for robot learning.The framework accommodates classical hardware and extended-reality applications.Source: §3.5 · p. 15

3.5.1. Classical Teleoperation Devices

• Keyboard and spacemouse interfaces provide delta-pose, multi-axis, and customizable control for arms, grippers, and selected humanoid actions.Spacemouse teleoperation can support higher-quality demonstrations for imitation learning.Source: §3.5.1 · p. 15

3.5.2. Extended Reality (XR) Device Teleoperation

• Apple Vision Pro teleoperation maps detected hand joints to end-effector poses and uses inverse kinematics for dexterous bimanual and humanoid control.The IK formulation adds waist and null-space tasks to infer whole-body motion and maintain nominal arm positions.Source: §3.5.2 · pp. 15–16
• CloudXR provides low-latency XR streaming with re-projection and augmented-reality overlays for more stable imagery and environmental awareness.The approach is contrasted with systems that directly stream stereoscopic robot-perspective video.Source: §3.5.2 · p. 16

3.6. Simulation Scene Generation

• Isaac Lab supports scripted procedural generation, scanned-mesh import, interactive editing, and hybrid workflows for simulation environments.The script interface enables terrain randomization, complex mesh construction, heightfields, and PhysX-efficient geometry.Source: §3.6.1 · pp. 16–17

3.6.1. Terrain Generation

• Terrain generation uses Trimesh and heightfields to create randomized primitive or composite structures while minimizing mesh complexity.Figure 11 presents examples of generated terrain assets.Source: §3.6.1 · pp. 16–17, Figure 11 · p. 16

3.6.2. Integration with External Asset Generation Frameworks

• Scene Synthesizer, Infinigen, and artist-generated assets can be combined into USD environments for cluttered loco-manipulation and manipulation training.Reported examples include sim-to-real door opening and generalizable humanoid pick-and-place skills trained in procedural kitchens.Source: §3.6.2 · p. 17

3.6.3. Multi-Asset Spawning and Randomization

• Assets can be cloned, swapped across parallel environments, and randomized geometrically and visually to expose policies to varied task-relevant scenarios.Object variation and randomized textures, materials, and colors are used to promote robustness and sim-to-real transfer.Source: §3.6.3 · p. 17

3.7.1. Manager-Based Workflow

• The manager-based workflow decomposes environments into reusable MDP managers, while handling simulation plumbing, vectorization, resets, and Gym-compatible spaces.Its explicit interfaces support term-level logging, configuration changes, and ablations, with modest CPU orchestration and kernel-launch overhead.Source: §3.7.1 · pp. 17–18

3.7.2. Direct Workflow

• The direct workflow exposes simulation, physics, learning components, and low-level data structures with minimal abstraction overhead for performance-sensitive training.It is suited to benchmarking and GPU-coupled pipelines while remaining extensible through custom tasks, observations, rewards, learning frameworks, and logging tools.Source: §3.7.2 · pp. 18–19

3.7.3. Environments

• Isaac Lab provides environments across classic, locomotion, manipulation, and navigation families, with unified APIs and data layouts supporting comparability and reuse.The suite includes contact-rich and dexterous tasks, hierarchical navigation, and diverse robot morphologies.Source: §3.7.3 · p. 19, Figure 12 · p. 19
• Locomotion environments span 11 robot morphologies, including humanoids, quadrupeds, and legged platforms, with robust sim-to-real results reported for Anymal and Spot.Isaac Lab also supports custom multi-agent environments for competition, cooperation, or mixed objectives through a PettingZoo Parallel API.Source: §3.7.3 · pp. 19–20

4.1. Environment Throughput

• Environment throughput is measured as FPS over environment steps divided by simulation time plus learning time, whereas sensor benchmarks isolate sensor update costs.The benchmarks cover three GPU platforms and distributed RTX Pro 6000 training.Source: §4.1 · p. 20, §4.2 · p. 22

4.1.1. State-based Environments

• With eight GPUs and 16,384 environments, DextrAH exceeds 900,000 FPS and Franka cabinet opening exceeds 1.6 million FPS, with distributed training scaling almost linearly.Newer Blackwell GPUs outperform the L40 across both state-based manipulation tasks.Source: §4.1.1 · p. 20, Figure 13 · p. 20

4.1.2. Perceptive Environments

• Perceptive-task throughput improves with newer GPUs and distributed training but remains lower than state-based throughput because rendering adds computational cost.The GeForce 5090 leads at smaller environment counts, whereas the RTX Pro 6000 narrows the gap and supports more parallel environments through larger VRAM.Source: §4.1.2 · pp. 21–22, Figure 14 · p. 21, Figure 15 · p. 21

4.1.3. Workflow Comparison

• The direct workflow averages 3.53% higher throughput on a single RTX Pro 6000, while the manager-based workflow becomes nearly equivalent at larger environment counts or when perception dominates.This supports using the manager-based workflow when structure and flexibility matter without a substantial efficiency penalty.Source: §4.1.3 · p. 22, Figure 16 · p. 22

4.2.1. Camera

• The USD-Camera provides the highest visual fidelity but runs out of memory beyond 48 parallel cameras, while Tiled-Camera and RayCasterCamera scale to several thousand environments.RayCasterCamera is more efficient at lower resolutions and has slightly lower memory use; Tiled-Camera performs better at higher resolutions and parallel multi-GPU workloads.Source: §4.2.1 · p. 23, Figure 17 · p. 23

4.2.2. Warp Ray-Caster Sensor

• Warp RayCaster throughput degrades sub-linearly as tracked assets increase, while mesh complexity has marginal impact across 20k–200k faces.GPU capability, mesh count, ray density, and parallel environment count dominate performance, making ray-density tuning more useful than mesh simplification for balancing accuracy and efficiency.Source: §4.2.2 · pp. 23–24, Figure 18 · p. 24

5.1. Reinforcement Learning

• Isaac Lab supports teacher-student distillation and end-to-end perception-in-the-loop training for policies using RGB, depth, or other sensory observations.Pre-trained visual encoders can support RGB students, while end-to-end training has produced sim-to-real systems and emergent active perception behaviors.Source: §5.1 · p. 24, §5.1.1 · p. 25, Figure 19 · p. 25

5.2. Population-Based Training

• Population-Based Training assigns workers independent hyperparameters and periodically replaces weaker workers with stronger weights and settings, while mutations preserve exploration.Isaac Lab’s DexPBT reproduces a 6D reposing task with eight workers and converges in approximately 16 hours on NVIDIA OVX L40 hardware.Source: §5.2 · p. 26, Figure 20 · p. 26

5.3. Domain Randomization

• Isaac Lab randomizes physics and rendering parameters, including friction, mass, textures, lighting, and backgrounds, to support sim-to-real robustness and generalization.Many parameters can change at runtime, although current PhysX and USD APIs require CPU access for simulation parameters and rendering parameters respectively; ADR adaptively changes environment difficulty.Source: §5.3 · pp. 26–27, Figure 21 · p. 27

5.4. Imitation Learning

• Isaac Lab integrates RoboMimic for demonstrations collected through teleoperation or synthetic generation, with standardized HDF5 storage and conversion to LeRobot formats.Reference scripts and conversion utilities streamline training-data integration with existing imitation-learning workflows.Source: §5.4 · pp. 27–28

5.5. Synthetic Data Generation

• Isaac Lab Mimic transforms object-centric segments from limited human demonstrations into diverse synthetic trajectories, enabling effectively unbounded generation from as little as one demonstration.Parallelized environments increase generation throughput and support different robot embodiments and controllers.Source: §5.5 · p. 28, Figure 22 · p. 28

5.5.1. Loco-Manipulation Data Generation

• Loco-manipulation data generation combines navigation, whole-body control, and segmented manipulation phases to synthesize coordinated walking and object-interaction demonstrations.Randomized pickup, drop-off, and obstacle locations expand data for humanoids and mobile manipulators.Source: §5.5.1 · p. 29

5.5.2. SkillGen-based Dataset Augmentation

• SkillGen combines human subtask segments with GPU-accelerated motion planning to produce collision-aware demonstrations that adapt to new object placements and scene layouts.Users can configure trials, parallel environments, compute devices, and planner parameters to balance speed, success rates, and motion quality.Source: §5.5.2 · p. 29

6.1. Locomotion

• Isaac Lab enables robust locomotion, including Spot sim-to-real transfer at running speeds up to 5.2 m/s and mobility across varied robotic platforms.Its applications also include depth-based visuomotor control and wheeled platforms with dynamic behaviors.Source: §6.1 · pp. 29–30

6.2. Whole-body control

• Whole-body-control research uses Isaac Lab for coordinated loco-manipulation, human-motion-driven humanoid skills, and interaction with articulated objects.Examples include dishwasher interaction, dynamic obstacle avoidance, and athletic humanoid movements.Source: §6.2 · pp. 31–32, Figure 24 · p. 31

6.3. Navigation

• Isaac Lab supports navigation through realistic sensors, multi-platform workflows, memory-based policies, forward-dynamics models, and cross-embodiment sim-to-real transfer.COMPASS combines imitation learning, residual reinforcement learning, and policy distillation for wheeled, quadruped, and humanoid platforms.Source: §6.3 · pp. 32–33, Figure 25 · p. 32, Figure 26 · p. 33

6.4. Industrial Manipulation

• Contact-rich manipulation achieves strong transfer through specialized simulation, with Factory environments reporting 83–99% zero-shot sim-to-real success and AutoMate around 80% success in simulation and reality.Isaac Lab also provides grasp-generation and evaluation pipelines for multiple gripper types and degrees of freedom.Source: §6.4 · pp. 33–34, Figure 28 · p. 34

6.5. Dexterous Manipulation

• Dexterous-control systems use Isaac Lab to train policies from RGB, egocentric vision, and spatial-memory representations for real-world manipulation.PDC reports better generalization and more natural behaviors when trained directly from pixels rather than distilled from state-based policies.Source: §6.5 · pp. 34–35, Figure 29 · p. 34, Figure 30 · p. 35

6.6. Healthcare

• Isaac Lab underpins healthcare simulation workflows involving surgical manipulation, ultrasound, digital twins, teleoperation, and zero-shot transfer to physical surgical robots.Its capabilities include photorealistic physics-based surgical simulation and GPU-accelerated ultrasound ray tracing.Source: §6.6 · pp. 35–36, Figure 32 · p. 36

6.7. Generalist Foundation Models

• Isaac Lab supplies synthetic data and safe simulation-based post-training for generalist robot foundation models such as GR00T N1.5.Online reinforcement-learning post-training can scale across new tasks using residual RL, sample-efficient methods, and learned reward models.Source: §6.7 · p. 37

7. Future Work and Discussion

• Newton is planned as a differentiable, GPU-accelerated engine for rigid, articulated, and deformable dynamics, with deeper rendering, scalability, and benchmarking support also planned.The integration is intended to support gradient-based learning and more photorealistic vision training.Source: §7 · p. 37, Figure 33 · p. 37, Figure 34 · p. 38

7.1.2. Architecture and Design Principles

• Newton separates models, states, and solvers, represents simulation data as flat tensors, exposes state explicitly, and integrates with PyTorch and JAX.Its modular design supports selective component use, batched access, and multiple imported asset formats.Source: §7.1.1 · pp. 38–39, §7.1.2 · p. 39

7.1.3. Solver Implementations

• Newton combines canonical and specialized solvers, including MuJoCo Warp for complex contact scenes, Kamino for closed loops, and extensions for cloth, tactile, and mixed-material simulation.Multiple solvers can manage rigid bodies, cloth, and particles, while two-way coupling remains under development.Source: §7.1.3 · pp. 39–40

7.1.4. Newton USD as a Staging Schema for USD Physics Standardization

• Newton’s USD staging schema aims to formalize engine-agnostic physical properties for future inclusion in the USD Physics standard.This aligns the schema effort with Newton’s solver-abstraction API.Source: §7.1.4 · p. 40

7.1.5. Newton Integration with Isaac Lab

• Isaac Lab’s Newton integration is experimental, currently covers a subset of environments, and is being evaluated through sim-to-sim and sim-to-real tests.The project aims to reach feature parity with PhysX, including additional sensor, actuator, and learning-environment support.Source: §7.1.5 · p. 40

7.2.1. Isaac Lab - Arena

• Isaac Lab Arena reduces the manual overhead of asset preparation, environment setup, and large-scale policy evaluation through composable tasks, metrics, data collection, and parallel GPU execution.It connects evaluation with data generation, training, and deployment workflows.Source: §7.2.1 · pp. 40–41, Figure 35 · p. 40

7.2.2. Assembly Benchmark

• Existing Isaac Lab assembly environments support real-robot transfer, while future work aims to extend coverage to the full NIST benchmark task set.The current environments correspond to a subset of the NIST Assembly Task Board 1.Source: §7.2.2 · p. 41

7.2.3. Dexterous Manipulation Suite

• The dexterous manipulation suite currently offers lifting, grasping, and reorientation with a KUKA Allegro hand, with planned expansion to humanoids and unstructured domestic tasks.The stated goal is broader coverage of industrial, logistics, and domestic-service scenarios.Source: §7.2.3 · p. 41

• Isaac Lab unifies high-fidelity physics, scalable rendering, and modular architecture with sensor simulation, actuation modeling, environments, motion generation, and imitation-learning tools.The framework supports direct and manager-based learning workflows, foundation-model training, reinforcement-learning fine-tuning, and large-scale perception systems.Source: §8 · p. 41
• Newton integration is planned to provide greater physical realism, performance, and flexibility for challenging tasks.Source: §8 · p. 41

A. Contributors and Acknowledgments

• The acknowledgments identify additional contributors from NVIDIA, NTNU, ETH, RAI, and UC Berkeley.Source: A. Contributors and Acknowledgments · p. 42, Core Contributors · p. 42

Core Contributors

• Core contributors are associated with XR Teleoperation, Arena, Physics, and Mimic development.The listed contributors include NVIDIA, NTNU, ETH, Flexion Robotics, RAI, and USC affiliations.Source: Core Contributors · p. 42

• Simulation and physics terms include USD, URDF, SDF, FEM, MPM, PBD, and VBD.Source: B. Acronyms · p. 44
• Learning and robotics terms include RL, IL, MARL, IK, DR, PBT, and PCA.Source: B. Acronyms · p. 44
• Computing, sensing, and interface terms include GPU, CPU, LiDAR, XR, API, DLSS, and DoF.Source: B. Acronyms · p. 44