Source-linked AI summary
CRESSim-Neo: A Batched GPU Simulation Engine for Surgical Robotics and Robot Learning
Yafei Ou, Ahnaf Naheen, Tleukhan Mussin, Hans Jarales, Melwin Moncy, Mahdi Tavakoli
TL;DR
Surgical robotics needs simulation that jointly handles diverse physical interactions and scales to robot learning and synthetic data generation. CRESSim-Neo combines PBD/XPBD physics, surgery-specific sensing and interactions, batched rendering, and GPU-resident dataflow, achieving high throughput across rigid-body, deformable-body, fluid, visual, and surgical tasks. The engine provides a unified platform for scalable surgical simulation, learning, and data generation, while retaining limitations in rigid-contact fidelity and platform coverage.
Problem
Surgical simulation must model rigid instruments, deformable tissues, fluids, and specialized sensing, but existing platforms lack an off-the-shelf combination of surgical functionality and scalable robotics support.
Method
CRESSim-Neo combines PBD/XPBD simulation with surgery-specific interactions and sensing, batched rendering, and GPU-resident pipelines with direct buffer access.
Results
Benchmarks and surgical task demonstrations show high-throughput simulation across rigid-body, deformable-body, fluid, visual, and surgical robot-learning scenarios.
Takeaways & Limitations
CRESSim-Neo provides a unified and scalable platform for surgical simulation, synthetic data generation, and surgical robot learning.
Takeaways & Limitations
PBD provides less physically faithful rigid contacts and joints than rigid-body engines, while cross-platform validation and some graphics capabilities remain limited.
Abstract
from arXiv · showhide
We introduce CRESSim-Neo, a batched GPU simulation engine for surgical robotics and robot learning. CRESSim-Neo combines position-based simulation of rigid bodies, deformable tissues, fluids, and strands with batched rendering, surgery-specific sensing, and a GPU-resident data pipeline. The engine supports applications including tissue manipulation, fluid suction, suturing, cable-driven robots, and ultrasound image synthesis. Direct access to physics and rendering buffers enables GPU-resident robot learning and zero-copy PyTorch integration using DLPack. We demonstrate CRESSim-Neo across rigid-body, deformable-body, and fluid simulation tasks, including vision-based and surgical robot-learning scenarios. On an NVIDIA RTX 4090, the engine achieves up to 2.03 million environment steps per second for 8192 parallel CartPole environments, and scales to batched surgical scenarios involving tissue deformation, fluid interaction, and ultrasound sensing. Overall, CRESSim-Neo provides a unified and scalable platform for surgical simulation, synthetic data generation, and surgical robot learning.
I. INTRODUCTION
Surgical simulation requires coordinated modeling of rigid instruments, deformable tissues, fluids, and surgery-specific interactions, while existing platforms often lack scalable robotics interfaces or require substantial customization. CRESSim-Neo addresses this gap with a GPU-accelerated, surgery-oriented engine that combines broad physical coverage, surgical features, batched rendering, and GPU-resident learning pipelines.
- Surgical robotics simulation must capture interactions among rigid instruments, deformable soft tissues, and fluids.
- Existing PBD-based surgical platforms typically target surgeon training and single-scene CPU simulation, limiting scalability for large-scale learning and data generation.
- Existing GPU-accelerated simulators require substantial domain-specific modification for capabilities such as tissue cutting and ultrasound image synthesis.
- CRESSim-Neo combines a PBD-based GPU runtime for rigid bodies, deformable soft bodies, fluids, and strands with batched rasterization-based rendering.
- Its surgery-centric features include suturing-style interactions, cauterization, cable constraints, and ultrasound image synthesis.
- The GPU-resident pipeline exposes simulation buffers and CUDA-interoperable outputs for high-throughput robot learning and data generation.
II. SYSTEM OVERVIEW
The system is designed to unify multiple physical domains and sensing modalities while keeping simulation, rendering, and task computations resident on the GPU. Custom GPU compute and buffer access support this dataflow.
- CRESSim-Neo targets simulation of rigid instruments, deformable tissues, fluids, and strand-like structures alongside color, depth, semantic segmentation, and ultrasound observations.
- The design keeps the physics solver, renderer, observation generation, reward calculation, post-processing, and randomization primarily GPU-resident.
- Interfaces for custom GPU compute and access to internal physics and render-state buffers are included to support user-defined processing.
- The system also considers cross-operating-system and cross-GPU support, modern graphics APIs, shading-language portability, and graphics–compute interoperability.
B. System Architecture
CRESSim-Neo separates a standalone C++ orchestration layer, GPU execution backend, and Python interface. Physics, rendering, sensing, and task logic run through GPU-oriented components, with DLPack enabling direct zero-copy access from learning frameworks.
- CRESSim-Neo is a standalone C++ engine with a low-level GPU execution layer and a high-level Python binding layer.
- C++ Engine Orchestration: The C++ runtime manages scene authoring, entity-component registration, resource uploads, and Vulkan compute and graphics command generation.
- GPU Execution: Diligent Engine executes heavy physics and rendering calculations on GPU-backed Vulkan or D3D12 pipelines, with custom task logic implemented in HLSL compute kernels.
- Python Extension: A pybind11-based Python layer exposes the runtime and integrates Vulkan-CUDA or D3D12-CUDA interoperability through DLPack.
- Python Extension: PyTorch and other Python learning frameworks can acquire direct, zero-copy access to GPU-resident simulation and sensor buffers.
C. Pipeline Overview
The runtime uses separate authoring and stepping phases, converting batched scene descriptions into GPU-resident layouts before executing a staged simulation, sensing, rendering, and custom-compute loop. Multiple environments share resources while retaining environment-specific dynamic state.
- Pipeline Overview: The runtime executes through two phases: scene authoring and steady-state frame stepping.
- Scene Authoring: Scene authoring registers entities, components, and constraints, then compiles shaders, allocates device memory, and creates layout caches.
- Frame Stepping: Each frame uploads dynamic updates, dispatches GPU PBD physics, processes sensors and rendering, and optionally runs custom compute passes.
- Frame Stepping: Explicit synchronization boundaries allow custom GPU computations to be interleaved with simulation.
- A. Batched Multi-Environment Scene Representation: Batched scenes assign each entity an environment index that determines its simulation and rendering state.
- A. Batched Multi-Environment Scene Representation: Shared meshes, textures, materials, and shaders coexist with environment-specific simulation state, lighting, cameras, fluids, and sensor execution.
B. GPU Physics Using PBD
CRESSim-Neo uses PBD and shared GPU pipelines to simulate diverse physical domains, including surgical interactions, while producing batched visual and ultrasound sensing outputs.
- Physics solver: PBD supports rigid colliders, particle-based dynamics, joints, tetrahedral and meshfree soft bodies, fluids, and task-specific constraints.The solver includes XPBD-style constraints where needed, plus cable, tool, needle, thread, and suturing attachments.
- Physics solver: Shared prediction, neighborhood and contact generation, and iterative constraint projection enable batched many-environment simulation.The same parallel pipeline underlies the supported physical features and example scenes.
- Sensing and rendering: Visual sensors run from the current GPU scene state, while computational sensors such as ultrasound execute in a separate sensor stage.Ultrasound image synthesis uses the COLE algorithm, and sensor outputs can remain on GPU for downstream processing.
- Sensing and rendering: The engine renders RGB-D, depth-only, and segmentation-with-depth outputs from GPU-updated softbody and strand states, with fluids composited separately.Matched camera configurations can render in batches into layered array textures spanning multiple environments.
D. Platform Support and CUDA Interoperability
The runtime supports Vulkan and D3D12 graphics backends, with optional CUDA interoperability for shared GPU buffers and downstream CUDA or PyTorch use.
- Platform support: Vulkan is the default cross-platform backend, while D3D12 is available on Windows and Vulkan support on macOS is provided through MoltenVK.These backends are supplied by Diligent Engine.
- CUDA interoperability: CUDA interoperability optionally exports shared structured GPU buffers through device pointers, with synchronization handled by external timeline semaphores or fences.Without exportable allocation, the APIs fall back to engine-only GPU buffers without interoperability.
A. Resource Exposure and PyTorch Interoperability
CRESSim-Neo exposes structured GPU resources and supports custom compute for constructing observations, rewards, actions, and termination data without leaving the device.
- Resource exposure: Layout mappings identify packed GPU-buffer contents through ids, environment indices, and per-object particle offsets and counts.These mappings let users interpret simulation state in batched environments.
- Resource exposure: Custom compute passes can read simulation buffers and render targets, write shared buffers, and expose those buffers through DLPack for Python torch.Tensor views.The pipeline supports RGB outputs and task-specific GPU-side data processing.
- Custom task computation: User-defined HLSL compute passes can produce and consume actions, observations, rewards, reset masks, and termination flags on GPU.Passes are dispatched through executeCustomComputePass() and inserted into the staged frame loop.
- Example RL task instantiations: The interface is demonstrated across rigid-body, particle-based manipulation, and image-based batched reinforcement-learning tasks.Examples include CartPole, SoftBodyPush, FluidPour, and TargetCenter.
V. EVALUATION
The evaluation demonstrates surgical robotics scenes and learning tasks spanning tissue manipulation, suction, ultrasound scanning, suturing, continuum robots, and thermal coagulation.
- Surgical robot-learning tasks: Three surgical robot-learning tasks are presented: TissueRetract, BloodSuction, and UltrasoundScan.UltrasoundScan moves a linear probe over deformable tissue to center a target dark area in the ultrasound image.
- Surgical robotics capability demonstrations: The simulated PSM robot from the dVRK includes a full robot, Large Needle Driver, and Suction/Irrigator configurations.These configurations support the surgical capability demonstrations shown in the evaluation.
- Surgical capability demonstrations: Additional scenes demonstrate suturing with a curved needle and thread, cable-driven continuum robots, and thermal coagulation under varying tissue properties and appearance.The scenes extend evaluation beyond the three surgical reinforcement-learning environments.
B. Simulation and RL Throughput
Throughput rises with parallel-environment count until task-dependent saturation, while end-to-end PPO throughput follows the same pattern and remains near pure stepping for expensive tasks.
- 2.03M env-steps/s: CartPole reaches this pure-stepping throughput at 8192 environments, compared with 8.86k env-steps/s for SoftBodyPush.SoftBodyPush saturates at around 512 environments.
- Throughput increases strongly with parallel environments before reaching task-dependent saturation points.The benchmark varies environment count on a single GPU for pure stepping and end-to-end RL training.
- Per-environment workload determines scaling: lighter tasks achieve higher env-steps/s and scale to larger environment counts.Saturation can be driven by physics, rendering, sensing, or GPU memory usage.
- Additional parallelism may continue amortizing fluid and particle computation in BloodSuction before saturation, unlike the earlier saturation observed for TargetCenter.Both tasks use RGB image observations, but their overall workloads differ.
- End-to-end PPO throughput remains close to pure stepping throughput for most heavier particle-based and surgical tasks.This indicates simulation dominates runtime when physics and sensing are sufficiently expensive.
VI. DISCUSSION
CRESSim-Neo is positioned as a focused foundational engine for manipulation-oriented surgical simulation rather than a general-purpose robotics platform.
- CRESSim-Neo provides a foundational engine on which surgical scenes and applications can be built, rather than another high-level platform.It is not intended to replace general-purpose engines such as Isaac Sim.
- The engine emphasizes PBD-based simulation of rigid bodies, deformable tissues, fluids, strands, and manipulation-oriented interactions.These priorities distinguish it from systems centered on rigid-body accuracy, differentiability, or runtime GPU kernel generation.
- The engine is better suited to manipulation-oriented surgical tasks than to general applications such as locomotion or navigation.
- CRESSim-Neo extends the iMSTK and PositionBasedDynamics approach with batched rendering, multi-environment execution, and robot-learning interfaces.Its strength is broad surgical capability within a simple PBD framework, not broad robotics-domain coverage.
B. Limitations
The discussion identifies limitations in physical fidelity, memory and topology management, platform support, rendering features, and the scope of the learning demonstrations.
- PBD makes CRESSim-Neo less suited to applications requiring accurate rigid-body contact dynamics.Rigid contacts and joints are less physically faithful than those from rigid-body engines such as MuJoCo.
- Memory usage scales roughly linearly with environment count because buffers follow per-environment capacity multiplied by the number of environments.Pre-allocation can mitigate topology changes but increases memory consumption.
- Topology-changing operations require buffer reallocation, scene rebuilding, or simulation-state re-upload.
- Cross-platform validation remains incomplete, and the actively supported graphics backends are Vulkan and D3D12.Validation on AMD GPUs, MoltenVK/macOS, and systems without CUDA remains limited.
- The renderer uses a rasterized forward pipeline without ray tracing, custom pipelines, or custom graphics shaders.
- The RL examples demonstrate the learning interface and computational performance rather than task-specific policy performance or transfer.Task-specific learning and transfer are outside the scope of the work.