PaperlayerView exact revision on arXiv

Source-linked AI summary

cuRoboV2: Dynamics-Aware Motion Generation with Depth-Fused Distance Fields for High-DoF Robots

Balakumar Sundaralingam, Adithyavairavan Murali, Stan Birchfield

arXiv:2603.05493v2cs.RO

TL;DR

Existing motion-generation methods struggle to satisfy hardware constraints and collision avoidance across global planning and reactive control. cuRoboV2 unifies GPU-native dynamics, perception, kinematics, and self-collision computation, enabling collision-free motion optimization on a 48-DoF humanoid and LLM-assisted development of GPU-accelerated robotics software.

  • Problem

    Existing motion-generation methods struggle to satisfy hardware constraints and collision avoidance simultaneously across global planning and reactive control.

    Source: §1 · p. 3, 1) The Feasibility Gap · p. 3

  • Method

    cuRoboV2 combines GPU-native dynamics-aware B-spline optimization, dense ESDF collision queries, scalable kinematics, and map-reduce self-collision computation.

    Source: §9 · p. 38

  • Results

    cuRoboV2 enables collision-free motion optimization on a 48-DoF humanoid and substantial LLM assistance in developing GPU-accelerated robotics modules.

    Source: Collision-Free IK · p. 25, §9 · p. 38

  • Takeaways & Limitations

    The unified stack supports physically valid reference motions for downstream locomotion policies and demonstrates productive human-LLM collaboration in robotics software development.

    Source: §9 · p. 38

    Single-camera scene coverage is partial, geometric robot segmentation is sensitive to depth-estimation errors, and interpreting some GPU compiler behavior still requires human judgment.

    Source: Limitations & Future Work · p. 38

Abstract

from arXiv · show

Effective robot autonomy requires motion generation that is safe, feasible, and reactive. Current methods are fragmented: fast planners output physically unexecutable trajectories, reactive controllers struggle with high-fidelity perception, and existing solvers fail on high-DoF systems. We present cuRoboV2, a unified framework with three key innovations: (1) B-spline trajectory optimization that enforces smoothness and torque limits; (2) a GPU-native TSDF/ESDF perception pipeline that generates dense signed distance fields covering the full workspace, unlike existing methods that only provide distances within sparsely allocated blocks, up to 10x faster and in 8x less memory than the state-of-the-art at manipulation scale, with up to 99% collision recall; and (3) scalable GPU-native whole-body computation, namely topology-aware kinematics, differentiable inverse dynamics, and map-reduce self-collision, that achieves up to 61x speedup while also extending to high-DoF humanoids (where previous GPU implementations fail). On benchmarks, cuRoboV2 achieves 99.7% success under 3kg payload (where baselines achieve only 72--77%), 99.6% collision-free IK on a 48-DoF humanoid (where prior methods fail entirely), and 89.5% retargeting constraint satisfaction (vs. 61% for PyRoki); these collision-free motions yield locomotion policies with 21% lower tracking error than PyRoki and 12x lower cross-seed variance than GMR. A ground-up codebase redesign for discoverability enabled LLM coding assistants to author up to 73% of new modules, including hand-optimized CUDA kernels, demonstrating that well-structured robotics code can unlock productive human-LLM collaboration. Together, these advances provide a unified, dynamics-aware motion generation stack that scales from single-arm manipulators to full humanoids. Code is available at https://github.com/NVlabs/curobo.

1. Introduction

Robot autonomy requires motion generation that is safe, efficient, responsive, hardware-feasible, and collision-free, yet global planning and reactive generation remain fragmented and struggle to satisfy these requirements simultaneously.

Problem framing

  • Global motion planning [43] computes paths from static starts to goals, while reactive motion generation [11] [60] tracks moving targets at high frequency.Both domains must satisfy robot hardware limits while avoiding collisions with the scene and the robot itself.Source: §1 · p. 3

1) The Feasibility Gap

Fast collision-free planning often ignores dynamics, producing physically unexecutable motions, while dynamics-aware optimization struggles with scalable collision avoidance under complex perception constraints.

The Feasibility Gap

  • Collision-free motion planning can be fast [60], but this speed often comes at the cost of ignoring dynamics.Source: 1) The Feasibility Gap · p. 3
  • Most planners output piecewise-linear paths that assume infinite torque, ignoring robot mass and momentum and often violating torque limits under heavy payloads.Aggressive post-processing can invalidate the original plans’ safety guarantees.Source: 1) The Feasibility Gap · p. 3
  • Dynamics-aware trajectory optimization respects torque limits but struggles to scale to non-convex collision-avoidance constraints from mesh or depth-based representations.Source: 1) The Feasibility Gap · p. 3

2) The Perception-Reactivity Trade-off

Existing reactive motion methods trade off safety against high-fidelity perception: analytic controllers provide safety guarantees but process raw depth slowly, while learning-based methods react quickly without strict collision guarantees or robust generalization.

Safety versus perception

  • Analytic controllers such as RMPs and MPC provide safety guarantees but process raw depth too slowly for reactive use, limiting them to simplified geometric primitives.Their computational cost prevents direct use of high-fidelity depth data.Source: 2) The Perception-Reactivity Trade-off · p. 3
  • Learning-based approaches process visual observations rapidly but fail to provide the strict collision guarantees required for safe deployment.Their speed comes at the expense of guaranteed collision safety.Source: 2) The Perception-Reactivity Trade-off · p. 3
  • Learning-based approaches also lack generalization, requiring extensive retraining for new environments.This further limits their deployment across changing environments.Source: 2) The Perception-Reactivity Trade-off · p. 3

3) The Scalability Wall

cuRoboV2 addresses the difficulty of scaling motion generation from single-arm robots to high-DoF bimanual manipulators and humanoids. Its unified framework combines smooth, torque-aware trajectory optimization, dense GPU-native perception, and scalable whole-body computation.

3) The Scalability Wall

  • High-DoF planners often converge slowly or fail, especially for collision-free IK in cluttered bimanual and humanoid applications.This scalability barrier affects real-world bimanual manipulation and humanoid motion retargeting.Source: 3) The Scalability Wall · p. 3

Contributions

  • B-spline control-point optimization enforces smoothness and torque limits while supporting non-static boundary conditions for reactive control.This provides a unified motion-generation representation for global planning and reactive control.Source: Contributions · p. 3
  • The GPU-native perception pipeline produces full-workspace dense ESDFs with O(1) distance queries, up to 10× faster, in 8× less memory than nvblox, and up to 99% collision recall.It fuses depth, meshes, and cuboids into millimeter-resolution TSDFs and enables millisecond-rate reactive updates.Source: Contributions · pp. 3–4
  • Topology-aware kinematics, differentiable inverse dynamics, and map-reduce self-collision scale to humanoids where prior GPU implementations fail, delivering up to 40× speedup.The increased throughput accelerates high-DoF optimization while enforcing torque limits.Source: Contributions · p. 4

2. Related Work

Existing motion-generation methods struggle with dynamics, reactivity, dense perception, hardware scalability, and high-DoF humanoid retargeting. cuRoboV2 addresses these gaps with constrained B-spline optimization, GPU-native perception and computation, and feasible collision-free humanoid motion generation.

Global Planning

  • cuRoboV2 extends manipulator trajectory optimization with B-splines that accommodate torque limits and non-convex collision constraints without sacrificing convergence.Prior waypoint-based methods rely on acceleration regularization, while existing manipulator B-spline formulations handle only convex constraints.Source: Global Planning · p. 4

Reactive Motion Generation

  • Reactive model-based methods can get stuck in local optima, while learning-based methods lack out-of-distribution generalization, require per-embodiment retraining, and cannot guarantee collision-free execution.These limitations motivate a unified approach for reacting to dynamic environments from visual observations.Source: Reactive Motion Generation · p. 4

Hardware Acceleration

  • GPU-parallelized optimization and related kinematic frameworks primarily target single-arm robots, while scaling to high-DoF systems degrades convergence and increases compute time.cuRobo introduced GPU-parallelized trajectory optimization, and PyRoki provides cross-platform kinematic optimization, but both leave high-DoF scalability gaps.Source: Hardware Acceleration · pp. 4–5

Visual Representations

  • cuRoboV2 provides GPU-native, millimeter-resolution collision queries, addressing dense-field and memory limitations of nvblox and the lack of real-time capability in CPU-based methods.KinectFusion represents scenes as TSDFs but lacks planning collision queries, while neural representations can fail on out-of-distribution scenes.Source: Visual Representations · p. 6

Human-to-Humanoid Motion Retargeting

  • cuRoboV2 provides collision-free, joint-limit-satisfying IK for high-DoF humanoids, addressing infeasible retargeted motions that destabilize policy training and increase cross-seed variance.GMR [1] uses per-frame local IK with only joint-limit constraints, while PyRoki [36] fails to converge on high-DoF robots.Source: Human-to-Humanoid Motion Retargeting · p. 6

3. Problem Formulation

This section formulates motion generation as constrained trajectory optimization from an initial state to target link poses, minimizing smoothness- and efficiency-related costs while enforcing collision, kinematic, dynamic, torque, and stopping requirements. The formulation uses B-spline trajectories and supports trajectory optimization, inverse kinematics, and reactive MPC-style control.

Problem formulation

  • Motion generation optimizes a trajectory from an initial state to one or more target link poses while minimizing smoothness and energy costs under collision and kinematic/dynamic constraints.The hard constraints include self- and environment-collision avoidance and robot limits.Source: §3 · p. 6
  • B-spline evaluation, joint-limit constraints, collision checks, terminal pose accuracy, inverse-dynamics torques, torque limits, and complete stopping are enforced across timesteps and links.The loss promotes smooth, efficient motion; collision constraints cover both the scene and the robot itself.Source: §3 · pp. 6–7
  • The same optimization framework covers local and global collision-free planning, inverse kinematics as a single-state special case, and reactive control through repeated short-horizon re-solving.Prior methods use rough trajectory seeds or parallel seeds with graph-based initialization for global planning.Source: §3 · p. 7

Optimization loop

  • Each optimization iteration interpolates B-spline control points, computes kinematics and torques, evaluates costs concurrently, backpropagates gradients, and updates control points with L-BFGS.World collision, self-collision, and state costs are evaluated in parallel on separate CUDA streams before aggregation.Source: Figure 1 · p. 7, §3 · p. 7

4. B-Spline Basis as Optimization Space

cuRoboV2 represents each joint trajectory with a uniform cubic B-spline whose control points provide a smooth, compact, well-conditioned optimization space. The formulation supports efficient GPU gradient computation and implicitly satisfies static or non-static boundary conditions.

4. B-Spline Basis as Optimization Space

  • Uniform cubic B-splines use control points as optimization variables, providing C2-continuous trajectories with local support and fewer decision variables than per-timestep position parameterizations.Their implicit smoothness supports acceleration and velocity regularization while keeping gradient-based optimization well-conditioned.Source: §4 · pp. 7–8

4.1. Gradients with Respect to Spline Control Points

  • Gradients for each control point sum contributions from its affected interpolation points and are computed through a fused GPU warp-level reduction that avoids intermediate global-memory writes.The implementation uses N_interp = 4 and assigns threads to knots in an interleaved pattern before reducing each local sum.Source: §4.1 · pp. 8–9, Figure 2 · p. 8

4.2. Boundary Conditions

  • Spline boundary handling implicitly satisfies static goals by repeating the final knot four times and handles non-static initial states with three fictitious knots anchored to the initial state.This eliminates constraint coupling and oscillatory corrections, allowing optimization to focus on the remaining objectives.Source: §4.2 · p. 9

5. ESDF for Scene Collision Avoidance

cuRoboV2 represents scenes with a persistent block-sparse TSDF and generates dense, task-resolution ESDFs on demand for collision queries. Its GPU pipeline integrates observations, recycles stale blocks, propagates exact distances, recovers signs, and supports swept-volume collision constraints.

5. ESDF for Scene Collision Avoidance

  • The two-stage TSDF-ESDF representation fuses depth and known primitives persistently, then generates dense task-specific distance fields with interior-sign recovery.The block-sparse TSDF supports world modeling, while the dense ESDF enables distance queries across the workspace.Source: §5 · p. 9

5.1. Block-Sparse TSDF Storage

  • Only near-surface 8^3-voxel blocks are allocated through hashed CAS insertion, while recycled low-weight blocks bound memory over scene duration.Each voxel stores separate depth and geometry channels, whose minimum forms the effective signed distance; a typical 100K-block scene uses ∼350 MB, roughly 11× less than a dense grid.Source: §5.1 · p. 9, Figure 4 · p. 10

5.2. Depth and Primitive Integration

  • Voxel-centric integration assigns one thread per voxel to project into the depth image and write signed distance without atomics, while primitives update the geometry channel by minimum SDF.The four phases discover, deduplicate, allocate, and integrate visible blocks; projected voxel area determines observation weight with w = max(c, 1).Source: Figure 5 · p. 10, §5.2 · p. 11

5.3. Frustum-Aware Decay and Block Recycling

  • Two-tier time and frustum decay rapidly adapts observations in dynamic scenes, then tombstones underweight blocks and returns them to a reusable free list.Frustum membership uses conservative block-level bounding-sphere tests, and decay is implemented as one fused multiplication without atomic operations.Source: §5.3 · p. 11

5.4. On-Demand ESDF with Sign Recovery

  • Task-specific ESDF resolution decouples fine TSDF integration from coarser collision queries: 5 mm suffices for 1–5 cm manipulation spheres, while larger spheres permit coarser fields.This reduces ESDF memory and computation while preserving reliable trilinear collision checks and enabling O(1) distance queries.Source: §5.4 · pp. 11–12

Stage 1: Site Seeding

  • Gather-based site seeding probes each ESDF cell at seven positions without atomics, uses fixed |G_esdf| work independent of scene content, and supports CUDA graph capture.The approach avoids scatter conflicts, though thin surfaces may be missed when the ESDF is much coarser than the TSDF; typical 2–4× ratios are adequately covered.Source: Stage 1: Site Seeding · p. 12, Figure 6 · p. 13

Stage 2: Distance Propagation

  • PBA+ [3] computes the exact Euclidean Voronoi diagram through separable Z-, Y-, and X-axis sweeps, requiring fixed five kernel launches compatible with CUDA graph capture.Maurer’s parabola test yields O(N)-work and O(1)-error distance transforms per axis, unlike approximate JFA, which can miss sites.Source: Stage 2: Distance Propagation · p. 12

Stage 3: Sign Recovery

  • Sign recovery treats depth-fused regions beyond truncation as exterior, resolves watertight geometry using adjacent geometry-SDF samples, and applies the resulting ESDF to swept-volume collision constraints.Missing adjacent samples fall back to the combined query SDF, with unobserved voxels defaulting to positive exterior; collision gradients use the CHOMP speed metric.Source: Stage 3: Sign Recovery · pp. 12–13

6. Scaling to High-DoF Robots

cuRoboV2 addresses high-DoF motion-optimization challenges from branching kinematic trees, mimic joints, quadratic self-collision pairs, and torque limits. Its GPU-native methods combine adaptive kinematics, topology-aware differentiation, sparse Jacobians, map-reduce collision handling, and differentiable inverse dynamics.

6. Scaling to High-DoF Robots

  • cuRoboV2 targets branching kinematic trees, mimic joints, quadratic self-collision growth, and torque-limit enforcement in high-DoF robots.The framework introduces five corresponding GPU-native innovations for scalable whole-body computation.Source: §6 · pp. 13–14

6.1.1. Forward Kinematics

  • For robots with ≤100 collision spheres, a fused kernel computes five outputs, while robots with >100 spheres use dual kernels to parallelize sphere, pose, center-of-mass, and Jacobian computation.The adaptive dispatch separates frame-transform computation from downstream outputs for complex robots such as bimanual systems and humanoids.Source: §6.1.1 · p. 14, Figure 8 · p. 15

6.1.2. Parallel Gradient Backpropagation

  • Topology caches provide O(1) ancestor-link lookup, enabling warp-level parallel gradient backpropagation that traverses only relevant kinematic chains and maps gradients through mimic joints.A link’s gradient affects only its ancestor joints, allowing the implementation to exploit kinematic sparsity.Source: Figure 7 · p. 14, §6.1.2 · p. 15

6.1.3. Sparse Jacobian Computation

  • Sparse Jacobians use O(1) coarse subtree filtering and fine-grained ancestor checks for mimic joints, supporting high-DoF Levenberg-Marquardt inverse kinematics with multiple tool frames.Collision-free IK first uses LM to reach the target, then L-BFGS refines the solution under self-collision and environment constraints.Source: §6.1.3 · pp. 15–16

6.2. Self-Collision via Map-Reduce

  • A two-stage map-reduce partitions self-collision pairs across blocks and reduces block maxima, addressing quadratic pair growth from 818 pairs on a 7-DoF Franka to 162k on a 48-DoF humanoid.The final kernel identifies the globally most-penetrating pair after per-block reductions.Source: §6.2 · p. 16

6.3. Torque Limits via Differentiable Inverse Dynamics

  • Differentiable inverse dynamics based on RNEA enforces actuator torque limits inside optimization rather than as a post-hoc check, with O(n) computation over the kinematic tree.RNEA performs forward velocity and acceleration propagation, force computation, and backward torque accumulation.Source: §6.3 · p. 16

11 else

The paper’s GPU-native RNEA implementation combines robot-generic kernels with compact representations, VJP-based differentiation, and tree-level parallelism to support scalable dynamics computation. It achieves substantial speedups while retaining runtime flexibility and humanoid-scale applicability.

Limitations of Existing GPU Implementations

  • Existing GPU RNEA implementations trade flexibility for throughput: Newton launches six kernels and becomes a batch-size bottleneck, while GRiD requires recompilation, lacks runtime inertial changes, and exceeds shared-memory limits on humanoids.These limitations motivate a robot-generic implementation that remains efficient across robot types and payload conditions.Source: Limitations of Existing GPU… · p. 18, Table 2 · p. 18

Our RNEA Implementation

  • Robot-generic GPU kernels parameterized by links, degrees of freedom, and batch size support any URDF through runtime-supplied topology, inertias, and joint types.The design uses compact spatial representations, VJP-based backward computation, and tree-level parallelism with warp-level synchronization.Source: Our RNEA Implementation · p. 18
  • The implementation reduces memory pressure by storing Featherstone-form spatial transforms and inertias in compact 12-float representations instead of full 6×6 quantities.Spatial products are recomputed analytically or on demand, trading arithmetic for lower memory use.Source: Our RNEA Implementation · p. 18
  • Direct VJP computation obtains gradients with respect to q, q̇, q̈, and external forces in O(n) time, avoiding GRiD’s O(n^2) Jacobian materialization.The adjoint pass also enables gradient flow through contact or interaction forces.Source: Our RNEA Implementation · p. 18
  • 14× speedup over Newton and within 1.5–2× of GRiD at trajectory-optimization batch sizes, while supporting runtime payload changes, PyTorch interoperability, and humanoid-scale robots.RNEA contributes only 24% of trajectory-optimization time and 29% of total motion-planning time, keeping dynamics-aware planning overhead modest.Source: Our RNEA Implementation · p. 19

7. Experimental Results

Experiments evaluate cuRoboV2 across dynamics-aware planning, high-DoF computation, humanoid applications, and ESDF generation, including planning, computational components, applications, and real-world depth-fusion manipulation. They examine executable trajectories, scalable solving, improved retargeting and policies, and faster dense distance fields with real-time collision avoidance from live stereo depth.

7.1. Dynamics-Aware Planning

  • 99.7% payload success with 3 kg versus 77.1% for cuRobo and 72–75% for sampling methods, while using 106 J and preserving executable trajectories.cuRoboV2 also achieves 99.7% zero-payload dynamics success, compared with 97.9% for cuRobo and 95.5–95.8% for sampling-based methods; planning takes 48 ms versus 36 ms for cuRobo.Source: §7.1 · p. 20, Figure 11 · p. 20

7.2. Runtime Breakdown

  • 42 ms end-to-end planning with dynamics versus 35 ms without it, as CUDA streams and graphs overlap added kernels and hide launch overhead.Dynamics adds 9.38 ms of kernel execution, including 5.12 ms from RNEA within trajectory optimization.Source: §7.2 · pp. 21–22

7.3. Inverse Dynamics Comparison

  • On the 48-DoF humanoid, cuRoboV2 runs inverse dynamics in 96.2 μs, while GRiD fails and Newton takes 1712.9 μs, making cuRoboV2 18× faster than Newton.On simpler robots, cuRoboV2 is 1.5–2× slower than GRiD but 14–18× faster than Newton.Source: §7.3 · p. 23, Figure 13 · p. 23

Fast Kinematics and Self-Collision

  • 61× speedup over cuRobo for humanoid self-collision computation, while cuRoboV2 achieves 100% standard IK success and 99.6% collision-free IK success on the 48-DoF humanoid.Parallel branching kinematics and map-reduce collision checking enable scalability; collision-free IK uses Levenberg–Marquardt seeding followed by L-BFGS refinement, whereas cuRobo and PyRoki achieve 0%.Source: Fast Kinematics and Self-Collision · p. 24, Figure 14 · p. 24, Global Inverse Kinematics · p. 25

7.5.1. Motion Retargeting Quality

  • 89.5% retargeting constraint satisfaction for cuRoboV2-IK versus 61.2% for PyRoki, 54.5% for mink, and 40.6% for GMR; MPC reaches 96.6%.The LM+L-BFGS strategy supports collision-free solutions, while MPC checks inter-frame collisions but sacrifices pose-tracking accuracy.Source: §7.5.1 · pp. 25–26

7.5.2. Locomotion Policy Training

  • Across both motions, cuRoboV2 yields 145.5 mm MPJPE, 99.6% alive rate, and 1.1 resets, outperforming mink, GMR, and PyRoki overall.On crawling, cuRoboV2 reaches 151.4±4.5 mm MPJPE versus 178.0±58.3 mm for GMR, whose cross-seed variance is 12× higher; on running, PyRoki averages 31% episode length.Source: §7.5.2 · pp. 26, 27, 29, 30, Figure 18 · p. 28

7.6. Fast and Safe ESDF at mm-Resolution

  • At 10 mm full coverage, cuRoboV2 generates ESDFs in 1.69 ms versus 12.68 ms for nvblox, uses 1.63 GB versus 11.87 GB, and achieves 92.5% versus 92.0% recall.At 20 mm, it reaches 1.52 ms and 97.0% recall versus 2.97 ms and 85.4% for nvblox; PBA+ propagation supplies the speedup, while gather enables deterministic CUDA-graph capture.Source: §7.6 · p. 30, Figure 20 · p. 31, Figure 21 · p. 31
  • ESDF seeding recall depends on r = v_esdf/v_tsdf: scatter is better at r ≥ 4, gather at r ≈ 1, and both are comparable near r ≈ 2.Gather dilates the seed band through seven probes, whereas scatter samples surface-containing cells and can achieve +2.5% recall at r=4.Source: §7.6 · pp. 30–32
  • cuRoboV2 generates a dense ESDF covering the full workspace, whereas nvblox provides signed distances only within allocated blocks.PBA+ propagation provides O(1) distance queries everywhere, though surface discrepancies remain due to TSDF weight-threshold tuning.Source: §7.6 · p. 32
  • 19% lower mean planning time, from 47 ms to 38 ms, with negligible success-rate impact (Δ = −0.12%) when known geometry is stamped into the TSDF.Collision queries become O(1) trilinear ESDF lookups instead of per-cuboid analytical SDF evaluations.Source: §7.6 · p. 32
  • 10× faster construction and 8× less memory than nvblox, with fast O(1) distance queries yielding a 19% reduction in motion-planning time.Decoupling TSDF and ESDF resolutions enables this manipulation-scale efficiency improvement.Source: §7.6 · p. 32

7.7. Real-World Manipulation

  • Real-time collision avoidance runs on the I2RT YAM within one control loop using segmented 1080p, 15 Hz stereo depth, a 2.5 mm TSDF, and a 2 cm ESDF.The ESDF drives an MPC trajectory optimizer that tracks pose commands while avoiding dynamically observed obstacles.Source: §7.7 · p. 32, Figure 23 · p. 33, Figure 24 · p. 33

8. LLM-assisted Development

cuRoboV2’s discoverability-focused codebase redesign enabled productive human–LLM collaboration, with LLMs authoring up to 73% of later module additions while humans supplied specifications, design decisions, and oversight. The approach supported implementations ranging from differentiable CUDA inverse dynamics to unified collision checking and faster ESDF propagation.

8.1. Codebase Discoverability

  • Typed configurations, predictable names, smaller documented files, executable tests, and self-documenting signatures made cuRoboV2’s APIs inspectable to LLMs and human researchers.The redesign addressed hidden configuration, opaque naming, oversized files, undocumented tests, and ambiguous tensor-shape semantics.Source: §8.1 · p. 34

Cost of the investment

  • Framework code grew 1.5×, inline documentation 2.7×, files 121→390, and tests 264→3,978 as discoverability investment expanded the codebase.CUDA C++ stayed nearly flat while GPU code migrated from CUDA C++ toward Warp, whose share increased from 34% to 57%.Source: Figure 25 · p. 34, Cost of the investment · pp. 34–35

8.2. Quantifying LLM Contributions

  • 73% of additions were LLM-authored in phase N2, while contributions were nearly 50/50 across the full development period.Discoverability refactoring and the switch to Claude Opus enabled increasingly complex LLM tasks after initial refactoring phases reached only 6–18%.Source: Figure 26 · p. 35, §8.2 · p. 35

Inverse Dynamics (RNEA) CUDA Implementation & Optimization

  • A human NumPy reference enabled the LLM to derive, test, translate, and optimize Featherstone RNEA forward and backward CUDA kernels, while humans resolved compiler-interpretation errors.The workflow combined analytical reverse-mode differentiation, Pinocchio verification, profiling, and human-directed optimizations such as tree-level parallelism and forward caching.Source: Inverse Dynamics (RNEA) CUDA… · pp. 35–36, Figure 27 · p. 36
  • The LLM-assisted migration replaced split collision backends and wrappers with a unified type-generic Warp system at roughly half the original 5,000 lines.The resulting architecture supported cuboids, meshes, voxels, and ESDF grids, but humans diagnosed a silent Warp overload-caching bug and supplied key design decisions.Source: Figure 28 · p. 36, Inverse Dynamics (RNEA) CUDA… · pp. 36–38
  • The LLM-generated PBA ESDF implementation was about 2× faster than the JFA baseline while producing exact signed distances and supporting nonuniform, arbitrary-dimensional grids.Humans selected the staged JFA-to-PBA strategy, while the LLM implemented banded indexing, CUDA integration, and mapper adaptation.Source: Inverse Dynamics (RNEA) CUDA… · p. 38

9. Conclusion … A.1.4. Web Visualization via Viser

cuRoboV2 combines GPU-native dynamics, dense distance fields, scalable whole-body computation, and a redesigned, discoverable software stack to support executable motion generation from arms to humanoids. The redesign also improves installation, robot configuration, perception integration, visualization, and LLM-assisted development while identifying remaining challenges in perception coverage and autonomous GPU-code reasoning.

9. Conclusion

  • cuRoboV2’s GPU-native stack compounds B-spline torque-aware optimization, dense ESDF collision queries, and scalable whole-body computation to produce executable, reactive, collision-free motion from 7-DoF arms to 48-DoF humanoids.Fast inverse dynamics makes torque constraints tractable, dense ESDFs support live depth-fused MPC, and scalable kinematics plus map-reduce self-collision enable humanoid optimization.Source: §9 · p. 38

Robot Segmentation / Limitations & Future Work

  • Robot segmentation removes depth points near forward-kinematics collision spheres, offering a faster but approximate alternative to mesh-based segmentation for TSDF reconstruction.The method uses calibrated camera extrinsics and intrinsics to project depth pixels into 3D; learned RGB segmentation and multi-camera fusion are proposed to improve robustness and scene completeness.Source: Limitations & Future Work · p. 38

A. Framework Design

  • A ground-up redesign replaced cuRoboV1’s monolithic pipeline with simpler, extensible architecture supporting custom costs, obstacle representations, optimization strategies, easier robot onboarding, and LLM-assisted development.The redesign prioritizes researcher experience and software architecture, accepting modest performance overhead where it improves usability.Source: §9 · p. 38, A. Framework Design · p. 46

A.1. Improving Researcher Experience

  • cuRoboV2 removes four researcher-experience barriers: version-sensitive installation, manual robot configuration, difficult perception integration, and dependence on external visualization and debugging tools.These barriers made installation, robot onboarding, real-perception integration, and plan inspection unnecessarily time-consuming in v1.Source: §A.1 · p. 46

A.1.1. Simplified Installation

  • Installation falls from approximately 20 minutes to seconds because CUDA Python compiles and caches GPU kernels on first use without requiring CUDA–PyTorch version matching.Kernels target the host GPU and are cached by source-content hash, while cuRoboV2 ships as a standard pip package.Source: Figure 30 · p. 46, §A.1.1 · p. 46

A.1.2. Robot Configuration Builder

  • RobotBuilder converts a standard URDF into a complete optimized planning configuration, automating collision-sphere fitting and self-collision pruning instead of requiring hand-authored YAML.GPU-batched Halton sampling identifies link pairs that never collide, and the resulting configuration can be exported, inspected, modified, and saved through web-based tools.Source: §A.1.2 · p. 47, Figure 31 · p. 47

A.1.3. Eye-to-Hand Utilities / Eye-to-Hand Calibration

  • Eye-to-hand utilities integrate GPU-accelerated ICP calibration into a web visualizer, providing a practical camera-to-robot alignment workflow for collision-free planning and pick-and-place.The workflow globally aligns robot-mesh samples with a scene bounding box, locally minimizes signed-distance residuals, and saves the calibrated pose.Source: §A.1.3 · p. 48, Eye-to-Hand Calibration · p. 48, Figure 32 · p. 48

A.1.4. Web Visualization via Viser / Limitations & Future Work

  • Viser [75] provides browser-based visualization of robot models, IK/MPC targets, collision spheres, obstacles, and TSDF/ESDF reconstructions, while LLM-assisted module development remains limited by human interpretation of compiler representations.The LLM workflow benefited from typed interfaces, predictable naming, small modules, and executable documentation; future work also includes warm-started whole-body MPC and learned RGB-based segmentation.Source: §9 · p. 38, Limitations & Future Work · p. 38, §A.1.4 · p. 48

A.2. Software Architecture … B. LLM Prompts

cuRoboV2 redesigns its software architecture around extensible obstacle and cost interfaces, unified GPU kernels, and explicit separation between stable CUDA components and modifiable Warp components. Persistent LLM prompts enforce the resulting structural, naming, formatting, and documentation conventions.

A.2. Software Architecture

  • cuRoboV2 adds extension points for custom obstacle types and cost functions without requiring changes to core code.Source: §A.2 · p. 48

A.2.1. Unified Scene Collision Checking

  • All scene collision checking is consolidated into one type-generic Warp kernel that parallelizes sphere–obstacle pairs and accumulates results into a shared collision buffer.Source: §A.2.1 · p. 49
  • New obstacle representations require only three interface functions, leaving the shared collision kernel unchanged.The functions determine obstacle activation, world-frame pose, and local signed distance with analytic gradient.Source: §A.2.1 · p. 49

A.2.2. Cost Manager & Warp-First Kernels

  • The CostManager registers standalone BaseCost subclasses, evaluates them in parallel on separate CUDA streams, and enables new costs without modifying the optimization pipeline.Each subclass implements forward(state, config), returns a cost tensor, and receives gradients through PyTorch backpropagation.Source: §A.2.2 · p. 49, Figure 33 · p. 49
  • Stable algorithms remain hand-tuned CUDA C++ kernels, while user-modifiable costs use Warp for direct editing, automatic differentiation, and no recompilation.PoseCost was moved from CUDA C++ to Warp because it is a commonly requested customization.Source: §A.2.2 · p. 49

B. LLM Prompts

  • Persistent rule files encode design, formatting, naming, and documentation standards so LLM-generated code follows the same structure as human-written code.These prompts are intended to reduce review friction and prevent stylistic drift over time.Source: B. LLM Prompts · pp. 49–50

Related papers

Loading 2603.05493v2…