PaperlayerView exact revision on arXiv

Source-linked AI summary

cuRoboV2: Dynamics-Aware Motion Generation with Depth-Fused Distance Fields for High-DoF Robots

Balakumar Sundaralingam, Adithyavairavan Murali, Stan Birchfield

arXiv:2603.05493v2cs.RO

TL;DR

Existing motion-generation methods are fragmented: planners can ignore dynamics, reactive controllers struggle with high-fidelity perception, and solvers fail to scale to high-DoF systems. cuRoboV2 unifies B-spline optimization with GPU-native perception and whole-body computation. Across planning, IK, retargeting, and downstream locomotion, it reports strong performance while extending motion generation from single arms to humanoids.

  • Problem

    Existing methods struggle to satisfy dynamics, collision, perception, and scalability requirements simultaneously across global planning and reactive motion generation.

  • Method

    cuRoboV2 combines B-spline trajectory optimization, GPU-native TSDF/ESDF perception, and scalable GPU-native kinematics, inverse dynamics, and self-collision computation.

  • Results

    The framework achieves strong results across payload-aware planning, 48-DoF collision-free IK, retargeting, and downstream locomotion, while supporting high-DoF humanoids where prior GPU implementations fail.

  • Takeaways & Limitations

    The paper presents a unified, dynamics-aware motion-generation stack spanning planning, reactive control, retargeting, and humanoid systems.

  • Takeaways & Limitations

    The system remains limited by depth-estimation sensitivity in geometric robot segmentation and incomplete scene coverage from a single camera.

Abstract

from arXiv · show

Effective robot autonomy requires motion generation that is safe, feasible, and reactive. Current methods are fragmented: fast planners output physically unexecutable trajectories, reactive controllers struggle with high-fidelity perception, and existing solvers fail on high-DoF systems. We present cuRoboV2, a unified framework with three key innovations: (1) B-spline trajectory optimization that enforces smoothness and torque limits; (2) a GPU-native TSDF/ESDF perception pipeline that generates dense signed distance fields covering the full workspace, unlike existing methods that only provide distances within sparsely allocated blocks, up to 10x faster and in 8x less memory than the state-of-the-art at manipulation scale, with up to 99% collision recall; and (3) scalable GPU-native whole-body computation, namely topology-aware kinematics, differentiable inverse dynamics, and map-reduce self-collision, that achieves up to 61x speedup while also extending to high-DoF humanoids (where previous GPU implementations fail). On benchmarks, cuRoboV2 achieves 99.7% success under 3kg payload (where baselines achieve only 72--77%), 99.6% collision-free IK on a 48-DoF humanoid (where prior methods fail entirely), and 89.5% retargeting constraint satisfaction (vs. 61% for PyRoki); these collision-free motions yield locomotion policies with 21% lower tracking error than PyRoki and 12x lower cross-seed variance than GMR. A ground-up codebase redesign for discoverability enabled LLM coding assistants to author up to 73% of new modules, including hand-optimized CUDA kernels, demonstrating that well-structured robotics code can unlock productive human-LLM collaboration. Together, these advances provide a unified, dynamics-aware motion generation stack that scales from single-arm manipulators to full humanoids. Code is available at https://github.com/NVlabs/curobo.

1. Introduction

Robot autonomy requires motion generation that is safe, efficient, and responsive, but global planning and reactive generation remain fragmented and struggle to satisfy hardware limits and collision avoidance simultaneously.

  • Global Motion Planning computes paths from static starts to goals, whereas Reactive Motion Generation tracks moving targets at high frequency.
  • Both domains must satisfy robot hardware limits while avoiding scene and self-collisions.
  • Existing methods struggle to meet these requirements simultaneously, leaving critical functionality gaps.

1) The Feasibility Gap

Fast collision-free planners often produce physically unexecutable motions because they ignore dynamics, while dynamics-aware optimizers struggle with complex collision constraints.

  • Collision-free motion planning can be fast, but often ignores dynamics and produces physically unexecutable motions.
  • Most planners output piecewise-linear joint paths that assume infinite torque and ignore robot mass and momentum.
  • Time-optimal plans can violate torque limits under heavy payloads, requiring post-processing that invalidates their original safety guarantees.
  • Dynamics-aware trajectory optimizers respect torque limits but struggle with non-convex mesh- or depth-based collision avoidance.

2) The Perception-Reactivity Trade-off

Reactive methods trade off safety against high-fidelity perception: analytic controllers are too slow for raw depth, while learning-based methods lack strict collision guarantees and generalization.

  • Analytic controllers provide safety guarantees but are often too slow for raw depth data, restricting them to simplified geometric primitives.
  • Learning-based approaches process visual observations rapidly but fail to provide the strict collision guarantees required for safe deployment.
  • Learning-based methods also lack generalization and therefore require extensive retraining for new environments.

3) The Scalability Wall

High-DoF systems expose a scalability wall: methods effective on single arms can fail in bimanual and humanoid settings. cuRoboV2 addresses this with unified B-spline optimization and GPU-native perception, kinematics, dynamics, and self-collision computation.

  • 3) The Scalability Wall: State-of-the-art planners can converge slowly or not at all for collision-free IK in cluttered bimanual and humanoid systems.
  • B-Spline Optimization: cuRoboV2 introduces B-spline optimization to provide a unified motion representation for global planning and reactive control while satisfying torque limits.
  • GPU-native Perception: Its GPU-native perception pipeline fuses depth, meshes, and cuboids into a millimeter-resolution block-sparse TSDF using voxel-centric projection without atomic contention.
  • Scalable Whole-Body Computation: Topology-aware kinematics, differentiable inverse dynamics, and map-reduce self-collision scale to humanoids where prior GPU implementations fail, with up to 40× speedup.
  • Unified Scope: Together, these GPU-native innovations support dynamics-aware motion generation from single-arm manipulators to full humanoids across planning, reactive control, and retargeting.

2. Related Work

Prior motion-generation systems trade off scalability, dynamics, reactivity, collision handling, and extensibility. Existing perception and retargeting methods particularly struggle with dense distance fields and high-DoF robots.

  • Search- and sampling-based planners struggle with high-dimensional spaces or dynamics-compatible post-processing, while trajectory optimization jointly handles smoothness, collision avoidance, and custom costs.
  • Reactive approaches provide safety guarantees but can get stuck in local optima or require simplified geometric primitives, while learning-based methods distill or warm-start classical planners.
  • GPU acceleration has been applied to MPC and optimization, but existing frameworks target limited robot scales and high-DoF systems degrade convergence and increase compute time.
  • cuRobo custom costs are complex to implement, its MPPI mode lacks L-BFGS motion quality, and PyRoki is slow because of inefficient collision checking.
  • nvblox computes ESDF only in allocated blocks and incurs high memory overhead at millimeter resolution, whereas CPU methods lack real-time capability.
  • GMR uses per-frame local IK with joint-limit constraints, while PyRoki supports global collision-free IK but fails to converge on high-DoF robots.

3. Problem Formulation

The paper formulates motion generation as trajectory optimization from an initial state to one or more link goals. The objective combines smooth, efficient motion with collision, state, kinematic, and dynamic constraints.

  • Motion generation seeks actions that drive a robot from an initial state to one or more link goal poses while minimizing an objective and satisfying hardware constraints.
  • Each joint trajectory is represented through state variables containing angles, velocities, acceleration, and jerk, with the number of joints defining the state dimension.
  • The formulation enforces smooth and efficient motion, joint and derivative limits, scene and self-collision avoidance, terminal pose accuracy, torque limits, and a complete stop.
  • The formulation extends prior trajectory-optimization methods and subsumes local and global collision-free motion-planning variants.
  • The optimization loop generates B-spline waypoints, computes kinematics and inverse dynamics, evaluates costs in parallel, aggregates them, backpropagates gradients, and updates control points.

4. B-Spline Basis as Optimization Space

cuRoboV2 uses uniform cubic B-splines as a smooth, compact optimization space. Their local support reduces decision variables, preserves conditioning, and enables efficient GPU gradient accumulation.

  • Uniform cubic B-splines represent each joint trajectory through control points that serve as optimization variables.
  • Cubic and higher-degree B-splines provide C2-continuous trajectories, while local support limits each control point’s influence to neighboring segments.
  • Far fewer decision variables than per-timestep position parameterization preserve smoothness regularization and keep the optimization problem well-conditioned.
  • Uniformly sampled spline segments provide consistent timesteps for computing velocity, acceleration, jerk, and the trajectory state.
  • Gradients: The backward pass applies the transpose of the spline forward mapping, and each control point’s gradient sums contributions from all relevant interpolation points.
  • Gradients: GPU warp-level reduction fuses gradient accumulation with computation and avoids intermediate global-memory writes.
  • Boundary Conditions: Boundary states are satisfied implicitly: repeated final knots enforce static stopping, while ghost control points anchor non-static initial states.

5. ESDF for Scene Collision Avoidance

cuRoboV2 represents depth and known geometry with a block-sparse TSDF, then generates a dense ESDF on demand for collision queries across the workspace. Its GPU pipeline combines voxel-centric integration, adaptive storage, site seeding, and distance propagation while supporting dynamic updates and task-appropriate resolutions.

  • Representation: A two-stage representation fuses depth and analytic primitives into a persistent block-sparse TSDF, then produces a dense ESDF on demand for collision avoidance.The TSDF handles persistent world modeling, while the ESDF supplies dense distance queries at task-appropriate resolution.
  • Storage and updates: Only blocks near observed surfaces are allocated, while hash-based indexing, recycled blocks, and decayed weights bound memory for dynamic scenes.Blocks below a weight threshold are tombstoned and returned to a free list for reuse.
  • Geometry fusion: The TSDF stores independent depth and geometry signed-distance channels, returning their minimum so observed surfaces and known primitives jointly support collision avoidance.Depth channels use weighted averages, while analytic geometry is stored separately.
  • Depth integration: Voxel-Project integration assigns one thread per visible voxel to project into the depth image and write signed distances without atomic operations.The four phases are block discovery, deduplication, block allocation, and voxel-centric integration.
  • Resolution: Task-appropriate ESDF resolution enables O(1) trilinear distance queries while separating fine TSDF integration from coarser ESDF computation.The paper motivates this decoupling because collision-sphere approximation limits the value of excessively fine ESDF resolution.
  • ESDF generation: Dense ESDF generation seeds surface sites from TSDF zero-crossings, propagates nearest-site assignments with PBA+, and resolves signs beyond the truncation band.Gather-based seeding avoids atomics and uses a fixed work dimension compatible with CUDA graph capture.

6. Scaling to High-DoF Robots

cuRoboV2 addresses high-DoF scalability with topology-aware kinematics, sparse Jacobians, differentiable inverse dynamics, and GPU-native self-collision computation. These components adapt execution to robot complexity and support collision-free IK for branching and mechanically coupled systems.

  • Challenges: High-DoF robots introduce branching kinematic trees, quadratic self-collision pairs, torque-limit enforcement, and mechanically coupled mimic joints.These challenges complicate parallel Jacobian computation, collision checking, and dynamics-aware optimization.
  • System design: Five GPU-native innovations combine adaptive forward-kinematics dispatch, topology caching, sparse Jacobian filtering, map-reduce self-collision, and differentiable inverse dynamics.The design targets complex robots while preserving efficient execution for simpler systems.
  • Forward kinematics: Adaptive forward kinematics uses one fused kernel for simple robots and two kernels for complex robots, separating frame transforms from sphere and Jacobian computation.For simple robots with ≤100 collision spheres, the fused kernel computes frame transforms, tool poses, collision-sphere positions, centers of mass, and Jacobians.
  • Kinematics and Jacobians: Topology caches provide O(1) ancestor lookups, while two-stage filtering retains only Jacobian contributions from joints and mechanically coupled links relevant to the target.The approach exploits kinematic sparsity and handles mimic joints by mapping gradients to actuated joints.
  • Inverse kinematics: Collision-free IK first uses Levenberg-Marquardt to reach the target pose, then L-BFGS refines the solution under self-collision and environment constraints.LM is suited to the nonlinear least-squares pose objective, while L-BFGS performs collision-resolution refinement from the near-goal seed.
  • Self-collision: Map-reduce self-collision partitions collision pairs across GPU blocks and reduces local maxima to find the globally most-penetrating pair.This replaces serialized pairwise processing with a two-stage GPU reduction for complex robots.
  • Dynamics: Differentiable inverse dynamics computes torques from q, q̇, and q̈ within optimization so actuator limits are enforced directly rather than checked post hoc.The method uses RNEA over the kinematic tree to support physically executable trajectories.

11 else

The GPU-native dynamics implementation is designed for robot-generic execution, efficient memory use, and differentiable optimization. Its RNEA kernels provide substantial speedups while retaining runtime payload changes, PyTorch interoperability, and humanoid-scale support.

  • Self-collision execution: Two-kernel self-collision execution is selected when collision pairs exceed 1000, distributing pair processing across GPU blocks before global reduction.The design targets the quadratic pair growth that makes single-block processing inefficient for high-DoF robots.
  • RNEA design: Robot-generic RNEA kernels accept runtime topology, spatial inertias, joint types, degrees of freedom, and batch size, so one compiled binary supports any URDF-described robot.The implementation uses compact spatial representations, VJP-based backpropagation, and tree-level warp parallelism.
  • Memory representation: Factored transforms and compact spatial inertias reduce storage, replacing full 6×6 representations with analytically computed products.Transforms use rotation and translation factors, while spatial inertias use a 12-float representation.
  • Differentiable dynamics: The VJP backward pass computes gradients with respect to q, q̇, and q̈ in O(n) time without materializing O(n^2) torque Jacobians.It also propagates gradients through external contact or interaction forces when those forces are provided.
  • Performance: 14× speedup over Newton is achieved while remaining within 1.5–2× of GRiD at trajectory-optimization batch sizes.The implementation additionally supports runtime payload changes, PyTorch interoperability, and humanoid-scale robots.
  • End-to-end cost: RNEA accounts for 24% of trajectory optimization time and 29% of total motion planning time, keeping dynamics-aware planning overhead modest.These proportions quantify the end-to-end cost of the dynamics component in the reported system.

7. Experimental Results

Across planning, kinematics, perception, and retargeting benchmarks, cuRoboV2 maintains feasible, collision-free performance while scaling to high-DoF robots. Its advantages include dynamics-aware trajectories, efficient GPU computation, and better downstream policy quality, although MPC trades tracking accuracy for inter-frame collision checking.

  • Dynamics-Aware Planning: 99.7% success under full 3 kg payload exceeds cuRobo’s 77.1% and sampling-based methods’ 72–75%.Without payload, cuRoboV2 maintains 99.7% while cuRobo and sampling-based methods decline to 97.9% and 95.5–95.8%.
  • Trajectory Quality: 106 J energy at the 75th percentile is lower than cuRobo’s 116 J and VAMP’s 131–160 J.B-spline optimization produces smooth trajectories, with 48 ms planning time versus 36 ms for cuRobo.
  • High-DoF Computation: 18× faster than Newton on the 48-DoF humanoid, cuRoboV2 runs at 96.2 μs while GRiD fails from shared-memory limits.Code-generated GRiD kernels are faster on simple arms but do not scale to the humanoid.
  • Collision-Free IK: 99.6% self-collision-free humanoid IK success contrasts with 0% for both cuRobo and PyRoki.Levenberg-Marquardt pose seeding followed by L-BFGS collision-constrained refinement achieves 2.4 μm position error in 533 ms.
  • Motion Retargeting: 89.5% retargeting constraint satisfaction exceeds PyRoki’s 61.2%, mink’s 54.5%, and GMR’s 40.6%.MPC reaches 96.6% by checking collisions between frames, but its pose-tracking accuracy is worse than cuRoboV2-IK’s.
  • Locomotion Policy Training: 145.5 mm overall MPJPE, 99.6% alive rate, and 1.1 resets make cuRoboV2 the strongest overall downstream policy result.On crawling, cuRoboV2 reaches 151.4±4.5 mm MPJPE, while GMR has 12× higher cross-seed variance.

8. LLM-assisted Development

cuRoboV2’s discoverable architecture supported substantial LLM assistance in developing new robotics modules, while human judgment remained necessary for interpreting GPU compiler behavior.

  • Codebase Discoverability: A discoverable codebase enabled an LLM coding assistant to author components ranging from API wrappers to hand-optimized CUDA kernels.The redesign emphasized typed interfaces, predictable naming, small modules, and executable documentation.
  • Cost of the Investment: Framework code grew 1.5×, inline documentation 2.7×, and the test suite expanded from 264 to 3,978 tests.Files increased from 121 to 390 as the codebase decomposed into single-responsibility modules.
  • LLM Contributions: 73% of additions in N2 were LLM-authored, after contributions rose from 6–18% during early refactoring to 50% in N1.The later phases developed new modules after the codebase had been restructured for discoverability.
  • Human–LLM Workflows: The RNEA workflow used a human-written NumPy reference while the LLM derived, tested, translated, and optimized differentiable CUDA kernels.Humans suggested optimizations, and the LLM profiled and implemented them iteratively.
  • Human Oversight: The LLM misread virtual versus physical register pressure and compiler lowering, so human intervention was required for some optimization judgments.These episodes exposed limits in interpreting intermediate compiler representations rather than in generating or profiling code.
  • Scene Collision Migration: An LLM-assisted migration replaced split collision backends and wrappers with a unified Warp system at roughly half the original 5,000 lines.The generalized kernel supported cuboids, meshes, voxels, and ESDF grids.

9. Conclusion

cuRoboV2 combines dynamics-aware trajectory optimization, dense GPU-native perception, scalable whole-body computation, and a more accessible software architecture. The resulting stack supports motion generation from single-arm robots to humanoids while exposing practical deployment tools and documented areas for future work.

  • Unified Motion Generation: GPU-native computation makes it practical to enforce smoothness, torque, collision, and whole-body constraints across motion-generation tasks.The paper describes these capabilities as compounding across trajectory optimization, perception, and scalable robot computation.
  • Unified Motion Generation: The framework scales collision-free motion optimization from 7-DoF arms to 48-DoF humanoids through B-splines, fast RNEA, dense ESDFs, and scalable self-collision.These components connect smooth trajectories, tractable torque evaluation, depth-driven collision avoidance, and whole-body optimization.
  • Software Architecture: Substantial LLM assistance contributed to RNEA CUDA kernels, unified scene collision, and PBA ESDF modules, enabled by typed interfaces, predictable naming, small modules, and tests.The conclusion presents discoverability as necessary for this style of LLM-assisted development.
  • Limitations and Future Work: Future work includes multi-camera fusion, learned RGB-based segmentation, global-plan warm starts for MPC, and reducing human dependence in compiler-representation analysis.The paper specifically identifies geometric segmentation sensitivity and single-camera scene coverage as limitations.
  • Deployment Improvements: cuRoboV2 reduces installation from roughly 20 minutes to seconds by compiling and caching CUDA Python kernels at first use without PyTorch-version matching.The kernels automatically target the host GPU and cache results by source-content hash.
  • Deployment Improvements: RobotBuilder converts URDFs into configurations with fitted collision spheres and automatically constructed self-collision ignore matrices.Sampling-based pruning evaluates joint configurations on the GPU to identify link pairs that never collide.

A.2. Software Architecture

cuRoboV2 replaces monolithic, hard-to-extend components with composable interfaces for obstacle types and costs, while enforcing consistent naming, documentation, and software-design conventions. These architectural choices support extension without modifying core optimization or collision code.

  • Extension Points: Users can add custom obstacle types and cost functions without modifying cuRoboV2’s core code.The architecture exposes extension points for both obstacle representations and cost terms.
  • Scene Collision: A single type-generic Warp kernel handles cuboids, meshes, voxels, and ESDF grids through three obstacle-specific interface functions.New representations require implementations of enablement, transform loading, and local signed-distance computation.
  • Cost Manager: Each standalone BaseCost subclass is registered by name, evaluated on its own CUDA stream, and differentiated through PyTorch backpropagation.Adding a cost such as a workspace or learned cost requires only a new subclass.
  • Kernel Organization: Mature algorithms remain hand-tuned CUDA C++ kernels, while user-modifiable cost functions are implemented as Warp kernels.The split prioritizes stability for infrequently changing components and modifiability for customization points.
  • Development Conventions: Persistent rule files encode formatting, naming, documentation, and design principles so LLM-generated code follows the project’s conventions.The rules include type hints, docstrings, shape documentation, category-first filenames, and software-design principles.
Loading 2603.05493v2…