PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Differentiable Simulation of Articulated Bodies

Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

arXiv:2109.07719v1cs.LGcs.GRcs.RO

TL;DR

Articulated-body simulation lacks efficient gradients for learning and optimization because autodiff incurs large computational and memory costs. The paper derives spatial-algebra adjoints and uses state checkpointing, achieving 10x faster differentiation with 100x lower memory, while accelerating reinforcement learning and inverse-problem optimization.

  • Problem

    Autodiff-based differentiation of articulated simulation is slowed by serialized dynamics and consumes prohibitive memory from long computation graphs.

  • Method

    The method derives adjoints for articulated dynamics and collision solving, while checkpointing initial states and reconstructing intermediates during backpropagation.

  • Results

    10x faster runtime and 100x smaller memory footprint are achieved versus existing autodiff tools, while reinforcement learning and inverse-problem optimization are accelerated.

  • Takeaways & Limitations

    Differentiable articulated dynamics supports faster reinforcement learning and gradient-based motion control and parameter estimation.

  • Takeaways & Limitations

    The simulator assumes every link is rigid and currently uses an iterative contact solver that can sacrifice accuracy for speed.

Abstract

from arXiv · show

We present a method for efficient differentiable simulation of articulated bodies. This enables integration of articulated body dynamics into deep learning frameworks, and gradient-based optimization of neural networks that operate on articulated bodies. We derive the gradients of the forward dynamics using spatial algebra and the adjoint method. Our approach is an order of magnitude faster than autodiff tools. By only saving the initial states throughout the simulation process, our method reduces memory requirements by two orders of magnitude. We demonstrate the utility of efficient differentiable dynamics for articulated bodies in a variety of applications. We show that reinforcement learning with articulated systems can be accelerated using gradients provided by our method. In applications to control and inverse problems, gradient-based optimization enabled by our work accelerates convergence by more than an order of magnitude.

1. Introduction

The paper targets efficient differentiation of articulated-body simulation, where missing analytic derivatives, long serialized computations, and autodiff memory overhead hinder gradient-based optimization. It derives adjoint gradients and adapts checkpointing, achieving faster computation and substantially lower memory use while supporting reinforcement learning and inverse problems.

  • Motivation: Articulated-body systems lack analytic dynamics derivatives, forcing gradient-based methods to compute them indirectly.
  • Motivation: Autodiff becomes impractical because articulated dynamics and iterative contact solvers create long computation graphs whose intermediate results consume prohibitive memory.The overhead of creating and storing autodiff variables also slows forward simulation.
  • Contributions: The proposed method derives gradients for the full articulated simulation workflow using the adjoint method and spatial algebra.
  • Contributions: Checkpointing stores each time step’s initial state and reconstructs intermediate variables during backpropagation, reducing memory consumption by two orders of magnitude.
  • Applications: Gradient information improves reinforcement-learning convergence and reward, while gradient-based control and parameter estimation accelerate convergence by more than an order of magnitude.

2. Related Work

Prior differentiable-physics work spans neural approximations, autodiff systems, and applications, but articulated-body differentiation remains costly and physically constrained. This paper adapts adjoints and checkpointing specifically to articulated dynamics.

  • Related approaches: Related work includes neural approximations of physics and differentiable applications in rendering, imaging, SLAM, design, control, and system identification.
  • Automatic differentiation: DiffTaichi and JAX MD use autodiff for high-performance simulation, particularly benefiting vectorizable systems such as fluids, deformable solids, or molecular dynamics.
  • Differentiable physics: Existing autodiff frameworks introduce substantial tracing, computation, and memory overhead when applied to articulated bodies.
  • Checkpointing: The paper adapts checkpointing to articulated dynamics, achieving dramatic memory reductions and enabling stable simulation of long experiences.
  • Adjoint method: Unlike Geilinger et al. (2020), which uses maximal coordinates and spring attachments, this work operates in reduced coordinates to preserve physical validity.

3. Preliminaries

The preliminaries formulate articulated dynamics as a time-coupled constraint system and motivate adjoint differentiation instead of explicitly forming the full state-to-control Jacobian. Local temporal sparsity makes the adjoint system tractable.

  • Articulated body dynamics: The simulator uses the recursive Articulated Body Algorithm, which has O(n) complexity and is widely used for articulated-body simulation.
  • Problem formulation: States and controls are concatenated across simulation steps, and the dynamics are represented as a global constraint f(x, u) = 0.
  • Adjoint formulation: Directly computing dx/du is prohibitively expensive because it forms a 2n_qn_t × n_un_t matrix.
  • Adjoint formulation: Differentiating the dynamics constraints yields an adjoint system whose sparse temporal structure facilitates solving for the adjoint variable.
  • Adjoint formulation: The adjoint of a variable is equivalent to its gradient under the paper’s substitution notation.

4. Efficient Differentiation

Efficient differentiation reverses the articulated simulation pipeline with adjoint operators and stores only compact per-step states. This avoids retaining the many intermediates produced by serialized dynamics and collision processing.

  • 4.1. Adjoint Method for Articulated Dynamics: Backpropagation reverses five simulation modules: kinematics, force updates, acceleration updates, collision handling, and time integration.
  • 4.1. Adjoint Method for Articulated Dynamics: Adjoint computation propagates gradients backward through time integration, collision resolution, and forward dynamics after reconstructing each step’s intermediates.
  • 4.1. Adjoint Method for Articulated Dynamics: The reverse projected Gauss-Seidel solver computes gradients for an iterative MLCP collision solve whose terminated solution may violate constraints.Implicit differentiation is unsuitable when the iterative solver does not guarantee constraint satisfaction.
  • 4.1. Adjoint Method for Articulated Dynamics: Each spatial-algebra forward operator receives a corresponding adjoint rule with the same time and space complexity as the original operation.
  • 4.2. Checkpointing for Articulated Dynamics: Many intermediate values make differentiable simulation memory-intensive over long runs, especially with small time steps required for accuracy and stability.
  • 4.2. Checkpointing for Articulated Dynamics: Checkpointing stores only each step’s initial state, frees intermediate results after the forward pass, and recomputes them during backpropagation.The chosen checkpoint granularity retains the smallest information sufficient to replicate each simulation step.

5. Reinforcement Learning

The paper uses computed dynamics gradients to enhance reinforcement learning through additional samples or physics-aware policy updates. These approaches support faster convergence, while the simulator also provides substantially lower memory use and faster forward simulation than autodiff baselines.

  • Sample enhancement: Sample enhancement generates approximately accurate neighboring transitions from first-order gradient approximations, giving the critic patchwise information that can accelerate learning and convergence.Random perturbations of the action produce additional next observations and rewards around an existing sample.
  • Simulation efficiency: The framework's peak memory footprint is more than two orders of magnitude lower than autodiff methods, while PyTorch crashes at 5,000 simulation steps.ADF is excluded because it fails to compute gradients within 10 minutes.
  • Simulation efficiency: The proposed simulator is at least an order of magnitude faster than autodiff tools for forward simulation, with CppAD the fastest baseline.The comparison is reported in milliseconds, and PyTorch crashes at 5,000 simulation steps.
  • Policy enhancement: Physics-aware policy enhancement embeds ground-truth action gradients into the policy loss, producing generally more accurate updates than relying on the value function alone and potentially higher reward.The method expands the value function one step forward and substitutes it into the policy objective, yielding the same action gradients while incorporating physics information.

6. Results

The method substantially improves differentiable articulated-body simulation efficiency and supports faster learning and gradient-based optimization across articulated control tasks.

  • Comparison with Autodiff Tools: Autodiff tools consume orders-of-magnitude more memory, with PyTorch crashing at 5,000 steps because of memory overflow.The comparison includes one Laikago robot over 50 to 5,000 simulation steps.
  • Comparison with Autodiff Tools: Our method is fastest for backward computation, while its performance gap over CppAD grows with simulation length in ten-robot scenes.Its backward pass reconstructs intermediate variables through forward simulation, yet remains much faster than CppAD.
  • Integration with Reinforcement Learning: For the n-link pendulum, our model reaches close to maximal reward across link counts, whereas MBPO degrades from four links onward.MBPO does not attain satisfactory results for six- and seven-link systems.
  • Integration with Reinforcement Learning: On MuJoCo Ant, generating nine extra samples per true sample with our differentiable simulator produces faster convergence than MBPO.The experiment repeats training four times, and the nearby samples improve the critic's local value-function estimation.
  • Motion Control: Gradient-based SGD reaches target accuracy in 20 and 50 steps for two motion-control tasks, while CMA-ES requires more than 500 and 1,500 steps without matching it.The tasks optimize torque vectors for throwing a ball with a 9-DoF arm and hitting a golf ball with an articulated linkage.

7. Conclusion

The paper develops an efficient differentiable simulator for articulated-body dynamics and demonstrates its utility for reinforcement learning, motion control, and parameter estimation.

  • Conclusion: The simulator runs 10x faster with a 100x smaller memory footprint than existing autodiff tools.The method derives adjoints for spatial algebra and the Gauss-Seidel solver and adapts checkpointing to articulated simulation.
  • Conclusion: Differentiable physics preliminarily accelerates reinforcement learning, while gradient-based optimization is effective for articulated motion control and parameter estimation.The conclusion frames these as demonstrated application scenarios rather than universal guarantees.

A. Discussion of RL Applications

The paper applies differentiable physics gradients to accelerate reinforcement learning, while identifying task-specific limitations in sample and policy enhancement.

  • In n-link pendulum experiments, the method maintained performance as system complexity increased, unlike MBPO, whose performance degraded.The policy can first optimize short-term reward gradients before the critic accurately estimates long-term values.
  • Faster convergence than MBPO was observed in the MuJoCo Ant experiment because nearby samples help the critic estimate values over a local patch.The paper reports that this improves value estimation and eventually the policy model.
  • Sample enhancement did not improve overall pendulum rewards, while policy enhancement did not accelerate convergence in the ant test.
  • Sample enhancement may not escape local minima, whereas policy enhancement assumes training-batch actions resemble current policy actions and has poor sample efficiency.
  • The techniques cannot be combined because generated sample-enhancement points lack accurate first-order gradients at their locations.

B. Ablation Study for Checkpointing Scheme

The checkpointing ablation compares memory and backpropagation time across checkpoint intervals and supports storing a checkpoint at every simulation step.

  • Checkpointing stores initial states at selected time steps and reconstructs intermediate variables by replaying forward simulation during backpropagation.
  • With interval k, average backpropagation time is (k + 1)/2 · tforawrd + tbackward, while peak memory is n/k · mcheckpoint + msimulation.
  • Because msimulation >> mcheckpoint, increasing the checkpoint interval does not significantly reduce memory but slows backpropagation linearly.
  • Larger checkpoint intervals reduce memory gradually but increase average backpropagation time linearly, so the method selects interval k = 1.The experiment simulates 10 Laikago robots for 5,000 steps, and memory remains low enough at k = 1.
  • The collision solver formulates an MLCP and uses projected Gauss-Seidel in the forward pass, while the adjoint computation propagates gradients from x to A and b.
  • The articulated-body simulator has MuJoCo-like physical and contact modeling, and a policy trained in it achieved non-trivial reward in MuJoCo without retraining.

E. Other Limitations and Future Directions

The paper identifies contact-model accuracy and rigid-body assumptions as boundaries of the current simulator, while outlining broader tooling and real-robot extensions. Profiling also shows ADF becomes impractical for deep articulated simulations.

  • Other Limitations and Future Directions: The iterative contact solver is fast but can be insufficiently accurate, motivating exact LCP or convex soft-contact models.The authors propose additional contact models to broaden the simulator’s applications.
  • Other Limitations and Future Directions: The simulator assumes every link is rigid, leaving deformable objects such as cloth for future extension.
  • Other Limitations and Future Directions: The authors plan more comprehensive tools, deeper reinforcement-learning sampling and optimization studies, and deployment on real-world robots.
  • Code for Profiling Autodiff Tools: ADF becomes impractical for articulated simulations because backpropagation slows sharply as the computation graph deepens, already taking a long time at 28 iterations.Articulated-body simulations have far greater computational depth than this profiling example.

G. Differentiation of Articulated Body Algorithm

The method manually differentiates articulated-body operations and contact resolution using adjoints, while experiments examine computational cost, reinforcement-learning enhancements, and optimization behavior. Its advantage depends on handling serialized articulated dynamics and the contact solver directly.

  • Differentiation of Articulated Body Algorithm: The derivation manually differentiates spatial-algebra operators and the articulated-body algorithm, including transforms, motion cross products, dyads, and quaternion operations.The section presents adjoints for the basic operators used in the implementation.
  • Differentiation of Articulated Body Algorithm: Gradient-based methods find the global minimum in the friction-coefficient loss landscape, supporting parameter estimation in the demonstrated setting.
  • Differentiation of Articulated Body Algorithm: DiffTaichi is not compared because its programming model does not readily express the articulated dynamics’ complex loops mixed with other statements.Implementing the articulated-body algorithm in DiffTaichi is left outside the paper’s scope.
  • Differentiation of Articulated Body Algorithm: 0.6 ms per step versus 2.9 ms for CppAD, 4.9 ms for JAX, 30.7 ms for Ceres, and 285.9 ms for PyTorch in one Laikago’s backward pass.The comparison uses 1 Laikago robot simulated for 50 steps and reports average backward time per step.
  • Differentiation of Articulated Body Algorithm: Sample enhancement generally converges faster but does not yield higher rewards, whereas policy enhancement is restricted to on-policy training.
Loading 2109.07719v1…