Source-linked AI summary
Learning Deployable Locomotion Control via Differentiable Simulation
Clemens Schwarke, Victor Klemm, Joshua Bagajo, Jean-Pierre Sleiman, Ignat Georgiev, Jesus Tordesillas, Marco Hutter
TL;DR
Contact discontinuities make analytic-gradient optimization difficult for locomotion, while soft contact models can sacrifice physical accuracy. This paper analytically smooths a hard-contact model to retain informative gradients and physical fidelity, then demonstrates zero-shot transfer of a policy learned entirely in a differentiable simulator to a real quadruped.
Problem
Contact discontinuities hinder FoG-based optimization for locomotion, while soft contact models can provide smooth gradients but often sacrifice physical accuracy.
Source: §1 · p. 2Method
The paper analytically smooths a hard-contact formulation to provide informative gradients while maintaining physical fidelity for deployment.
Source: §1 · p. 2, §3.3 · p. 4Results
The policy learned entirely in the differentiable simulator using analytic gradients transferred zero-shot to a real quadrupedal robot, the authors’ first such legged-locomotion transfer to their knowledge.
Source: §1 · p. 2Takeaways & Limitations
The successful transfer establishes the feasibility of using differentiable simulation to learn real-world locomotion control.
Source: §5 · p. 8FoG-based optimization with SHAC takes more wall-clock time per iteration than PPO, which might negate its sample-efficiency benefits in low-dimensional problems.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Differentiable simulators promise to improve sample efficiency in robot learning by providing analytic gradients of the system dynamics. Yet, their application to contact-rich tasks like locomotion is complicated by the inherently non-smooth nature of contact, impeding effective gradient-based optimization. Existing works thus often rely on soft contact models that provide smooth gradients but lack physical accuracy, constraining results to simulation. To address this limitation, we propose a differentiable contact model designed to provide informative gradients while maintaining high physical fidelity. We demonstrate the efficacy of our approach by training a quadrupedal locomotion policy within our differentiable simulator leveraging analytic gradients and successfully transferring the learned policy zero-shot to the real world. To the best of our knowledge, this represents the first successful sim-to-real transfer of a legged locomotion policy learned entirely within a differentiable simulator, establishing the feasibility of using differentiable simulation for real-world locomotion control.
1 Introduction
Analytic gradients may improve robot-learning efficiency, but contact discontinuities make them difficult to use for locomotion. The paper introduces an analytically smoothed contact model and demonstrates zero-shot transfer of a locomotion policy learned entirely in a differentiable simulator.
1 Introduction
- Contact discontinuities complicate FoG-based optimization for locomotion, while soft models can provide smooth gradients at the cost of physical accuracy.Hard contact offers higher fidelity but produces discontinuous, less informative gradients.Source: §1 · p. 2
- The proposed contact model is designed to combine informative optimization gradients with the physical fidelity needed for real-world deployment.It is analytically smoothed from a hard-contact formulation and compared with soft and hard contact alternatives.Source: §1 · p. 2
- A locomotion policy learned entirely in the differentiable simulator transferred zero-shot to a real quadrupedal robot, demonstrating the first such legged-locomotion transfer to the authors’ knowledge.The policy used analytic gradients from the proposed contact model and SHAC [12].Source: §1 · p. 2
- Analytic gradients are presented as a route to high-sample-efficiency learning, and the paper reports using them to learn legged locomotion in its differentiable simulator.The authors identify high sample efficiency with analytic-gradient learning as a contribution.Source: §1 · p. 2
2 Related Work
Prior work has explored ways to improve gradient quality around discontinuities and to integrate analytic gradients into learning. For locomotion, the trade-off between smooth gradients and physical fidelity has left transfer to hardware challenging.
2 Related Work
- Prior work has explored smoothing collision geometries, leaky gradients, randomized smoothing, analytically smoothed dynamics, and smoothed contact dynamics for model-predictive control.Other approaches target long-trajectory gradient divergence through concurrent controllers, per-step action optimization, or hybrid FoG/ZoG methods.Source: §2 · p. 2
- SHAC [12] integrates FoGs into an actor-critic framework and has demonstrated competitive performance on tasks including MuJoCo ant locomotion.The cited work is presented as a prior example of FoG-based learning.Source: §2 · p. 3
- A prior sim-to-real locomotion success [28] used a non-differentiable forward simulator and a separate simplified model for gradient computation, unlike this work’s single differentiable framework.The two-model approach avoids the direct gradient-quality and physical-realism trade-off within one simulator, but requires managing and aligning separate models.Source: §2 · p. 3
- Soft contact models provide smooth gradients but may lack physical accuracy, whereas hard contact models generally offer higher fidelity with less informative gradients.The paper positions its contact model as addressing this trade-off for sim-to-real transfer.Source: §2 · p. 3
3 Method
This method addresses contact-induced discontinuities that undermine gradient optimization by analytically smoothing hard contact while retaining informative gradients, then combines differentiable simulation with training choices for sim-to-real transfer.
3.1 The Challenge with Discontinuities
- Hard contact creates discontinuous dynamics, while stochastic sampling smooths objectives but yields biased gradients; discrete-time contact resolution can also invert gradient direction.The stochastic gradient estimate may be zero or misleading near discontinuities.Source: §3 · p. 3, §3.1 · p. 4
3.2 Differentiable Rigid-Body Simulation
- The differentiable simulator uses NVIDIA Warp, Moreau time stepping, and an adapted Gauss-Seidel solver to produce smooth dynamics.Warp supports automatic differentiation; the time-stepping scheme is described as more stable than semi-implicit Euler integration.Source: §3.2 · p. 4
3.3 Analytic Smoothing of Contact Dynamics
- The model’s smoothing can be reduced by increasing κ, but scheduling was unnecessary for the studied task, whose training and transfer were not sensitive to smoothing level.As κ grows, the model converges to hard contact.Source: §3.3 · p. 5
- Analytic smoothing mitigates contact discontinuities while providing informative, unbiased gradients, as shown against hard contact and stochastic smoothing.The model scales contact impulses using sigmoid functions of penetration depth, creating forces at a distance.Source: §3.3 · pp. 4–6, Figure 3 · p. 5
3.4 Considerations for Successful Learning and Sim-to-Real Transfer
- SHAC limits FoG backpropagation to short horizons, while the transfer setup uses a differentiable reward, fitted PD actuator model, and domain randomization.SHAC approximates long-term return with a value function trained using ZoGs.Source: §3.4 · p. 6
4 Experimental Results
The experiments compare contact models and learning algorithms, then evaluate velocity tracking for real-world deployment. Analytic smoothing supports transfer to hard contact, while SHAC improves sample efficiency over PPO and the learned policy transfers to a real quadruped.
4 Experimental Results
- The experiments progress from comparing contact models and SHAC against PPO to training an ANYmal velocity-tracking task.The initial quadruped task walks forward at a constant 1.0 m/s.Source: §4 · p. 6
4.1 Comparison between Contact Models
- Analytically smoothed contact achieves the highest performance in its training environment and after transfer to hard contact, with more natural and efficient gaits.Its return is approximately 10% higher than training with hard contact; soft-contact policies fail to transfer to hard contact, while hard-contact training produces erratic, unnatural walking.Source: §4.1 · p. 7
4.2 Comparison between RL Algorithms
- SHAC and PPO reach nearly identical final rewards, but SHAC uses over an order of magnitude fewer samples; PPO is faster per iteration at 0.18 s versus 0.33 s.SHAC's learning phase is slower because it backpropagates through the rollout.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3 Training a Velocity-Tracking Policy for Real-World Deployment
- The proposed method trains locomotion policies demonstrating zero-shot transfer and robust real-world execution.Figure 5 reports mean gradient norms during velocity-tracking training and final returns after 1,000 iterations across horizon lengths for both robots.Source: §4.3 · p. 8, Figure 5 · p. 8
5 Conclusion and Future Work
The paper concludes that analytic smoothing can support physically faithful contact modeling for differentiable locomotion learning and reports zero-shot transfer to a real quadruped. Future work would extend the approach to higher-dimensional systems and differentiable rough terrain.
5 Conclusion and Future Work
- Figure 6 compares target forward/backward walking velocities with measured velocities on the real robot and in simulation.Source: Figure 6 · p. 8
- The analytically smoothed contact model balances informative gradients with physical fidelity, and the learned locomotion skill transfers zero-shot from the differentiable simulator to a real quadruped.The authors describe this as validating the feasibility of FoG-based optimization for contact-rich tasks.Source: §5 · p. 8
- Future work could scale the approach to humanoids or image observations and incorporate differentiable rough terrain.The authors identify these directions as ways to leverage sample efficiency and improve competitiveness with current RL methods.Source: §5 · p. 8
6 Limitations
The approach has computational, actuator-modeling, and differentiability constraints that may affect its suitability across tasks.
6 Limitations
- SHAC's sample efficiency comes with higher wall-clock time per iteration than PPO, which may outweigh the benefit for low-dimensional optimization problems.Source: §6 · p. 9
- A simple PD actuator model was used for tractability instead of a learned network, potentially reducing fidelity for tasks requiring high actuator precision.Source: §6 · p. 9
- Full differentiability requires additional effort in simulation design or reward engineering compared with ZoG methods.Source: §6 · p. 9
Appendix
The appendix provides additional details on the simulation framework, experimental setups, and supporting results.
Appendix
- The appendix covers simulation-framework details, experimental setups, and supporting results.Source: Appendix · p. 12
A.1 Simulation Implementation
The simulator advances rigid-body dynamics with Moreau time stepping and resolves contact through iterative impulse updates. Fixing the solver iteration count makes the contact routine differentiable in Warp, while a soft-contact baseline trades physical accuracy against stability and gradient smoothness.
A.1 Simulation Implementation
- Moreau’s scheme starts with an explicit Euler half-step to the midpoint, computes dynamics and contact impulses, then updates positions with a trapezoidal rule.RNEA computes inverse dynamics and CRBA constructs the inertia matrix; the final state feeds the next simulation step.Source: §A.1 · pp. 12, 14
- The contact solver projects the equations of motion into contact coordinates and directly solves for impulses that satisfy contact constraints at the step’s end.The contact Jacobian relates generalized-coordinate variations to local contact-coordinate variations; it includes all active contacts.Source: §A.1 · p. 12
- The Delassus matrix G expresses inverse inertia in contact coordinates and is computed using factors of H rather than explicitly forming H^-1.Explicit inversion is avoided for its computational cost and potential numerical instability.Source: §A.1 · p. 13
- Gauss-Seidel updates sequentially adjust each contact impulse to reduce constraint violation, then proximal projection keeps contact forces within the friction cone.Unlike, this work does not set tangential impulses to zero, yielding more accurate results with a limited number of iterations.Source: §A.1 · pp. 13–14
- Warp differentiates the unrolled Gauss-Seidel solver by fixing its iteration count at N = 10, avoiding hand-derived analytical sensitivities.The approach is presented as simpler to implement than alternatives based on the implicit function theorem.Source: §A.1 · p. 14
- The soft-contact baseline computes forces directly, with kp, kd, and kf tuned to trade physical accuracy against simulation stability and gradient smoothness.Tangential friction depends on relative velocity in the contact frame and friction coefficient µ.Source: §A.1 · p. 14
A.2 Environment Setup for Quadrupedal Locomotion
The quadruped environment trains forward walking at a constant target velocity of 1.0 m/s. Joint position commands are tracked with a PD law, and episodes end on a low base height or at the time limit.
A.2 Environment Setup for Quadrupedal Locomotion
- The quadruped is trained to walk forward at a constant velocity of 1.0 m/s in an environment adapted for comparability and reproducibility.The setup is closely related to the ant environment studied in [12].Source: §A.2 · p. 14
- The policy receives the observations used in [12] and outputs joint position commands tracked by a PD law, with actuator torque capped at 20 Nm.The observation formulation is listed for completeness in Tab. 5.Source: §A.2 · p. 14
- Episodes terminate when the quadruped’s base center falls below 0.25 m or the 10 s maximum episode length is reached.Reward and training parameters are specified in Tabs. 4 and 6.Source: §A.2 · p. 14
A.3 Environment Setup for Velocity-Tracking with ANYmal
The ANYmal velocity-tracking setup replaces a non-differentiable foot-swing reward with differentiable foot-height tracking. Domain randomization and real-data fitting of actuator parameters are also used to support sim-to-real transfer.
A.3 Environment Setup for Velocity-Tracking with ANYmal
- A sinusoidal foot-height reference, phase-shifted by π for alternating feet, replaces the non-differentiable long-swing reward from [2].The differentiable reward significantly enhances transfer to ground that is not perfectly flat, and can be removed for rough-terrain training.Source: §A.3 · p. 15
- The foot-height reward is an addition to the velocity-tracking task’s reward and observation formulations, detailed in Tabs. 7 and 8.The task remains learnable without this reward.Source: §A.3 · p. 15
- Domain randomization varies observation noise, robot mass, friction, and base velocity, while real-world data fit the PD tracking and joint-armature parameters.Velocity commands are sampled per environment, and the randomized values are uniformly sampled from ranges in Tab. 9.Source: §A.3 · p. 15
A.4 Additional Experimental Results
The additional experiments report close agreement between simulated and real ANYmal joint trajectories. They also show that SHAC’s backpropagation increases iteration time, and relate robot-property differences and contact smoothing to locomotion behavior.
A.4 Additional Experimental Results
- SHAC uses fewer samples per iteration than PPO, but backpropagation makes its total training iteration longer despite faster rollouts.The benefit of FoG-based methods is expected to matter in higher-dimensional settings where sample generation is a bottleneck.Source: §A.4 · p. 15, Table 2 · p. 15
- ANYmal’s base-to-shank mass ratio is over an order of magnitude higher than the Warp Quadruped and MuJoCo Ant, and its higher actuator torque contributes to less smooth dynamics.The comparison motivates evaluation on physically realistic models and real robots.Source: §A.4 · p. 15, Table 3 · p. 16
- Smoothed-contact training yields smoother, more periodic joint trajectories and smaller joint motion than hard-contact training evaluated under hard contact.The smaller motion contributes to the higher return reported in Tab. 1.Source: §A.4 · pp. 15–16, Figure 7 · p. 16
- ANYmal’s simulated joint trajectories closely match real-world execution, while smoothed- and hard-contact simulations are also highly similar.The authors suggest the remaining sim-to-real gap mainly reflects factors beyond contact modeling, such as actuator dynamics or system parameters.Source: Figure 8 · p. 16, §A.4 · p. 16