Source-linked AI summary
A Differentiable Physics Engine for Deep Learning in Robotics
Jonas Degrave, Michiel Hermans, Joni Dambre, Francis wyffels
TL;DR
Robotic controller optimization often relies on derivative-free methods because robots and existing engines are treated as non-differentiable, limiting efficient optimization of many parameters. The paper introduces a differentiable 3D rigid-body physics engine with analytical gradients, showing faster optimization and applications to neural-network controllers while identifying implementation and sim-to-real limitations.
Problem
Robotic controller optimization commonly treats robots as non-differentiable black boxes, making derivative-free methods prevalent and limiting optimization speed and controller parameter counts.
Method
The paper implements a 3D rigid-body physics engine that differentiates control parameters through BPTT and optimizes neural-network policy weights with stochastic gradient descent.
Results
The engine makes gradient evaluation tractable and speeds optimization even for six-parameter problems; it also supports GPU batches, differentiated cameras, and neural-network controllers.
Takeaways & Limitations
Differentiable physics can support gradient-based robotics optimization and joint learning of computer vision and control policies.
Takeaways & Limitations
The implementation sacrifices capabilities because GPU branching was unavailable, restricting contact constraints mainly to spheres and sphere–ground interactions; model gradients may also differ from physical-system gradients.
Abstract
from arXiv · showhide
An important field in robotics is the optimization of controllers. Currently, robots are often treated as a black box in this optimization process, which is the reason why derivative-free optimization methods such as evolutionary algorithms or reinforcement learning are omnipresent. When gradient-based methods are used, models are kept small or rely on finite difference approximations for the Jacobian. This method quickly grows expensive with increasing numbers of parameters, such as found in deep learning. We propose the implementation of a modern physics engine, which can differentiate control parameters. This engine is implemented for both CPU and GPU. Firstly, this paper shows how such an engine speeds up the optimization process, even for small problems. Furthermore, it explains why this is an alternative approach to deep Q-learning, for using deep learning in robotics. Finally, we argue that this is a big step for deep learning in robotics, as it opens up new possibilities to optimize robots, both in hardware and software.
1 Introduction
Robotic controller optimization commonly treats the robot as a non-differentiable black box, favoring derivative-free methods. The paper proposes differentiable physics engines to make gradient-based optimization tractable for modern robotic models.
- Derivative-free methods—including particle swarms, reinforcement learning, genetic algorithms, and evolutionary strategies—are commonly used to optimize robotic controllers.
- Deep Q-learning requires many evaluations, while gradient optimization can be faster and more efficient when controllers have many parameters.
- Robots are usually treated as non-differentiable black boxes because hardware and current physics engines cannot provide gradients for the complete system.
- Neural-network surrogate gradients can address the problem, but they tend to poorly approximate gradients of the original system.
- Recent physics engines can differentiate action-to-state transitions but cannot compute derivatives with respect to model parameters.
- The paper introduces analytical gradients for modern 3D rigid-body engines and tests whether these gradients are differentiable and computationally tractable.
2 Materials and methods
The paper implements a differentiable 3D rigid-body physics engine whose analytical gradients support gradient-based optimization of robot parameters and neural-network controllers.
- 2.1 A 3D Rigid Body Engine: The engine differentiates robot-model parameters with respect to simulation fitness, enabling gradient-descent optimization.It targets modern 3D rigid-body simulation rather than spring-damper models.
- 2.1 A 3D Rigid Body Engine: The physics engine is implemented as a Theano mathematical expression whose computational graph is compiled for CPU and GPU evaluation and differentiation.The implementation avoids branching because of limited GPU support for conditionals, restricting contact constraints to spheres and sphere–ground interactions.
- 2.1 A 3D Rigid Body Engine: Rotation matrices replace quaternions, reducing graph complexity through matrix multiplications while requiring renormalization to maintain orthogonality.The update uses A′ = 3A − A ◦ (A · A), where A′ is the renormalized rotation matrix.
- 2.1.1 Throwing a Ball: The ball experiment optimizes initial linear and angular velocities by backpropagating final simulation errors through time.Each simulation timestep is treated like a recurrent-neural-network pass, and the unfolded sequence supplies gradients to the initial parameters.
- 2.1.1 Throwing a Ball: 88 iterations and 16.3 s with gradient descent and backpropagation through time beat CMA-ES at 2422 iterations and 59.9 s for the six-parameter ball problem.The result demonstrates an optimization advantage even with only six parameters.
- 2.2 Policy Search: For policy search, a recurrent deep neural controller maps sensor observations, hidden state, and task parameters to actions, with weights optimized using automatically differentiated dL/dW.The complete physics-controller system is optimized as an RNN unfolded over time, requiring sufficient samples for generalization.
3 Results
The experiments apply differentiable physics and backpropagation through time to robot-control tasks, including quadrupedal locomotion, arm reaching, and camera-based pendulum control. Results show fast optimization and GPU scalability, while identifying objective and modeling constraints.
- Quadrupedal Robot: Computing Speed: The engine computes the quadruped controller’s traveled-distance Jacobian by backpropagating through 10 seconds of simulation.Both forward-only and forward-plus-backward computation times are measured for comparison with conventional physics engines.
- Quadrupedal Robot: Computing Speed: Backpropagation slows physics computation by roughly 10×, but this factor barely increases with controller parameter count; batching yields significant GPU acceleration.A single GPU can simulate about 864 000 model seconds per day, or 86 400 000 model states.
- Reaching a Fixed Point: For fixed-point reaching, the controller comes within 4 cm of the target in 100 evaluations and within 1 cm in 150 evaluations.The controller uses a 17 284-parameter dense network and optimizes distance averaged over the 8-second simulation.
- Reaching a Fixed Point: CMA-ES failed to improve the fixed-point controller after 60 000 evaluations over a week, changing performance from 0.995 ± 0.330 m to 0.933 ± 0.369 m without significant difference.The paper therefore discontinued CMA-ES as a benchmark in later experiments.
- Reaching a Random Point: For random-point reaching, 576 updates achieve average error below 10 cm, while 2 563 updates reduce it below 5 cm.The task minimizes average end-effector distance to a randomly sampled reachable target.
- A Quadrupedal Robot: revisited: For the quadruped, 500 model evaluations, or about one CPU hour, produce a hopping gait reaching 1.17 m/s despite a 4 Nm servo-torque limit.The resulting gait includes a somersault every three steps; evaluating energy-efficiency improvements is outside the paper’s scope.
4 Discussion
The differentiable physics engine supports gradient-based robot-controller optimization, with benefits extending to GPU batching, complex sensors, system identification, and possible hardware optimization. The authors report speedups even for small problems while noting that transfer to physical robots remains an open dependency.
- Contributions: The engine differentiates control parameters in modern 3D rigid-body simulations using BPTT and supports GPU execution for large robot batches.It can also differentiate through complex sensors such as cameras, enabling joint learning of computer vision and control policies.
- Results: Gradient evaluation is tractable even with six parameters because its cost depends mainly on physics-model complexity and only slightly on parameter count.The authors attribute the optimization benefit to batch gradient descent, GPU acceleration, and fewer model evaluations.
- Optimization challenges: Controller optimization corresponds to training an RNN, so noisy parameter spaces and exploding or vanishing gradients make optimization highly non-trivial.The paper relates each simulation time step to a neural-network pass and points to recurrent-network optimization methods for more complex tasks.
- Limitations: The method’s physical-robot relevance remains uncertain because the reported results depend largely on how controllers perform on physical counterparts of the simulated models.The authors conjecture that model gradients may be informative but imperfect and more sensitive to modeling noise than physical-system gradients.
- Future applications: The approach could support efficient system identification by differentiating masses, lengths, textures, and potentially state-dependent parameters simultaneously.A neural network could map the current state to parameters such as the friction coefficient.
- Future applications: The engine could distill model knowledge into forward or backward neural networks, which might provide more robust training targets and faster approximate evaluation.The proposed transfer uses gradients through an exact model and a relevant model error.
- Future applications: Hardware-parameter optimization and robot-model adjustment from hardware measurements are proposed extensions, while adversarial robotics training is suggested for competing robots and swarm robotics.These directions are presented as possibilities rather than results established by this paper.
Funding
The research received funding from the Agency for Innovation by Science and Technology in Flanders, and NVIDIA donated the GTX 1080 used in the research.
- The research received funding from the Agency for Innovation by Science and Technology in Flanders, while NVIDIA donated the GTX 1080 used for the research.