Source-linked AI summary
Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, Yuval Tassa
TL;DR
Robotics needs model-based control that is more accessible and practical alongside data-intensive learning-based alternatives. MJPC addresses this with an interactive MuJoCo framework supporting established derivative-based planners and simple Predictive Sampling, which proved surprisingly competitive. The paper prioritises performant algorithms, simple code, and accessible tooling rather than algorithmic advances.
Problem
Model-based optimisation is difficult to implement, often depends on elaborate algorithms, and remains generally inaccessible despite enabling real-time behaviour synthesis.
Method
MJPC provides an open-source interactive MuJoCo framework with iLQG, Gradient Descent, and the simple zero-order Predictive Sampling planner.
Results
Predictive Sampling proved surprisingly competitive with more established algorithms while MJPC supports real-time predictive control and interactive task authoring.
Takeaways & Limitations
Interactive, open-source, responsive tooling can lower barriers to predictive control and accelerate robotics research.
Takeaways & Limitations
Predictive control is myopic beyond its fixed horizon and can control only what the simulator advances much faster than real time.
Abstract
from arXiv · showhide
We introduce MuJoCo MPC (MJPC), an open-source, interactive application and software framework for real-time predictive control, based on MuJoCo physics. MJPC allows the user to easily author and solve complex robotics tasks, and currently supports three shooting-based planners: derivative-based iLQG and Gradient Descent, and a simple derivative-free method we call Predictive Sampling. Predictive Sampling was designed as an elementary baseline, mostly for its pedagogical value, but turned out to be surprisingly competitive with the more established algorithms. This work does not present algorithmic advances, and instead, prioritises performant algorithms, simple code, and accessibility of model-based methods via intuitive and interactive software. MJPC is available at: github.com/deepmind/mujoco_mpc, a video summary can be viewed at: dpmd.ai/mjpc.
1. Introduction
The introduction contrasts data-intensive learning-based methods with real-time model-based control and identifies accessibility as a central barrier. MJPC addresses this barrier through interactive, open-source predictive-control tooling, including the simple Predictive Sampling planner.
- Learning-based methods offer flexible representations but require large amounts of data and computation, whereas model-based methods can synthesise behaviour in real time.
- Model-based optimisation remains difficult to implement, often relies on elaborate algorithms, and is generally inaccessible.
- MJPC is an open-source interactive MuJoCo framework for authoring and solving complex robotics tasks with predictive-control algorithms in real time.
- MJPC includes derivative-based iLQG and Gradient Descent planners alongside Predictive Sampling, a simple zero-order sampling-based method that is easy to understand.
- Asynchronous simulation slowdown lets the planner run faster relative to simulation time, enabling behaviour generation on older, slower machines.
- The project aims to lower barriers to predictive control and accelerate research through responsive, customisable, open-source interactive tooling.
2. Background
The background introduces finite-horizon predictive control as real-time trajectory optimisation that repeatedly measures state, replans, and applies the current nominal action. It then distinguishes trajectory-optimisation approaches and motivates derivative-free sampling methods for complex, nonsmooth systems.
- Trajectory Optimisation: Trajectory optimisation solves a fixed-length optimal-control problem and divides methods into direct methods, which optimise states and controls jointly, and shooting methods, which optimise controls through forward simulation.
- Predictive Control: An asynchronous predictive-control loop applies the current nominal action while a planner measures state and optimises the finite-horizon objective.
- Predictive Control: Predictive control optimises a finite-horizon plan online as system dynamics evolve, using the current measured or estimated state and repeatedly replanning.
- Predictive Control: Predictive control benefits from faster computation and warmstarting, which reuses the previous plan so optimisation can adapt to the newly measured state.
- Derivative-Free Optimisation: Derivative-free sampling methods evaluate sampled policy parameters without gradients and can discover complex behaviours despite not using problem structure or gradient information.
- Derivative-Free Optimisation: Sampling-based methods suit nonsmooth or discontinuous dynamics and are trivially parallelisable, although they are usually considered sample inefficient.
3. MuJoCo MPC (MJPC)
MJPC is an open-source MuJoCo framework for interactive real-time predictive control, combining accessible objective design with derivative-based and derivative-free planners. Its compressed control representations, parallel simulation, and interactive tooling support efficient experimentation on complex systems.
- Framework: MJPC is an open-source predictive-control framework with real-time interaction, composable objectives, and Gradient Descent, iLQG, and Predictive Sampling planners.The framework is implemented in C++ and uses multi-threaded parallel rollouts.
- Objective: MJPC provides composable base costs built from weighted, twice-differentiable norms and residuals that are small when the task is solved.The framework also provides utilities for designing costs and computing derivatives.
- Objective: Risk sensitivity transforms the base cost with ρ(l; R), where R = 0 is risk-neutral, R > 0 risk-averse, and R < 0 risk-seeking.For negative R, the transformed cost is bounded and can yield an inverted-Gaussian cost under a quadratic norm.
- Splines: Spline representations compress action trajectories into time-indexed knots, reducing the search space while smoothing controls and propagating gradients to spline parameters.MJPC supports cubic Hermite, piecewise-linear, and zero-order-hold splines; bounded parameters also bound the trajectory for zero and linear interpolation.
- Planners: The derivative-based planners use gradients or second-order approximations, while Predictive Sampling samples candidate parameterizations and selects the lowest-cost rollout.Predictive Sampling is a simple zero-order baseline that proved surprisingly competitive with the derivative-based planners.
4. Results
MJPC provides an interactive environment for designing and observing predictive-control behaviours, demonstrating real-time locomotion and manipulation across several simulated systems.
- Interactive environment: The GUI lets researchers change cost parameters or planner settings and immediately inspect effects in simulation and live plots.This supports rapid debugging and understanding of factors influencing behaviour.
- Interactive environment: Planning updates take 1–20 milliseconds across the examples, which run on a single CPU.
- Humanoid: A 27-DOF humanoid stands from the floor, withstands large disturbances, and can stand again after being knocked down.
- Quadruped: A Unitree A1 quadruped traverses uneven terrain, including a steep slope, with simulation slowdown providing additional planning time on slower machines.
- Hand: A Shadow Hand performs real-time in-hand cube reorientation while the researcher interactively sets the target orientation.The paper describes this high-DoF contact task as difficult to solve and previously unsolved from scratch in real time, to the authors’ knowledge.
5. Discussion
The discussion frames MJPC as customizable, transparent tooling for making predictive control more accessible, while positioning Predictive Sampling as a simple baseline whose effectiveness reflects the replanning demands of predictive control.
- Accessibility: The paper’s thrust is making predictive control accessible through customizable, interactive, open-source tooling.
- Predictive Sampling: Fast approximate optimisation can be competitive with more sophisticated methods when predictive control values frequent replanning over convergence on an outdated problem.
- Accessibility: MJPC’s GUI exposes task and agent settings, live cost and action predictions, planner information, compute times, trajectories, and interactive simulation controls.
- Predictive Sampling: Predictive Sampling is presented as a baseline rather than a novel algorithm, establishing a lower bound for sampling-based shooting performance.The paper describes it as the simplest possible sampling-based shooting method.
1. Task design. MJPC makes it easy to add new
MJPC supports configurable tasks and interactive behaviour generation, while exposing limitations and future directions for predictive control. Its scope is bounded by simulation capability, finite planning horizons, and unresolved hardware-control challenges.
- Task design: MJPC supports task parameters through its GUI and can quickly generate desired behaviour for reimplementation in another framework.Time-dependent tasks are not yet implemented, though the paper states they should be possible.
- Task design: Figure 4 shows generated behaviours progressing over time from left to right.The figure presents behaviours produced with MuJoCo MPC.
- Task design: MJPC can switch planners on-the-fly, while asynchronous execution enables fair comparison by accounting for faster planners.
- Limitations: Predictive control is limited to systems that MuJoCo can simulate, especially when simulation cannot advance much faster than real time.Very-high-DoF systems such as fluids, cloth, and soft bodies are identified as difficult examples.
- Limitations: Predictive control is myopic because it cannot see beyond its fixed horizon, motivating learned policies, value functions, or high-level agents.These approaches are presented as ways to increase the effective horizon or provide long-horizon task information.
- Limitations: Direct hardware control remains an open challenge requiring transfer learning or real-time state estimation, particularly for free-moving bodies and contact-rich tasks.The paper states that contact-aware estimation is possible but remains to be demonstrated generally.
A. Interpolation
The interpolation appendix describes zero-order, linear, and cubic interpolation schemes for parameter values, with query indices located efficiently by binary search.
- Interpolation: Query-point indices are found by binary search in O(log(n)) over the domain-variable set.
- Interpolation: Zero-order interpolation returns the parameter values at the lower-bound index.
- Interpolation: Linear interpolation uses the normalized interval position q = (τ−τ_j)/(τ_j+1 − τ_j).
- Interpolation: Cubic interpolation uses finite-difference approximations of the slope at interval points.
B. Tasks
The task appendix specifies objective terms for humanoid, quadruped, and hand examples, alongside the Predictive Sampling settings reported in Table 1.
- Tasks: The humanoid objective comprises M = 6 cost terms covering posture, height, velocity, and related alignment quantities.Reported weights include 100 for lateral center-of-mass and feet alignment, 1 for torso-to-center-of-mass alignment, 100 for height difference, and 10 for lateral velocity.
- Tasks: The quadruped objective comprises M = 4 cost terms, including body-height and feet-height difference relative to a target.
- Tasks: The hand objective comprises M = 3 cost terms for cube position, cube orientation, and hand-palm position relationships.The position residual uses weight 20, while the orientation residual is quadratic.
- Planner settings: Table 1 provides the settings used for Predictive Sampling.
C. Predictive Sampling Algorithm
The appendix identifies the Predictive Sampling procedures for policy optimisation and action extraction within the predictive-control algorithm.
- Predictive Sampling Algorithm: Predictive Sampling defines an OptimizePolicy procedure.
- Predictive Sampling Algorithm: The appendix presents PredictiveSampling as an algorithmic procedure within the predictive-control framework.
- Predictive Sampling Algorithm: The algorithm defines an ActionFromPolicy procedure for obtaining actions from a policy.
D. Compute Resources
Experiments used a Lenovo Think-Station P920 with 48GB of memory and an Intel Xeon Gold 6154 72-core CPU, with additional experiments on a 2021 Apple MacBook Pro with 16GB of memory and an M1 Pro CPU.
- Experiments were performed on a Lenovo Think-Station P920 with 48GB of memory and an Intel Xeon Gold 6154 72-core CPU.
- Additional experiments used a 2021 Apple MacBook Pro with 16GB of memory and an M1 Pro CPU.