PaperlayerView exact revision on arXiv

Source-linked AI summary

Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing

Haoru Xue, Chaoyi Pan, Zeji Yi, Guannan Qu, Guanya Shi

arXiv:2409.15610v1cs.RO

TL;DR

Full-order legged control is difficult, and sampling-based MPC can be high-variance in high-dimensional non-convex problems. DIAL-MPC uses diffusion-inspired annealing to refine control sequences online. The paper reports efficient full-order control, diverse solutions, and training-free generalization across tasks and dynamics, while noting a planning-horizon constraint.

  • Problem

    Full-order legged control is challenging because sampling-based MPC can yield high-variance, suboptimal solutions in high-dimensional non-convex problems, while NMPC is often limited to reduced-order models.

    Source: §I · p. 1

  • Method

    DIAL-MPC iteratively refines control sequences using diffusion-inspired annealing, with sampling schedules that move from broad exploration toward more concentrated search.

    Source: §I · p. 1, B. Diffusion-Inspired Annealing · p. 4, C. Diffusion-Inspired Annealing for… · p. 4

  • Results

    DIAL-MPC solves full-order control efficiently and generates diverse solutions, including in crate climbing and humanoid jogging tasks.

    Source: §V · p. 6, A. Convergence and Coverage · p. 6

  • Takeaways & Limitations

    The method is described as generalizable across tasks and dynamics in a training-free manner.

    Source: §V · p. 7

    DIAL-MPC requires fast simulation to generate samples, limiting its application to longer planning horizons.

    Source: §V · p. 7

Abstract

from arXiv · show

Due to high dimensionality and non-convexity, real-time optimal control using full-order dynamics models for legged robots is challenging. Therefore, Nonlinear Model Predictive Control (NMPC) approaches are often limited to reduced-order models. Sampling-based MPC has shown potential in nonconvex even discontinuous problems, but often yields suboptimal solutions with high variance, which limits its applications in high-dimensional locomotion. This work introduces DIAL-MPC (Diffusion-Inspired Annealing for Legged MPC), a sampling-based MPC framework with a novel diffusion-style annealing process. Such an annealing process is supported by the theoretical landscape analysis of Model Predictive Path Integral Control (MPPI) and the connection between MPPI and single-step diffusion. Algorithmically, DIAL-MPC iteratively refines solutions online and achieves both global coverage and local convergence. In quadrupedal torque-level control tasks, DIAL-MPC reduces the tracking error of standard MPPI by $13.4$ times and outperforms reinforcement learning (RL) policies by $50\%$ in challenging climbing tasks without any training. In particular, DIAL-MPC enables precise real-world quadrupedal jumping with payload. To the best of our knowledge, DIAL-MPC is the first training-free method that optimizes over full-order quadruped dynamics in real-time.

I. INTRODUCTION

High-dimensional, contact-rich legged control makes full-order optimization difficult, while sampling-based MPC can be high-variance and suboptimal. DIAL-MPC uses diffusion-inspired iterative refinement for full-order torque-level locomotion and is evaluated in real-world quadruped control.

I. INTRODUCTION

  • Full-order legged control is challenging because articulated systems are high-dimensional, underactuated, and contact-rich, producing non-convex, non-smooth optimization landscapes.NMPC is often limited to reduced-order models, while sampling-based MPC can have high variance and yield suboptimal solutions in high-dimensional non-convex problems.Source: §I · p. 1
  • Sampling range entails a trade-off: larger ranges improve global coverage but may miss the optimum, while smaller ranges improve local search but increase susceptibility to local minima and initial guesses.The latter can lead to increased variance.Source: §I · p. 1
  • DIAL-MPC iteratively shifts from smooth but inaccurate objectives to more accurate local objectives, balancing global coverage and local convergence.Its annealing framework is inspired by the connection between sampling-based MPC and diffusion processes.Source: §I · pp. 1–2
  • Full-order torque-level DIAL-MPC is evaluated on a quadruped, with real-world validation reporting 50 Hz control, robustness, and efficiency.The paper also reports precise jumping with payload and performance that outperforms RL methods, without quantifying that comparison here.Source: §I · pp. 1–2

II. RELATED WORK

Prior legged-locomotion approaches balance model fidelity, agility, and generalization differently. Sampling-based optimization offers flexibility but faces high variance and dimensionality, while parallel simulation supports sample-hungry methods.

A. Agility in Legged Locomotion

  • Reduced-order NMPC eases planning over full-order hybrid dynamics but can cause suboptimal performance or constraint violations; a full-order solver remains constrained by predefined contact sequences.The paper presents its method as allowing real-time feedback and environmental adaptation without redundant constraints.Source: A. Agility in Legged Locomotion · p. 2
  • Goal-conditioned RL improves task-level generalization but can struggle with unseen dynamics and novel task classes; the paper describes its approach as training-free online optimization.The authors position it as combining RL-like agility with MPC robustness and generalization.Source: A. Agility in Legged Locomotion · p. 2

B. Sampling-Based Optimization

  • MPPI and its variants are used for online motion planning because they are parallelizable and flexible; zeroth-order methods are reported to have better convergence properties on high-stiffness legged-locomotion problems.The cited convergence comparison is described as supported empirically and theoretically.Source: B. Sampling-Based Optimization · p. 2
  • Sampling-based methods avoid requiring gradient information and address non-convex, non-smooth problems, but face high variance and the curse of dimensionality under constrained online sampling budgets.This challenge is especially pronounced in high-dimensional, contact-rich environments.Source: B. Sampling-Based Optimization · p. 2

C. Parallel Robot Simulation

  • Massively parallelizable simulators facilitate rapid data generation for sample-hungry algorithms such as PPO, supporting efficient training of complex policies.Examples named are Isaac Gym, Brax, and MuJoCo.Source: C. Parallel Robot Simulation · p. 2

III. METHOD

The method connects MPPI to a single-stage diffusion process and uses that connection to motivate diffusion-inspired annealing. Its sampling formulation iteratively refines control sequences across noise levels.

III. METHOD

  • DIAL-MPC is introduced as diffusion-inspired annealing for sampling-based MPC, with the connection to diffusion motivating its annealing process.The method section establishes MPPI's equivalence to a single-stage diffusion process before presenting the annealing technique.Source: §III · p. 2

A. Sampling-Based MPC as Single-stage Diffusion

  • MPPI draws Gaussian perturbations, evaluates each perturbed control sequence by simulating dynamics and accumulating costs, then updates the sequence.The control problem constrains state transitions by system dynamics and includes state, control, and terminal costs and constraints.Source: A. Sampling-Based MPC as Single-stage… · p. 2
  • The optimal-control distribution is convolved with Gaussian noise because directly sampling from its narrow support is impractical.As λ approaches zero, the target distribution concentrates around the optimal control sequence; the resulting corrupted distribution provides a sampling context for MPPI.Source: A. Sampling-Based MPC as Single-stage… · p. 3
  • MPPI's update is described as a one-step ascent using the score function of the noise-corrupted target distribution, with covariance Σ as the learning rate.The paper interprets this as a denoising step at a fixed noise level; diffusion models instead refine samples across different noise levels.Source: A. Sampling-Based MPC as Single-stage… · p. 3

B. Diffusion-Inspired Annealing

MPPI’s single-stage diffusion connection explains its exploration–convergence trade-off: broader kernels improve coverage but can distort the optimum, while narrower kernels risk local minima. DIAL-MPC applies decreasing-noise annealing to move from broad exploration toward convergence.

B. Diffusion-Inspired Annealing

  • MPPI’s single-stage diffusion moves particles toward a stationary point of the noise-smoothed target distribution.The paper notes that this update differs from diffusion models by lacking a scaling factor and extra noise.Source: B. Diffusion-Inspired Annealing · p. 3
  • Larger kernels improve exploration and global-optimum access but can shift the optimum and compromise the original problem’s optimality.Smaller kernels improve local optimality but risk trapping the search in local minima.Source: B. Diffusion-Inspired Annealing · p. 3
  • Contact-rich locomotion’s sharp, asymmetric landscapes make the coverage–convergence trade-off especially consequential for MPPI.In this setting, even a small increase in det Σ can degrade performance, and score ascent on p1(U) cannot guarantee global optimality.Source: B. Diffusion-Inspired Annealing · p. 3
  • Decreasing-noise sampling starts broadly for global exploration, then concentrates toward the target distribution to improve convergence.The method uses an exponential noise schedule and incorporates the multi-stage process into MPPI.Source: B. Diffusion-Inspired Annealing · p. 4

C. Diffusion-Inspired Annealing for Sampling-Based MPC

DIAL-MPC combines MPPI with multi-stage annealing in a receding-horizon MPC framework. Its dual-loop covariance design schedules sampling across both the full control sequence and individual actions at different horizon positions.

C. Diffusion-Inspired Annealing for Sampling-Based MPC

  • DIAL-MPC combines MPPI with multi-stage annealing and receding-horizon MPC through a dual-loop covariance design.Each control action is updated in both trajectory-level and action-level annealing loops before it is applied.Source: C. Diffusion-Inspired Annealing for… · p. 4
  • Trajectory-level annealing progressively narrows the sampling distribution toward the target as the iteration index decreases.The trajectory-level schedule uses temperature parameter β1.Source: C. Diffusion-Inspired Annealing for… · p. 4
  • Action-level annealing allows a larger sampling region for future controls that have received fewer updates.The covariance design assumes isotropic matrices as a practical realization and combines both schedules to adjust exploration and exploitation.Source: C. Diffusion-Inspired Annealing for… · p. 5

IV. EXPERIMENT

DIAL-MPC reduces walking tracking error relative to MPPI and outperforms GCRL on tasks requiring both precision and agility, especially under significant model mismatch. The evaluation includes walking tracking, sequential jumping, and crate climbing.

IV. EXPERIMENT

  • 3.9 times lower tracking error in walking tasks than MPPI; DIAL-MPC outperforms GCRL on precision-and-agility tasks, especially with significant model mismatch.The tested tasks include walking tracking, sequential jumps onto randomly placed 10 cm-radius platforms, and climbing a 60 cm crate.Source: §IV · p. 5

A. Convergence and Coverage

DIAL-MPC combines broad solution coverage with convergence to produce stronger results than the compared sampling-based MPC methods. It also solves agile, non-smooth tasks where NMPC may fail to converge and generates diverse motions.

Convergence of DIAL-MPC

  • The comparison includes MPPI with large and small sampling kernels, CMA-ES, NMPC, and GCRL; sampling-based methods use 2048 samples and 20 optimization steps.GCRL training takes approximately 31 minutes, and two reward terms were added to regularize its unstable exploration.Source: A. Convergence and Coverage · p. 5
  • 13.4 times lower tracking error and 107.7% higher contact reward than MPPI and CMA-ES; DIAL-MPC also achieves 3 times the success rate of the best MPPI schedule in crate climbing.It is the only sampling-based MPC method reported to consistently generate feasible crate-climbing solutions.Source: A. Convergence and Coverage · p. 5, Table I · p. 5
  • DIAL-MPC solves sequential jumping and crate climbing, where NMPC is more likely to get stuck in local minima and fail to converge.Source: A. Convergence and Coverage · p. 6
  • DIAL-MPC consistently outperforms GCRL across tasks, despite GCRL using a lower-level leg-position controller while DIAL-MPC directly outputs torque.The authors note that this comparison may be unfair because DIAL-MPC is training-free.Source: A. Convergence and Coverage · p. 6

Coverage of DIAL-MPC

  • The annealing process supports diverse solutions in high-dimensional spaces, including crate climbing and humanoid jogging with a 19-dimensional action space.In the humanoid crate-pushing task, the generated motion requires less effort when the crate weight is reduced from 30 kg to 15 kg.Source: Figure 5 · p. 5, A. Convergence and Coverage · p. 6

B. Test-Time Generalizability

DIAL-MPC generates solutions for new tasks or dynamics models in real time without finetuning. The reported evaluations include different walking goals, a 10 kg payload, and real-world jumping with a 7 kg payload.

Test-Time Generalizability

Task-level generalizability

  • DIAL-MPC reduces walking tracking error by 3.9 times and improves contact reward by 3.5% versus GCRL across different goals.DIAL-MPC explicitly conditions on each task, whereas GCRL uses a universal policy.Source: B. Test-Time Generalizability · p. 6

Dynamic-level generalizability

  • With a 10 kg payload, GCRL performs poorly in tracking and fails, while DIAL-MPC outperforms it by a larger margin in jumping.The crate-climbing task was excluded because the payload made solutions infeasible.Source: B. Test-Time Generalizability · p. 6, Table II · p. 6

Test-Time Generalizability

  • On a real Go2 quadruped, DIAL-MPC performs precise sequential jumping with a 7 kg payload using direct torque control.Source: Figure 7 · p. 6

C. Robustness to Model Mismatch

Under simulated 2 kg base-mass mismatch, DIAL-MPC outperforms the best sampling-based MPC baselines in walking and jumping, and it also demonstrates real-world walking and jumping. The conclusion notes that fast sample-generation simulation limits longer-horizon planning.

C. Robustness to Model Mismatch

V. CONCLUSION

  • DIAL-MPC is training-free and generalizes across tasks and dynamics, but its need for fast simulation limits its use in longer planning-horizon tasks.Source: §V · pp. 6–7

APPENDIX

The appendix describes the quadruped hardware and controller settings, computation setup, and task specifications used in the evaluations.

A. Hardware and Software Setup

  • The Unitree GO2 has 18 degrees of freedom and 12 actuated joints; its abduction and hip joints produce about 24 N·m and its knees about 45 N·m.The torque-control method uses a derivative gain of 0.65, as does the simulation.Source: A. Hardware and Software Setup · p. 8
  • Algorithms run in JAX with Brax; inference uses an RTX 4090 GPU and i9-13900KF CPU, with 50 Hz algorithm control and 200 Hz motor commands.Sampling-based MPC uses 2048 parallel environments and a 20-step, 0.4 s horizon.Source: A. Hardware and Software Setup · p. 8
  • Quadruped walking-tracking and sequential-jumping tasks largely share reward specifications between DIAL-MPC and GCRL.Source: Table IV · p. 8
  • In sequential jumping, the robot has 1 s at each stage to reach the next target, and contacts outside the target are penalized.Source: Figure 8 · p. 8

B. Task Implementation Details

The section details reward design and evaluation setups across quadruped tracking, jumping, crate climbing, and humanoid crate pushing. It reports a 30-second, training-free crate-climbing plan and whole-body control across robots with different degrees of freedom.

Quadruped Velocity Tracking

  • For velocity tracking, DIAL-MPC used six reward terms and demonstrated robust walking, while GCRL used two additional terms and was trained with PPO for 500 million steps.GCRL training took 31 minutes and used domain randomization of base mass, actuator gains, friction, and velocity targets; DIAL-MPC's energy term was joint torque times joint velocity.Source: B. Task Implementation Details · p. 8

Quadruped Sequential Jumping

  • The sequential-jumping task required transitions between randomized contact targets every 1 second across 10 cm plates, with targets specifying four-foot placements and center-of-mass location.The reported contact-reward measure aggregates the worst-case contact score across 10 stages in 8 seconds; each algorithm ran five trials with a 10 kg payload.Source: B. Task Implementation Details · p. 9

Quadruped Crate Climbing

  • The crate-climbing task targeted a 0.6 m crate, more than twice the robot's standing height; the paper describes DIAL-MPC as achieving RL-level agility in 1/15 the speed of real world.The authors note that integration with perception remains future work and compare the current method to a teacher policy with privileged environment information.Source: B. Task Implementation Details · p. 9
  • 30 seconds of synchronous rollout produced a 2-second full-state crate-climbing motion plan with zero prior knowledge or training.The task used 4096 parallel samples, a 40-step (1 s) horizon, four annealing steps, and contacts enabled on the base and thighs.Source: B. Task Implementation Details · p. 9

Humanoid Crate Pushing

  • DIAL-MPC tracked 0.8 m/s while pushing 30 kg and 15 kg crates, demonstrating whole-body control on 25-degree-of-freedom humanoid and 18-degree-of-freedom quadruped systems.The task rewarded speed and upper-end-effector contact; its reward terms were described as straightforward, with no training required.Source: B. Task Implementation Details · p. 9

Related papers

Loading 2409.15610v1…