PaperlayerView exact revision on arXiv

Source-linked AI summary

Bundled Contact Gradients: Stabilizing Differentiable Simulation for Deployable Dynamic Tasks

Dyuman Aditya, Jin Cheng, Clemens Schwarke, Quan Nguyen, Gaurav Sukhatme, Stelian Coros, Gabriele Fadini

arXiv:2609.30951v1cs.RO

TL;DR

Softened contact can provide smooth gradients for differentiable policy learning, but it limits physical fidelity, while stiff contact can make gradients high-variance. BCG averages gradients from nearby perturbation rollouts at stiff contacts; the paper reports more efficient motion learning and zero-shot transfer to a real Unitree G1.

  • Problem

    Differentiable policy learning has largely relied on softened contact, while increasing stiffness toward deployable contact can make gradients high-variance.

    Source: §I · p. 1, B. Differentiable Simulation · p. 3

  • Method

    BCG locally samples and simulates perturbations at detected stiff contacts, then aggregates their gradients without softening forward dynamics.

    Source: §I · p. 2, A. Contact-Local Perturbations · p. 4, C. Gradient Propagation · p. 5

  • Results

    BCG uses over an order of magnitude fewer environment samples than PPO, improves tracking accuracy after transfer to MuJoCo, and enables zero-shot deployment on a real Unitree G1.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The results show that analytic gradients from differentiable simulation can remain useful in the stiff-contact regime required for real-world transfer.

    Source: §VI · p. 7

    BCG adds computation and memory for contact branches, and sustained stiff contact may incur greater overhead than PPO; perturbation scales and bundle size also require tuning.

    Source: §VI · p. 7

Abstract

from arXiv · show

Differentiable simulation provides analytic gradients of robot dynamics, enabling fast and sample-efficient first-order policy optimization. However, obtaining smooth and informative gradients through rigid-body contact typically requires softened contact models, often at the expense of physical fidelity and thereby limiting learned policies largely to simulation. This trade-off becomes particularly consequential for dynamic humanoid motions, where accurate contact dynamics are critical for transferring policies to the real world. Increasing contact stiffness in rigid-body simulation improves the fidelity of interactions, but also makes the dynamics increasingly sensitive to small state perturbations, producing high-variance gradients that can destabilize first-order policy learning. To address this, we propose \emph{Bundled Contact Gradients (BCG)}, a contact-local randomized smoothing framework for differentiable policy learning. When stiff contact is detected, our method evaluates a local bundle of randomized perturbation rollouts around the stiff contact configuration and aggregates their gradient signal thereby reducing gradient variance. We demonstrate the effectiveness of our method by successfully training and transferring dynamic motions zero-shot onto a real-world Unitree G1 humanoid platform. Videos and supplementary information can be found at https://bundledcontactgradients.github.io/

I. INTRODUCTION

Model-free RL handles nonsmooth contact but is sample-inefficient, while differentiable simulation offers efficient first-order learning that has largely relied on softened contact. BCG stabilizes gradients locally at stiff contacts, and the paper reports efficient training and zero-shot transfer of dynamic motions to humanoid hardware.

Motivation

  • Model-free RL handles nonsmooth contact but its zero-order gradient estimates make long-horizon, high-dimensional policy optimization sample-inefficient.Training often requires billions of environment interactions and substantial wall-clock time.Source: §I · p. 1
  • Soft contact can yield smooth gradients but sacrifices physical fidelity, whereas stiff, deployable contact makes gradients high-variance and can destabilize first-order optimization.Existing first-order methods also soften contact or shorten the differentiable horizon around contact events, discarding gradient information there.Source: §I · pp. 1–2

Method

  • BCG samples nearby perturbations only at stiff contacts, simulates them in parallel, and aggregates their gradients to stabilize first-order learning without softening forward dynamics.This local smoothing retains stiff, deployable contact in the forward rollout.Source: §I · p. 2

Results

  • BCG learns motion-tracking policies with fewer samples and less training time than PPO, while achieving lower tracking error after sim-to-sim transfer and supporting zero-shot transfer to real humanoid hardware.The authors describe the hardware transfer as the first successful zero-shot transfer of a policy trained in differentiable simulation onto a real humanoid platform.Source: §I · p. 2

II. RELATED WORK

Prior work spans robust but sample-inefficient model-free RL and more efficient first-order learning through differentiable simulation. Stiff contact remains difficult because physical fidelity, gradient quality, and long-horizon propagation impose competing demands.

A. Reinforcement Learning for Dynamic Robot Control

  • Model-free RL, including PPO [13], is widely used for dynamic, contact-rich robot skills, but its high-variance gradients reduce sample efficiency, especially for humanoids.Its robustness to discontinuous contact physics has supported sim-to-real results in legged locomotion.Source: A. Reinforcement Learning for Dynamic… · p. 2

B. Differentiable Simulation and First-Order Policy Optimization

  • Differentiable simulation enables first-order policy optimization, with prior work extending it to deployable locomotion, agile flight, and multiphysics settings.Related approaches include quadruped methods using a rigid-body surrogate or frozen trotting policy, and zero-shot transfer using analytically smoothed contact, [28],,, [32].Source: B. Differentiable Simulation and… · p. 2

C. Gradients through Stiff Contact

  • Stiff-contact methods face trade-offs: greater stiffness improves fidelity but can impair gradient accuracy, while hard-contact gradients and contact-truncated horizons have their own limitations.Adaptive integration can improve gradient fidelity at added computational cost and with risks from longer differentiable rollouts; horizon truncation discards direct gradient information through contact events.Source: C. Gradients through Stiff Contact · pp. 2–3

D. Randomized Smoothing and Bundled Gradients for Contact

This section contrasts model-free policy gradients with differentiable simulation and reviews randomized smoothing for contact. BCG applies smoothing locally at stiff contacts for first-order policy learning rather than globally across trajectories.

D. Randomized Smoothing and Bundled Gradients for Contact

  • Prior randomized-smoothing and bundled-gradient methods largely target trajectory optimization or quasi-static manipulation and smooth contact globally, whereas BCG applies smoothing locally at stiff contacts for dynamic motion-tracking.The section situates this approach alongside contact-implicit trajectory optimization, and work addressing smoothing bias.Source: D. Randomized Smoothing and Bundled… · p. 3

A. Reinforcement Learning

  • The score-function policy gradient differentiates the policy while treating dynamics as a black box, avoiding physics derivatives but incurring variance that grows with horizon and action dimension.PPO [13] is a widely used instance.Source: A. Reinforcement Learning · p. 3

B. Differentiable Simulation

  • Differentiable simulation backpropagates through an unrolled rollout, typically reducing gradient variance, but stiff contact can make transition Jacobians high-variance across contact events.The state sensitivity is propagated recursively through time.Source: B. Differentiable Simulation · p. 3

C. Analytically Smoothed Contact Model

The contact model smooths impulse scaling as a function of penetration depth, while a large but finite stiffness preserves a response close to hard contact for BCG.

C. Analytically Smoothed Contact Model

  • Penetration depth scales contact impulses through a continuous, analytically smoothed response that permits gradients before contact.The model replaces abrupt contact activation and allows forces at small separations.Source: C. Analytically Smoothed Contact Model · p. 3
  • Larger κ reduces smoothing and approaches hard contact, but makes derivatives increasingly sensitive; BCG therefore uses large, finite κ.Smaller κ produces a wider, softer transition; stiff contact is not the discontinuous hard-contact limit.Source: C. Analytically Smoothed Contact Model · p. 3
  • At stiff contact, B perturbed branches advance for H steps under shared actions and are averaged to continue the rollout.The policy-learning framework uses the averaged rollout with ADD imitation rewards.Source: Figure 2 · p. 4

D. First-Order Policy Optimization with SHAC

SHAC uses short differentiable rollouts and a learned critic for first-order policy optimization; BCG adds contact-local bundles to stabilize gradients at stiff contacts.

D. First-Order Policy Optimization with SHAC

  • SHAC collects short N-step differentiable rollouts from E parallel environments and bootstraps remaining return with a learned critic.The short horizon limits unstable gradient propagation, but SHAC remains sensitive to individual stiff-contact events.Source: D. First-Order Policy Optimization… · p. 4

IV. BUNDLED CONTACT GRADIENTS

  • At stiff contact, BCG creates nearby configurations, advances them in parallel, and aggregates their states while retaining large, finite contact stiffness.Combining sensitivities from several contact realizations reduces reliance on any single derivative.Source: §IV · p. 4

A. Contact-Local Perturbations

  • A bundle triggers when a contact’s normal impulse exceeds threshold τ; sampled Cartesian position and velocity offsets map to joint-space increments.The mapping uses the contacting chains’ final-link positions and a damped Jacobian pseudoinverse; sampled offsets are treated as constants during backpropagation.Source: A. Contact-Local Perturbations · p. 4

B. Multi-Step Bundled Rollout

During a bundled rollout, perturbed branches share policy actions, evolve for H control steps, and their averaged state continues the nominal rollout.

B. Multi-Step Bundled Rollout

  • Each perturbed branch starts from its perturbed state and initially uses the nominal rollout’s current action.At later control boundaries, the same policy action is applied to every branch.Source: B. Multi-Step Bundled Rollout · p. 5
  • Each transition holds its action constant for S simulation substeps before branch states are averaged using arithmetic means for positions and velocities.Even H = 1 permits S substeps of branch evolution before averaging.Source: B. Multi-Step Bundled Rollout · p. 5
  • The averaged state supplies the observation and reward, and the nominal rollout resumes from that state after H steps.Bundled environment transitions run in parallel on the GPU to reduce overhead.Source: B. Multi-Step Bundled Rollout · p. 5

C. Gradient Propagation

BCG propagates policy sensitivities along each perturbed branch and averages them, creating local smoothing while retaining feedback through the bundled rollout.

C. Gradient Propagation

  • Without bundling, sensitivities propagate along a single trajectory; BCG instead propagates them along each perturbed branch using transition Jacobians.Branch Jacobians are evaluated at each branch state and the shared action.Source: C. Gradient Propagation · p. 5
  • For arithmetically aggregated state components, the next-state sensitivity averages the corresponding branch sensitivities.State sensitivity Zt and action sensitivity Ut are total derivatives with respect to policy parameters.Source: C. Gradient Propagation · p. 5
  • Averaging sensitivities across nearby trajectories can attenuate sharp contact-dependent variations while retaining policy feedback through all H · S simulation substeps.These sensitivities enter the imitation-objective gradient through the chain rule.Source: C. Gradient Propagation · p. 5

D. Policy Learning and Imitation Objective

The method pairs contact-bundled rollouts in SHAC with ADD as the motion-imitation objective, using the discriminator’s differentiable reward to update the policy.

D. Policy Learning and Imitation Objective

  • SHAC uses bundled rollouts to minimize LFO, evaluating rewards and critic inputs on the bundle-averaged states.This places BCG’s contact-local gradient treatment inside the policy-learning objective.Source: D. Policy Learning and Imitation… · p. 5
  • ADD supplies a differentiable imitation reward that supports backpropagation through both the objective and simulated rollouts.Its learned aggregation avoids manually tuning individual tracking-reward weights.Source: D. Policy Learning and Imitation… · p. 5
  • The discriminator distinguishes ideal zero residuals from policy-generated residuals using simulated and phase-matched reference features.The reward’s gradients pass through the discriminator input and bundled simulator rollout to update the policy.Source: D. Policy Learning and Imitation… · p. 5

V. EXPERIMENTS

The experiments test BCG under stiff contact across simulator transfer, learning efficiency, tracking accuracy, and hardware deployment, using four dynamic motions on the Unitree G1.

V. EXPERIMENTS

  • The evaluation asks whether stiff contact supports dynamic motions and whether BCG stabilizes gradients and transfers policies across simulators and to hardware.It also compares SHAC+BCG, vanilla SHAC, and PPO on efficiency, convergence, and post-transfer tracking accuracy.Source: §V · p. 6

A. Experimental Setup

  • The setup trains Unitree G1 policies on four 15-second LAFAN1 motions: Run, Jump, Fight, and Dance.SHAC+BCG uses a 32-step rollout, bundle size 10, duration H = 2, and triggers bundling above 400 N normal force.Source: A. Experimental Setup · p. 6

1) Q1: Effect of Contact Stiffness on Transfer:

  • As contact stiffness κ increases, fewer trained policies fall after transfer to MuJoCo.The stiffness ablation also measures the difference in foot penetration between the training simulator and MuJoCo.Source: 1) Q1: Effect of Contact Stiffness on… · p. 6

2) Q2: Gradient Variance and Contact Sensitivities:

BCG reduces gradient variability under stiff contact and improves learning efficiency relative to PPO, while bundling’s computational graph carries some overhead.

2) Q2: Gradient Variance and Contact Sensitivities:

  • With BCG, policy-gradient variance is lower across Jump training environments, indicating greater agreement among environment-wise gradient directions.The figure’s shading reports standard deviation across resamples of the 128 training environments.Source: 2) Q2: Gradient Variance and Contact… · p. 6, Figure 3 · p. 6

3) Q3: Learning Efficiency and Convergence:

2) Q2: Gradient Variance and Contact Sensitivities:

  • Averaging gradients across perturbed stiff-contact branches attenuates extreme sensitivities and reduces variation in the full state-gradient vector.The point-cloud comparison shows individual branches spread more widely than bundled gradients, whose ellipse is smaller; the authors report more stable gradients over longer rollout horizons than previous work.Source: Figure 4 · p. 6, 2) Q2: Gradient Variance and Contact… · p. 7

3) Q3: Learning Efficiency and Convergence:

  • SHAC+BCG uses 64 environments rather than 128, offsetting extra branch samples, but its larger backpropagation graph adds computational overhead.The methods consume comparable sample budgets at a given iteration count.Source: 3) Q3: Learning Efficiency and… · p. 7

4) Q4: Tracking Performance across Methods:

SHAC+BCG improves tracking across most tested motions, transfers learned dynamic motions zero-shot to a real Unitree G1, and carries computational costs that motivate further development.

4) Q4: Tracking Performance across Methods:

5) Q5: Sim-to-Real Transfer:

VI. CONCLUSION

  • Over an order of magnitude fewer environment samples than PPO, SHAC+BCG improved tracking accuracy after MuJoCo transfer and enabled zero-shot deployment on a real Unitree G1.Source: §VI · p. 7
  • BCG adds computation and memory for multiple contact branches, with potentially greater overhead than PPO on tasks with sustained stiff contact.Source: §VI · p. 7
  • Future work could adapt bundle parameters to local contact sensitivity and extend BCG to humanoid loco-manipulation and dexterous manipulation.Source: §VI · pp. 7–8
Loading 2609.30951v1…