Source-linked AI summary
Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning
Kousheek Chakraborty, Chandan K. Rajendra, Ayham Alharbat, Abeje Y. Mersha
TL;DR
The paper addresses narrow-viability robot-learning tasks, where early termination leaves standard policy-gradient training without useful signal. It introduces an Actuator Dynamics Curriculum that anneals elevated simulated stiffness toward identified hardware dynamics, motivated by cart-pole viability-kernel analysis. The curriculum converges on a Spot handstand transition in simulation and supports sim-to-real transfer, while the authors identify theoretical, task-scope, scheduling, and hardware-evaluation boundaries.
Problem
Narrow-viability tasks are those where most exploration trajectories terminate early, leaving policy-gradient methods without sufficient learning signal to converge.
Method
The Actuator Dynamics Curriculum anneals simulated joint stiffness from a high initial value to its system-identified hardware value while keeping closed-loop dynamics critically damped.
Results
The curriculum converges on the Spot handstand task in simulation and demonstrates sim-to-real transfer of the resulting transition policy on a physical Spot robot.
Takeaways & Limitations
Simulated closed-loop joint dynamics is a useful curriculum axis for tasks whose exploration is bottlenecked by termination conditions rather than reward signal.
Takeaways & Limitations
The analytical result is established only for a representative cart-pole, the curriculum is demonstrated on one task, and hardware deployment is reported qualitatively.
Abstract
from arXiv · showhide
Reinforcement learning has produced capable controllers across a broad range of legged-robot tasks, but a subset of these tasks fail to converge under standard training: those for which most exploration trajectories terminate before producing useful gradient signal. To address such tasks we introduce the \emph{Actuator Dynamics Curriculum}, a procedure that initializes joint stiffness at a high value and anneals it toward the system-identified value as completed episode lengths grow. Using a cart-pole system as a representative example, we show that higher closed-loop joint natural frequency under critical damping enlarges the viability kernel of the underlying Markov Decision Process, increasing the fraction of initial states from which the task is feasible. We validate the kernel monotonicity on the cart-pole and apply the curriculum to a quadrupedal-to-handstand transition on the Boston Dynamics Spot, a narrow-viability task where training under fixed identified stiffness plateaus at a policy that never completes the transition. The trained policy executes the transition in simulation across 10 seeds and transfers to hardware. More broadly, our results suggest that simulated actuator dynamics is a useful axis along which to design curricula for tasks in which exploration is bottlenecked by termination conditions rather than by reward signal.
1 Introduction
The paper targets narrow-viability robot-learning tasks, where useful reward-seeking trajectories are extremely rare because most exploration rollouts terminate early. It introduces an Actuator Dynamics Curriculum that increases early-training recovery margin by using elevated stiffness, then returns simulated dynamics to identified hardware values.
- Motivation: Narrow-viability tasks have so few reward-seeking, non-terminating trajectories that practical sampling budgets provide insufficient policy-gradient signal.The paper characterizes these tasks through early termination during exploration.
- Motivation: The Spot handstand requires a 90° forward pitch transition in roughly one second while the hind legs lose contact and the front legs stabilize the body.During the mid-pitch, zero-contact, rotating phase, most actions cause a fall and early termination.
- Approach: The Actuator Dynamics Curriculum anneals simulated joint stiffness from a high initial value to the system-identified hardware value while maintaining critical damping.The method changes actuator dynamics during training rather than providing external reference information or a motion-prior dataset.
- Theory: Higher closed-loop natural frequency under critical damping enlarges the representative cart-pole’s viability kernel and the fraction of feasible initial states.The result motivates elevated stiffness as a way to increase the set of states from which task constraints can be satisfied.
- Evaluation: The method is evaluated on the Spot handstand task in simulation and transferred to a physical Spot robot.The paper also situates the approach among work on low-level controller gains and emphasizes scheduling closed-loop joint dynamics toward deployment values.
2 Problem Formulation
The paper formulates actuator-parameterized robot learning as an MDP whose termination constraints shape viability, then shows that higher critically damped natural frequency enlarges the feasible initial-state set. This links actuator dynamics to the narrow-viability exploration problem.
- 2.1 Setup: The robot task is modeled as an actuator-parameterized MDP in which κ changes closed-loop dynamics and the state distribution visited by a policy.The termination set encodes safety violations, and entering it ends the episode and truncates the return.
- 2.1 Setup: The policy outputs bounded relative joint-position commands, qd = q0 + αa, with each action component clipped to [−1, 1].This defines the bounded-command interface used in the experiments and cart-pole analysis.
- 2.1 Setup: Each joint uses PD torque control with proportional gain Ki and derivative gain Bi, while external effects are collected in τext,i.The closed-loop parameters are expressed through natural frequency and damping ratio.
- 2.1 Setup: The curriculum varies stiffness Ki while maintaining ζi = 1, changing closed-loop bandwidth without changing the damping regime.Actuator parameters affect both the viability kernel and the policy-gradient landscape through the transition dynamics.
- 2.2 Propositions: The finite-horizon viability kernel contains states from which some policy can avoid the termination set for T steps with probability one.States outside the kernel cannot be guaranteed to remain safe for the full horizon under any policy.
- 2.2 Propositions: A narrow-viability task has at most ε probability of entering the finite-horizon viability kernel under uniform random exploration from ρ0.The visitation distribution is defined over states reached within T steps under the bounded-action uniform policy.
- 2.2 Propositions: Under the stated bounded-command assumptions, the cart-pole viability kernel is monotone non-decreasing with natural frequency.The result applies to the representative linearized cart-pole system Mωn over a finite horizon.
- 2.2 Propositions: Higher natural frequency moves initial-state mass into the kernel, increasing the fraction of initial states from which the task is feasible.The proposed mechanism is greater controller authority and responsiveness for returning the state toward a reference before termination.
3 Method
The Actuator Dynamics Curriculum begins in a large-kernel, high-stiffness regime and progressively restores the system-identified actuator dynamics as completed episodes become longer. Critical damping is maintained while the schedule moves from K0 to K∗.
- 3 Method: The curriculum holds training at high stiffness K0 until episode lengths grow, then anneals stiffness toward the identified hardware value K∗.This keeps early training in the large-kernel regime while matching deployment actuator characteristics later.
- 3 Method: The stiffness schedule uses curriculum progress β(t) to interpolate between K0 and K∗, with p controlling the annealing-curve shape.The experiments use p = 1, corresponding to linear interpolation.
- 3 Method: Damping is recomputed at each iteration to maintain critical damping, so the schedule changes closed-loop bandwidth rather than damping properties.The per-joint damping is set from the current stiffness and effective inertia.
- 3 Method: Curriculum progress is computed from a running estimate of completed episode length maintained as an exponential moving average.Lmin and Lmax define the episode-length range over which the schedule moves between the two stiffness values.
4 Experimental Results
The experiments validate the curriculum on cart-pole and Spot handstand tasks, comparing annealed stiffness against fixed identified stiffness and ablations. The curriculum improves training behavior and produces successful simulated and hardware transitions.
- Cart-pole validation: The cart-pole evaluation sweeps natural frequency and records viability and success over initial pole states under closed-loop control.The admissible region is {|x| ≤1 m, |θ| ≤π/2}.
- Curriculum procedure: The Actuator Dynamics Curriculum updates policy training while annealing stiffness from an initial value toward the target value using completed episode lengths.The associated damping is recomputed as Bi = 2√KMi for each joint.
- Spot handstand experiment: The Spot experiment compares curriculum training with fixed system-identified stiffness across 10 seeds while holding other hyperparameters identical.The handstand task rotates Spot 90° from a four-legged stance to a balanced front-leg handstand.
- Training outcomes: Without the curriculum, mean episode length plateaus around 400 steps after 10,000 updates, whereas curriculum training reaches maximum reward and episode length within 1.5 hours.The curriculum schedule reaches the system-identified stiffness by convergence.
- Simulation and transfer: The curriculum-trained policy executes a clean simulated handstand transition, while the baseline fails; all 10 trained policies transfer to physical Spot hardware.Hardware trials included disturbances applied by prodding the robot with a pole.
- Ablations: The ablation study trains five variants across 10 seeds and evaluates each over 1000 episodes under system-identified deployment dynamics.Reported outcomes are final mean episode length and mean reward.
5 Limitations
The theoretical result is established on a representative cart-pole rather than the full multi-rigid-body dynamics of a legged robot, and the empirical demonstration is limited in scope.
- Scope: The analytical result is proved for a representative cart-pole, leaving its extension to full legged-robot dynamics as an open theoretical gap.The curriculum is demonstrated on the handstand transition task alone, with further scope for broader narrow-viability tasks.
- Method design: The current schedule uses one uniformly varied stiffness scalar and hand-chosen hyperparameters, rather than per-joint or principled adaptive schedules.Possible extensions include joint-specific targets and adaptive hyperparameter selection.
- Hardware evaluation: Hardware transfer is reported qualitatively, without quantitative success metrics or explicit failure-mode analysis.More rigorous evaluation would further validate sim-to-real transfer.
6 Conclusion
The paper frames failed fixed-dynamics training as a narrow-viability problem and addresses it with an actuator-dynamics curriculum. Cart-pole analysis motivates the approach, while Spot experiments demonstrate simulation convergence and hardware transfer.
- Problem: Training under system-identified actuator dynamics failed to converge because most handstand rollouts terminated before producing useful gradient signal.The paper names these tasks narrow-viability tasks.
- Approach: The Actuator Dynamics Curriculum starts with high joint stiffness and anneals it toward the system-identified hardware value as completed episode lengths grow.The method keeps the deployment target aligned with the hardware actuator parameters.
- Analysis: Higher closed-loop natural frequency under critical damping enlarges the cart-pole viability kernel and the fraction of feasible initial states.This result motivates training with elevated stiffness.
- Outcome: The curriculum converges on the Spot handstand task and transfers zero-shot to hardware across different seeds.The conclusion presents simulated closed-loop joint dynamics as a curriculum-design axis for termination-bottlenecked exploration.
7.1 Proofs
The proof models a cart-pole with critically damped PD cart control and shows that, under bounded-command assumptions, increasing natural frequency expands the finite-horizon viability kernel.
- System model: The representative cart-pole uses bounded cart-position commands and PD gains K = Mωn^2 and B = 2Mωn to impose critical damping.The admissible state box bounds cart position and pole angle.
- Assumptions: The proof assumes bounded-command feasibility and a closeness condition controlling the inversion-correction difference between natural frequencies.The closeness condition requires Δ(ωn,1, ωn,2) ≤ δ.
- Linearized dynamics: Linearization shows that natural frequency enters the cart equation, while the pole equation depends on actuator dynamics through cart acceleration.The command is obtained by inverting the cart equation for a desired cart acceleration.
- Trajectory construction: For ωn,1 ≤ ωn,2, the proof reproduces a viable trajectory at the higher frequency and keeps the adjusted command within bounds under the closeness assumption.Identical initial conditions yield the same cart and pole accelerations along the reproduced trajectory.
- Kernel monotonicity: The resulting inclusion VT(Mωn,1) ⊆ VT(Mωn,2) establishes monotonic growth of the viability kernel with natural frequency.The conclusion follows because the initial state was arbitrary.
7.2 Task MDP Specification
The task specification combines proprioceptive observations, bounded relative joint-position actions, a low-level control interface, reward terms for handstand quality and efficiency, and domain randomization.
- Observations: The policy receives a 45-dimensional concatenated observation containing base velocities, projected gravity, and joint positions relative to the default standing pose.
- Actions: Actions are clipped to [-1, 1] and converted into joint-position setpoints relative to the nominal standing configuration using a fixed action scale.
- Rewards: The reward combines handstand height and orientation, hind-foot clearance, and smooth, low-effort actions.
- Domain randomization: Physical parameters are randomized at startup, episode resets, and random intervals during episodes to improve simulation-to-hardware transfer.
7.3 Training and Simulation Parameters
The training setup is specified through IsaacSim, PPO, network, and Actuator Dynamics Curriculum configurations.
- Environment: The simulation environment is configured through the IsaacSim Environment Configuration listed in Table 5.
- Optimization: The learning procedure uses the PPO configuration listed in Table 6.
- Policy: The policy architecture is specified by the Network Configuration in Table 7.
- Curriculum: The Actuator Dynamics Curriculum hyperparameters are listed in Table 8.
7.4 Additional Ablations
Additional controlled ablations test whether the curriculum’s benefit can instead be explained by action scaling or exploration noise.
- Ablation design: The ablations compare retuned action scaling at K*, action-normalized ADC, and an episode-length-driven action-noise curriculum.
- Results: The normalized-action ADC performs close to the original ADC, whereas retuned action scaling and action-noise curricula provide only modest gains over baseline.
- Evaluation: Final ablation results are reported as mean ± standard deviation across 10 seeds, with each policy evaluated over 1000 episodes under identified dynamics K*.