Source-linked AI summary
Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior
Gabriel B Margolis, Pulkit Agrawal
TL;DR
Learned locomotion policies can fail in out-of-distribution environments where retraining is slow or environment properties are difficult to model. The paper introduces MoB, a single policy with tunable behavior parameters, and demonstrates a robust controller whose diverse behaviors support generalization and downstream tasks. Its main limitation is reduced flat-ground sprinting performance and a current reliance on manual behavior tuning.
Problem
Locomotion policies lack a fast tuning mechanism when deployment environments differ from training, making iterative reward and environment redesign costly.
Method
MoB trains a conditional policy to solve the same locomotion tasks through multiple behavior-parameterized strategies based on gait, foot-swing, posture, and contact properties.
Results
The released controller executes diverse structured behaviors and enables real-time tuning across unseen terrains, dynamics, and downstream tasks.
Takeaways & Limitations
A single learned controller can provide interpretable behavior knobs for adapting motor skills and composing locomotion behaviors without training a new policy.
Takeaways & Limitations
MoB can reduce flat-ground sprinting performance, and the current system requires a human pilot to tune behavior during deployment.
Abstract
from arXiv · showhide
Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more. Video and code release: https://gmargo11.github.io/walk-these-ways/
1 Introduction
Out-of-distribution deployment can expose locomotion failures that are costly or impossible to fix through repeated simulation and retraining. Multiplicity of Behavior (MoB) instead encodes diverse strategies in one tunable policy, allowing behavior selection for unseen environments and tasks.
- Motivation: Simulation restrictions can leave deployment scenarios unmodeled, making reward or environment redesign followed by retraining a tedious adaptation loop.Some properties, such as thick-bush compliance and sensing, may also be difficult to model or identify accurately.
- Motivation: Under-specified locomotion tasks admit multiple equally successful training solutions whose generalization differs across out-of-distribution scenarios.Flat-ground walking constrains body velocity but does not uniquely determine leg motion or torso height.
- Multiplicity of Behavior: MoB conditions one policy on behavior parameters so the same observation history can produce different walking behaviors for real-time testing in unseen scenarios.Varying parameters offers quicker iteration than retraining and supports collecting online demonstrations from a human pilot.
- Multiplicity of Behavior: The approach assumes that some behaviors successful in the restricted training environment will also succeed in the out-of-distribution target environment.The paper evaluates a policy trained on flat ground across uneven ground, stairs, external shoves, and constrained spaces.
- Contribution: The released controller executes diverse structured quadruped behaviors and provides an interpretable high-level interface for locomotion applications and demonstrations.The authors position MoB as a practical tool for out-of-distribution generalization while noting that its implementation uses locomotion expertise.
- Contribution: The controller demonstrates transitions among trotting, pronking, pacing, and bounding while alternating frequencies between 2Hz and 4Hz.Figure 2 reports timing references and foot-sensor contact states for the displayed gait phases.
2 Background
Prior locomotion research uses auxiliary rewards, parameterized styles, diversity objectives, and hierarchical controllers to shape or reuse different behaviors. These approaches motivate MoB while differing in how behaviors are learned, represented, and controlled.
- Auxiliary Rewards: Auxiliary rewards bias learned locomotion toward contact schedules, smooth actions, energy consumption, or foot clearance to help compensate for the sim-to-real gap.The passage gives foot clearance as an example of a bias that can improve robustness to uneven terrain or body sinking.
- Parameterized Style: Parameterized-style methods include gait parameters in auxiliary rewards for bipedal locomotion, such as contact timing offsets and target foot placements.The cited approach inspired the paper’s reward structure but has a smaller command-parameter space that does not support the same compositional applications.
- Diversity Objectives: Quality-diversity methods learn libraries of high-performing, diverse policies by enforcing trajectory novelty, typically with evolutionary optimization.Prior work reports benefits for optimization and online skill reuse.
- Hierarchical Control: Hierarchical gait-parameter controllers have been applied to downstream objectives such as energy minimization and vision-guided foot placement.These systems used model-predictive low-level controllers rather than gait-conditioned learned controllers.
- Reward Structure: Table 1 organizes the paper’s reward structure into task rewards, augmented auxiliary rewards, and fixed auxiliary rewards.This categorization distinguishes task objectives from behavior-dependent and behavior-independent shaping terms.
3 Method
MoB trains a conditional locomotion policy to complete the same velocity-tracking task through different behavior parameters controlling gait, posture, foot placement, and foot swing. Training combines task rewards with behavior-conditioned auxiliary rewards, domain randomization, and sim-to-real modeling, while the resulting controller is compared with a gait-free baseline.
- Task Structure for MoB: The policy π(·|c_t, b_t) conditions locomotion on task commands and behavior parameters, enabling multiple solutions to the same task.Behavior parameters are designed around useful gait properties rather than an unsupervised diversity metric.
- Task Structure for MoB: The 8-dimensional behavior vector specifies contact-pattern timing, stepping frequency, body height and pitch, stance width, and footswing height.Timing offsets express pronking, trotting, bounding, pacing, and continuous interpolations such as galloping.
- Task Structure for MoB: Augmented auxiliary rewards increase when realized motion matches the requested behavior while remaining compatible with the velocity-tracking task.The Raibert Heuristic adjusts desired foot positions to account for body velocity and contact schedule, avoiding penalties for necessary foot motion during turning.
- Learning Diversified Locomotion: Behavior and task commands are resampled during episodes, with stable symmetric contact patterns sampled first and remaining behavior parameters sampled independently and uniformly.This training procedure is intended to support graceful transitions among diverse useful gaits.
- Materials: The policy uses observation, command, behavior, action, and timing histories, while domain randomization and actuator, latency, and system-identification models support sim-to-real transfer.Training uses flat ground without terrain-geometry randomization; deployment runs on a Unitree Go1 at 50 Hz.
- Design Choices for Sim-to-Real Transfer: Compared with the gait-free baseline, structured behavior tuning enables interventional comparisons of gait properties and performance criteria within one policy.Several structured gaits surpass the unconstrained gait in power efficiency across the speeds shown in Table 3.
4 Experimental Results
Experiments show that a single MoB controller can switch among structured gaits and tune behavior parameters for novel tasks, unseen terrains, disturbances, and agile maneuvers.
- Flat-ground evaluation: After sim-to-real transfer, the controller consistently tracks gait parameters while switching among trotting, pronking, bounding, and pacing.Transitions alternate command frequencies between 2 Hz and 4 Hz.
- Tuning for new tasks: MoB behavior parameters can tune performance on a new objective: selected contact schedules, body heights, and frequencies consume less simulated energy than the gait-free policy across all speeds.The metric is mechanical power consumption.
- Tuning for new environments: On platform terrain with platform heights up to 16 cm, modulating contact schedule or footswing height outperforms the gait-free policy.The evaluation reports mean reward and mean survival time relative to a 10 s maximum episode.
- Tuning for new environments and tasks: MoB supports novel physical tasks, including smooth stair and curb traversal, guided movement through thick bushes, crouching under a 22 cm bar, and bracing against shoves.The robot body thickness is 13 cm, leaving 9 cm of clearance under the bar.
- Speed and behavior adaptation: Behavior transitions can occur rapidly at high speed, enabling an agile leap demonstration that crosses a gap wider than the robot’s body length.The robot accelerates to 3 m/s, switches from trotting to pronking, and completes a 60 cm leap phase.
- Speed and behavior adaptation: Precisely timed gait, body-height, and velocity modulation produces a dance routine synchronized to music at 90 bpm.The routine uses gait phases and frequencies of 1.5 Hz and 3 Hz to create multiple beat intervals.
5 Discussion and Limitations
MoB provides a structured, controllable space of locomotion behaviors, but its current deployment requires manual human tuning and can reduce flat-ground sprinting performance.
- Limitations: Adding MoB can limit in-distribution flat-ground sprinting performance because the behavior parameterization is restrictive for combinations of high linear and angular velocity.The authors identify controlling the tradeoff between task performance and reward shaping as future work.
- Discussion: MoB gives one learned policy controllable behavioral alternatives that can be used as tunable parameters for motor skills in unseen test environments.The paper describes these alternatives as a structured space of diverse behaviors for each training-distribution state and task.
- Discussion: The current system requires a human pilot to manually tune behavior during deployment.The authors suggest future automation through imitation from demonstrations or hierarchical self-tuning.
A Training Details
Training samples gait-centered curricula, velocity commands, stepping frequencies, and body heights, while early learning favors pronking and trotting over pacing and bounding.
- Training details: Pronking and trotting are easier to learn and tend to dominate early training, whereas pacing and bounding can later become preferred for downstream tasks.The latter gaits can yield good performance once discovered.
- Training details: The appendix provides ranges for domain and command randomization and hyperparameters for PPO and curriculum training.
- Training details: The curriculum samples around trotting, pronking, bounding, and pacing, then samples velocity commands, stepping frequency, and body height before rollout and reward computation.The curriculum updates after policy and simulator rollouts.
B Teleoperation Interface
The teleoperation interface maps remote-control inputs to gait parameters and supports switching gaits at any time, with additional parameters available beyond the displayed mapping.
- Teleoperation interface: Front bumpers toggle control modes for the large parameter set, while rear bumpers can trigger preprogrammed sequences such as dancing and leaping.
- Teleoperation interface: Remote-control inputs map to gait parameters during teleoperation, allowing the user to change gaits at any time.Continuous interpolation between contact patterns is supported but not mapped in the figure.
- Teleoperation interface: The controller supports lateral velocity, although it is excluded from the displayed mapping.
C Extended Performance Analysis
The extended analysis examines how gait parameters affect tracking, terrain traversal, and real-world robustness. Diverse behaviors improve generalization to unseen settings, while some gait constraints reduce flat-ground performance.
- Higher gait frequency is necessary for good tracking performance during high-speed running.Figure 10 evaluates the impact of gait frequency on high-speed locomotion.
- Higher footswing heights improve platform traversal on out-of-distribution terrain, outperforming the gait-free policy.The comparison concerns time to failure on platform terrain.
- Backward walking produces fewer failures than forward walking in the random platforms environment.Possible explanations include recovery strategies depending on knee orientation and the robot’s weight distribution.
- The controller traversed stairs, granular and slippery terrain, and external perturbations during several hours of real-world testing.During shove tests, the state estimator inferred lateral velocity changes and adapted joint torques and contact timing to stabilize the robot.
D Contact Schedule Parameterization
The controller parameterizes contact schedules through commanded gait timing and smooth contact-state functions. These parameters support multiple gaits and expose tradeoffs between gait constraints, tracking performance, and terrain robustness.
- At each control timestep, the controller advances global timing using commanded stepping frequency relative to control frequency.Separate per-foot timing variables are then clipped between 0 and 1.
- The desired contact state is computed from per-foot timing variables using smooth functions that transition between stance and swing.The formulation approximates a Von Mises distribution to create smooth contact transitions.
- The architecture conditions the policy and simulator on gait parameters, with a curriculum engine periodically resampling them according to reward.The training setup uses PPO and curriculum hyperparameters listed in the supplementary tables.
- Higher footswing height improves time to failure on platform terrain, while higher gait frequency improves high-speed tracking.These results show that different behavior parameters support different generalization objectives.
- Pronking, pacing, and bounding preserve lower-range flat-ground performance while sacrificing some training-task performance to obtain behavioral diversity.The gait-free policy performs slightly better at the lowest and highest speeds in the frequency comparison.