Source-linked AI summary
Multi-expert learning of adaptive legged locomotion
Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, Zhibin Li
TL;DR
Adaptive quadruped locomotion requires rapid responses to unseen situations and coordinated transitions among locomotion modes. The paper proposes MELA, which learns from pretrained expert policies and dynamically fuses them through a gating network. On a real robot, MELA demonstrated multimodal locomotion with agile adaptation and fast responses, while its current training and sensing scope remains limited.
Problem
Legged robots generally lack autonomous adaptive actions for unexpected situations, where rapid coordination is needed to prevent or recover from failures.
Method
MELA uses pretrained fall-recovery and trotting experts, co-trains them with a gating neural network, and dynamically fuses their parameters into adaptive feedback policies.
Results
MELA achieved multimodal locomotion with agile adaptation and fast responses across standing balancing, trotting, turning, and fall recovery on a real robot.
Takeaways & Limitations
Dynamically synthesising policies from specialised experts supports responsive adaptive behaviours in changing locomotion situations.
Takeaways & Limitations
MELA lacks visual and haptic perception, and simulation-to-real discrepancies may become problematic as the number of locomotion tasks increases.
Abstract
from arXiv · showhide
Achieving versatile robot locomotion requires motor skills which can adapt to previously unseen situations. We propose a Multi-Expert Learning Architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialised by a distinct set of pre-trained experts, each in a separate deep neural network (DNN). Then by learning the combination of these DNNs using a Gating Neural Network (GNN), MELA can acquire more specialised experts and transitional skills across various locomotion modes. During runtime, MELA constantly blends multiple DNNs and dynamically synthesises a new DNN to produce adaptive behaviours in response to changing situations. This approach leverages the advantages of trained expert skills and the fast online synthesis of adaptive policies to generate responsive motor skills during the changing tasks. Using a unified MELA framework, we demonstrated successful multi-skill locomotion on a real quadruped robot that performed coherent trotting, steering, and fall recovery autonomously, and showed the merit of multi-expert learning generating behaviours which can adapt to unseen scenarios.
INTRODUCTION
The paper addresses adaptive quadruped locomotion by learning reactive behaviours that recombine trained skills for unforeseen situations. MELA uses hierarchical deep reinforcement learning and multi-expert fusion to support responsive recovery, multimodal transitions, and robust locomotion.
- Motivation: Legged robots still lack autonomous adaptive actions for unexpected situations, although falls can occur within a second and prevention may have only 0.2–0.5 seconds.The paper frames immediate coordination across locomotion modes as critical for preventing or recovering from failures.
- Approach: MELA combines multiple expert neural networks with a gating neural network that dynamically fuses them into an adaptive feedback policy.The architecture reuses and recombines trained motor skills rather than selecting only one expert at a time.
- Approach: The framework is trained in two stages, first learning distinct policies such as fall recovery and trotting, then co-training initialized expert groups with the gating network.Random initial configurations during simulation help expose the policy to diverse robot states.
- Results: The learned recovery policy handled varied fall scenarios through online feedback, while trotting remained steady across hard, slippery, and obstacle-covered surfaces.Recovery strategies included rolling, active righting, standing up, and stepping.
- Results: On the robot, MELA demonstrated multimodal locomotion involving standing, trotting, turning, steering, target following, fall recovery, and dynamical transitions.The reported behaviours include both non-cyclic recovery motions and rhythmic trotting, with user-command following.
Note S1. Data analysis of MELA learning results
MELA’s expert weights changed smoothly at locomotion-mode boundaries and correlated coherently with the corresponding modes. Different expert groups were selectively activated for trotting, fall recovery, and directional trotting.
- Changing expert weights around locomotion-mode boundaries indicate smooth and quick transitions across successive modes.
- The summed weights of collaborating experts correlated with locomotion modes, indicating coherent expert activation.
- Experts 3 and 7 were highly weighted during trotting but weakly weighted during standing and fall recovery.
- Experts 5, 6, and 8 peaked during fall recovery while remaining low in other locomotion modes.
- Experts 1 and 4 showed complementary activation during left and right trotting.
Note S2. Additional Materials and Methods
The section provides additional training and control-framework details required to reproduce the presented policies.
- The supplementary section supplies training-procedure and control-framework details needed to reproduce the presented policies.
Sample collection procedure
MELA augments standard sample collection to expose the agent to diverse and challenging robot states, improving the opportunity for policy generalisation.
- Two techniques augment standard sample collection: reference state initialisation and early termination.
- Episodes begin from randomly selected reference configurations to diversify sampled robot states and expose the agent to difficult cases more often.
Control framework
The control framework combines neural-network position references with low-level impedance control, while smoothing the references to encourage gentle, compliant torque profiles.
- Joint torque control configures impedance for all joints on the real Jueying robot because mechanical impedance is robust during physical contact.
- The neural-network policy outputs joint-position references at 25 Hz, which are interpolated to 1000 Hz for low-level impedance control.
- Medium low-level PD gains allow desired or lower impedance to be rendered by adjusting high-level position set-points to buffer impacts.
- MELA regulates the impedance-controller set-point to achieve compliant behaviour through active adjustment of position references.
Nomenclature
The nomenclature defines robot state, motion, goal, foot, orientation, and joint variables used in the reward equations.
- Robot orientation is represented by the projection of the normalised gravity vector into the robot base frame.
- The notation includes robot base height, linear velocity, local heading-frame velocity, yaw orientation, and yaw angular velocity.
- Joint configuration is represented by vectors of all joint positions and joint velocities.
- Foot-related variables describe each foot’s height, horizontal velocity, and horizontal placement in the world frame.
- Goal-related notation specifies the goal’s horizontal world-frame position, the robot base’s horizontal world-frame position, and the base-frame unit vector toward the goal.
Soft Actor Critic
The policy uses Soft Actor Critic, extending reward maximisation with an entropy objective to balance exploration and exploitation. Actions are sampled from a Gaussian policy, squashed by tanh, and scaled into joint-position references.
- Soft Actor Critic optimises the expected reward together with an additional maximum-entropy objective.
- The temperature parameter α controls policy stochasticity and exploration by weighting the entropy term, with higher α producing more stochastic policies.
- The squashed action is scaled by the joint range to form target joint-position references for the robot.
- The Gaussian policy generates action means and covariances from a neural network, then applies tanh to constrain samples to (−1, 1).
Initialisation and early termination
Episodes are initialised from task-specific reference states and terminated using contact, orientation, or time conditions, with fall recovery retaining only the time limit.
- Initialisation: Fall-recovery episodes begin from nine distinct poses spanning standing, sprawling, lying, crouching, and kneeling configurations.
- Early termination: Locomotion episodes terminate when a non-foot body part contacts the ground, body orientation exceeds 90°, or the time limit is reached.
- Early termination: Fall recovery uses only the time-limit criterion because ground contact and extreme orientation are states required for autonomous recovery learning.
Reward design
Rewards use bounded radial basis functions for continuous quantities and separate terms for discrete contacts. Reference gait imitation and foot-placement terms promote stable, symmetric locomotion.
- Continuous rewards: A radial basis function provides a bounded reward for continuous physical quantities, using the desired value and a width-controlling parameter.
- Discrete rewards: Discrete properties, including foot-ground contact, body-ground contact, and ground-contact imitation, use separately designed reward terms.
- Gait imitation: Reference joint-position and foot-contact rewards imitate an existing trotting gait, helping the agent learn stable trotting more efficiently and effectively.
- Locomotion stability: Swing-and-stance rewards discourage stance-foot movement and prevent slippage, while foot-placement rewards encourage symmetric and stable placement around the robot body.
Note S3. Expert imbalance phenomenon
The MoE comparison shows that weighted expert outputs and shared training can produce faster learning in some phases, but expert imbalance limits adaptive and transitional behaviours. MELA instead maintains more diversified expert representations and performs better on unseen combinations of steering and target following.
- MoE formulation: MoE synthesises its policy by taking a weighted sum of outputs from multiple expert networks using gating-network weights.The final output is formed from expert outputs indexed by n and variable weights α_n.
- Learning comparison: MELA achieved higher reward than MoE during episodes 120–300, although both policies were trained to convergence for comparison.The policies used the same pre-trained experts and training procedure, and training continued to 600 episodes.
- Trained scenarios: For trained fall-recovery and forward-trotting scenarios, MoE and MELA showed similar performance because they shared pre-trained experts and training settings.Their differences emerged in behaviours that were not pre-trained.
- Adaptive behaviours: MELA actively steered left and right and followed the moving target, whereas MoE failed to walk straight toward the target and remained misoriented.The comparison targets newly emerged behaviours and transitional skills rather than the pre-trained tasks.
- Expert imbalance: MoE developed asymmetric steering, with unresponsive right steering and a persistent heading offset, while MELA produced symmetric left-right steering around zero yaw.The analysis attributes MoE’s imbalance to some steering experts being favoured and insufficiently trained.
- Expert representations: t-SNE analysis found approximately half of MoE experts imbalanced and under-trained, limiting diversification of skills needed for adaptive motor behaviours.Only five experts formed distinct clusters for fall recovery and four for trotting; the remaining experts were scattered.
Figures
The supplementary figures compare learning, engineered-controller baselines, expert specialisation, simulated unseen scenarios, and real-robot adaptive locomotion. Together they document steering, target following, fall recovery, terrain adaptation, expert-weight transitions, and posture recovery.
- Learning curves: Using more than 8 expert networks did not improve task performance and resulted in slower convergence.The learning-curve comparison identifies more than eight experts as unnecessary for the tested setting.
- Baselines: The engineered fall-recovery controller used a fixed motion sequence and took more than 12 seconds to stand, while the manufacturer gait supplied trotting control.These experiments provide baseline behaviours for comparison with learned locomotion.
- Expert specialisation: Experts 1–8 specialised respectively in right turning, balance stabilisation, large-step trotting, left turning, posture control, back righting, small-step trotting, and lateral rolling.The activation patterns identify each expert’s highest-activation motor skill.
- Simulation behaviours: Simulation figures show MELA adapting trotting speed to a moving target and producing steering, standing, fall recovery, and seamless transitions in unseen situations.The transition demonstrations use smooth changes in desired joint positions during steering and recovery-to-trotting sequences.
- Fall recovery: In the real fall-resilient experiment, torque saturation caused stumbling and tripping, followed by fall recoveries; roll and pitch peaks reached 0.47 rad and 0.7 rad, with recovery within 1 second.The saturated hip-pitch torque prevented the affected leg from moving as intended.
- Robustness tests: Simulated robustness tests varied terrain geometry and robot mass, including mass changes of ±25%, ±30%, and ±40% from the original 40 kg.The tested terrains included gravel, inclined surfaces, a moving slope, and rough terrain.
- MELA–MoE comparison: The real target-following comparison shows MELA steering left and right to follow the moving target, while MoE yaw drifted because of a degenerated steering expert.The associated global-position and yaw analysis contrasts the two policies during target following.
Tables
The supplementary tables specify expert specialisations, robot hardware, reward construction, task-dependent state inputs, controller parameters, and SAC hyperparameters. The listed equations and variables describe locomotion targets and control-related quantities.
- Expert specialisations: Table S1 reports the distribution of expert specialisations over motor skills and highlights the two dominant experts activated for each skill.The table accompanies evidence that multiple experts can activate for a single motor skill.
- Reward design: Table S3 details the individual reward terms used to train the locomotion policies.The supplied table description identifies the reward terms but does not enumerate them.
- Control quantities: The control variables include robot heading, goal position, swing and stance, and foot placement quantities.These entries are presented as task or locomotion-related quantities in the supplementary material.
- Task configuration: Table S4 gives task-specific reward-term weights, with trotting and fall recovery using subsets and multimodal MELA locomotion using all reward terms.The weighting differs according to the task being trained.
- State inputs: Table S5 specifies task- and network-dependent state inputs, including gravity, angular and linear velocities, and goal position for the target-following gating network.The synthesised MELA network receives feedback inputs except the goal position.
- Implementation parameters: Tables S6 and S7 provide the proportional-derivative controller parameters and SAC hyperparameters used by the implementation.The supplied passages identify these tables’ contents without listing their parameter values.