PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-expert learning of adaptive legged locomotion

Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, Zhibin Li

arXiv:2012.05810v1cs.ROcs.AIcs.LG

TL;DR

Robot locomotion must adapt rapidly to unforeseen situations, but conventional systems often target narrow skills. MELA recombines trained expert policies through a gating network to synthesize adaptive behaviours, and demonstrated multimodal locomotion with responsive transitions on a real quadruped robot.

  • Problem

    Robot locomotion requires rapid coordination of multiple skills under unforeseen situations, including preventing or recovering from falls within a brief time window.

    Source: INTRODUCTION · p. 3

  • Method

    MELA learns adaptive feedback policies by dynamically blending and synthesising neural networks from pre-trained locomotion experts through a gating network.

    Source: DISCUSSION · p. 15

  • Results

    MELA demonstrated multimodal locomotion on a real quadruped robot, including smooth transitions among standing balance, trotting, turning, and fall recovery.

    Source: RESULTS · p. 10, DISCUSSION · p. 15

  • Takeaways & Limitations

    The unified architecture generated diverse adaptive motor skills and responsive behaviour across changing situations and perturbations.

    Source: DISCUSSION · p. 15

    MELA lacks visual and haptic perception, which are important for long-term motion planning.

    Source: DISCUSSION · p. 15

Abstract

from arXiv · show

Achieving versatile robot locomotion requires motor skills which can adapt to previously unseen situations. We propose a Multi-Expert Learning Architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialised by a distinct set of pre-trained experts, each in a separate deep neural network (DNN). Then by learning the combination of these DNNs using a Gating Neural Network (GNN), MELA can acquire more specialised experts and transitional skills across various locomotion modes. During runtime, MELA constantly blends multiple DNNs and dynamically synthesises a new DNN to produce adaptive behaviours in response to changing situations. This approach leverages the advantages of trained expert skills and the fast online synthesis of adaptive policies to generate responsive motor skills during the changing tasks. Using a unified MELA framework, we demonstrated successful multi-skill locomotion on a real quadruped robot that performed coherent trotting, steering, and fall recovery autonomously, and showed the merit of multi-expert learning generating behaviours which can adapt to unseen scenarios.

INTRODUCTION

This section presents MELA, a hierarchical multi-expert architecture that combines pre-trained locomotion specialists through a gating network to dynamically synthesise policies for changing and unforeseen situations. It covers the system’s multimodal, transitional, and fall-resilient locomotion capabilities on a real quadruped, along with its adaptation to unseen simulated conditions and noted perception and sim-to-real limitations.

INTRODUCTION

  • Reactive locomotion is necessary because robots can fall within a second, leaving only about 0.2–0.5 seconds for prevention, while existing systems generally lack autonomous recovery from unexpected situations.The paper motivates reusing and recombining trained feedback policies to improve robustness against failures and changing tasks.Source: INTRODUCTION · pp. 2–3

Our approach

  • The approach uses biologically inspired equilibrium-point control with low-level joint impedance and high-level neural networks producing joint set-points for posture and torque modulation.This separates torque-level control from learning-based motor intelligence and avoids requiring inverse-dynamics computation.Source: Our approach · pp. 4–5

Contributions

  • The framework combines specialised experts in high-dimensional parameter space rather than selecting one policy, avoiding disjointed switching and supporting responsive, versatile feedback control.MELA changes reference joint angles to induce torques through environmental dynamics and is designed to mitigate MoE expert imbalance.Source: Contributions · p. 5, Our approach · p. 5

RESULTS

  • MELA continuously fuses eight expert networks through a gating network that weights their parameters at each timestep, enabling adaptive behaviours across unseen locomotion scenarios.Experts are first trained on distinct tasks, then co-trained with the gating network; eight experts represent seven anticipated situations plus a redundant expert.Source: RESULTS · pp. 6–7, Figure 2 · p. 6

Analysis of skill adaptation and transfer

  • MELA generated emerged steering, turning, and variable-speed-trotting skills, while smoothly coordinating transitions between failure poses and trotting in previously untrained situations.Real-robot tests increased from single-mode recovery to combined steering, trotting, and target-following locomotion.Source: Analysis of skill adaptation and… · p. 12
  • MELA completed real-robot trotting, steering, target following, and fall recovery, including recovery within one second and continued locomotion without human intervention.The synthesised expert produced coordinated movements, smooth transitions, and reactive responses on level and sloped outdoor terrain.Source: Analysis of skill adaptation and… · pp. 13, 15
  • MELA adapted to unseen gravel, inclined, moving-slope, rough-terrain, mass-variation, and motor-failure scenarios while learning nonlinear transitions beyond its eight basic skills.These results support flexible behaviour across unforeseeable situations and locomotion-mode transitions.Source: Analysis of skill adaptation and… · pp. 13–14

DISCUSSION

  • The discussion identifies MELA’s contribution as learning adaptive multimodal locomotion from specialised experts, with agile responses and smooth transitions among balancing, trotting, turning, and fall recovery.Compared with MoE, parameter fusion avoided expert imbalance; MELA expanded diverse skills, blended experts conditionally, and partially preserved original skills, whereas MoE developed degenerated steering experts.Source: DISCUSSION · p. 15

MATERIALS AND METHODS

  • The methods use SAC on the 12-DoF Jueying quadruped with task-specific rewards and minimal state observations comprising body motion, joints, gait phase, and relative goal position.Reward terms combine common physical-stability quantities with task-specific terms for recovery, trotting, and target-following.Source: MATERIALS AND METHODS · pp. 16–17

Action space

  • The action space supplies joint references and impedance control, while action filtering and smoothing loss constrain outputs toward smooth, physically feasible motions.These techniques address compliant interaction and real-actuator constraints rather than relying on unrestricted simulated torque control.Source: Action space · p. 17, Smoothing loss · p. 18

Action filtering

  • Action filtering uses a 5 Hz simulation cutoff and a conservative 3 Hz real-robot cutoff to restrict risky high-frequency actions while preserving necessary dynamic movements.The resulting policies exhibited smooth motions within the actuator bandwidth and were directly executable on the robot.Source: Action filtering · pp. 17–18

Smoothing loss

  • The smoothing loss minimises differences between target joint references and measured positions, encouraging low-effort torque profiles and smoother actions.Adding it directly to the SAC training loss lets action-causality information backpropagate without waiting for accurate Q-function approximation.Source: Smoothing loss · pp. 18–19

MELA training procedure

  • MELA trains eight expert networks and one gating network in two stages: pre-training trotting and fall recovery, followed by co-training experts with the gating network to fuse weights dynamically.Softmax-normalised gating weights construct a nonlinear synthesised policy during runtime, while differentiable connections permit joint backpropagation.Source: MELA training procedure · pp. 19, 21, Figure 6 · p. 20

Notes · Note S1. Data analysis of MELA learning results

MELA’s expert weights correlated with locomotion modes, enabling coherent expert collaboration and smooth, rapid transitions between successive behaviors.

Note S1. Data analysis of MELA learning results

  • MELA produced smooth and quick transitions across successive locomotion modes, as indicated by changing expert weights near mode boundaries.The boundary-weight changes are shown in fig. S12A.Source: Note S1. Data analysis of MELA… · p. 30
  • Expert 3 and 7 weights were jointly high during trotting but low during standing and fall recovery, while experts 5, 6 and 8 peaked during fall recovery.These mode-specific activation patterns are shown in fig. S12C-E.Source: Note S1. Data analysis of MELA… · p. 30
  • Experts 1 and 4 showed complementary activation during left and right trotting, supporting coherent collaboration among trained MELA experts.Expert 1 was activated and expert 4 inhibited during right trotting, with the reverse pattern during left trotting.Source: Note S1. Data analysis of MELA… · p. 30

Note S2. Additional Materials and Methods

This section provides additional training and control-framework details needed to reproduce the presented policies.

Sample collection procedure

The training procedure augments standard sample collection with reference state initialisation and early termination to diversify robot-state samples and improve policy generalisation.

Sample collection procedure

  • These two techniques augment standard collection to produce more varied robot-state samples for better policy generalisation.The paper motivates augmentation by noting that training-sample distributions affect policy-learning outcomes.Source: Sample collection procedure · p. 30
  • Reference state initialisation randomly selects episode starts from reference configurations, exposing the agent to diverse and difficult robot states.This broadens the sample distribution and increases exposure to challenging configurations.Source: Sample collection procedure · p. 30
  • Early termination ends training episodes when the robot reaches undesirable or unrecoverable states, preventing irreversible failures from skewing samples.The procedure is applied during training when the robot encounters a failure state from which it cannot recover.Source: Sample collection procedure · p. 30

Control framework

The control framework uses joint-level impedance control, with MELA regulating high-level position set-points to synthesize compliant behaviour. Neural policies update references at 25 Hz, which are interpolated to 1000 Hz for low-level control.

Control framework

  • Joint torque control gives the real Jueying robot an impedance mode across all joints, while the synthesized DNN generates spring-damper-like trajectory attractors.Mechanical impedance is used because it is robust during physical contact and interaction (59, 60).Source: Control framework · p. 30
  • 25 Hz neural-network position references are interpolated into smooth 1000 Hz trajectories for the low-level impedance controller.The 25 Hz policy update serves as a high-level motion-planning layer, while interpolation and speed limiting generate low-level references (27, 28, 31, 61).Source: Control framework · p. 31
  • 700 Nm/rad stiffness and 10 Nms/rad damping gains provided sufficient swing-leg and stance-foot control, while learned position adjustments enabled active compliance.Experiments showed compliant interaction rather than the stiffness expected from pure position control with the same gains.Source: Control framework · p. 31
  • MELA regulates impedance-controller set-points to actively limit joint torque, reduce stiffness, and produce compliant behaviour through feedback of measured joint positions.This mechanism follows active-compliance and series-elastic-actuator principles by adjusting desired positions relative to measured positions.Source: Control framework · p. 31

Nomenclature

This section defines the mathematical notation used to explain the reward terms. It covers robot orientation, motion, joint states, foot states, and goal-related positions and directions.

  • ϕ_base denotes the orientation vector obtained by projecting the normalised gravity vector into the robot base frame.Source: Nomenclature · p. 32
  • h_world is robot base height, while v_world and v_local denote base linear velocity in world and local heading frames, respectively.Source: Nomenclature · p. 32
  • The notation includes the robot body's world-frame yaw orientation and yaw angular velocity, plus joint-position vector q and joint-velocity vector q˙.Source: Nomenclature · p. 32
  • Goal-related notation covers desired quantities, horizontal goal and base positions, and the base-frame unit vector pointing from the robot base to the goal.Source: Nomenclature · p. 32
  • For each foot, z_foot,n is height, horizontal velocity is represented in the world frame, and horizontal placement is defined in the world frame.Source: Nomenclature · p. 32

Soft Actor Critic

The method uses Soft Actor Critic (SAC), augmenting reward maximization with a maximum-entropy objective. SAC controls policy stochasticity through an automatically tuned temperature parameter and generates bounded robot joint-position references from a Gaussian policy.

Soft Actor Critic

  • SAC optimizes reward maximization together with an additional maximum-entropy objective.Source: Soft Actor Critic · p. 33
  • The temperature parameter α regulates policy stochasticity and exploration, with higher α producing more stochastic policies; SAC tunes α automatically to balance exploration and exploitation.Source: Soft Actor Critic · p. 33
  • A tanh transformation bounds sampled actions within (−1, 1), after which joint-range scaling produces robot joint-position reference targets.Source: Soft Actor Critic · p. 33
  • The policy is Gaussian, with its mean and covariance generated by a neural network.Source: Soft Actor Critic · p. 33

Initialisation and early termination

Training episodes were initialized from task-specific reference states, combined across fall recovery and trotting for multimodal locomotion, with goals sampled within 6 m of the robot. Termination rules differed by task, allowing fall recovery to proceed through failure states needed for autonomous recovery.

Initialisation

  • Multimodal locomotion episodes combined fall-recovery and trotting reference-state datasets, with target goals initialized within a circular area of 6 m radius around the robot.Fall recovery used nine poses spanning common and difficult contact configurations, while trotting used a factory-setting gait sample for imitation.Source: Initialisation and early termination · p. 33

Early termination

  • Locomotion episodes terminated when non-foot body contact occurred, body orientation exceeded 90°, or the episode reached its time limit.These criteria applied to locomotion-related tasks generally.Source: Initialisation and early termination · pp. 33–34
  • Fall-recovery episodes used only the time-limit termination criterion so the robot could experience failure states required for autonomous recovery.The robot therefore proceeded through non-foot contacts and orientations exceeding the 90° threshold during training.Source: Initialisation and early termination · p. 34

Reward design

The reward design uses bounded radial basis functions for continuous physical quantities and separate terms for discrete contact properties. Additional imitation and contact-constraint rewards promote efficient stable trotting, prevent slippage, and encourage symmetric foot placement.

Reward design

  • A bounded radial basis function evaluates reward terms involving continuous physical quantities, while discrete contact properties use separately designed terms.The discrete properties include foot-ground contact, body-ground contact, and ground-contact imitation; individual terms are listed in tables S3 and S4.Source: Reward design · p. 34
  • Reference joint-position and foot-contact rewards imitate an existing trotting gait, enabling the agent to learn stable trotting more efficiently and effectively.The first twelve terms in table S3 directly evaluate task-related physical quantities.Source: Reward design · p. 34
  • Swing-and-stance rewards enforce contact constraints that encourage swing motion and discourage stance-foot movement, thereby preventing slippage.Foot-placement rewards guide feet to distribute around the robot body, promoting more symmetric and stable placement during locomotion.Source: Reward design · p. 34

Note S3. Expert imbalance phenomenon

MoE combines pre-trained expert outputs through gating weights but exhibits expert imbalance, limiting adaptive and transitional skills compared with MELA. Although both policies perform similarly on trained fall-recovery and trotting tasks, MELA learns more balanced steering and follows changing targets more effectively.

MoE implementation

  • MoE synthesises experts by taking a gating-weighted sum of their network outputs, with each expert weight constrained to [0, 1].The final MoE output is formed from expert outputs and variable gating-network weights.Source: Note S3. Expert imbalance phenomenon · p. 34

Training comparison

  • MELA learned faster than MoE during training, achieving higher reward across episodes 120–300 before both policies were trained through 600 episodes for convergence.MoE used the same pre-trained experts and training procedure as MELA, and both were benchmarked after continued training to 600 episodes.Source: Note S3. Expert imbalance phenomenon · pp. 34–35

Qualitative comparison

  • MELA learned adaptive steering and target-following behaviours that MoE failed to acquire, despite similar performance in trained fall-recovery and forward-trotting tasks.In the shared target-following test, MELA actively steered left and right and followed the moving target, whereas MoE could not walk straight toward it or orient its body toward travel.Source: Note S3. Expert imbalance phenomenon · p. 35

Expert imbalance

  • MoE’s asymmetric gait and drifting yaw arose from unbalanced left and right steering skills, while MELA maintained symmetric steering and progressively matched the target’s heading.MELA turned left and right equally well with a yaw profile symmetric around zero; MoE showed unresponsive right steering, heading offsets, and an offset body pose.Source: Note S3. Expert imbalance phenomenon · p. 35
  • Approximately half of MoE’s experts were imbalanced and under-trained, producing limited skill diversification that disadvantaged adaptive motor-skill learning.t-SNE showed only five distinct expert clusters for fall recovery and four for trotting, with remaining experts overlapping or scattered.Source: Note S3. Expert imbalance phenomenon · pp. 35–36

Figures

The figures show that MELA outperforms MoE in target-following with fall recovery, steering, and trotting, while synthesising specialised expert behaviours across transitions and disturbances. They also document expert specialisation, real-robot recovery, and training choices supporting adaptive locomotion.

  • Using more than 8 expert networks did not improve task performance and slowed convergence.Source: Figure S1 · p. 36
  • The eight experts specialised in right turning, balance stabilisation, large-step trotting, left turning, posture control, back righting, small-step trotting, and lateral rolling.Source: Figure S3 · p. 37
  • MELA synthesised smooth adaptive transitions for steering on the spot, steering during fall recovery to trotting, slipping recovery, large-impact recovery, and cliff-fall recovery.The synthesised policy produced outputs with characteristics distinct from all eight basic experts during stable trotting, impact, and recovery, indicating interpolated nonlinear synthesis.Source: Figure S4 · p. 38, Figure S10 · p. 41, Figure S11 · p. 42
  • During real fall-resilient locomotion, torque saturation caused stumbling and tripping, but recovery occurred within 1 second despite roll peaks of 0.47 rad and pitch peaks of 0.7 rad.The robot also maintained steady posture while trotting on uneven grass at approximately 0.25 m/s.Source: Figure S8 · p. 40, Figure S9 · p. 41, Figure S10 · p. 41
  • Collaborating expert groups activated together during real multimodal locomotion, including experts (3, 7) for trotting, (5, 6, 8) for fall recovery, and (1, 4) for turning.Source: Figures · p. 44
  • MELA successfully followed a moving target with fall recovery, steering, and trotting, whereas MoE developed yaw drift from a degenerated steering expert and imbalanced expert representations.MELA formed eight unique clusters, while MoE had only half favoured experts with discernible clusters and others scattered or overlapped.Source: Figures · p. 54

Tables

The tables document expert specialisation, robot and training specifications, reward design, task-dependent reward usage, and state-input selection across MELA components.

Expert specialisation

  • Table S1 identifies the top two dominant experts activated for each motor skill, reflecting that skills typically activate multiple experts.The distribution matrix complements Fig. 4A by summarising expert specialisations over motor skills.Source: Table S1 · p. 59

Experimental specifications

  • The supplementary tables specify the Jueying quadruped, joint-level proportional-derivative controller parameters, and SAC hyperparameters used in the experiments.These tables provide implementation specifications for the robot, low-level controller, and reinforcement-learning algorithm.Source: Table S2 · p. 59, Table S6 · p. 62, Table S7 · p. 62

Reward components

  • The reward formulation includes terms for robot heading, goal position, swing and stance behaviour, and foot placement.These terms are expressed through the corresponding task-oriented reward definitions.Source: Tables · p. 60

Reward design

  • Table S4 reports task-specific reward-term weights: trotting and fall recovery use subsets, whereas multimodal MELA locomotion uses all reward terms.Table S3 provides detailed descriptions of the individual reward terms.Source: Table S3 · p. 60, Table S4 · p. 61

State inputs

  • For target-following locomotion, the MELA gating network receives normalised gravity, angular and linear velocities, and goal position as task-oriented inputs.The synthesised MELA network receives all feedback except the goal position.Source: Table S5 · p. 62

Related papers

Loading 2012.05810v1…