PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou

arXiv:2610.37441v1cs.ROcs.LG

TL;DR

Hip-exoskeleton controllers must account for a wearer adapting to assistance, while human-in-the-loop tuning requires repeated per-user experiments. MARL-Exo co-adapts a human model and exoskeleton policy in simulation using a physiological metabolic reward. The policy transferred without per-user tuning and lowered metabolic cost relative to the passive device in the tested adults.

  • Problem

    Exoskeleton assistance changes gait and muscle recruitment, while human-in-the-loop optimization requires long experiments repeated for each user and condition.

    Source: §I · p. 1

  • Method

    MARL-Exo co-adapts a full-body human model and hip-exoskeleton policy in simulation, with a physiological muscle-energetics term in the exoskeleton reward.

    Source: §I · p. 2, C. Two-Stage Training · p. 4

  • Results

    18.3 ± 6.6% lower net metabolic rate with MARL-Exo than Passive in eight participants; six of eight also had lower net cost than No-Exo, with a cohort mean reduction of 7.4 ± 10.0%.

    Source: §V · p. 10

  • Takeaways & Limitations

    The metabolically informed policy transferred to eight healthy adults without per-user tuning and reduced net metabolic rate relative to the passive device.

    Source: §VI · p. 11

    The human study tested eight healthy adults walking on a treadmill at one speed after 10 minutes of familiarization with each controller.

    Source: §V · p. 11

Abstract

from arXiv · show

Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARL-Exo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 $\pm$ 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 $\pm$ 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their No-Exo cost. Conclusion: A metabolically informed policy co-adapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-the-loop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing.

I. INTRODUCTION

Hip exoskeleton assistance must account for a wearer who adapts to the device, while per-user human-in-the-loop tuning requires repeated experiments. This study uses staged co-adaptation and a physiological metabolic reward to test stable training and transfer to users.

A. Human–Exoskeleton Coupling

  • Human–device interaction changes gait mechanics and muscle recruitment, making controller tuning difficult as both wearer and device adapt.Human-in-the-loop optimization tunes directly on the wearer but requires long experiments repeated for each user and condition.Source: §I · p. 1

I. INTRODUCTION

  • Prior work leaves mutual adaptation largely implicit and commonly uses activation or work proxies rather than a physiological metabolic-rate term in the exoskeleton reward.The authors identify metabolic energy as the quantity a wearer ultimately experiences and note that simulation can compute it directly.Source: §I · pp. 1–2
  • A full-body human model and hip-exoskeleton policy co-adapt in two stages, using a physiological muscle-energetics term in the exoskeleton reward.The model has 304 Hill-type muscle actuators; the policy uses delayed hip kinematics and its recent commands.Source: §I · p. 2
  • The study compares metabolic-only and metabolic-plus-power rewards; removing explicit power shaping reduced modeled muscle metabolic power by 13.8%, versus 8.8% with shaping.On hardware, the deployed policy reduced net metabolic rate by 18.3% relative to the passive device in eight healthy adults.Source: §I · p. 2

C. Two-Stage Training

Training first establishes device-free locomotion, then introduces the exoskeleton for joint policy optimization. The two exoskeleton rewards retain metabolic power but differ in whether they also shape mechanical power.

C. Two-Stage Training

  • Training separates gait acquisition from co-adaptation: the human policy first learns stable walking without the device, then both policies optimize jointly with the device introduced.This ordering avoids training the exoskeleton against a human policy that has not yet acquired stable gait.Source: C. Two-Stage Training · p. 3
  • Stage 1 trains the human policy alone at varied speeds; its reward combines locomotion, velocity, and foot-placement terms with activation-based effort and auxiliary regularizers.The activation effort term is distinct from the physiological energetics term used in the exoskeleton reward.Source: C. Two-Stage Training · pp. 3–4
  • The exoskeleton reward includes action smoothness and magnitude penalties, metabolic power, human-reward coupling, and optional joint-power shaping.The Metabolic controller sets the power-term weight to zero; Metabolic + Power uses a weight of 0.005, while both retain the other listed terms.Source: C. Two-Stage Training · p. 4
  • The metabolic reward sums an Umberger/Uchida muscle-energetics model over 92 actuators spanning hip flexion and extension.Each muscle’s metabolic rate combines heat and mechanical work, scaled by its muscle mass.Source: C. Two-Stage Training · p. 4

D. Simulation Evaluation

The simulation evaluation compares unassisted walking with two active controllers that differ in power shaping. It analyzes muscle effort and metabolic power across selected lower-limb actuators and gait cycles.

D. Simulation Evaluation

  • Three conditions were analyzed: NoExo, Metabolic + Power (PowerExo), and Metabolic without the power term (EnergyExo).EnergyExo is the policy later deployed on hardware and called MARL-Exo in the human experiment.Source: D. Simulation Evaluation · p. 5
  • The analysis covers 31 actuators per leg representing 15 anatomical muscles, grouped by functional direction and by joint.Biarticular muscles are included in groups at more than one joint.Source: D. Simulation Evaluation · p. 5
  • Outcomes include muscle-contributed joint torques, activation, and metabolic power, summarized bilaterally across gait cycles for one deterministic episode per condition.The analyzed muscle set is sagittal-plane only, so absolute metabolic values underestimate whole-body cost, although conditions use the same partial-muscle total.Source: D. Simulation Evaluation · p. 5

A. Hip Exoskeleton Hardware and Deployment

The trained MARL-Exo policy was deployed on a bilateral hip exoskeleton with IMU-derived hip motion as its input. It ran onboard at 50 Hz with the simulated observation interface and online filtering.

A. Hip Exoskeleton Hardware and Deployment

  • The trained policy ran on a custom bilateral hip exoskeleton weighing 5.94 kg, with one actuated flexion/extension degree of freedom per side.A passive abduction/adduction hinge supports fitting and lateral compliance.Source: A. Hip Exoskeleton Hardware and… · p. 5
  • Three wireless IMUs measured pelvis–femur hip angle and angular velocity; the Raspberry Pi 5 policy consumed the same 18-dimensional observation used in simulation.The measured signals were low-pass filtered online at 3 Hz, matching the training observation filter.Source: A. Hip Exoskeleton Hardware and… · p. 5
  • At 50 Hz, the onboard actor produced normalized torque commands that were scaled by preset per-hip limits and sent to the actuators.Trials used a software torque limit and emergency stop, with an operator supervising throughout.Source: A. Hip Exoskeleton Hardware and… · p. 5

B. Human Validation Framework

Eight healthy adults tested MARL-Exo and SMAT using a symmetric treadmill protocol that included repeated No-Exo and Passive reference conditions. Indirect calorimetry measured net metabolic rate, with Passive retaining device mass but no commanded torque.

B. Human Validation Framework

  • Passive walking retained the full device mass with zero commanded torque and back-drivable actuators, isolating the hardware burden from active assistance.No-Exo, Passive, and standing-rest conditions were repeated before and after the active trials, and repeats were averaged to assess fatigue and drift.Source: B. Human Validation Framework · p. 6
  • MARL-Exo used a 12 N·m torque limit; SMAT used 15 N·m because its output saturated near 0.8, matching both controllers at about 12 N·m realized assistance.The controllers used identical hardware and differed only in policy.Source: B. Human Validation Framework · p. 6
  • Eight healthy adults completed approximately 6-minute treadmill trials at 1.25 m/s, with expired gases measured breath-by-breath by indirect calorimetry.The MARL-Exo and SMAT active trials followed familiarization; active-controller order was randomized.Source: B. Human Validation Framework · p. 6
  • Net metabolic rate was calculated from the last 2 minutes of each condition after subtracting standing-rest rate, and comparisons assessed device penalty and assistance benefit.The contrasts were Passive versus No-Exo, MARL-Exo versus Passive, and MARL-Exo versus No-Exo.Source: B. Human Validation Framework · p. 6

C. Biomechanical and Statistical Analysis

Biomechanical analysis compared simulated hip motion and assistance with a biological reference and summarized per-cycle kinematics, assistance output, and participant-level statistical comparisons. Both active controllers altered hip kinematics, while their assistance torque and power summaries differed.

C. Biomechanical and Statistical Analysis

  • Hip kinematic metrics were averaged across cycles within each subject and across hips; cadence and stride length were reciprocal by construction at fixed belt speed.Cycles were segmented at peak hip flexion because ground-reaction-force sensors were not used.Source: C. Biomechanical and Statistical… · p. 6
  • Estimated assistance power was commanded torque multiplied by IMU-derived hip angular velocity; the positive-power ratio describes assistance timing relative to hip motion, not metabolic efficiency.Comparisons used paired two-tailed t-tests across eight participants after Shapiro–Wilk screening, with Bonferroni correction within outcome families.Source: C. Biomechanical and Statistical… · p. 6

1) Hip joint kinematics and kinetics:

  • Both active controllers reduced peak hip extension versus NoExo, with a larger reduction under PowerExo, and slightly deepened peak flexion, most under PowerExo.EnergyExo had τRMS 13.62 vs. 13.12 N·m and MPP 19.98 vs. 20.64 W compared with PowerExo; the plotted assistance used τmax = 18 N·m.Source: 1) Hip joint kinematics and kinetics · pp. 6–7, Figure 3 · p. 7

2) Muscle activation and metabolic energy by functional group:

In simulation, metabolic power fell more under EnergyExo than PowerExo, with savings concentrated proximally. In the experiment, active assistance reduced metabolic rate relative to Passive for every participant, while the average reduction relative to No-Exo was smaller and variable.

2) Muscle activation and metabolic energy by functional group:

  • −13.8%: EnergyExo reduced cycle-averaged metabolic power to 1.578 W/kg per leg from 1.830, versus −8.8% to 1.670 W/kg per leg for PowerExo.The table summarizes cycle-averaged metabolic power by functional group and across involved muscle actuators.Source: 2) Muscle activation and metabolic… · p. 7, Table III · p. 8
  • EnergyExo reduced hip-muscle metabolic power by 29.8% and knee-muscle power by 12.8%, while ankle-muscle power fell 3.9%; PowerExo reductions were 16.0%, 11.9%, and 2.5%, respectively.Biarticular muscles appear in multiple joint rows, so those rows overlap and their savings can exceed the total reduction.Source: 2) Muscle activation and metabolic… · p. 7, B. Metabolic Cost of Walking from… · p. 8
  • Both controllers reduced hip-extensor and knee-extensor activation, while hip-flexor activation rose during stance and fell during swing even as summed hip-flexor energy fell.Activation is an unweighted group mean, whereas metabolic power is summed with muscle-mass weighting, which may partly explain the divergence.Source: 2) Muscle activation and metabolic… · p. 7, Figure 4 · p. 8

B. Metabolic Cost of Walking from Experiment

  • Repeat No-Exo and Passive trials differed by 6.9 ± 5.2% and 3.5 ± 2.7% of gross metabolic rate, respectively, without a consistent direction across participants.Averaging repeats reduces, but does not remove, this variability.Source: B. Metabolic Cost of Walking from… · p. 8
  • 13.1 ± 4.9%: carrying the unpowered device raised net metabolic rate above unassisted walking; the data do not separate added mass from structural constraint.Active assistance was compared against this penalized Passive condition to isolate its contribution.Source: B. Metabolic Cost of Walking from… · p. 8
  • 18.3 ± 6.6%: MARL-Exo lowered net metabolic rate from 3.514 to 2.888 W/kg versus Passive, with reductions in all eight participants.Individual reductions ranged from 7.2% to 27.7%.Source: B. Metabolic Cost of Walking from… · p. 8
  • 7.4 ± 10.0%: net metabolic rate was lower than No-Exo for six of eight participants, with individual reductions up to 24.2%.The paired difference was −0.221 W/kg (unadjusted 95% CI [−0.474, +0.033], corrected p = 0.24).Source: B. Metabolic Cost of Walking from… · p. 8

C. Hip Biomechanics and Assistance Power Delivery

On hardware, MARL-Exo reduced metabolic rate relative to Passive in all eight participants and enlarged hip range of motion. The discussion connects simulated outcomes to reward design and identifies hardware and evaluation boundaries.

V. DISCUSSION

  • −18.3 ± 6.6% relative to Passive; six of eight participants also beat No-Exo, with a cohort-mean reduction of 7.4 ± 10.0%.All eight participants reduced net cost under assistance relative to Passive; the No-Exo contrast includes the device-wearing penalty.Source: Figure 5 · p. 9, §V · p. 10

C. Hip Biomechanics and Assistance Power Delivery

  • 36.8%: hip range of motion increased from 44.0 ± 4.3° without the device to 60.2 ± 6.6° under MARL-Exo.The increase was almost entirely flexion: peak flexion rose 59.8%, while peak extension was statistically unchanged.Source: C. Hip Biomechanics and Assistance… · p. 9, Figure 6 · p. 10
  • 11.79 ± 0.39 N·m in flexion and −11.81 ± 0.42 N·m in extension: peak torques were similar, with a positive-power ratio of 0.985 ± 0.006.Assistance occurred in both directions of the stride; the policy was trained without explicit power shaping.Source: C. Hip Biomechanics and Assistance… · p. 9, Table V · p. 10

V. DISCUSSION

  • −13.8% versus −8.8%: removing positive-power shaping further lowered modeled metabolic power, with savings and controller rankings specific to the Uchida model.Further training of Metabolic + Power over a comparable number of steps did not improve it.Source: §V · pp. 9–10
  • At the same belt speed, the larger hip excursion suggests some metabolic benefit may accompany altered gait as well as reduced muscle workload.Cadence increased 4.8% and stride length shortened, but neither change was significant after correction.Source: C. Hip Biomechanics and Assistance… · p. 9, §V · p. 10
  • 19.7% for SMAT versus 18.3% for MARL-Exo: both reduced cost relative to Passive, and the exploratory comparison detected no difference.The policies showed similar assistance mechanics, although MARL-Exo had higher RMS torque and a slightly lower mean peak torque.Source: §V · p. 11
  • A shared ~12 N·m hardware ceiling may partly explain similar controller benefits; the human study also covered only eight healthy adults at one treadmill speed.The authors identify higher torque limits, lighter hardware, and evaluation across speeds, slopes, and gait-impaired populations as next steps.Source: §V · p. 11
  • The 12 N·m hardware limit was below the ±18 N·m simulated peak; a lighter, better-fitting interface may permit higher torque without cuff motion.The authors also note that soft-tissue compliance and strap looseness make IMU-derived hip angles approximate.Source: §V · p. 11

VI. CONCLUSION

MARL-Exo co-adapts a full-body musculoskeletal human policy and hip-exoskeleton policy in simulation using a physiological muscle-energetics reward. The policy transferred to eight users without per-user tuning and reduced metabolic rate relative to Passive.

VI. CONCLUSION

  • −18.3 ± 6.6%: MARL-Exo reduced net metabolic rate relative to Passive in eight healthy adults, without per-user tuning.Six of eight participants walked below their No-Exo cost, with a cohort-mean reduction of 7.4 ± 10.0%.Source: §VI · p. 11
  • MARL-Exo co-adapted full-body human and hip-exoskeleton policies in two stages, using a physiological muscle-energetics term in the exoskeleton reward.The human model had 304 muscle actuators, with anatomy, gait, and speed randomized.Source: §VI · p. 11
  • The results support screening candidate reward designs in simulation before human testing, while larger and more diverse evaluations remain needed.The conclusion identifies lighter hardware and testing across speeds, slopes, and gait-impaired populations as next steps.Source: §VI · p. 11

I. AUXILIARY TERMS OF THE HUMAN REWARD

The human reward uses three auxiliary terms to regularize learned gait, with identical settings across the baseline and both co-adaptation runs. The supplied terms penalize knee hyperextension and encourage smoother desired joint torques.

I. AUXILIARY TERMS OF THE HUMAN REWARD

  • Three auxiliary terms regularize learned gait, using identical settings in the human baseline and both co-adaptation runs.Source: §I · p. 13

A. Knee Joint-Limit Penalty

  • Knee hyperextension is penalized quadratically beyond a soft limit set inside the extension limit; with the stated settings, the penalty applies for any hyperextension, not excessive flexion.The upper flexion limit is not penalized.Source: A. Knee Joint-Limit Penalty · p. 13

B. Joint-Torque Smoothness

  • The bounded torque-smoothness term decreases with abrupt desired-torque changes and combines first-, second-, and third-order backward differences.It ranges from 0 to 1.7 and is applied once four control steps of torque history are available.Source: B. Joint-Torque Smoothness · p. 13

C. Foot-Clearance Penalty

The foot-clearance penalty targets scuffing during horizontal foot motion below a clearance height. Simulation muscle analysis averages left and right legs and groups actuators by function and joint, with biarticular muscles creating overlap across joints.

C. Foot-Clearance Penalty

  • Foot scuffing is penalized when a fixed foot point is below the clearance height while moving horizontally, mainly during swing.The penalty is zero above the 0.03 m clearance height or when stationary on the ground.Source: C. Foot-Clearance Penalty · p. 13

II. MUSCLE GROUPS USED IN THE SIMULATION ANALYSIS

  • The analysis uses 31 actuators per leg, representing 15 sagittal-plane muscles, and reports left–right averages across six functional groups and three joint groups.Source: §II · p. 14
  • Group activation is the unweighted mean over member actuators, while group metabolic power is their sum; biarticular muscles make groups overlap across joints.Each joint group is the disjoint union of its two direction groups, but biarticular muscles belong to groups at two joints.Source: §II · p. 14
Loading 2610.37441v1…