PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Speed-adaptive Hip Exoskeleton Control Policy Via Sim-to-real Reinforcement Learning

Bin Li, Zhimin Hou, Jiacheng Hou, Zenian Liang, Tong Wu, Teng Ma, Chenglong Fu

arXiv:2609.28027v1cs.RO

TL;DR

Personalized exoskeleton assistance across varying walking speeds remains challenging, and prior approaches have focused on discrete speeds or specific locomotion conditions. The paper combines simulation-trained assistance timing with a distilled, onboard-sensing policy and online preference-based magnitude personalization. In free walking, the personalized method reduced negative mechanical power by 55% and resistance ratio by 27%, with a 9% reduction in RMS torque.

  • Problem

    Personalized assistance that remains effective across continuously varying walking speeds is underexplored, while prior sim-to-real RL often focuses on specific speeds and human-in-the-loop methods on specific locomotion conditions.

    Source: §I · p. 1, B. Personalized Exoskeleton Control… · p. 2

  • Method

    The framework learns assistance timing across speeds in simulation, distills the policy for onboard-sensing deployment, and uses Gaussian-process preference learning with pairwise comparisons to personalize assistance magnitude.

    Source: §I · p. 1, D. Exoskeleton Control Policy Online… · p. 4

  • Results

    Negative mechanical power fell by 55% (p = 0.030) and resistance ratio by 27%, despite a 9% reduction in RMS torque during free walking.

    Source: 2) Free Walking · p. 6

  • Takeaways & Limitations

    The learned speed-adaptive structure compresses online personalization into a low-dimensional latent space, allowing subject-specific adaptation without reoptimizing the full assistive profile.

    Source: §VII · p. 7

    Validation was limited to level-ground walking and a hip exoskeleton.

    Source: §VII · p. 7

Abstract

from arXiv · show

Providing personalized exoskeleton assistance across varying walking speeds remains challenging. Existing online optimization methods are sample-inefficient, requiring extensive human-in-the-loop (HIL) evaluations to optimize the entire assistive torque profile. Sim-to-real reinforcement learning (RL) offers a promising alternative but cannot directly account for individual user preferences. We propose a framework integrating sim-to-real RL with online preference learning for personalized exoskeleton assistance. Specifically, assistance timing is learned in simulation by training RL policies with human musculoskeletal models across varying walking speeds. The learned policies are then distilled and deployed on a physical hip exoskeleton using onboard sensory observations. Gaussian-process-based preference learning further personalizes the assistance magnitude through pairwise user comparisons. By decoupling assistance timing learning in simulation from magnitude optimization in real-world experiments, our framework substantially reduces the online optimization space. Human-subject experiments demonstrate efficient identification of personalized assistive torque profiles across varying walking speeds with fewer real-world evaluations.

I. INTRODUCTION

Personalized hip-exoskeleton assistance across continuously varying walking speeds remains underexplored, while existing sim-to-real RL and human-in-the-loop optimization have limitations for this setting. The proposed framework learns assistance timing in simulation and personalizes magnitude online after deploying a distilled policy.

I. INTRODUCTION

  • The framework learns assistance timing in simulation across varying speeds, distills the learned policy into a TCN using onboard sensing, and personalizes assistance magnitude through user preferences after deployment.The overview describes four stages, including human-policy learning and exoskeleton-policy training through interaction with a simulated human musculoskeletal model.Source: §I · p. 1, Figure 1 · p. 2, §III · p. 2

A. Sim-to-real RL Techniques for Hip Exoskeleton Control

  • Existing RL-based human control policies typically rely on predefined reference motions, limiting their ability to represent natural locomotion under continuously varying walking speeds.Source: A. Sim-to-real RL Techniques for Hip… · p. 2

B. Personalized Exoskeleton Control Optimization

  • Personalized assistance that remains effective across continuously varying walking speeds is underexplored, and existing human-in-the-loop methods primarily optimize for specific locomotion conditions.Existing methods can require extensive interaction trials when optimizing high-dimensional assistive policies.Source: §I · p. 1, B. Personalized Exoskeleton Control… · p. 2

III. OVERVIEW FRAMEWORK

  • The simulation framework uses a musculoskeletal model and a virtual hip exoskeleton to learn control policies for varying-speed locomotion and reduce muscle effort.Source: §III · p. 2

IV. METHOD

The human control policy is trained in two phases: first to track a fixed-speed gait, then to produce locomotion across variable speeds without explicit motion tracking.

A. Human Control Policy with Variable Walking Speed

  • Phase 1 trains a stable fixed-speed gait by tracking reference joint angles and velocities; Phase 2 uses target velocity and commanded acceleration for variable-speed locomotion, removing reference-motion imitation terms.The resulting policy is intended to generate adaptive locomotion across velocity conditions without explicit motion tracking.Source: A. Human Control Policy with Variable… · p. 3

1) Observation Space:

2) Reward Function:

The human-policy reward combines motion objectives with terms that regularize effort and physical constraints. In Phase 2, it tracks commanded velocity during holds and commanded acceleration during speed transitions.

2) Reward Function:

  • Phase 1 combines reference-motion imitation and velocity tracking with terms that regularize muscle effort, activation, and joint-specific motion.The velocity reward is defined to keep the current walking speed.Source: 2) Reward Function · p. 3
  • Auxiliary reward terms regularize muscle effort and activation, smoothness, joint constraints, and ground-contact forces.The constraint term penalizes maximum joint constraint force, while the foot term penalizes total bilateral contact force exceeding 1.2 times body weight.Source: 2) Reward Function · p. 3
  • Phase 2 tracks commanded velocity during initial and final holds and commanded acceleration during speed transitions.The transition duration is constrained to 1–5 s; initial and target speeds are selected from 0.5–1.7 m/s, with acceleration/deceleration levels in [−0.5, 0.5] m/s2.Source: 2) Reward Function · p. 3

B. Exoskeleton Control Policy Learning

The exoskeleton teacher policy is trained through interaction with a simulated human to reduce muscle effort. Training first optimizes assistance with the human policy fixed, then jointly optimizes both policies for co-adaptation.

B. Exoskeleton Control Policy Learning

  • The exoskeleton teacher policy is trained with continuous-action reinforcement learning to reduce human muscle effort.A virtual bilateral hip exoskeleton is coupled to the musculoskeletal model and controlled by the policy.Source: B. Exoskeleton Control Policy Learning · p. 3
  • In Phase 1, the pretrained human policy is frozen while the exoskeleton policy learns assistance that reduces effort and adapts to the learned locomotion.The exoskeleton and human policies share a critic network during training.Source: B. Exoskeleton Control Policy Learning · p. 4
  • In Phase 2, the human and exoskeleton policies are jointly optimized to further refine assistance through co-adaptation.The policy outputs normalized bilateral hip commands that are scaled by the maximum actuator torque to generate assistive torques.Source: B. Exoskeleton Control Policy Learning · p. 4

1) Observation Space:

The observation-space passages distinguish Phase 1 from Phase 2 and identify command history and prior torque commands. The reward passages describe phase-specific assistance objectives and torque limits.

1) Observation Space:

  • Phase 1 uses a three-step history of bilateral normalized exoskeleton commands; the Phase 2 fully observed state includes the previous bilateral torque commands.The supplied passages also identify the human state used in Phase 2 human-policy learning.Source: 1) Observation Space · p. 4

2) Reward Function:

  • Phase 1 trains separate steady-walking, acceleration, and deceleration policies with assistance limited to 6 Nm, favoring torque that generates mechanical power in the desired direction.The power term penalizes assistance in the opposite direction; the steady teacher has no speed-transition terms.Source: 2) Reward Function · p. 4
  • Phase 2 jointly optimizes the human and exoskeleton actors with a 12 Nm torque limit, including rewards that penalize hip-muscle activation and suppress excessive torque.The reward also regularizes temporal variations in commanded torque.Source: 2) Reward Function · p. 4

C. Exoskeleton Control Policy Distillation

The trained teacher behaviors are distilled into a student policy that uses onboard sensory observations. Teacher rollouts provide observation–action training data for supervised learning.

C. Exoskeleton Control Policy Distillation

  • The three fully observed teacher policies are distilled into a student policy that relies only on onboard sensory observations.A temporal convolutional network is used as the student policy to handle the partially observed onboard state.Source: C. Exoskeleton Control Policy… · p. 4
  • Training pairs are generated by rolling out each teacher policy under the specified curriculum scheme.The observation–action pairs are interpolated from 30 Hz to 100 Hz to construct the training dataset.Source: C. Exoskeleton Control Policy… · p. 4
  • The student policy parameters are updated using supervised learning.Source: C. Exoskeleton Control Policy… · p. 4

D. Exoskeleton Control Policy Online Tuning

The exoskeleton policy sets assistance timing from measurable human state, while a separate parameter controls hip extension and flexion magnitudes. Pairwise user comparisons inform Gaussian-process preference learning to select torque profiles.

D. Exoskeleton Control Policy Online Tuning

V. SIMULATION VALIDATION

Simulation was used only to evaluate assistance timing; the protocol used PPO with fixed, empirically selected reward weights and a 0.0333-second control step.

V. SIMULATION VALIDATION

  • Simulation was used only to evaluate the performance of assistance timing.Source: §V · p. 5

A. Simulation Protocol

  • The simulation control time step was 0.0333 seconds, and PPO was used for all policy learning.Continuous-action algorithms such as TD3 and SAC were noted as alternatives.Source: A. Simulation Protocol · p. 5
  • Reward weights were empirically selected and held fixed because the work aimed to demonstrate framework effectiveness rather than optimize hyperparameters.Source: A. Simulation Protocol · p. 5

1) Performance of Learned Human Control Policy:

The learned assistance profile produced higher positive power than spline and DOFC controllers at comparable torque amplitudes and consistently reduced rectus femoris activation across evaluated speeds. The learned human policy reproduced speed-adaptive locomotion despite reference-motion training only at 1.25 m/s; treadmill and free-walking trials assessed assistance timing and personalized magnitude.

1) Performance of Learned Human Control Policy:

  • The learned human policy reproduced speed-adaptive locomotion across 0.5–1.7 m/s despite reference-motion training only at 1.25 m/s.At 1.25 m/s, pooled RMSEs were 17.06° for joint angles and 211.93°/s for angular velocities; metrics were calculated for the left limb.Source: 1) Performance of Learned Human… · pp. 5–6

2) Performance of Learned Assistance Profile:

  • At torque amplitudes comparable to DOFC and the spline controller, the learned profile yielded higher positive power and consistently reduced rectus femoris activation across evaluated speeds.The power comparison suggests improved assistance timing.Source: 2) Performance of Learned Assistance… · p. 6

VI. EXPERIMENTAL VALIDATION

  • Treadmill and free-walking trials evaluated learned assistance timing and personalized assistance magnitude, respectively.Source: §VI · p. 6

1) Hardware System:

  • A Raspberry Pi 5 ran the distilled TCN exoskeleton policy, while bilateral hip kinematics were measured by wearable IMUs at 100 Hz.Online optimization used a GUI and voice interaction.Source: 1) Hardware System · p. 6

2) Experimental Protocol:

The experiments evaluate assistance during treadmill walking at three set speeds and free walking at varying speeds, then discuss the framework’s scope and limitations.

2) Experimental Protocol:

  • Six able-bodied participants completed treadmill tests at 0.9, 1.25, and 1.6 m/s, with muscle activation measured at those speeds.Mechanical power was evaluated across walking conditions; torque and power metrics included RMS and maximum torque, mean positive power, and mean negative power.Source: 2) Experimental Protocol · p. 6
  • Online magnitude optimization ran for 30 iterations using 50 predefined comparison pairs, with participants walking 30 seconds per profile before stating a preference.The two profile walks were separated by unassisted walking periods of 5 and 10 seconds, respectively.Source: 2) Experimental Protocol · p. 6

1) Treadmill Walking:

  • The method produced higher positive mechanical power than DOFC during treadmill walking, with online personalization increasing it further.Several participants’ BF activation changes differed from simulation; increased positive power did not necessarily correspond to reduced muscle activation.Source: 1) Treadmill Walking · p. 6, Table I · p. 7

2) Free Walking:

  • 55% lower negative mechanical power and 27% lower resistance ratio, despite only 9% lower RMS torque, were observed with the personalized method during free walking.Four participants walked freely for 20 seconds at self-selected, continuously varying speeds.Source: 2) Free Walking · p. 6, Figure 4 · p. 7

VII. DISCUSSION AND CONCLUSION

  • The framework combines continuous-speed sim-to-real RL with preference-based personalization, and its distilled policy addresses partial observability during onboard deployment.The discussion reports better performance than MLP-PPO and LSTM-PPO baselines; prior approaches,, [21] are described as relying on reflex-based baseline gaits or discrete-speed deployments.Source: §VII · p. 7
  • Validation was limited to level-ground walking with a hip exoskeleton; slope walking, stair climbing, and other lower-limb joints remain future directions.Source: §VII · p. 7

VIII. APPENDIX

The appendix includes reward-weight and PPO-hyperparameter tables, alongside an algorithm for online preference-based optimization.

  • Table II lists reward weights for the different policy-learning stages.Source: Table II · p. 7
  • The algorithm initializes a latent utility function with a Gaussian-process prior and maintains preference data over parameters in a defined search space.Source: §VIII · p. 8
  • Each iteration selects a current best parameter and an exploratory challenger, evaluates their generated assistance profiles, and records the user’s pairwise preference.Source: §VIII · p. 8
  • The procedure returns the parameter with the highest posterior mean utility as the best personalized setting.Source: §VIII · p. 8
  • Table III lists PPO hyperparameters used for policy learning.Source: Table III · p. 8
Loading 2609.28027v1…