Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulation

Jun Hyuk Lee; Chihyeong Lee; Jooeun Ahn

Source-linked AI summary

Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulation

Jun Hyuk Lee, Chihyeong Lee, Jooeun Ahn

arXiv:2608.17030v1Aug 17, 2026cs.ROcs.GRcs.LG

TL;DR

Training muscle-actuated models to produce human-like motion is difficult because redundant, high-dimensional muscle control makes reinforcement-learning exploration inefficient. The paper introduces a physiology-inspired λ-hold controller that sets per-muscle equilibrium-point thresholds and holds them between gait-phase decisions, enabling human-like sprinting from a minimal reward within an hour of training.

Problem
Redundant muscle actuation makes control ill-posed and reinforcement-learning exploration inefficient because many activation patterns can produce the same movement.
Method
The λ-hold controller outputs per-muscle equilibrium-point threshold lengths that a stretch-reflex law converts into excitation while holding thresholds between gait-phase decisions.
Results
About 4.7 m/s: coordinated, human-like sprinting emerges from a forward-velocity reward alone without mirroring or a symmetry reward.
Takeaways & Limitations
λ-hold control provides a physiology-grounded route to more efficient exploration and a learnable model of human motor control.
Takeaways & Limitations
The emergent sprint does not reach elite human sprinters’ top speed, partly because the model’s muscle force–velocity relation limits propulsive force.

Abstract

from arXiv · show

The massive overactuation in the human musculoskeletal system makes it challenging to train musculoskeletal models to generate human-like motion via reinforcement learning, primarily because exploration in the resulting high-dimensional and redundant action space is extremely inefficient. To address this problem, we propose the $λ$-hold controller, inspired by the equilibrium-point (EP) hypothesis, which has been widely supported by extensive evidence from human motor control studies. The policy's control variable is the per-muscle EP threshold length $λ$, from which a stretch-reflex recruitment law computes the muscle excitations automatically. Holding each $λ$ over an interval of the gait phase also sharply reduces the frequency at which the policy must be queried. Consequently, the controller, to our knowledge for the first time, enables a muscle-actuated skeletal model to learn human-like sprinting using only a minimal reward within an hour of training. The efficient exploration through the proposed $λ$-hold controller is not merely an engineering trick but an approach grounded in physiology, bringing together the EP hypothesis, intermittent control, and optimal feedback control. Beyond encapsulating human-like behavior in predictive simulation, this achievement contributes to developing a learnable model of the human motor controller.

1 Introduction

The paper addresses inefficient exploration in predictive musculoskeletal simulation by controlling per-muscle equilibrium-point threshold lengths λ rather than excitations. This λ-hold controller enables human-like sprinting with only a forward-velocity reward while connecting muscle-level control to simple neural-circuit modelling.

Background: The work sits between muscle and neural-circuit control levels, using predictive forward dynamics to generate behavior without an experimental trajectory as input.Predictive simulation generates human-like behavior from a body model and controller, unlike inverse simulation, which begins with measured motion and estimates internal quantities.
Problem: Muscle-level reinforcement learning is inefficient because redundant muscles create a many-to-one excitation-to-movement map, causing independent action perturbations to cancel in joint space.This redundancy also creates a high-dimensional exploration problem when muscle excitations are perturbed independently at every timestep [13] [14].
Contribution: The policy commands per-muscle threshold lengths λ, while a stretch-reflex law converts muscle length and velocity into excitations that covary through skeletal coupling.Holding λ over gait-phase intervals makes the policy operate at sparse, phase-locked decision points while the reflex fills the intervening control gaps.
Contribution: The λ-hold controller produced, to the authors’ knowledge, the first coordinated, human-like sprinting in predictive musculoskeletal simulation using only a forward-velocity reward.The approach replaces motion imitation, effort or metabolic objectives, and hand-designed locomotion-specific reflex circuitry as the primary basis for this result.

2 Methods · 2.1 Musculoskeletal model and simulation · 2.2 The λ-hold controller

The study simulates a 21-degree-of-freedom, 90-muscle musculoskeletal model and controls it through per-muscle equilibrium-point threshold lengths λ. A stretch-reflex law converts held λ commands into continuously varying excitations, reducing policy-query frequency while coordinating recruitment through skeletal mechanics.

2.1 Musculoskeletal model and simulation: The H2190 model has 21 mechanical degrees of freedom and 90 muscle–tendon units, simulated in SCONE with the Hyfydy engine.All methods use a 0.01 s simulation step; the engine exposes normalized muscle lengths and velocities and receives muscle excitation as input.
2.1 Musculoskeletal model and simulation: Excitation is the simulator’s neural command, while activation is the resulting muscle state produced by activation dynamics and determining force with force–length and force–velocity properties.
2.2 The λ-hold controller: The λ-hold controller outputs a threshold length λi for each muscle, and a stretch-reflex recruitment law converts each threshold into excitation ei.λi represents the equilibrium point in the EP hypothesis, defined as the muscle length where recruitment begins.
2.2 The λ-hold controller: The recruitment law combines tonic length feedback and rectified phasic velocity feedback, adding phasic drive only during muscle stretching and never during shortening.The result is clamped to the excitation range [0, 1].
2.2 The λ-hold controller: Shared reflex gains are fixed at Gtonic = 50 and Gphasic = 0.1, while each policy threshold λi is bounded to [0.6, 1.2].Muscles below threshold and not being stretched are silent; stretched muscles are recruited in proportion to threshold excess.
2.2 The λ-hold controller: Although each muscle responds only to its own length and velocity, skeletal mechanics make those lengths covary, coordinating recruitment without an explicit coupling term or predefined synergy basis.
2.2 The λ-hold controller: The policy emits λ only at gait-phase-locked decision points and holds it across many simulation steps while the reflex law continues generating excitation as muscle length and velocity evolve.Reward accumulates over each hold interval, whose duration varies with gait phase.
2.2 The λ-hold controller: Querying the policy once per hold interval rather than every simulation step reduces the number of training decisions and wall-clock training time.Because excitation continues varying through the reflex loop, λ-holding is not equivalent to action repeat [37].

2.3 Decision timing · 2.4 Reinforcement-learning setup · 2.5 Task: sprinting with a minimal reward

The study uses gait-synchronized λ decisions, physiology-grounded SAC training, and a minimal forward-sprinting reward. This setup enables human-like sprinting within about an hour of training.

2.3 Decision timing: Decisions occur at foot strikes and toe-offs, defined by vertical GRF crossing 0.05 body weight, with one sub-point inserted between consecutive events.Every hold is bounded between 0.05 and 0.15 s.
2.3 Decision timing: The interposed sub-point is placed halfway through an estimated interval based on the most recent preceding interval of the same stance-or-flight type.The ongoing interval cannot be measured before its closing event, and stance and flight durations differ.
2.3 Decision timing: GRF-triggered timing synchronizes commands with gait phase while leaving stride period under policy control, unlike fixed-time holds that can lock stride duration to the hold interval.The method relies on gait’s cyclic structure; suitable decision placement for noncyclic tasks remains open.
2.5 Task: sprinting with a minimal reward: Human-like sprinting emerges within about an hour when reinforcement learning controls per-muscle λ with a stretch-reflex recruitment law and only forward velocity reward.The controller holds λ between intermittent decisions while muscle excitation is computed automatically.
2.4 Reinforcement-learning setup: Training uses SAC [38] with gSDE [21] on Stable-Baselines3, while the DEP-RL baseline combines DEP with MPO [41].The λ-hold controller and SAC-based baselines share the same configuration; exploration noise remains constant throughout each hold.
2.4 Reinforcement-learning setup: The 432-dimensional observation contains sensed muscle, force, contact, kinematic, vestibular, and held-λ signals, while excitation and activation are excluded as downstream quantities.The observation is restricted to signals the nervous system can sense, and excitation and activation are determined by λ, muscle length, and velocity.
2.5 Task: sprinting with a minimal reward: Sprinting permits a minimal reward because maximizing forward speed is unambiguous, unlike walking or balance tasks whose objectives encode contestable assumptions.The reward uses uncapped forward velocity, attenuates it with a narrow lateral-velocity Gaussian, and terminates episodes after low COM or head height.

2.6 Baselines · 2.7 Quantifying learning speed: simulation steps and decision steps

The study compares λ-hold with four baselines trained on the identical task and model, while evaluating learning speed separately by simulation steps and policy decision steps. These measures distinguish environment experience from policy-query cost because controllers differ in command representation and query frequency.

2.6 Baselines: λ-hold is compared with four baselines, all trained on the identical task and model.
2.6 Baselines: Plain SAC directly commands 90 muscle excitations at every decision and serves as the null baseline.
2.6 Baselines: Excitation-hold directly commands excitations but holds them between λ-hold’s decision points, isolating intermittency from the stretch-reflex recruitment law.It is an ablation using action repeat [37], so any λ-hold advantage over it is attributed to reflex recruitment rather than holding alone.
2.6 Baselines: DEP-RL [13] directly commands excitations through DEP + MPO, using the prior implementation with default hyperparameters and rescaled step-counted quantities.The 0.025 s configuration is adapted to the 0.01 s simulation step by scaling windows and delays by 2.5, rates by 1/2.5, and replacing γ with γ1/2.5.
2.6 Baselines: Synergy [24] commands a 30-dimensional latent representation that expands through a fixed shared NMF basis into leg-muscle excitations, with 10 additional trunk-control dimensions.
2.7 Quantifying learning speed: simulation steps and decision steps: Training cost is reported on separate simulation-step and decision-step axes because policy commands and query frequencies differ across controllers.
2.7 Quantifying learning speed: simulation steps and decision steps: A simulation step is one 0.01 s engine step, whereas a decision step is one policy query; Plain SAC and Synergy equate them, but held-command controllers use fewer queries.DEP-RL’s exploration rule occasionally takes over during data collection, so its policy-query count is also distinct from simulation steps.
2.7 Quantifying learning speed: simulation steps and decision steps: Simulation steps measure performance per unit of environment experience, while decision steps measure performance per policy query and therefore reflect training and deployment computation.

2.8 Joint-velocity coverage · 2.9 Validation of human-likeness

The study quantifies exploration through 15 anatomical joint angular velocities using entropy and grid coverage, then evaluates gait human-likeness through sagittal kinematics, vertical GRF, and muscle activity against human running data.

2.8 Joint-velocity coverage: Exploration is measured over the 15 anatomical joint velocities, excluding six pelvis-root degrees of freedom that represent global body motion.The retained joints span the hip, knee, ankle, subtalar, and lumbar joints.
2.8 Joint-velocity coverage: Velocity-distribution entropy quantifies movement variety using a k-nearest-neighbour estimator with k = 4 and equal sample counts across methods.Velocities are standardized per dimension within each method to estimate differential entropy comparably.
2.8 Joint-velocity coverage: Grid coverage measures explored velocity-space volume by counting distinct visited cells in a fixed pooled-data grid.Each dimension uses four bins spanning its 1st to 99th percentile, producing 4^15 ≈ 1.1 × 10^9 cells, with equal samples per method.
2.9 Validation of human-likeness: Human-likeness is evaluated along sagittal joint kinematics, vertical ground-reaction force, and muscle activity against openly available human running data.These three axes provide complementary comparisons of emergent gait mechanics and muscle activation.
2.9 Validation of human-likeness: All human-likeness comparisons align cycles at right initial contact, time-normalize each cycle to 101 points, and compare averaged mean waveforms.Model cycles are segmented at the rising edge of the right vertical GRF, while human EMG uses marker-based heel-strike segmentation.
2.9 Validation of human-likeness: Kinematics and vertical GRF are compared with Fukuchi et al. at 2.5, 3.5, and 4.5 m/s by grouping model cycles within ±0.35 m/s of each reference speed.Because the model accelerates from standing, lower-speed groups represent accelerating rather than steady-state gaits.
2.9 Validation of human-likeness: Muscle activations are compared with surface EMG from 19 participants running at 5.0 m/s, using model cycles with mean COM speed at least 4.3 m/s.The human EMG is band-pass filtered at 20 to 450 Hz, rectified, and low-pass filtered at 10 Hz to obtain linear activation envelopes.

3 Results

Across equal simulation-step budgets, λ-hold rapidly and persistently outperforms all baselines while requiring far fewer policy queries. It produces coordinated, partly human-like sprinting with moderately matching muscle-activation timing, although training can oscillate and muscle-level agreement is uneven.

Sprint learning: Within the first 10 × 10^6 simulation steps, λ-hold separates from every baseline and remains above them for the rest of training.Figure 3 reports mean returns over two random seeds, with the logarithmic return axis understating the plotted performance gap.
Sprint learning: Excitation-hold reaches a return about an order of magnitude above Plain SAC but remains more than an order below λ-hold, while both use about an order of magnitude fewer policy queries than the other methods.Excitation-hold holds direct muscle excitations between decision points, whereas Plain SAC, Synergy, and DEP-RL are queried every simulation step.
Training behavior: A sprint reaching about 4.0 m/s emerges in roughly one hour on one RTX 5070 workstation, but λ-hold’s curve periodically loses and relearns the fast sprint late in training.The paper does not report a head-to-head wall-clock comparison because hardware, simulation-to-decision ratios, optimizers, parallel environments, and simulator throughput differ.
Exploration: λ-hold yields the highest entropy from the first evaluation and finishes with about 2.5 times DEP-RL’s joint-velocity coverage and about seven times Plain SAC’s and Excitation-hold’s.Synergy has higher entropy than Plain SAC but the lowest coverage because its fixed muscle co-variation confines movement to its basis span.
Human-likeness: The emergent sprint reaches about 4.7 m/s, shows left–right alternation and swing-limb flexion without symmetry constraints, and largely reproduces human sprint kinematic waveform shapes and vertical GRF.The model’s sagittal-angle amplitudes diverge from human data increasingly with speed, despite a single stance pulse and human-like GRF magnitude.
Muscle activity: Model muscle activations have mean waveform correlation r = +0.42 with measured EMG, with six of eleven muscles achieving r > 0.5.The comparison uses peak-normalized waveforms for timing and shape; human recordings are at 5.0 m/s, above the model’s 4.7 m/s top speed, and agreement is weakest in several swing muscles.

4 Discussion

The λ-hold controller improves exploration through mechanically coordinated, sustained referent-threshold perturbations while combining EP-based control with intermittent command updating. The discussion argues that predictive simulation should capture both muscle physiology and human motor-control principles, while identifying speed, timing, and generalization limitations.

Control mechanism: λ-space perturbations move muscles in mechanically coordinated ways that transfer into joint space, unlike excitation-level perturbations that can cancel through the many-to-one excitation-to-joint map.The controller’s coordination arises from the mechanical coupling of synergist muscles and the reflex law’s dependence on each muscle’s length and velocity relative to its held threshold.
Control mechanism: Holding commands sustains movement deviations and shortens credit-assignment chains; excitation-hold therefore achieves about an order of magnitude higher return than Plain SAC.Excitation-hold differs from Plain SAC only by holding commands, isolating the benefit of temporal holding.
EP hypothesis: The sprint model uses referent commands updated only eight times per gait cycle, with one reflex law and identical gains across all 90 muscles.This supports the discussion’s claim that sprinting need not require finely shaped or continuously updated referent commands.
Control architecture: The learned policy supplies EP referent commands, addressing the hypothesis’s command-source limitation, while optimization and EP reflex control operate as complementary levels in a hierarchical architecture.The policy performs state-feedback optimization over referent commands, which the EP-based reflex mechanism translates into muscle activation through body mechanics.
Intermittent control: λ-hold combines EP control’s specification of what variable to control with intermittent control’s specification of when to update it, but the GRF trigger remains task-specific.A task-agnostic timing mechanism is left open, including whether it could be learned from experience or derived from prediction error.
Limitations and outlook: The emergent sprint is fast and human-like in coordination but below elite human top speed because model limitations, including the muscle force–velocity relation, constrain performance.Future tests should examine balance, jumping, and walking, especially regimes without clear cyclic events, and ultimately perturbation responses and cross-condition generalization.

Appendix · A Training hyperparameters

The appendix provides the complete configuration for both learning stacks in Table 1.

A Training hyperparameters: Table 1 lists the full configuration for both learning stacks.

B Extending the sprint: stabilizing training and raising the speed ceiling

This section reports two additional ingredients that extend the λ-hold controller beyond the main-text evaluation: one stabilizes late-training return, while the other raises the top speed.

B Extending the sprint: stabilizing training and raising the speed ceiling: Two further ingredients extend the λ-hold controller beyond the main-text characterization.They are reported separately because neither belongs to the controller evaluated in the main text.
B Extending the sprint: stabilizing training and raising the speed ceiling: One added ingredient stabilizes the late-training return.
B Extending the sprint: stabilizing training and raising the speed ceiling: The other added ingredient raises the top speed.

B.1 Reference-state initialization stabilizes the late-training oscillation

Late-training oscillation occurs because one policy must learn both low-speed acceleration and steady high-speed sprinting, causing it to forget one sub-task while improving the other. Reference-state initialization (RSI) addresses this by restarting some episodes from archived fast-sprint states.

Cause of oscillation: The oscillation reflects multi-task learning: the policy alternately loses and recovers fast sprinting while learning acceleration.The baselines do not show this oscillation because they never reach the fast sprint.
Reference-state initialization: RSI [4] occasionally resets episodes to previously reached fast-sprint states instead of the default standing posture.After a warm-up of 3 × 10^6 decision steps, reset states are sampled from a self-generated archive of highest-return episodes, following the spirit of Go-Explore.
Evaluation: Figure 7 compares λ-hold alone, λ-hold with RSI, and λ-hold with both RSI and a maximum-shortening-velocity curriculum.The comparison reports return against both simulation steps and decision steps using two seeds, with means and seed ranges shown.

B.2 The speed ceiling is the muscle force–velocity relation

The controller’s speed ceiling arises from the muscle force–velocity relation: rapidly shortening, fully activated hip flexors generate little propulsive force. Raising maximum shortening velocity from 10 to 20 optimal fiber lengths per second increases top speed from about 4.7 to 6.5 m/s.

B.2 The speed ceiling is the muscle force–velocity relation: At the speed ceiling, hip flexors are fully activated but rapid shortening reduces their force to one-quarter to one-third of isometric force.Thus, full activation produces little propulsive force during the concentric swing drive that would accelerate the limb.
B.2 The speed ceiling is the muscle force–velocity relation: Figure 8 shows activation saturating at 1 while hip-flexor shortening velocity reaches −2 to −3 and normalized force remains strongly modulated.These measurements come from the right psoas major and iliacus during the top-speed λ-hold episode over the last few gait cycles.