PaperlayerView exact revision on arXiv

Source-linked AI summary

DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems

Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, Georg Martius

arXiv:2206.00484v2cs.ROcs.LG

TL;DR

RL performance on large musculoskeletal models has remained limited, and common exploration noise is inadequate for large overactuated action spaces. The paper uses differential extrinsic plasticity to generate embodiment-adapted correlated exploration and alternates DEP with an RL policy during episodes. DEP-RL outperforms all baselines on reaching and locomotion tasks without motion-capture data or training curricula.

  • Problem

    RL performance on large musculoskeletal models has remained limited, and common exploration noise is inadequate for large overactuated action spaces.

  • Method

    The paper uses differential extrinsic plasticity to generate embodiment-adapted correlated exploration and alternates DEP with an RL policy during episodes.

  • Results

    DEP-RL outperforms all baselines on reaching and locomotion tasks without motion-capture data or training curricula.

  • Takeaways & Limitations

    Effective exploration is a key issue in applying RL to muscular control, and DEP-RL provides a robust approach for high-dimensional muscle-level learning.

  • Takeaways & Limitations

    A critical bug in ostrich-run affected the baseline results and was fixed for the experiments.

Abstract

from arXiv · show

Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount of muscles. Reinforcement learning (RL) on large musculoskeletal models, however, has not been able to show similar performance. We conjecture that ineffective exploration in large overactuated action spaces is a key problem. This is supported by the finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. We identify differential extrinsic plasticity (DEP), a method from the domain of self-organization, as being able to induce state-space covering exploration within seconds of interaction. By integrating DEP into RL, we achieve fast learning of reaching and locomotion in musculoskeletal systems, outperforming current approaches in all considered tasks in sample efficiency and robustness.

1 INTRODUCTION

The paper argues that ineffective exploration in highly overactuated muscle spaces limits RL, then introduces DEP-RL to learn muscle-level control without common simplifications. DEP produces correlated stimulation patterns, and DEP-RL outperforms existing approaches on reaching and locomotion tasks.

  • 1 INTRODUCTION: Overactuated muscle control requires coordinated stimulation because antagonistic muscles can cancel forces and individual twitches may not generate adequate joint motion.Many muscles and muscle-specific dynamics make exploration especially difficult.
  • 1 INTRODUCTION: DEP-RL outperforms current approaches on unsolved reaching and running tasks involving up to 120 muscles.The method avoids motion-capture data, training curricula, reward shaping, and expert demonstrations.
  • 1 INTRODUCTION: DEP generates strongly correlated stimulation patterns adapted to the embodiment, recruiting muscle groups for large joint-space motions within seconds and with minimal prior knowledge.The method uses only an identity mapping between control signals and contracted muscle lengths rather than hand-designed geometry information.
  • 1 INTRODUCTION: The paper contributes correlated-action exploration, DEP-based exploration, alternating DEP and RL within episodes, and improved robustness under out-of-distribution perturbations.The robustness result covers three locomotion tasks.
  • 1 INTRODUCTION: The work controls a 7-DoF human arm with 50 individually controllable muscles and achieves the highest measured simulated-ostrich top speed using RL without reward shaping.These claims are presented as firsts or best results to the authors’ knowledge.

2 RELATED WORKS

Prior musculoskeletal RL studies commonly simplify control through low-dimensional models, grouped muscles, demonstrations, curricula, reward shaping, or learned action reductions. This paper instead targets large continuous muscle-level action spaces without those aids.

  • 2 RELATED WORKS: Earlier musculoskeletal RL studies often use low-dimensional systems, manually grouped muscles, or simplified torque control.These choices reduce the effective control problem before RL is applied.
  • 2 RELATED WORKS: High-performing prior solutions commonly rely on demonstrations, explicit reward shaping, curricula, or complex ensemble architectures requiring substantial hand-crafting.The paper contrasts these practices with a standard two-layer agent without such additions.
  • 2 RELATED WORKS: Existing large-action-space methods either grow actions iteratively, require muscle-group structure, target discrete actions, or bootstrap from expert data.The paper positions DEP as learning useful coordination without demonstrations or predefined muscle groupings.
  • 2 RELATED WORKS: Action-reduction approaches learn torque-to-muscle mappings, correction networks, or synergies but require specific data, prior knowledge, human data, or expert demonstrations.The proposed approach is described as simpler and applicable across systems with minimal tuning.

3 BACKGROUND

The background formalizes RL and goal-conditioned rewards, describes simplified and elastic-tendon muscle simulation settings, and explains why uncorrelated noise loses effectiveness as actuator count grows. Correlated noise can preserve effective exploration in overactuated systems.

  • 3 BACKGROUND: RL learns a policy maximizing expected discounted return in a discounted episodic Markov Decision Process.Goal-reaching policies are conditioned on goals and receive 0 at the goal and -1 elsewhere.
  • 3 BACKGROUND: MuJoCo uses a computationally efficient simplified muscle model with non-elastic tendons, while HyFyDy provides experiments with elastic tendons.MuJoCo’s non-elastic tendon assumption makes the task fully observable because muscle length uniquely maps to joint configuration.
  • 3 BACKGROUND: For i.i.d. noise with fixed actuator variance, effective torque variance decreases with 1/n and approaches zero as actuator count grows.The maximum variance of realistic actuators is bounded, so simply increasing per-actuator noise cannot solve the problem.
  • 3 BACKGROUND: Correlated noise introduces n^2 − n cross terms that decay with 1 − 1/n, allowing effective variance to avoid vanishing.This provides the background rationale for exploring action-correlated noise in overactuated systems.

4 METHODS

DEP adapts sensor-action connections by amplifying temporally correlated state changes, normalizing controller magnitudes, and discouraging joint-limit saturation. DEP-RL alternates DEP and the RL policy within episodes to obtain exploration without chaotic action summation.

  • 4.1 DIFFERENTIAL EXTRINSIC PLASTICITY (DEP): DEP updates a control matrix using an inverse prediction model of current sensor velocities and delayed sensor velocities, with a time scale Δt for learned behaviors.The rule incorporates future consequences rather than only simultaneous action-sensor activity.
  • 4.1 DIFFERENTIAL EXTRINSIC PLASTICITY (DEP): DEP normalizes controller connections to retain activity across state-space regions and uses a time-dependent bias to prevent excessive actions at joint limits.The control matrix also includes first-order decay, while τ tunes its adaptation time scale.
  • 4.1 DIFFERENTIAL EXTRINSIC PLASTICITY (DEP): For muscle-driven systems, DEP uses the relationship between excitation and muscle-length change, including f(˙s_t) := −˙s_t because muscles contract with increasing excitation.The choice of f embeds sensor-actuator connectivity and assumes an approximately linear relationship.
  • 4.1 DIFFERENTIAL EXTRINSIC PLASTICITY (DEP): DEP can quickly coerce systems into correlated motions by strengthening connections between state dimensions whose velocity changes correlate across time.In high-dimensional chaotic environments, reset randomization and RL interaction help prevent behavioral deprivation or persistent limit cycles.
  • 4.2 INTEGRATING DEP AS EXPLORATION IN REINFORCEMENT LEARNING (DEP-RL): DEP-RL alternates complete control between DEP and the RL policy because summing their actions produced chaotic behavior that hindered learning.DEP takes over stochastically for a fixed horizon H_DEP, and an optional DEP-only phase pre-fills the off-policy replay buffer.

5 EXPERIMENTS

The experiments test DEP-MPO across synthetic overactuated systems, sparse-reward reaching tasks, and bipedal locomotion. DEP improves exploration, learning performance, and robustness across the considered settings.

  • 5.1 ENVIRONMENTS: The environments span torque-driven arms, muscle-driven arms, sparse reaching and foraging, and 18- or 120-muscle bipedal locomotion.The experiments use SCONE for bipedal humans and MuJoCo for arm-reaching tasks, with 10 random seeds unless stated otherwise.
  • 5.2 EXPLORATION WITH OVERACTUATED SYSTEMS: Only DEP covers the full endeffector space across the tested torque- and muscle-driven action spaces; other noise strategies degrade with increasing redundancy.The muscle-driven system requires exploration correlated across time and actions, which DEP generates.
  • 5.2 EXPLORATION WITH OVERACTUATED SYSTEMS: DEP-MPO remains effective with 600 available actions, whereas MPO performance strongly decreases as action-space size grows.This action count approximates the number of muscles in the human body.
  • 5.3 SPARSE REWARD TASKS WITH UP TO 52 ACTUATORS: DEP-MPO and DEP-HER-MPO quickly solve sparse tasks, with DEP-MPO outperforming MPO and DEP-HER-MPO providing the best data-efficiency on the harder task.DEP-MPO also finds non-zero rewards in the non-goal-conditioned human-hop task, whereas MPO does not.
  • 5.4 APPLICATION TO BIPEDAL LOCOMOTION: DEP-MPO produces alternating human gaits in 5 out of 5 seeds, compared with 1 out of 5 for MPO, after a performance increase around 10^6 steps.The learned DEP-MPO gait has symmetric leg extensions and higher speed than the asymmetric gait.
  • 5.4 APPLICATION TO BIPEDAL LOCOMOTION: DEP-MPO achieves the largest measured ostrich top speed despite late-training performance decay, while TD4 learns a suboptimal hopping gait.Top speeds use the fastest checkpoint from each method, averaged over 50 test episodes.
  • 5.4 APPLICATION TO BIPEDAL LOCOMOTION: DEP-MPO is the most robust under stepdown and sloped-step perturbations across the considered ostrich and human tasks.For human-hopstacle, 4 out of 5 DEP-MPO seeds achieve robust hopping; the remaining seed is non-hopping and not robust.
  • 5.4 APPLICATION TO BIPEDAL LOCOMOTION: The study reports robust high-speed running with 120 muscles without reward shaping, curriculum learning, or expert demonstrations.The authors also report first controlling the 7-DoF human arm at muscle-stimulation level with 50 individually controllable actuators.

6 CONCLUSION

The paper identifies inadequate exploration as a key issue in musculoskeletal RL and shows that integrating DEP into RL enables successful reaching and locomotion without motion-capture data or training curricula. It also reports simulator and coupling limitations that motivate future work.

  • DEP-RL outperformed all baselines on the investigated reaching and locomotion tasks without motion-capture data or training curricula.
  • Common exploration noise strategies performed inadequately on overactuated systems, whereas DEP induced state-space-covering exploration.
  • The MuJoCo muscle simulation is simplified relative to OpenSim and other software, and motions in the more realistic HyFyDy simulator are not yet consistent with human motor control.
  • Further work should integrate natural cost terms or other incentives for natural motion.
  • DEP-RL may not be feasible for every application, motivating more principled coupling between DEP and an RL policy.

7 REPRODUCIBILITY STATEMENT

The paper provides extensive implementation, environment, algorithm, and hardware details, while making its environments and learning algorithms available through public or submitted code resources.

  • The paper provides experimental details, environment descriptions, hyperparameters, optimization graphs, and hardware requirements in the supplementary material.
  • The used RL algorithms are available from the TonicRL package, and the ostrich and human-run environments are publicly available.
  • The submitted source code includes the environments, variations, and learning algorithms, while MuJoCo is freely available and a curated repository is planned.

Supplementary Material

The supplementary material defines state-space coverage and the exploration-noise baselines, then describes the muscle model and its activation dynamics. It also notes simplifications in the simulated muscle mechanics and omitted biological delays and input constraints.

  • A.1 STATE-SPACE COVERAGE: State-space coverage is the fraction of visited grid points, with less than 100% expected when geometry prevents reaching every point.The metric is computed from discretized joint values and normalized by total grid size.
  • A.2 ORNSTEIN-UHLENBECK NOISE: OU noise generates temporally correlated signals through drift toward a mean plus stochastic Gaussian forcing.The drift and stochastic strengths, along with the mean, are tunable; actions outside [-1, 1] are clipped.
  • A.3 COLORED NOISE: Colored noise is characterized by a power spectral density whose frequency exponent β determines the noise color and temporal-frequency emphasis.White, pink, and red noise correspond to β = 0, 1, and 2, respectively, while β and a scaling parameter σ are tunable in optimization.
  • A.4 MUSCLE MODELING: The simulated muscle model uses nonlinear force-length and force-velocity relationships, inelastic tendons, and activation dynamics that low-pass filter control signals.MuJoCo determines total muscle-tendon length from muscle-fiber length under the inelastic-tendon assumption.
  • A.4 MUSCLE MODELING: The muscle model sets τact = 0.01 and τdeact = 0.04, so activation increases faster than it decreases, while real biological sensor and actuator delays are omitted.These choices simplify control relative to biological agents that must infer tendon-related configuration and operate under additional delays and input restrictions.

B EXPERIMENTAL DETAILS

The experiments use repeated episodes, resets, multiple random seeds, and evaluation without exploration to measure coverage, learning, speed, robustness, and gait behavior.

  • B.1 GENERAL DETAILS: State coverage uses 50 episodes of 1000 iterations, computes the metric over five episodes, and resets DEP's internal state thereafter.Training curves average 10 random seeds, with each point based on 10 evaluation episodes without exploration.
  • B.1 GENERAL DETAILS: Robustness is evaluated from final training checkpoints over 100 episodes with stepdown and slopetrotter perturbations, using task-specific success or distance measures.Maximum speed instead selects the fastest checkpoint from all runs and records the fastest velocity across 50 test episodes.

B.2 ENVIRONMENTS

The environments span torque- and muscle-driven reaching, locomotion, sparse-reward hopping and foraging, and perturbed locomotion, with varied action dimensions and task constraints.

  • B.2 ENVIRONMENTS: The benchmark includes a 2-DoF torque-arm comparative task, a 2-DoF six-muscle arm26 reaching task, and a 7-DoF humanreacher actuated by 50 muscles.For humanreacher, effective exploration requires temporal correlation and coordinated stimulation across connected muscle groups because of activation dynamics, asymmetric muscular geometry, and joint-limit cul-de-sac states.
  • B.2 ENVIRONMENTS: Ostrich-foraging uses 52 muscles to reach sparse randomly appearing goals, whereas ostrich-run maximizes horizontal running speed using only a weak velocity reward and no motion-capture data.Foraging keeps the last pose across episode resets, so an agent with inadequate exploration is very unlikely to encounter a goal.
  • B.2 ENVIRONMENTS: The robustness variants add a drop to ostrich-run or seven half-sloped obstacles, with success or traveled distance measuring performance under perturbations.The slopetrotter obstacles are spaced at 5 m intervals and cover 50 m in total.
  • B.2 ENVIRONMENTS: Human-run, human-hop, human-stepdown, and human-hopstacle vary locomotion rewards and obstacles, including sparse hopping rewards and parcours or slopes that penalize falling.Human-run uses an 18-leg-muscle planar model, whereas human-hop rewards a center-of-mass height above 1.08 m.
  • B.2 ENVIRONMENTS: Virtual action multipliers expand the planar reaching action spaces while multiplying muscle-related state data by the same factor.Tables summarize the joint, state, action, and observation dimensions for the environments and their variants.

B.3 DEP IMPLEMENTATION

The supplementary implementation details describe DEP inputs and history-dependent control, its alternation with RL, training infrastructure, tuning procedures, and state-coverage experiments.

  • B.3 DEP IMPLEMENTATION: DEP uses actuator-related proprioception from joint angles or normalized muscle lengths and forces, retaining actuator-linked information while discarding the rest of the RL state.A separate history-dependent controller is maintained for each parallel environment, with minimal overhead from the small control matrix.
  • B.4 RL IMPLEMENTATION: The RL agents use a slightly modified TonicRL implementation, and the most computationally intensive ostrich-run training required about 48 hours for 10^8 iterations on one V100 GPU and 20 CPU cores.The reported learning steps include repeated training updates for environment interactions.
  • B.6 HYPERPARAMETERS: Hyperparameter optimization did not improve default MPO performance, so most MPO parameters remained at TonicRL defaults while DEP integration settings were tuned by task group.DEP settings were shared across arm-reaching tasks, whereas ostrich-run settings were optimized and reused for ostrich-foraging.
  • B.6 HYPERPARAMETERS: Exploration baselines were tuned separately for each task and action space, while DEP was tuned once on humanreacher and then held fixed across arm-reaching experiments.The ablation experiments trained for 5 × 10^7 iterations and averaged over 10 random seeds.
  • C.1 STATE-COVERAGE: State-coverage visualizations compare trajectories from different noise processes on torquearm and arm26, while Figure 11 spans original and expanded action spaces.Figure 11 records hand trajectories during 50 pure-exploration episodes of 1000 iterations and reports end-effector-space coverage.

C.2 ABLATIONS

The ablations compare DEP-RL integration strategies, action correlations, baseline agents, and locomotion patterns across challenging overactuated tasks. Stochastic DEP switching and correlated exploration support stronger control and more regular ostrich gaits than conventional alternatives.

  • C.2 ABLATIONS: The stochastic ablation switches DEP over for HDEP iterations with probability pswitch, while continuing to adapt DEP and storing its actions for RL training.This variant was selected for the main results because the stoch-force version performed well across all tasks.
  • C.3 ACTION CORRELATION MATRIX: DEP produces strong action correlations and anticorrelations across ostrich muscle groups, whereas uniform, colored, and OU noise do not.The correlation patterns emerge after only 50 seconds of data while DEP learns from scratch.
  • C.4 ADDITIONAL BASELINES: DEP-MPO achieves the largest returns in ostrich-run and humanreacher compared with MPO, OU-MPO, and β-MPO baselines.OU-MPO and β-MPO improve over MPO, but DEP-MPO performs best in both tasks.
  • C.5 OSTRICH GAIT VISUALIZATION: DEP-MPO produces the most widespread and regular ostrich gait, with symmetric ≈1m steps versus ≈0.5m for MPO and ≈0.3m for TD4.MPO shows limited leg extension, while TD4 produces an irregular and asymmetric gait.

C.6 ADDITIONAL EVALUATION WITH DENSE AND SPARSE REWARDS IN LARGE VIRTUAL ACTION SPACES

DEP-MPO remains effective in dense and sparse reaching tasks with large virtual action spaces, while a simplified DEP analysis explains exploration through delayed velocity correlations. Performance also depends on switching probability, especially for unstable locomotion.

  • C.6 ADDITIONAL EVALUATION WITH DENSE AND SPARSE REWARDS IN LARGE VIRTUAL ACTION SPACES: DEP-MPO reaches good performance across all tested virtual action spaces for dense rewards and quickly solves sparse tasks with or without HER.Vanilla MPO degrades as action dimensionality increases, failing to learn reliably with 600 actions; HER improves learning but remains slower or ineffective at that scale.
  • C.7 DESCRIPTION OF A ONE-DIMENSIONAL SYSTEM: The simplified DEP construction uses a learned control matrix driven by current and delayed sensor-velocity correlations, under explicit assumptions about normalization, inverse modeling, bias, and update dynamics.The one-sensor mountain-car formulation extracts position from the RL state and applies the scalar control relation to that sensor.
  • C.7 DESCRIPTION OF A ONE-DIMENSIONAL SYSTEM: For the mountain car, delayed velocity correlations reverse actions intermittently, building momentum that drives the system across the state space.The simplified analysis reports full exploration for ∆t ∈{5, ..., 28}, with ∆t = 27 producing increasing oscillation amplitudes.
  • C.8 SENSITIVITY TO CHANGES IN pswitch: DEP-MPO is less sensitive to pswitch for humanreacher than for ostrich-run, where larger switching probabilities often cause falls and hurt learning.The ablation averages results over five seeds and compares against MPO without DEP.
Loading 2206.00484v2…