PaperlayerView exact revision on arXiv

Source-linked AI summary

RLMuscle: Muscle-Driven Character Animation via Reinforcement Learning

Chunlei Li, Siyuan Yu, Yang Gao, Shuai Li, Peng Yu, Aimin Hao, He Wang

arXiv:2610.30033v1cs.GR

TL;DR

Hand-authored muscle activations are chosen for visual rather than physical plausibility, so their physical plausibility is not guaranteed. RL learns activations on a 1D Hill-type surrogate, then transfers them to a 3D volumetric simulation using the same constitutive law and an EMG prior. The trained walk averages 10.5° joint-angle error and 49 mm end-effector error; recovered activations achieve mean in-band 91% and mean correlation 0.97 against sEMG.

  • Problem

    Hand-authored muscle activations are chosen for visual rather than physical plausibility, so their physical plausibility is not guaranteed.

    Source: §I · p. 1

  • Method

    RL learns activations on a 1D Hill-type surrogate, then transfers them to a 3D volumetric simulation using the same constitutive law and an EMG prior.

    Source: A. Overview · p. 3, 4) EMG Prior · p. 4, §VI · p. 12

  • Results

    The trained walk averages 10.5° joint-angle error and 49 mm end-effector error; recovered activations achieve mean in-band 91% and mean correlation 0.97 against sEMG.

    Source: A. Full-Body Demonstration · p. 8, B. Validation Against EMG · p. 9

  • Takeaways & Limitations

    The system produces volumetric muscle deformation from activations that drive the tracked motion, with full-body simulation reported at 18 ms/frame.

    Source: C. Comparison with Baselines · p. 10, §VI · p. 12

    The 3D volumetric solver is kept outside RL training because integrating it into the loop was slow, unstable, and uncontrollable in the authors’ experiments.

    Source: §V · p. 11

Abstract

from arXiv · show

Body deformation caused by muscle contraction plays a vital role in the visual realism of character animation, yet in most 3D animation pipelines artists have to manually specify muscle activations, which is time-consuming and gives no guarantee of physical realism. To address this, we propose a new method in which a reinforcement learning (RL) controller discovers per-muscle activations that track reference motions without manual tuning. Existing RL muscle controllers stop at 1D muscles; we go one step further, keeping the 1D model inside the RL control loop as a fast surrogate, and using a 3D volumetric simulation to produce the final deformation. Because both stages share the very same constitutive law, the muscle deformation is produced by the same activations that generated the motion. We further add a measured electromyography (EMG) prior that steers the redundant activation solution toward real human muscle recruitment. To achieve stable, efficient, and realistic simulation, we propose a novel geometric multigrid for Extended Position-Based Dynamics (XPBD) that is built on a coarse tetrahedral cage with barycentric-coordinate prolongation and closes a V-cycle on tetrahedral meshes. Within this GPU-parallelized solver, we inject the same Hill-type force-length law as an energy-based fiber constraint that turns per-fiber activation into realistic muscle contraction and bulging. We demonstrate volumetric muscle simulation on a 194-muscle human body with a diverse set of motions (walk, kick, swim, overhead-squat, etc.), simulating the full-body model at 18 ms per frame on a single RTX 4090. Our work takes a step toward character animation that is not only visually convincing but also biomechanically grounded. The website with code and videos is at https://chunleili.github.io/project-page-rlmuscle/.

I. INTRODUCTION

Hand-authored muscle activations are labor-intensive and chosen for visual rather than physical plausibility, leaving open how to recover activations that produce a target motion. The paper combines 1D RL control, an EMG prior, and 3D volumetric simulation with a geometric multigrid solver.

I. INTRODUCTION

  • Hand-authored tension lines and keyframed activations prioritize visual plausibility, and the workflow is labor-intensive rather than solving which activations produce a target motion.The paper frames recovering hidden controls from observed motion as the muscle inverse problem.Source: §I · p. 1
  • Existing RL muscle controllers use simplified 1D Hill-type models, while nonlinear, high-dimensional control makes analytical inversion intractable and naive RL sample-inefficient.The target also requires 3D volumetric deformation for visual realism.Source: §I · p. 1
  • A 1D Hill-type surrogate trains muscle activations through RL, then a 3D model uses the same Hill–Millard fiber law to produce deformation consistent with the motion dynamics.This separates 3D simulation from the control loop; the authors report that direct volumetric optimization was unstable and failed to converge in their tests.Source: §I · pp. 1–2
  • A measured surface-EMG prior steers the redundant activation solution toward human muscle recruitment, matched by synergist group and aligned to a common gait cycle.The prior addresses activation drift away from measured EMG when redundancy is resolved using torque and effort alone.Source: §I · p. 2
  • A tetrahedral geometric multigrid with barycentric prolongation and Petrov–Galerkin restriction closes a full XPBD V-cycle while retaining local per-constraint solves.The framework also includes muscle–bone collision handling based on a signed distance field.Source: §I · p. 2

II. RELATED WORK

  • Prior volumetric muscle work includes FEM energy potentials and position-based elastic rods, while prior 1D RL controllers actuate line-segment Hill muscles rather than volumetric deformation.The paper distinguishes its forward-dynamic RL approach from static volumetric activation methods that cannot guarantee the recovered activations drive the target motion.Source: §II · pp. 2–3

III. METHOD

The system learns muscle-driven motion on a 1D asset, then transfers motion and activations to a 3D character for volumetric simulation. Its RL controller jointly trains policy and muscle networks, using compact joint-target actions rather than directly exploring hundreds of activations.

III. METHOD

  • The method couples a 1D Hill-type model with an RL controller and a 194-muscle volumetric model using the same constitutive law.The method section covers the RL inverse problem, 1D dynamics, 3D simulation, and the mapping between models.Source: §III · p. 3

A. Overview

  • The 1D RL stage learns reference-motion bone movement and muscle activations, which are retargeted and mapped to a 3D character for a single XPBD forward simulation.The 1D track serves as a fast surrogate; 3D simulation is performed at inference.Source: A. Overview · p. 3

B. Reinforcement Learning Control

  • A policy network and muscle network co-train in one PPO loop so the controller discovers 284 activations that track the reference motion.A training-only value network acts as PPO’s critic and does not participate in control.Source: B. Reinforcement Learning Control · p. 3

1) Two-Network Co-Training:

  • Instead of exploring 284 activations directly, the policy outputs per-joint PD-target offsets that Stable PD converts to desired torques for the muscle network.The 1D skeleton has 50 actuated degrees of freedom; the muscle network maps desired torque to activations.Source: 1) Two-Network Co-Training · p. 3

2) Policy Network:

The policy is trained to track reference motion with a reward built from pose, velocity, end-effector, center-of-mass, and pelvis-orientation terms. Balance-critical motions reweight balance terms because pose matching alone can leave the character unstable.

2) Policy Network:

  • The motion-mimicking reward combines pose, velocity, and end-effector tracking with center-of-mass and pelvis-orientation terms for balance-critical motions.Pose error uses the SO(3) geodesic distance between current and target joint frames.Source: 2) Policy Network · p. 3, Figure 3 · p. 4
  • End-effector errors are measured relative to the center of mass, and their multiplicative reward suppresses local joint credit when limbs are grossly misplaced.The exponential kernel bounds each reward term to [0,1] and smoothly saturates as tracking error grows.Source: 2) Policy Network · p. 4
  • For squats and single-leg kicks, weights shift toward center-of-mass and pelvis-up terms, with root pose added; without these terms the policy can match pose but lose balance and fall.The pelvis-up term measures yaw-free torso tilt.Source: 2) Policy Network · p. 4

3) Muscle Network:

A supervised muscle network resolves redundant muscle activations while an EMG prior steers its solutions toward measured recruitment. The prior aligns coarse electrode recordings with the model by matching synergist-group means over a common gait cycle.

3) Muscle Network:

  • The muscle network maps desired joint torques to per-muscle activations, resolving the underdetermined inverse problem with torque, minimum-effort, and EMG losses.Torque and effort losses are standard; the EMG loss steers redundant solutions toward measured human recruitment.Source: 3) Muscle Network · p. 4

4) EMG Prior:

  • Four right-leg EMG electrodes represent synergist groups, so each channel is matched to its group’s cross-section-weighted mean activation.The group mean is weighted by maximal isometric force, which reduces to cross-sectional area when maximal stress is uniform.Source: 4) EMG Prior · p. 5
  • The EMG signals and model activations are aligned on a common gait cycle and compared by shape and timing, not magnitude.The prior uses a phase-indexed MSE against a peak-normalized human envelope; its scale and channel weights are specified separately.Source: 4) EMG Prior · p. 5
  • The EMG prior modifies only the muscle-network loss, leaving the policy’s motion objective unchanged.It is enabled for walking, where a measured envelope exists, and switched off for other motions.Source: 4) EMG Prior · p. 5

C. 1D Hill-type Muscle Surrogate

The 1D Hill-type model serves both as the RL training surrogate and as the constitutive law for the 3D muscle simulation. Its force-to-torque mapping is differentiable and closed form.

C. 1D Hill-type Muscle Surrogate

  • The 1D Hill-type model has a dual role: it drives RL training through MuJoCo tendon paths and supplies the constitutive law for the 3D XPBD muscle.The 1D asset uses a rigid-body skeleton with spherical and revolute joints and polyline muscle paths.Source: C. 1D Hill-type Muscle Surrogate · p. 5, Figure 5 · p. 5
  • Each fiber’s active force depends on activation and normalized length, with the active force–length curve peaking at optimal fiber length.The model places a tendon spring in series with a muscle belly containing active and passive elements in parallel.Source: C. 1D Hill-type Muscle Surrogate · p. 5, Figure 6 · p. 6
  • The model assumes an infinitely stiff tendon and omits velocity-dependent force so its forces admit an energy potential.Muscle-specific peak isometric force scales the normalized fiber force.Source: C. 1D Hill-type Muscle Surrogate · p. 6
  • The anchor-polyline path length determines normalized fiber length, and point Jacobians yield the closed-form torque map τ_musc(a_M) = La_M + b.L gives per-unit-activation moment arms and b is passive torque; the differentiable map trains the muscle network and drives MuJoCo.Source: C. 1D Hill-type Muscle Surrogate · p. 6

D. 3D Volumetric Muscle Simulation

The volumetric simulation combines anisotropic fiber contraction, volume conservation, and muscle–bone attachment within XPBD. A geometric multigrid solver is introduced to accelerate stable simulation.

D. 3D Volumetric Muscle Simulation

  • The 3D simulation addresses fiber contraction, volume conservation, and muscle–bone coupling, with tendons attached hard and muscle bellies attached softly.Fiber contraction drives shortening along fibers, while volume conservation produces bulging in orthogonal directions.Source: D. 3D Volumetric Muscle Simulation · p. 6
  • A geometric multigrid solver is proposed to accelerate XPBD simulation under the high stiffness needed for muscle volume conservation.Under-converged solves can leave visible artifacts and inverted tetrahedra.Source: D. 3D Volumetric Muscle Simulation · p. 6

1) XPBD Update:

  • The volumetric deformation is driven by XPBD position constraints for volume and shape preservation, fiber contraction, attachment, and collision.Each constraint is solved with the standard XPBD update using its constraint function and gradient.Source: 1) XPBD Update · p. 6

2) Geometric Multigrid:

The method builds a two-level geometric multigrid for tetrahedral XPBD, using a coarse cage to guide fine-mesh motion. Its Petrov–Galerkin restriction exactly recovers coarse vectors after prolongation.

2) Geometric Multigrid:

  • The tetrahedral cage hierarchy and Petrov–Galerkin restriction support a complete geometric multigrid for XPBD, unlike HPBD’s cloth-focused hierarchy that does not close a V-cycle.Fine vertices are embedded in coarse tetrahedra using barycentric coordinates; coarse corrections are prolonged to the fine mesh.Source: 2) Geometric Multigrid · pp. 6–7
  • Restriction finds the least-squares coarse state for the fine state, and its left-pseudoinverse property ensures RP = I.This exactly recovers any coarse-level vector after prolongation.Source: 2) Geometric Multigrid · pp. 6–7
  • The hierarchy uses a 31,044-tetrahedron cage and a 468,162-tetrahedron fine simulation mesh.Figure 7 shows the coarse cage on the left and the fine mesh on the right.Source: Figure 7 · p. 7
  • The per-step simulation uses a two-level V-cycle, with restriction, coarse solve, and prolongation at steps 4, 5, and 6.These steps use Eq. (13), Eq. (10), and Eq. (11), respectively.Source: Figure 8 · p. 7

3) Volume and Shape Preservation Constraints:

Volume and shape constraints preserve muscle volume and resist shear. An anisotropic fiber constraint places the Hill law’s energy in the 3D XPBD model, linking fiber contraction to bulging.

3) Volume and Shape Preservation Constraints:

  • Stable Neo-Hookean constraints primarily enforce incompressibility, while an explicit volume constraint and ARAP term preserve volume and resist shear.The ARAP term uses an SVD-based formulation with non-negative clamping, found more robust than the polar-decomposition formulation.Source: 3) Volume and Shape Preservation… · p. 7

4) Anisotropic Fiber Constraints:

  • The anisotropic fiber constraint encodes Hill strain energy in XPBD using 3D stretch along the rest fiber direction, identical to the 1D normalized fiber length.The piecewise quintic Hill curve yields a closed-form degree-10 energy polynomial per segment, avoiding numerical quadrature; along-fiber shortening and volume conservation produce orthogonal bulging.Source: 4) Anisotropic Fiber Constraints · p. 7

5) Attachment Constraints:

Muscles attach to bones through hard pins or soft springs, while inter-muscle springs bundle neighboring muscles. The method also handles bone collisions and transfers motion and muscle activations between 1D and 3D assets.

5) Attachment Constraints:

  • Pins impose attachment positions directly, while springs allow deformation; a painted mask assigns pins near tendon origins and insertions and springs to the belly.Bone attachments are one-way, with rigid bones driving muscles; inter-muscle springs are two-way.Source: 5) Attachment Constraints · p. 7
  • Too many pins increase tetrahedron inversions and surface roughness, while too few let muscles lag or detach, especially thin, slender muscles.These muscles are dominated by boundary conditions rather than elasticity.Source: 5) Attachment Constraints · p. 8

6) Collision Handling:

  • Bone collisions use a precomputed signed-distance field, while explicit muscle–muscle collisions are omitted because inter-muscle springs hold neighbors apart and residual penetration is minor.The RL and volumetric assets are also bridged by retargeting 50 actuated DOFs and mapping 284 1D muscles to 194 3D muscles by anatomical similarity, then nearest match if needed.Source: 6) Collision Handling · p. 8, E. Motion Retargeting and Muscle Mapping · p. 8

IV. RESULTS

The system trains muscle activations on a 1D surrogate, then uses them in a single 3D volumetric simulation. Across the tested motions, it tracks references and the EMG prior brings recovered activations closer to measured recruitment.

A. Full-Body Demonstration

  • The 194-muscle body was simulated for walking, kicking, swimming, and squatting, with each motion training in under 2 hours.The volumetric simulation is a single forward pass after RL training on the 1D Hill surrogate.Source: A. Full-Body Demonstration · p. 8, §IV · p. 8, Figure 9 · p. 9
  • In a deterministic walking rollout, joint-angle error averaged 10.5°, end-effector error 49 mm, relative torque residual 17.8%, and mean muscle activation 0.061.The generated motion tracks closely but not exactly because activations must keep it physically feasible.Source: A. Full-Body Demonstration · p. 8, Figure 11 · p. 9

B. Validation Against EMG

  • With the EMG prior, activations reached 91% mean in-band agreement and 0.97 mean correlation with sEMG while preserving stable, faithful gait.Without the prior, mean in-band agreement was 20% and mean correlation was −0.04.Source: B. Validation Against EMG · p. 9, Figure 12 · p. 9

C. Comparison with Baselines

The volumetric results are close to production and neural-system references, while the ablations show gains from multigrid and collision handling. Multigrid improves convergence on both the bunny benchmark and the full-body muscle model.

C. Comparison with Baselines

D. Ablation Study

  • The ablation study isolates geometric multigrid, collision handling, and the Hill-type fiber constraint in the volumetric pipeline.The multigrid evaluation includes both a squashed-bunny benchmark and the full-body muscle model.Source: D. Ablation Study · p. 10, 1) Effect of Multigrid · p. 10

1) Effect of Multigrid:

2) Effect of Collision:

3) Effect of the Fiber Constraint:

The Hill–Millard fiber constraint produces activation-dependent shortening and bulging, while the system’s current training and multigrid choices have defined scope limits.

3) Effect of the Fiber Constraint:

V. LIMITATIONS AND FUTURE WORK

  • The 3D simulation is not included in RL training, and slender muscles near the distal forearm are excluded from multigrid and solved at fine level.The authors attribute these limits to unstable, slow coupled training and difficulty coarsening thin elongated geometry.Source: §V · p. 11

VI. CONCLUSION

  • The system learns activations on a 1D Hill-type surrogate and applies the same constitutive law in GPU-parallel XPBD volumetric simulation.The conclusion describes an EMG prior steering redundant activations toward measured human recruitment, with geometric multigrid supporting the stiff solve.Source: §VI · pp. 11–12
Loading 2610.30033v1…