PaperlayerView exact revision on arXiv

Source-linked AI summary

SMR: Learning streaming motion retargeting for humanoids from human semantic

Chun Gu, Juntong Zhang, Jingyu Li, Li Zhang

arXiv:2610.29169v1cs.RO

TL;DR

Kinematic geometric matching does not establish dynamic executability, and prior physics-supervised neural retargeters still require full-sequence context. SMR trains a causal retargeter from semantic-conditioned physics experts, improving closed-loop results over kinematic and physics-refined baselines under a shared controller.

  • Problem

    Retargeting must preserve human motion semantics in physically executable robot references, while prior physics-supervised neural retargeters depend on prescribed kinematic solutions and full-sequence context.

    Source: §1 · pp. 1–2

  • Method

    SMR tracks robot-aware task-space semantics with physics experts and trains a strictly causal Transformer on accepted executed state rollouts.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    In the shared 400-motion Unitree G1 evaluation, SMR completes 97.0% of motions under a frozen whole-body controller, versus 90.0% for GMR and 92.0% for NMR, while lowering MPJPE and E-MPJPE.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Accepted expert rollouts support streaming references that improve closed-loop execution under the tested simulator and controller.

    Source: §5 · p. 10

    Evaluation centers on the Unitree G1 in one simulation and control setting; transfer to other morphologies, controllers, and hardware remains untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context. We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller. Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.

1 INTRODUCTION

Streaming humanoid retargeting must preserve human semantics while producing physically executable references from current and past observations. SMR uses semantic-conditioned physics experts to train a causal retargeter for this setting.

1 INTRODUCTION

  • Human-to-humanoid transfer must preserve motion semantics in physically executable references, but geometric agreement alone does not establish dynamic executability.Humans and robots differ in morphology, joint structure, mass distribution, contact mechanics, and actuation.Source: §1 · p. 1
  • NMR refines prescribed kinematic trajectories through simulated tracking but uses bidirectional full-sequence modeling, requiring future observations and preventing streaming use.The proposed alternative specifies robot-aware task-space semantics directly and uses physical execution for supervision.Source: §1 · pp. 1–2
  • SMR tracks robot-aware landmark positions, orientations, and velocities directly in simulation, then trains a strictly causal Transformer on accepted executed state rollouts.The student uses only current and past human observations to generate references for a downstream whole-body controller.Source: §1 · p. 2
  • Under the same frozen whole-body controller, SMR improves closed-loop performance over kinematic and physics-refined baselines, while its experts outperform direct optimization-based retargeting in semantic fidelity and physical validity.The paper also reports whole-body execution on a physical Unitree G1.Source: §1 · p. 2

2 RELATED WORK

Related work spans optimization-based retargeting, learned cross-skeleton mappings with physics supervision, and physics-based motion tracking. SMR differs by learning streaming motion references from accepted physics-expert rollouts.

2.1 OPTIMIZATION-BASED MOTION RETARGETING

  • Optimization-based methods progress from constrained character motion transfer [Gleicher, 1998; Popović & Witkin, 1999; Tak & Ko, 2005] to dynamics-aware optimization, while SMR learns references without deployment-time retargeting optimization.Methods incorporate geometric matching, physical constraints, interaction preservation, or dynamics-aware optimization.Source: §2.1 · p. 2

2.2 LEARNING-BASED AND PHYSICS-SUPERVISED RETARGETING

  • Learning-based retargeters use latent spaces, skeleton-aware models, and motion translation; NMR tracks GMR trajectories, whereas SMR experts directly track human semantics.SMR then uses accepted state rollouts to train a human-only, strictly causal reference generator.Source: §2.2 · p. 3

2.3 PHYSICS-BASED MOTION TRACKING

  • Physics-based tracking policies use robot-state feedback to produce actions, whereas SMR produces references from human observations and evaluates them with a frozen whole-body controller.This evaluation is intended to isolate the effect of retargeting on downstream execution.Source: §2.3 · p. 3

3 METHOD

SMR turns human motion into robot-aware task-space semantics, trains physics experts to execute those targets, and distills accepted rollouts into a strictly causal retargeter. The method separates task-space target construction from robot trajectory generation and uses only current and past human observations at streaming inference.

3.2 SEMANTIC-CONDITIONED PHYSICS EXPERTS

  • SMR converts human motion into robot-aware task-space commands for 15 landmarks, with positions, orientations, and finite-difference velocities.The transport produces task-space targets, not an IK solution or robot joint trajectory.Source: §3.2 · pp. 3–4
  • More than 400 semantic-conditioned physics experts track the commands in simulation, with motion clusters organizing training pools.Their rewards combine semantic tracking with regularization for action changes, joint velocity and limits, self-collision, and survival; offline per-frame IK only initializes episodes.Source: §3.2 · pp. 4–5
  • Accepted expert rollouts complete the motion and pass semantic-fidelity and physical-validity gates before supervising the student.The recorded states are executed under the specified robot, contact, actuator, and simulator models.Source: §3.2 · p. 5

3.3 NEURAL RETARGETING

  • A single student maps human observations to robot references using 29 joint positions and pelvis translation and rotation offsets anchored to the human pelvis.The anchored representation avoids integrating predicted root velocity, though the reconstructed root can inherit drift, heading noise, and ground-height error from the human input.Source: §3.3 · pp. 5–6
  • The strictly causal Transformer attends to the current and 127 preceding tokens, processes one arriving frame at a time, and uses zero lookahead.Causal windows are trained on accepted expert rollouts across 30, 50, and 90 Hz input rates, with noise and causal holds applied to input history.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments assess reference quality, closed-loop execution, scaling and ablations, and streaming hardware deployment on the Unitree G1. SMR improves closed-loop execution under a shared frozen controller and produces stable motion in the demonstrated physical teleoperation sequences.

4.1 EXPERT AND STUDENT REFERENCE QUALITY

  • The semantic expert achieves much lower landmark error than GMR at similarly low collision, while SMR slightly improves landmark accuracy but has worse self-collision and foot-slip metrics.The authors suggest the student’s lack of simulator feedback for explicitly correcting collisions or slipping contacts may explain the latter difference.Source: §4.1 · p. 7, Table 3 · p. 7

4.2 CLOSED-LOOP EXECUTION

  • On the shared 400-motion evaluation, SMR completes 97.0% of motions versus 90.0% for GMR and 92.0% for NMR, with the lowest MPJPE (0.039 m) and E-MPJPE (0.080 m).The comparison uses the same frozen whole-body controller; the selected crouching, kneeling, and floor-level poses also show closer posture preservation than GMR and NMR.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 3 · p. 8

4.3 ABLATIONS

  • Held-out error falls markedly through 50% of training data, gains little at 100%, and slightly worsens for models wider than 12.7M parameters despite lower training loss.The 12.7M-parameter architecture is used for the final model.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • By iteration 4,000, neural references yield higher MimicLite reward and longer episodes, although reward curves cross earlier and reference-processing protocols differ.These curves describe training on each reference distribution, not a controlled comparison of retargeters alone.Source: §4.3 · pp. 8–9
  • Auxiliary losses reduce reference foot slip and both frozen-controller tracking errors in matched pairs with little success change; augmentation lowers both controller errors but has mixed clean-input reference effects.The auxiliary terms are velocity, foot-penetration, and support-foot-slip losses; augmentation uses correlated pose noise and causal holds.Source: Table 5 · p. 8, §4.3 · p. 9

4.4 STREAMING REAL-ROBOT DEPLOYMENT

  • At 50 Hz, SMR takes 0.37 ms per frame versus 4.87 ms for GMR’s iterative solve; physical demonstrations show stable SMR-driven motion, unlike GMR’s unstable kneeling and out-of-phase arm motion.The latency values average three warm-run medians; XLarge was measured on a separate GPU.Source: §4.4 · p. 9

5 CONCLUSION

SMR trains a strictly causal retargeter from semantic-conditioned physics-expert rollouts, separating the semantic task from its physical realization. In the shared Unitree G1 evaluation, it improves motion completion over kinematic and physics-refined baselines under a frozen controller.

5 CONCLUSION

  • Semantic-conditioned physics experts track robot-aware task-space commands directly, and accepted simulator state rollouts train a strictly causal retargeter from current and past human observations.IK initializes episodes rather than prescribing the tracking trajectory.Source: §5 · p. 10
  • The method separates the semantic task from its physical realization while preserving a streaming input contract.Source: §5 · p. 10
  • SMR completes 97.0% of motions, versus 90.0% for GMR and 92.0% for NMR, while lowering MPJPE and E-MPJPE under a frozen whole-body controller.The comparison uses a shared 400-motion Unitree G1 evaluation.Source: §5 · p. 10

6 LIMITATIONS AND FUTURE WORK

The evaluation is limited to the Unitree G1 in one simulation and control setting, and transfer to other embodiments and conditions remains untested. The appendix describes the semantic-command and expert-rollout pipeline, while the reproducibility statement points to implementation and experimental details.

6 LIMITATIONS AND FUTURE WORK

  • Evaluation centers on the Unitree G1 under one simulation and control setting; transfer to other morphologies, actuation, contact conditions, controllers, and hardware remains untested.The student also lacks the robot-state and limited future semantic context available to the physics expert, and its performance ceiling may be lower than a bidirectional full-sequence model.Source: §6 · p. 10
  • Simulation acceptance does not establish hardware safety, and the qualitative demonstration does not establish safety across robots, environments, or human interactions.Physical deployment requires separate risk assessment, supervised testing, and appropriate safeguards.Source: ETHICS STATEMENT · p. 10

A SEMANTIC-CONDITIONED PHYSICS EXPERTS

A.1 ROBOT-AWARE SEMANTIC COMMAND DETAILS

The semantic command represents human motion through robot-aware task-space targets for a defined set of landmarks. Its transport uses the robot’s neutral configuration and supplies landmark positions, orientations, and velocities to the expert pipeline.

A.1 ROBOT-AWARE SEMANTIC COMMAND DETAILS

  • The 15-landmark set includes the pelvis, neck, head, and bilateral hips, knees, ankles, shoulders, elbows, and wrists; toes determine vertical alignment but are excluded.Each landmark has position, orientation, linear velocity, and angular velocity fields.Source: §A.1 · p. 14, Table 7 · p. 14
  • Transport scales the human root by the ratio of robot-to-human leg rest lengths, aligns the lowest toe, and assembles remaining landmarks using robot rest-pose segment lengths.It produces task-space targets, not an inverse-kinematics solution or robot joint trajectory.Source: §A.1 · p. 14
  • The command archives transported positions, orientations, and their linear and angular velocities in the world frame at 50 Hz.Velocities are used by the expert critic and tracking rewards, not as additional student inputs.Source: §A.1 · p. 14

A.2 MOTION GROUPING AND EXPERT ASSIGNMENT

The motion corpus is partitioned into offline training pools before physics experts are fitted. Periodic-autoencoder descriptors and source-specific clustering assign motions to experts, while the student receives no cluster or expert identity.

A.2 MOTION GROUPING AND EXPERT ASSIGNMENT

  • The corpus is partitioned into offline training pools before physics experts are fitted; motions shorter than the 101-frame, two-second window are padded by holding the terminal pose.Source: §A.2 · pp. 14–15
  • A Periodic Autoencoder encodes 101-frame windows into 64 phase channels, whose harmonic amplitudes and power-weighted frequencies form a 256-dimensional descriptor per motion.Descriptors aggregate mean and standard deviation across sliding windows; motions are not split into separate descriptors.Source: §A.2 · p. 15
  • Mini-Batch K-means assigns BONES-SEED and 100STYLE clusters to experts, while smaller specialized sources use one expert per motion; the Unitree G1 roster has 465 experts for 145,970 candidate motions.Assignment sets offline training pools only; the student does not receive cluster or expert identity.Source: §A.2 · p. 15, Table 8 · p. 15

A.3 PHYSICS EXPERT TRAINING

Each physics expert is a closed-loop PPO policy that tracks semantic commands in simulation, with IK used only to initialize the robot state. The training setup combines sparse proprioception, tracking rewards, and adaptive tracking widths.

A.3 PHYSICS EXPERT TRAINING

  • Experts use a 50 Hz semantic command window with current and past samples plus four future samples, while proprioception includes sparse history and recent policy actions.The actor sees position and orientation fields in the command window; landmark velocities are reserved for the privileged critic and tracking rewards.Source: §A.3 · p. 15
  • Each policy outputs a 29-dimensional joint-position target, and its eight tracking terms use exponential kernels with separate widths.Tracking covers the root and all landmarks except the head for position, and the torso, wrists, and ankles for orientation.Source: §A.3 · p. 15
  • Tracking widths are tightened from 1.25 to 0.5 times their base values over 1,050 iterations, while reward weights remain fixed.Episodes terminate when root or landmark errors exceed thresholds for consecutive control steps, or when the motion ends.Source: §A.3 · p. 15
  • Offline differential IK supplies only the initial simulator state; it uses neither contact reasoning nor cross-frame continuity and is not an expert tracking target.The actor and critic are MLPs trained with PPO across 16,384 parallel environments; single-motion experts run 1,500 iterations and cluster experts 4,000.Source: §A.3 · pp. 15–16

A.4 EXPERT ROLLOUT SUPERVISION

Expert supervision is produced by replaying each assigned motion and retaining only rollouts that complete the motion and pass the label-production checks. Motion-group splits hold out student distillation examples, not expert generalization.

A.4 EXPERT ROLLOUT SUPERVISION

  • On Unitree G1, the semantic expert release accepts 105,634 of 145,970 candidate motions, while the GMR-conditioned release accepts 106,019.Rejected candidates remain in the candidate-level acceptance denominator and are excluded from positive student labels.Source: §A.4 · p. 16
  • An accepted rollout is a deterministic 50 Hz mean-policy replay that reaches the final frame without termination and passes every Table 9 check.The replay starts from the same offline IK reset state used during training and pairs each recorded simulator state with its frame's semantic command.Source: §A.4 · p. 16, Table 9 · p. 16
  • Accepted rollouts are split by motion group 90/5/5 into train, validation, and test sets, so holdout evaluation measures student distillation on unseen groups.Experts are trained before the split, and no motion contributes frames to more than one split.Source: §A.4 · p. 16

B NEURAL RETARGETING

The student maps only current and past human observations to a robot motion reference, which a downstream whole-body controller executes. Its causal Transformer runs one arriving frame at a time, with training labels aligned to the causal input stream.

B NEURAL RETARGETING

  • The 35-D output contains 29 Unitree G1 joint positions and pelvis translation and relative-rotation offsets expressed in the current human-pelvis frame.Composing those offsets with the current human pelvis pose reconstructs the robot root; the output is a reference, not a torque or action.Source: B NEURAL RETARGETING · p. 16
  • At time t, the input includes a 79-D human pose and a 6-D root twist, with no simulator state, expert action or identity, or future human observation.The twist is a backward difference at the native observation rate and is zero at the first session frame.Source: B NEURAL RETARGETING · p. 16
  • The four-block, 12.7 M-parameter Transformer uses a causal 128-token context and a bounded ring KV cache to process one arriving frame per streaming call.RoPE positions are scaled by 50/f so the context spans the same physical duration across input rates; the cache resets at session boundaries.Source: B NEURAL RETARGETING · p. 16
  • Strict causality applies to the retargeter interface, while the downstream whole-body controller may use an 80-ms future-reference buffer.The buffer is reported separately and is not retargeter lookahead.Source: B NEURAL RETARGETING · pp. 16–17
  • Training resamples 256-frame windows and recomputes human features and root-offset labels under causal alignment, sometimes corrupting only the input history.The input corruption combines temporally correlated pose noise with one- or two-frame causal holds; offset labels remain based on the uncorrupted expert rollout.Source: B NEURAL RETARGETING · p. 17

Related papers

Loading 2610.29169v1…