PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training

Raphael Memmesheimer, Sven Behnke

arXiv:2609.26420v1cs.ROcs.CV

TL;DR

Human-space text-to-motion models do not account for robot dynamics, and tracking controllers cannot replan infeasible references. The paper samples from a frozen generator and uses simulated rollouts of the deployment policy to select a candidate. Physics selection reaches the generator’s any-of-8 ceiling, while the results also identify limits that selection cannot resolve.

  • Problem

    Human-motion generators lack a model of robot dynamics, while whole-body trackers can execute references but cannot replan infeasible ones; prior trained systems do not isolate the value added by training.

    Source: §I · p. 1

  • Method

    S3 samples candidates from a frozen text-to-motion model, retargets and simulates them with the deployment policy, and selects the candidate it executes best.

    Source: §I · p. 1

  • Results

    89.5% upright execution is reached from 83.5% on 200 prompts and from 80.5% on the complete test split; all 177 gate-selected clips completed standing on the real G1.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Selection recovers feasible motions already sampled but cannot create motions the generator does not produce; pelvis-lowering motion remains infeasible for the frozen generator.

    Source: §V · p. 7, §VI · p. 8

    The pipeline is offline and open-loop, has no exteroceptive sensing or object-interaction model, and evaluates motion on flat ground without props.

    Source: §V · p. 7

Abstract

from arXiv · show

Text-to-motion models generate plausible human motion but do not model a robot's dynamics; whole-body tracking controllers execute robot references reliably but cannot replan an infeasible one. Recent language-to-humanoid systems bridge this gap by training. We measure how much of the gap closes with no training at all, by putting the deployment controller itself in the loop. Sample-simulate-select (S$^3$) draws $N$ motions per prompt from a frozen text-to-motion model, retargets each to a Unitree G1 by direction-matching inverse kinematics, rolls all of them out under full rigid-body dynamics with the pretrained SONIC tracking policy, and keeps the candidate the policy executed best. Because the verifier is the deterministic simulator itself, S$^3$ attains the any-of-$N$ ceiling by construction; what we measure is where that ceiling lies and what falls short of it. On 200 stratified HumanML3D test prompts with $N=8$, upright execution rises from 83.5% to 89.5% and hardware-gate passes from 33 to 85; on the complete test split (4,184 prompts) it rises from 80.5% to 89.5%. A kinematic verifier that predicts falls well (AUROC 0.90) recovers only a quarter of this gain: ranking a prompt's own candidates is harder than classifying the population. What selection cannot fix is one class, prompts that lower the pelvis, which a generator trained on retargeted robot data does execute. We further score the semantic fidelity of the executed motion with the standard text-motion evaluator, with a real-mocap control that attributes the loss to the robot projection, ablate the retargeter against GMR (complementary failures: the any-of-8 ceiling rises to 95.0% over both), and execute all 177 gate-selected clips on the real G1: every one completes standing, with hardware tracking error matching simulation ($r=0.94$).

I. INTRODUCTION

Human-space motion generators lack robot dynamics, while trackers cannot replan infeasible references, and prior language-to-humanoid systems train without isolating what training adds. This paper measures the training-free gap with sample-simulate-select, which uses the deployment policy in simulation to choose among frozen-generator candidates.

I. INTRODUCTION

  • Language-to-humanoid systems train generators on retargeted robot data, language-conditioned policies, or human-space generators with tracker feedback, leaving the value of training unisolated.The paper frames its question as how much of the gap is sampling variance versus motion the generator never produces feasibly.Source: §I · p. 1
  • S3 samples motions from a frozen generator, retargets and simulates them with the deployment policy, then selects the candidate it executed best.With a deterministic simulator as verifier, selection reaches the any-of-N ceiling by construction.Source: §I · p. 1, Figure 2 · p. 2
  • 89.5% of prompts execute upright after physics selection, up from 83.5% on 200 stratified prompts; the complete test split rises from 80.5% to 89.5%.The kinematic predictor recovers 1.5 points, while the any-of-8 ceiling is 89.5%.Source: §I · p. 2
  • The unrecoverable remainder is pelvis-lowering motion, which a generator trained on retargeted robot data executes under the same verifier.A kinematic predictor has AUROC 0.90 but recovers only 1.5 points.Source: §I · p. 2
  • The evaluation also measures semantic fidelity, compares retargeting with GMR, and tests 177 gate-selected clips on the real G1.The hardware tracking error matches simulation.Source: §I · p. 2

II. RELATED WORK

Related work spans human-space motion generation, humanoid tracking and language-conditioned systems. The paper positions its training-free rollout verifier against modular pipelines that train a generator or distill a verifier.

a) Text-to-motion in human space

  • HumanML3D anchors human-space text-to-motion comparisons, but these models do not represent contact, balance, or actuation.Graphics tracking controllers address feasibility for simulated characters whose morphology matches the motion data; robots add retargeting and hardware.Source: §II · p. 2

b) Retargeting and tracking on humanoids

  • Humanoid trackers use inverse kinematics against SMPL keypoints, and SONIC is the unchanged G1 policy used in this paper.GMR is adopted for the retargeting ablation.Source: §II · p. 2

c) Language-to-humanoid systems

  • Language-to-humanoid systems train robot-space generators, language-conditioned policies, or physically aligned generators, while modular systems separate generation and tracking.Examples include UH-1, Humanoid-LLA, FRoM-W1, LangWBC, RoboGhost, RLPF, SafeFlow, TextOp, ECHO, and TEXEDO.Source: §II · p. 2
  • Unlike TEXEDO, which trains a G1-joint-space generator and ranks samples with a distilled verifier, this paper keeps the generator frozen and uses rollout physics directly.It reports the any-of-N ceiling and evaluates the complete HumanML3D test split and 177 real-robot prompts.Source: §II · p. 2

A. Overview and notation

The overview defines a pipeline that samples motion candidates for a prompt, retargets and simulates them, then selects by rollout outcome. Its frozen text-to-motion stage draws multiple joint-position sequences from MoMask.

A. Overview and notation

  • For prompt p, the generator yields N joint-trajectory candidates, which are mapped to robot references and rolled out under the deployment policy.The rollout produces an achieved trajectory, gate status, and tracking error; S3 selects by gate and error in lexicographic order.Source: A. Overview and notation · pp. 2–3
  • Table I groups recent language-to-humanoid systems by generation space, feasibility mechanism, and what is trained.Source: Table I · p. 2

B. Sample: frozen text-to-motion

  • MoMask uses its public HumanML3D checkpoints to generate 22-joint positions at 20 fps, with N samples drawn using default sampling settings and foot-contact postprocessing.Joints are rotated to Z-up and grounded, without scaling.Source: B. Sample: frozen text-to-motion · p. 3

C. Retarget: direction-matching IK

The retargeter maps human limb directions to robot-scaled targets and solves for G1 joint angles. The simulator then evaluates tracking and fall criteria, with a separate conservative gate for hardware export.

C. Retarget: direction-matching IK

  • Direction-matching IK retargets limb directions without importing human limb lengths, using damped least squares regularized toward the G1 stand and previous frame.The deterministic map costs about 15 CPU seconds per clip and reproduces limb orientation to 4.2° on 1,979 AMASS test sequences.Source: C. Retarget: direction-matching IK · p. 3

D. Simulate: the deployment policy as verifier

  • SONIC's released G1 checkpoint runs in MuJoCo with its own G1 model, PD actuation, and observation pipeline, at about three CPU-seconds per clip.The simulation reproduces its C++ deployment stack.Source: D. Simulate: the deployment policy as… · p. 3
  • A rollout is fallen if the pelvis drops below 0.25 m, tilts more than 60° from vertical, or sinks 0.30 m below its reference for over 0.5 s.The reference-tracking criterion also requires pelvis orientation within 45° of reference and mean joint error below 0.3 rad.Source: D. Simulate: the deployment policy as… · p. 3
  • The hardware gate rejects fallen rollouts, marks caution for an initial deviation above 0.6 rad or joint rate above 8 rad/s, and otherwise passes them.Export additionally requires pelvis height of at least 0.5 m throughout the clip.Source: D. Simulate: the deployment policy as… · p. 3

E. Select, and what to compare it with

S3 selects by deployment-policy gate and tracking error, while semantic fidelity and gate-screened real-G1 execution are assessed separately.

E. Select, and what to compare it with

F. Semantic fidelity of the executed motion

  • The HumanML3D text-motion evaluator scores generated joints, robot-projected references, and executed motion; real test motions provide a control for projection effects.Matching score measures embedding distance to the prompt, while R-precision tests whether it is nearest among 32 prompts.Source: F. Semantic fidelity of the executed… · p. 3

G. Hardware: batched, gate-screened sessions

  • Hardware clips are first tested on a fall-arresting gantry, then clean clips are re-executed without it; sessions of eight are risk-ordered and re-screened at export.Each clip uses a 3 s blend from stance and a 2 s blend back to stance.Source: G. Hardware: batched, gate-screened… · pp. 3–4

IV. EXPERIMENTS

The experiments use a stratified HumanML3D test-prompt set and also evaluate the pipeline across the usable complete test split.

IV. EXPERIMENTS

  • The study draws 200 captions from the HumanML3D test split, taking the shortest caption per motion, with captions 3–20 words long.The source split contains 4,384 motions and 12,584 whole-clip captions.Source: A. Prompt set and protocol · p. 4
  • Keyword rules stratify prompts into six categories: locomotion (60), turning (30), upper-body (50), squat/bend (30), ballistic (20), and other (10).The 128 prompts requiring non-upright motion are excluded; 31 low-pelvis prompts are labeled separately and remain scorable.Source: A. Prompt set and protocol · p. 4
  • All 200 prompts receive eight generated, retargeted, and simulated samples; the complete test split adds 4,184 usable prompts and 33,472 rollouts.Captions are used verbatim, and the categorization rules are auditable and released.Source: A. Prompt set and protocol · p. 4

B. Main result: the ceiling, and how far selection reaches

Physics-in-the-loop selection reaches the frozen generator’s any-of-eight ceiling, while kinematic ranking captures less and squat/bend remains constrained.

B. Main result: the ceiling, and how far selection reaches

  • With N=8, upright execution rises from 83.5% to 89.5% on 200 prompts and 80.5% to 89.5% on the full split; hardware-gate PASS clips rise from 33 to 85.The selected samples’ mean joint tracking error falls from 0.171 to 0.138 rad.Source: B. Main result: the ceiling, and how… · p. 4
  • TEXEDO’s trained generator reaches 97.0% first-sample execution and 93% squat/bend, where its references keep the pelvis higher (0.55 vs. 0.35 m).Its executed-motion semantic scores are indistinguishable from MoMask’s on the reported metrics.Source: B. Main result: the ceiling, and how… · p. 4

C. Why a kinematic verifier is not enough

  • Despite combined AUROC 0.90, kinematic ranking reaches only 85.0% and 56 PASS clips because within-prompt ordering differs from population fall classification.The physics rollout observes candidate outcomes directly.Source: C. Why a kinematic verifier is not… · pp. 4–5
  • Selection closes the gap in every behavior category except squat/bend, whose ceiling is 57%.Selected clips from all four illustrated categories ran on the real G1 and completed standing at 0.08–0.10 rad.Source: Figure 5 · p. 5, Figure 4 · p. 5

D. Where selection stops: the failure class

Selection leaves a concentrated residual class of pelvis-lowering motions, while semantic fidelity drops mainly during robot retargeting. The executed-motion scores approach a floor also seen with real mocap, implicating the robot projection and evaluator domain.

D. Where selection stops: the failure class

  • 50% to 57%: squat/bend remains the weakest category, while ballistic prompts rise from 75% to 95% after selection.On the complete split, 94% of 3,742 upright-applicable prompts pass, versus 53% of 442 low-pelvis prompts.Source: D. Where selection stops: the failure… · p. 5

E. Semantic fidelity: where meaning is lost

D. Where selection stops: the failure class

  • Among 21 unrecoverable prompts, 13 are squat/bend; 10 survive both retargeters—seven squat/bend, a rise from seated, and a stair climb with no stairs in the world—while TEXEDO’s retargeted-data training reaches 93% on that class.Source: D. Where selection stops: the failure… · p. 5

E. Semantic fidelity: where meaning is lost

  • Real mocap falls from R@1 0.26 to 0.07 to 0.04 through the same pipeline, supporting a floor tied to robot morphology, FK projection, and evaluator domain.The real-motion control covers 74 captioned clips, and the authors note that it is small.Source: E. Semantic fidelity: where meaning is… · p. 6

F. Retargeting ablation: direction IK vs. GMR

Neither direction-matching IK nor GMR dominates across categories, and their complementary failures let selection across both retargeters raise the any-of-8 ceiling.

F. Retargeting ablation: direction IK vs. GMR

  • GMR leads direction IK overall on first samples, 86.0% versus 83.5%, but the confidence intervals overlap and category strengths differ.Direction IK is better on locomotion (95.0 vs. 83.3%) and tracks more tightly where the robot stays up; GMR is better on squat/bend (66.7 vs. 50.0%) and ballistic (95.0 vs. 75.0%).Source: F. Retargeting ablation: direction IK… · p. 6
  • 95.0% [91.0, 97.3] (190/200): selecting across both retargeters raises the any-of-8 ceiling above either individual retargeter.GMR’s ceiling is 92.5% [88.0, 95.4].Source: Table V · p. 6, F. Retargeting ablation: direction IK… · p. 6
  • On squat/bend, the any-of-8 ceiling rises from 56.7% to 76.7% across both retargeters, leaving seven prompts unrecoverable under either.The seven remaining prompts are sits, kneels, and deep bends.Source: F. Retargeting ablation: direction IK… · p. 6

G. Hardware sessions

All 177 gate-selected clips completed standing on the real G1, and hardware tracking error closely matched simulation. The discussion bounds these results to the tested stack and to offline, scene-unaware free-space motion.

G. Hardware sessions

  • All 177 exported clips completed standing on the G1, with no falls or aborts; PASS clips succeeded 85/85 and CAUTION clips 92/92.Two kneeling clips below the 0.5 m export floor were excluded; 23 sessions were run over two days.Source: G. Hardware sessions · pp. 6–7, Figure 8 · p. 7
  • For 134 logged clips, mean joint tracking error was 0.114 rad on hardware versus 0.115 rad in simulation (r = 0.94).Maximum pelvis tilt separated unstable from stable clips with AUROC 0.94 in simulation and 0.92 on hardware.Source: Figure 9 · p. 7, G. Hardware sessions · p. 7

V. DISCUSSION

  • Retargeted-data training raises first-sample execution from 83.5% to 97% and closes squat/bend, while selection raises the frozen generator to 89.5%.Selection raises the trained generator by one point; training improves sample feasibility, while selection recovers feasible samples already present.Source: §V · p. 7
  • Keeping generation in human space supports transfer to other humanoids through a chain definition, failure attribution, and scoring on the human benchmark’s scale.These are stated properties of the paper’s pipeline, not a comparison of transfer results across robots.Source: §V · p. 7
  • The measured rates bound two generators, one retargeter family, one controller, and one robot, rather than the broader design space.Simulation uses SONIC’s MuJoCo model without system identification, so a simulated PASS is evidence about the deployment stack, not a guarantee.Source: §V · p. 7
  • The pipeline is offline and scene-unaware: chair, step, or object prompts are evaluated as free-space motion without modeled object interaction.No visual or other exteroceptive measurement enters the loop.Source: §V · p. 7

VI. CONCLUSION

Sample-simulate-select raises upright execution to the frozen generator’s any-of-8 ceiling, while leaving prompts that lower the pelvis infeasible. All 177 gate-selected clips completed standing on the real G1.

VI. CONCLUSION

  • Upright execution reaches 89.5% from 83.5% on 200 HumanML3D prompts and 89.5% from 80.5% on the complete test split.These are the frozen generator’s any-of-8 ceiling results.Source: §VI · p. 7
  • All 177 gate-selected clips completed standing on the real G1, with no training anywhere.Source: §VI · p. 8
  • Selection cannot fix prompts that lower the pelvis, a class that a generator trained on retargeted robot data does execute.Source: §VI · p. 8
Loading 2609.26420v1…