PaperlayerView exact revision on arXiv

Source-linked AI summary

METATRON: A Promptable Whole-Body Execution Layer for Humanoids

Chen Tessler, Yifeng Jiang, Alec Otto, Yi Shi, Umar Iqbal, Gal Chechik, Xue Bin Peng

arXiv:2610.38027v1cs.ROcs.AI

TL;DR

Humanoid controllers often need separate task-specific training despite sharing balance and whole-body coordination needs. Metatron combines a frozen generative policy that turns partial goals into coordinated motion with stateful programs that supply those goals. On G1, it supports multiple demonstrated behaviors and reports higher reaching success and push robustness than tested reference-tracking pipelines.

  • Problem

    Humanoid tasks commonly use separate learned controllers that duplicate balance and coordination capabilities, while underspecified prompts can admit multiple motions.

    Source: §I · p. 1, §IV · p. 3

  • Method

    Metatron distills motion supervision into a frozen generative whole-body policy, while stateful skills update partial-goal prompts to guide its behavior.

    Source: D. Training · p. 3, §IV · p. 4

  • Results

    Metatron reports higher whole-body reaching success and push robustness than the tested reference-tracking pipelines, and one frozen checkpoint per embodiment supports demonstrated tasks on physical G1 and simulated SOMA.

    Source: §I · p. 2

  • Takeaways & Limitations

    Within the learned repertoire, new tasks require goal-generating programs rather than policy retraining; skills specify goals, timing, and transitions while the frozen policy coordinates the body.

    Source: §VIII · p. 8

    The repertoire depends on training coverage, and task systems still select goals and handle long-horizon planning; autonomous approach followed by sitting is not demonstrated.

    Source: §VII · p. 8

Abstract

from arXiv · show

Humanoid controllers often require task-specific training despite sharing the need for balance and whole-body coordination. We present Metatron, a promptable whole-body execution layer combining a generative base policy with a programmable skill layer. The base policy receives partial goals, collectively called a prompt, that can combine timed body-part targets, velocity commands, and style cues. It produces diverse whole-body motions consistent with these goals. Training combines motion capture, synthetic recovery trajectories, and retargeted scene-interaction motions. After training, the policy is frozen. Stateful programs called skills express task intent as prompts at each control step, drawing rough geometric sketches that the policy turns into coordinated movement. Skills are ordinary code, refined by coding agents that analyze motion data and fit compact rules without task-specific policy training. On the physical Unitree G1, one frozen checkpoint supports walking, recovery, teleoperated object pickup, and sitting on furniture. Comparative evaluations show higher reaching success and push robustness than the tested reference-tracking pipelines. The same recipe produces climbing and acrobatics on a simulated character. Metatron enables coding agents to turn learned motor capabilities into new skills through code, without further policy training.

I. INTRODUCTION

Metatron separates task intent from whole-body coordination: a frozen policy responds to partial motion goals, while task-specific programs express those goals. It builds on motion-tracking and partial-goal controllers, and reports higher reaching success and push robustness than tested reference-tracking pipelines.

I. INTRODUCTION

  • The proposed recipe trains one general-purpose whole-body policy to coordinate the body and lets task-specific programs specify what motion should accomplish.Prompts are partial goals, such as an object’s shape and location for sitting or a style cue drawn from example poses.Source: §I · p. 1
  • Metatron reports higher whole-body reaching success and push robustness than the tested reference-tracking pipelines.One frozen checkpoint per embodiment supports demonstrated tasks on physical G1 and simulated SOMA.Source: §I · p. 2

II. RELATED WORK

  • Unlike reusable motion representations that may require downstream task-specific policy training, Metatron combines optional goal streams with explicit skill programs refined as ordinary code.Its related-work discussion also distinguishes a direct partial-goal interface from test-time optimization of generated motion.Source: §II · p. 2

III. METATRON: BASE POLICY

  • The base policy is distilled from frozen whole-body trackers trained on complete reference motions, then maps deployment-available observations and partial prompts to joint actions.Stateful skills construct prompts without retraining the frozen policy.Source: §III · p. 2

A. Dense Motion Tracker

  • Dense motion trackers learn whole-body coordination from complete reference states, with PPO optimizing tracking rewards and action-rate regularization.Their rewards train the trackers, not the distilled student.Source: A. Dense Motion Tracker · p. 2

B. Prompt Interface

The prompt interface lets skills specify only selected aspects of a motion, including timed body targets, pose goals, velocity, style, and scene geometry. A generative policy uses these partial goals and observation history to produce a motion completion, with training-only reference information excluded from deployment.

B. Prompt Interface

  • Masks mark which bodies, properties, and times are constrained, allowing other goal components to remain unspecified.The prompt combines optional goal streams with scene context.Source: B. Prompt Interface · p. 2
  • Spatio-temporal goals specify optional body positions or orientations in world coordinates together with a time-to-arrival.A prompt can request a timed hand target or a planar location while leaving height unconstrained.Source: B. Prompt Interface · p. 2
  • Canonicalized pose goals describe a desired pose independently of its world placement, with horizontal positions relative to the target torso and heading.Heights are measured above the target pose’s lowest body point; these goals are not offsets from the robot’s current pose.Source: B. Prompt Interface · pp. 2–3
  • Velocity goals specify planar travel velocity in the agent’s heading frame without specifying an endpoint.Style cues pool canonicalized example frames into one token, discarding temporal order and arrival times.Source: B. Prompt Interface · p. 3
  • Scene elements are represented as bounding boxes whose corner positions are observed in the agent’s heading frame.Deployment requires pre-measured dimensions and an object-pose estimate.Source: B. Prompt Interface · p. 3

C. Generative Policy

  • A partial prompt can admit many valid motions, so the policy uses a conditional VAE with a training-only posterior conditioned on the full reference and a deployable prior conditioned on observations and prompts.The stochastic latent selects a completion while a deterministic condition encodes observed context.Source: C. Generative Policy · p. 3
  • The prior transformer predicts a deterministic condition and latent distribution from sensor and prompt tokens, while a separate training-only posterior uses privileged reference information.The sampled latent is sphere-projected and fused with the condition before the decoder maps it to an action; a predictor and temporal consistency loss regularize training representations.Source: C. Generative Policy · p. 3, Figure 2 · p. 3

D. Training

METATRON distills whole-body coordination into a frozen policy trained from tracker rollouts, then uses stateful skills to express task intent as partial goals. Skills sketch motions with compact targets, and coding agents can refine them through simulation feedback without policy retraining.

D. Training

  • The student learns from mixed posterior-guided student and frozen-tracker rollouts, with tracker labels from complete references and randomly masked prompts as student inputs.The objective imitates the tracker while aligning prior and posterior distributions and regularizing the latent representation and action sequence; deployment retains only the prior, fusion module, and decoder.Source: D. Training · p. 3

IV. METATRON: SKILL LAYER

  • A skill is a stateful program that updates prompts at each control step, adding goals that guide the policy toward task behavior.Skills draw rough geometric sketches rather than planning precise trajectories; the policy coordinates balance, weight transfer, and joint actions.Source: §IV · p. 4
  • A fitted periodic foot-trajectory rule uses travel speed and facing angle to set swing amplitude, cadence, foot lift, and lateral widening.The resulting rule supports forward walking, side-stepping, and turning; cadence and step length both increase with speed.Source: §IV · p. 4
  • A human and coding agent define evaluation scenarios, then the agent iteratively refines skill code using simulation feedback.For visible artifacts, the agent can analyze motion clips, fit compact rules, score candidates across task variation, and iterate on failures.Source: Figure 3 · p. 4, §IV · p. 4
  • Furniture sitting uses three static targets derived from the seat bounding box, while standing raises and advances the torso target to elicit coordinated steps.No intermediate waypoints or prescribed stepping pattern are required.Source: §IV · p. 4
  • Recovery uses a hand-authored rule that requests a canonical upright pose in simulation but only torso height on G1.The target arrival time scales with the remaining height deficit, and separate entry and exit thresholds prevent repeated switching near the boundary.Source: §IV · p. 4, Figure 4 · p. 5

V. TRAINING DATA

METATRON training combines motion-capture data, scene interactions, and embodiment-specific synthetic recoveries, with retargeting designed to preserve scene contacts. On G1, one frozen policy supports multiple hardware demonstrations, including steering, object pickup, and sitting.

V. TRAINING DATA

  • Training supervision combines motion-capture data, scene interactions, and synthetic recoveries, with G1 data divided among four specialist trackers.Scene-interaction retargeting accounts for body–object contacts and furniture collisions; synthetic recoveries are generated for each embodiment.Source: §V · p. 5, A. Experimental Setup · p. 5

A. Experimental Setup

  • The G1 policy is distilled from four specialist trackers, while the SOMA policy distills a scene-aware tracker; skills use the resulting frozen policies without retraining.Reference-pipeline comparisons use a separate G1 policy trained without synthetic recovery data.Source: A. Experimental Setup · p. 5

B. Hardware Demonstrations

  • The physical G1 uses one frozen policy across control modes, with mode switches changing only goal observations and masks.Model weights and sensor processing remain unchanged.Source: B. Hardware Demonstrations · p. 5
  • Timed foot-placement goals make steering steps more decisive and improve walking quality, especially on hardware, compared with steering commands alone.Without these goals, foot swings are less committed and steps more hesitant; no policy retraining is required.Source: B. Hardware Demonstrations · p. 5
  • A PICO headset maps the user’s relative hand motions to robot hand targets, enabling teleoperated object pickup.The demonstrations include directing the robot to pick an apple.Source: B. Hardware Demonstrations · p. 5
  • A sitting skill uses the nearby sofa seat’s bounding box to generate torso and foot targets, allowing the G1 to sit and stand back up.The robot is positioned near the sofa before the skill runs.Source: B. Hardware Demonstrations · p. 5

C. Recovery and Compliance

METATRON supports responsive physical interactions and recovery from falls, and its frozen policy also executes dynamic behaviors in simulation. In the tested push comparison, it outperforms reference-tracking pipelines; the reaching evaluation distinguishes methods by reached rate rather than arm precision.

C. Recovery and Compliance

  • Sparse goals leave room for physical guidance, while synthetic recovery training supports G1 get-ups from diverse fallen poses, including configurations absent from the original mocap data.Recovery is triggered automatically when torso height falls below a safety threshold.Source: C. Recovery and Compliance · pp. 5–6, Figure 7 · p. 7

D. Simulation Capabilities

  • In simulated G1, dense body targets produce a front roll, while unordered style frames guide crawling without imposing their original timing.The dynamic skills are demonstrated in MuJoCo to avoid potential damage to the physical G1 and its three-finger hands.Source: D. Simulation Capabilities · p. 6
  • On simulated SOMA, a skill uses velocity, heading, and terrain-relative pelvis-height goals for locomotion; climbing leaves hand and foot placement unconstrained.The frozen policy executes the ledge fall, recovery, and climb without reference trajectories or full-body landing and recovery poses.Source: D. Simulation Capabilities · pp. 6–7

E. Comparison with Reference-Tracking Pipelines

  • Across the 54-target wrist-reaching grid, methods achieve comparable arm precision, while reached rate separates them, especially for high counterbalance and low hip-fold targets.Targets span three distances, three heights, and both hands; success requires holding the wrist within 10 cm while upright.Source: E. Comparison with Reference-Tracking… · p. 7, Table I · p. 7
  • Both reference-tracking pipelines have zero success above 600 N, while METATRON succeeds in 75% of trials at 800 and 1000 N.The test applies a 100 ms torso force pulse during a 1 m/s forward walk; the comparison measures complete pipelines rather than isolating a cause.Source: Table II · p. 7, E. Comparison with Reference-Tracking… · p. 7

F. Ablation Study

The ablation study finds higher observed push success for the full model than key ablations and adapted baselines, while the limitations and conclusion delimit the demonstrated scope. Results remain weak under strong perturbations, and new tasks still depend on externally supplied goals and learned motion coverage.

F. Ablation Study

  • The full model exceeds the recovery-data and auxiliary-loss ablations at every tested force; versus Global only, it leads at 0–200 N but not at higher forces.Against adapted MaskedMimic+ and HOVER+, it has higher observed success at every nonzero force, while MaskedMimic+ leads slightly without pushes; all methods fall below 10% at 600–800 N.Source: F. Ablation Study · p. 8, Table III · p. 8

VII. LIMITATIONS

  • METATRON’s repertoire depends on training coverage, and underspecified goals or incidental style cues can produce unintended motions.An abrupt 2 m/s command from rest often triggers a front roll; additional motion capture and targeted synthetic trajectories could broaden coverage, while refining skill-generated goals can improve selection without retraining.Source: §VII · p. 8
  • The task system remains responsible for goal selection and long-horizon planning; autonomous approach followed by sitting is not demonstrated.For the sitting demonstration, the robot is placed near the sofa and given the seat bounding box as an input condition.Source: §VII · p. 8
  • The hardware study shows diverse behaviors with one frozen controller, but further trials are needed to quantify recovery reliability, gait stability, contact safety, actuation limits, and failure modes.The reported gait improvements are qualitative, and recovery is demonstrated only from tested configurations.Source: §VII · p. 8

VIII. CONCLUSION

  • METATRON turns dense motion supervision into a generative whole-body controller for partial goals; within its learned repertoire, new tasks use goal-generating programs rather than policy retraining.Skills specify goals, timing, and transitions, while the frozen policy coordinates the body.Source: §VIII · p. 8

APPENDIX I IMPLEMENTATION DETAILS

The appendix describes a three-stage pipeline from PPO-trained motion trackers through supervised prompt-conditioned distillation to runtime skills. It details training groups, data, expert configurations, and how the dense trackers’ rewards and optimization differ from student distillation.

APPENDIX I IMPLEMENTATION DETAILS

  • The implementation separates PPO training of motion-tracking experts, supervised distillation of a prompt-conditioned student, and runtime skill construction of prompts for the frozen student.G1 distills four frozen experts—BONES, AMASS+LAFAN, recovery, and SAMP—with relative training weights of 0.6, 0.2, 0.1, and 0.1.Source: Appendix I · p. 10, A. Training Setup · p. 10

A. Training Setup

  • The student uses BONES and synthetic recovery data, with AMASS+LAFAN and SAMP added for G1; SOMA training includes wall-climbing and ground-sliding scene motions.G1 operates at 50 Hz control and SOMA at 30 Hz; runtime skills use the trained student without further weight updates. Student training uses 20 GPUs, while ablations use eight A100 GPUs for 5,000 iterations with matched parameter counts, excluding expert training.Source: A. Training Setup · p. 10

B. Dense Reference Trackers

  • Dense trackers learn from complete reference motions and privileged state; the SOMA teacher includes climbing and sliding with terrain context, while G1 SAMP supplies scene interactions.The G1 BONES expert uses finite scalar quantization and predicts 29 joint commands; the SOMA teacher has 66 action outputs. Their expert PPO settings are distinct from student distillation.Source: B. Dense Reference Trackers · p. 10, Table IV · p. 10
  • Tracker rewards combine body tracking terms and action-rate regularization, with SOMA using speed-gated precision rewards that the G1 BONES expert does not use.The tracker rewards are distinct from the student’s supervised losses; the G1 BONES expert also scales its rewards by 0.8 before PPO optimization.Source: B. Dense Reference Trackers · p. 10

C. Masked On-Policy Distillation

Masked on-policy distillation trains a prompt-conditioned student from frozen trackers, with a prior path for deployment and a posterior-guided path used during training. The method combines imitation, latent regularization, and temporal objectives while supervising states visited under student and expert control.

C. Masked On-Policy Distillation

  • The student’s prior encodes prompt streams and sensor history into tokens, while fusion and an action decoder combine the condition, latent representation, current observations, and previous action.G1 reconstructs body poses with forward kinematics and outputs 29 commands; SOMA uses simulator body states and a height map and outputs 66 commands.Source: C. Masked On-Policy Distillation · pp. 10–11
  • Training-only posterior and dynamics networks support distillation, while deployment uses the prior, fusion network, and action decoder.The posterior receives full reference and prompt features; the dynamics predictor maps the posterior representation to the next condition.Source: C. Masked On-Policy Distillation · pp. 11–12
  • The latent samples are projected onto the unit sphere before fusion, and SOMA additionally trains for latent diversity by encouraging distinct latents to produce different actions.G1 omits the latent-diversity loss.Source: C. Masked On-Policy Distillation · pp. 11–12
  • Each rollout assigns control to the posterior-guided student with probability 0.9 or the frozen tracker otherwise, while the tracker labels visited states under either controller.The selected controller acts for the whole episode, and deployment actions use the prior path.Source: C. Masked On-Policy Distillation · pp. 11–12
  • Students use supervised losses: imitation averages squared student–expert action differences over action coordinates (λact = 5 × 10^-2), while dynamics and temporal-loss weights are each 10^-3.Temporal losses compare consecutive steps within the same episode and exclude pairs crossing a reset.Source: C. Masked On-Policy Distillation · p. 11

D. Prompt and Style-Cue Sampling

Prompt sampling varies the amount and timing of body-target information, independently samples several goal streams, and treats scene context separately. Style cues encode example-motion character without arrival times or global placement.

D. Prompt and Style-Cue Sampling

  • Body-target streams use five future slots spaced 0.1–2.0 seconds apart, with dense, intermediate, or sparse schedules sampled at probabilities 0.2, 0.6, and 0.2.Masks select body identities and target properties; position constraints can separately specify planar and vertical components, and orientation can be heading-only or a full rotation.Source: D. Prompt and Style-Cue Sampling · p. 12
  • World-target, canonicalized-target, planar-velocity, and style streams are independently active with probability 0.5, making all 16 presence combinations equally likely, including no motion goal.Style cues sample up to five frames from a two-second sliding window; the visible-frame count is zero in half of episodes.Source: D. Prompt and Style-Cue Sampling · p. 12
  • Scene context is separate from the four prompt-stream masks: G1 accepts one object represented by up to two boxes, while SOMA receives scene geometry through a local height map.Valid G1 boxes remain visible for SAMP; unused boxes and boxes in scene-free groups are masked out, allowing the same student to operate with or without furniture.Source: D. Prompt and Style-Cue Sampling · p. 12
  • Style frames are order-invariant example-based cues, with no arrival times or global placement, distinguishing them from timed body targets.A shared frame encoder and masked mean pooling form the style representation.Source: D. Prompt and Style-Cue Sampling · p. 12

E. Observation Construction

Observation construction combines embodiment-specific state inputs with body-pose features for comparing current states and prompted goals. Targets use either a local canonicalized pose convention or world-frame differences that require localization registration.

E. Observation Construction

  • The G1 reconstructs torso orientation from pelvis IMU and joint measurements, then uses forward kinematics to obtain body poses in the heading frame.SOMA instead supplies body positions, orientations, velocities, root height, and a local 16×16 height map.Source: E. Observation Construction · p. 12
  • Each prompted body receives nine pose features; the local convention sets the lowest body point to zero height but does not estimate ground height.The prompted set includes torso, hands, and feet, with rotations represented in 6D.Source: E. Observation Construction · p. 12
  • The history encoder uses five sensor steps with elapsed times, while training perturbs joint and IMU measurements but leaves localization, base velocity, and ground clearance unnoised.The previous action also enters the current-state encoder and action decoder.Source: E. Observation Construction · p. 12
  • G1 training randomizes action offsets, material properties, torso center of mass, hand masses, and periodic velocity pushes; SOMA disables domain randomization.G1 training adds no sensor or action latency, and its velocity perturbations differ from evaluation force pulses.Source: E. Observation Construction · p. 12
  • Canonicalized targets use each target pose’s own torso position and heading, so their values depend on the requested pose rather than the robot’s current state.Training canonicalizes the complete target pose before masking; runtime skills use the same convention.Source: E. Observation Construction · pp. 12–13
  • World-frame position differences require the target and robot poses to share an origin; planar goals need x–y localization, while absolute-height goals also need vertical registration.Subtracting the same floor offset from both poses leaves their difference unchanged.Source: E. Observation Construction · p. 13

F. Recovery-Data Generation

Recovery clips are generated offline from settled randomized poses, filtered by endpoint tracking error, and added to the motion library for the shared training stages. Harness skills then construct prompts as stateful code, including steering and height-triggered recovery without training a new motor policy.

F. Recovery-Data Generation

  • The recovery-data generator extends HIL with MaskedMimic-style spatio-temporal conditioning to produce five-second clips offline in 1024 parallel environments.It masks body constraints and future times while generating recoveries from randomized ground poses.Source: F. Recovery-Data Generation · p. 13
  • Rollouts begin only after sampled root and joint targets settle for 150 physics steps with velocities zeroed.This lets the sampled poses settle under contact dynamics before recording begins.Source: F. Recovery-Data Generation · p. 13
  • A recovery clip is accepted only if it avoids reset and ends below 0.25 m mean position error, a filter that does not test sustained balance, contact quality, or hardware safety.Each rollout receives a sparse timed source-motion target with at least one position-constrained body; termination conditions are disabled during collection.Source: F. Recovery-Data Generation · p. 13
  • Accepted clips store body and joint poses, velocities, and contacts, then join captured motions in the same library and supervise the same tracking and distillation stages.Only initial root x–y translation is removed; the generator itself is not part of the deployed controller.Source: F. Recovery-Data Generation · p. 13

G. Harness Skills

  • Harness skills are stateful functions or finite-state machines that construct prompts from task inputs, and none requires training a new motor policy.A harness runs the skills and manages transitions; each embodiment uses one frozen policy across supported skills.Source: G. Harness Skills · p. 13
  • The steering skill plans timed foot targets in world coordinates, while at speeds at or below 0.05 m/s it substitutes stationary-support targets.It evaluates targets at 0.1–0.5 s arrival times; the learned policy determines joint commands and physical contacts.Source: G. Harness Skills · p. 13
  • Simulation recovery suspends the task prompt below 0.65H and restores it above 0.9H, requesting canonicalized upright poses across future slots.On exit, the skill restores the suspended task prompt; only the prompt changes between task and recovery states.Source: G. Harness Skills · p. 13
  • G1 recovery requests calibrated upright torso height alone, with recovery thresholds adjusted for headset-controlled crouching; dense tracking and unconditional generation disable automatic recovery.The skill begins below 0.65H and ends above 0.90H; during headset crouching, entry is based on commanded torso height with a 0.65 m lower bound.Source: G. Harness Skills · p. 13

H. Deployment on the G1

The G1 deployment stack runs a frozen policy with a fixed joint-control interface and uses runtime commands from an Xbox controller or PICO headset. Evaluation separates reference reconstruction from prior-policy task execution and matches conditions for task comparisons.

H. Deployment on the G1

  • Sensor acquisition to joint-command transmission takes 1.3 ms on the TensorRT/CUDA-graph laptop stack; this is processing latency, not the control period.The hardware is equipped with an NVIDIA RTX 5090 GPU.Source: H. Deployment on the G1 · p. 13
  • Hardware demonstrations use the deterministic prior mean with sampling noise set to zero, although stochastic sampling is supported but not used for reported demonstrations.The mean is projected onto the unit sphere and decoded together with the time elapsed since measurement.Source: H. Deployment on the G1 · p. 14
  • The G1 policy updates nominal-pose-relative joint offsets every 20 ms, and deployment must preserve joint order, offsets, scales, and gains.World-position skills additionally consume localization; canonicalized skills use the local pose convention.Source: H. Deployment on the G1 · p. 14
  • Sitting demonstrations provide a nearby-object cue externally, so the setup does not estimate object location or autonomously plan an approach.Control-mode selection changes goal-related observations and masks without changing policy weights or the joint-control interface.Source: H. Deployment on the G1 · p. 14

I. Evaluation Protocol

  • Evaluation uses the posterior with privileged reference information for reconstruction but the prior for task and hardware tests; task comparisons match initial states, commands, perturbations, horizons, and success criteria.The reference-tracking comparison excludes synthetic recovery data, whereas the four-expert policy includes recovery training.Source: §I · p. 14

Related papers

Loading 2610.38027v1…