PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: Typed Reference-Aware Conversational Editing for Humanoid Motion Diffusion

Zejian Kang, Yifu Luo, Boyang Shen, Jingxuan Li, Zekai Chen, Zekun Shi, Yongxue Xu, Zhenyu Hou, Bingqian Wu, Shaolei Zhang, Wentao Yang, Xiangru Huang

arXiv:2610.21515v1cs.RO

TL;DR

Language-conditioned motion generation must turn underspecified instructions into concrete structure, and editing must relate requested changes to a current motion while preserving the rest. TRACE uses motion-grounded typed events for generation and scoped, source-relative goals for editing. The reported evaluations show stronger generation results and more reliable successive edits than the compared methods under shared preservation constraints.

  • Problem

    A caption may not determine when support occurs or how high a hand rises, and relative edits must use the current motion as a baseline while preserving the rest.

    Source: §1 · p. 2

  • Method

    TRACE predicts typed support, root, and hand events from motion-derived supervision for diffusion generation, and uses a separate reference-conditioned editor with scoped source-relative goals and projection.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    TRACE leads compared methods in retrieval, MM-Dist, FID, and human ratings, and achieves higher all-turn editing success than adapted learned editors under shared scopes and reference projection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support typed motion quantities as a shared basis for language-conditioned generation and measurable revisions of an evolving reference.

    Source: §6 · p. 10

    The system addresses text instructions and flat-ground kinematics without scene perception, and its contact objectives do not guarantee zero slip.

    Source: §6 · p. 10

Abstract

from arXiv · show

Language-conditioned humanoid motion generation enables users to create and refine robot movements through natural language. The central challenge is to connect an underspecified request to concrete motion structure that can subsequently be revised relative to an accepted reference. We present TRACE, a generation and editing framework for the Unitree G1 humanoid robot built around typed, measurable kinematic quantities. Our insight is to ground generation conditions in paired training motions and editing goals in the current motion. A language model learns to predict support, root, and hand events from motion-derived supervision; their time-aligned encodings complement the original text in a diffusion generator. Gravity-referenced states and trajectory--contact objectives improve kinematic consistency. For editing, typed requests define body--time scopes and source-relative goals. A separate reference-conditioned diffusion editor uses these goals and revised events to realize changes, while sampling-time projection preserves protected state coordinates. Quantitative evaluation and human assessment show improved semantic alignment and motion quality over the compared methods. Controlled ablations support the value of motion-grounded conditioning and kinematic consistency, while multi-turn editing evaluations demonstrate more reliable execution of successive relative requests under shared preservation constraints. Together, these results support typed kinematic structure as a common basis for motion generation and reference-aware revision.

1 Introduction

TRACE connects language-conditioned generation to selective revision by grounding motion plans in measurable events and editing goals in the current reference. Its evaluations report stronger generation metrics and human ratings, alongside more reliable successive edits under shared preservation constraints.

1 Introduction

  • The motivating challenge is turning underspecified language into a coherent motion whose intended properties can be changed selectively while retaining the rest as a reference.A paired demonstration grounds generation targets, while the current reference supplies editing baselines and protected coordinates.Source: §1 · p. 2
  • TRACE derives typed support, root, and hand event targets from training motions and combines their time-aligned conditions with caption features in a diffusion generator.Gravity-referenced states and trajectory and support-selective contact objectives constrain kinematic consistency.Source: §1 · p. 2
  • For editing, typed requests compile into body–time scopes and source-relative goals; a separate diffusion editor uses revised events while projection restores protected coordinates.Each edited output becomes the reference for the next request.Source: §1 · p. 2
  • TRACE leads compared methods in retrieval, MM-Dist, FID, and human ratings, and achieves higher all-turn success than adapted learned editors under shared scopes and reference projection.Successive hand-height changes are evaluated against an evolving motion reference.Source: §1 · pp. 2–3
  • Figure 1 illustrates a jumping-jack refinement in which a deeper preparatory crouch accompanies a higher jump while the overall action is retained.The height profiles compare initial and edited motions over normalized time.Source: §1 · p. 2, Figure 1 · p. 3

2 Related Work

Prior work covers text-to-motion generation, structured controls, kinematic consistency, and source-conditioned editing. TRACE distinguishes its approach by grounding typed events in paired motions and compiling relative edits into scoped, reference-preserving goals.

2 Related Work

  • Text-to-motion research spans discrete codes, continuous trajectories, structured body-part and temporal conditions, and language-model planning; TRACE grounds event conditions in the paired training motion.Its planner predicts support, root, and hand events from text for inference.Source: §2 · p. 3
  • Prior kinematic and humanoid-generation methods address plausibility or downstream execution at different stages, while TRACE targets upstream reference quality with gravity-referenced states and trajectory and contact objectives.Its contact losses are support-selective and based on forward kinematics.Source: §2 · p. 3
  • Earlier editing work includes inpainting, compositional control, and source-conditioned editors; TRACE compiles typed relative changes into body–time scopes and source-relative goals with reference projection.The editor also uses revised source events.Source: §2 · p. 3

3 Method

TRACE uses typed motion events to condition G1 generation and source-relative goals to edit accepted motions while protecting unselected coordinates. Its separate generator and editor share a motion representation and kinematic vocabulary.

3.1 Overview

  • Typed event plans structure generation, while body–time scopes, source-relative goals, and reference projection let a separate editor revise selected regions and preserve protected coordinates.Each edited output becomes the reference for the next request.Source: §3.1 · p. 4, §3.4 · pp. 5–6

3.2 Motion-Grounded Sparse Planning and Conditioning

  • Motion-derived support, root, and hand events provide realization-aligned supervision; predicted events complement caption semantics as time-aligned numerical conditions for diffusion.Training mixes target and predicted conditions, falling back to predicted events for uncertain targets; inference uses predicted events only.Source: §3.2 · pp. 4–5

3.3 Gravity-Gauged and Contact-Consistent Diffusion

  • Gravity-referenced height and tilt, integrated-trajectory objectives, and support-selective contact penalties target accumulated root-motion error and foot sliding.Support masks come from clean training motion; predicted plan probabilities condition the denoiser but do not replace those masks.Source: §3.3 · p. 5

3.4 Automatic Masking and Reference-Aware Motion Editing

  • A frozen LLM and deterministic compiler localize each request to a body–time mask and compile signed wrist-height offsets relative to the source motion.The editor conditions on source features, scope, offsets, and revised source events; sampling projects protected normalized coordinates back to the reference.Source: §3.4 · p. 6

4 Motion Generation Experiments

On BONES-SEED, TRACE leads the compared methods on retrieval, MM-Dist, and FID, with user-study evidence of perceptual quality and text alignment. Matched controls assess event conditioning and motion-derived supervision, while kinematic ablations examine trajectory and contact errors.

4.1 Experimental Setup

  • The BONES-SEED evaluation uses 13,302 test text–motion pairs and compares TRACE with retrained human-motion methods and native G1 methods on a shared test set.The frozen evaluator reports R-Precision, MM-Dist, FID, and Diversity; physical errors are assessed only in matched component controls.Source: §4.1 · p. 6

4.2 Generation Performance Analysis

  • TRACE achieves the best retrieval, MM-Dist, and FID results, while the user study provides complementary evidence of perceptual quality and text alignment.Table 1 reports automatic metrics on test motions and user-study scores for 100 test prompts.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Ablation Study

  • Matched controls assess aligned event inputs against text-only inputs and motion-derived against caption-derived training targets, with predicted Plans retained at test time.The aligned Plan checkpoint is the main-table model; text-only retains branch capacity and raw-caption features while zeroing event inputs.Source: §4.3 · p. 8
  • Kinematic controls show complementary effects: Gravity-Gauged reduces contact-height error, Geometry improves trajectory diagnostics but increases penetration, and Contact reduces penetration.The Contact model has lower errors than Heading-local across all reported measures.Source: §4.3 · p. 8

5 Motion Editing Experiments

The experiments evaluate scope localization and successive hand-height edits under shared preservation constraints. TRACE matches prescribed masks on reviewed requests and leads the learned editor adaptations in multi-turn success.

5.1 Experimental Setup

  • Hand-height editing tests source-relative goals by measuring goal achievement and preservation without requiring a unique target motion.The controlled request specifies a measurable FK height goal while allowing multiple valid joint trajectories.Source: §5.1 · p. 8
  • The study uses 199 held-out G1 source motions and three four-turn patterns per source, totaling 597 chains.Requests specify a hand, a signed 5 cm height change, and an explicit time interval; all-turn success requires change and preservation at every turn.Source: §5.1 · p. 9

5.3 Multi-Turn Editing Analysis

  • At four turns, all-turn success is 30.82% for TRACE, versus 6.48% for TMED-G1 and 2.74% for SimMotionEdit-G1.All 597 chains remain in the denominator, with each request evaluated relative to the method’s preceding output.Source: Figure 7 · p. 9, §5.3 · p. 9

5.2 Scope-Construction Diagnostics

  • The full scope-construction pipeline matches prescribed masks on all 199 reviewed requests.Removing components yields overly broad whole-body or full-duration masks, or wrist-only masks that omit proximal arm coordinates.Source: §5.2 · p. 9, Figure 6 · p. 9

6 Conclusion and Limitations

TRACE combines typed motion-grounded events, gravity-referenced diffusion, and source-relative editing for Unitree G1 motion. The conclusion reports reliable successive hand-height changes under shared scopes and projection, while identifying limits in scene and contact assumptions.

6 Conclusion and Limitations

  • Under shared scopes and reference projection, TRACE sustains successive hand-height changes more reliably than the adapted learned editors.The conclusion also reports strong semantic alignment and perceptual quality, with matched controls supporting temporal event conditioning, motion-grounded supervision, and kinematic consistency.Source: §6 · pp. 9–10
  • The system addresses text instructions and flat-ground kinematics without scene perception, and contact objectives do not guarantee zero slip.Retrieval and distributional metrics assess generated motions rather than exact adherence to each predicted event.Source: §6 · p. 10

A Real-Robot Qualitative Demonstrations

B.1 Motion-Derived Sparse Plan Construction

  • Motion-derived plans use clean-motion support, root, and palm signals to form confidence-weighted 16-channel targets on a four-frame patch grid.The plan is sparse in vocabulary and dense in time, with motion length supplied separately.Source: §B.1 · p. 14
  • Support labels use hysteresis and mask uncertain frames; root and hand categories use smoothed signals with threshold-near frames marked uncertain.Contact runs shorter than three frames are confidence-masked, and lack of stable support is not labeled airborne.Source: §B.1 · p. 14
  • Each patch pools up to four valid frames, with its target formed from confident-frame histograms and confidence set by their fraction.Patches with no confident frame receive zero weight, not a negative event; the padded grid has 128 patches.Source: §B.1 · p. 14
  • Training labels supervise a frozen LLM planner, while diffusion training mixes target and predicted conditions using the confidence fallback.At test time, conditions use captions and predicted durations.Source: §B.1 · p. 14

B.2 Main-Model Matched Plan Controls

Matched controls show that patchwise event conditions improve semantic criteria over event-free and globally averaged controls. The section also specifies the gravity-gauged state and geometric and contact objectives used to support kinematic consistency.

B.2 Main-Model Matched Plan Controls

  • Aligned Plan improves the displayed semantic criteria over both Text Only and Global Events controls.The controls share the same training budget and other listed components; the comparison isolates event content and timing.Source: §B.2 · p. 15
  • Motion-derived event targets improve all reported retrieval and distributional criteria over caption-derived targets under a fixed inference planner.Caption-derived events can disagree with the paired diffusion target because captions admit multiple valid support, root, and hand trajectories.Source: §B.2 · p. 15

C.1 Gravity-Gauged State: Encoding and Decoding

  • The gravity-gauged state retains height, tilt, yaw and planar increments, and 29 joint angles while omitting initial global planar position and yaw.Recovering the original global frame requires the initial planar translation and yaw as external metadata.Source: §C.1 · pp. 15–16

C.2 Generation Objective Groups and Contact Implementation

  • Generation objectives combine motion reconstruction with geometric consistency losses on integrated position and yaw, body-up direction, and yaw-pair norm.Geometric inputs are decoded and integrated from zero initial planar position and yaw, without rigid alignment or target headings for predicted translations.Source: §C.2 · p. 17
  • Contact objectives penalize support-foot sliding, drift, and height error, plus ground penetration across valid frames.Clean motion selects support masks; predicted Plan probabilities condition the denoiser but do not select those masks.Source: §C.2 · pp. 17–18

C.3 Motion-Consistency Ablation Protocol

The ablation protocol incrementally adds geometry and contact objectives to a gravity-gauged core. The section also describes schema-based language interfaces and the source-relative scope used for supported editing requests.

C.3 Motion-Consistency Ablation Protocol

  • Geometry increases penetration before the Contact bundle reduces it.The ablation retains reconstruction, velocity, and acceleration losses across variants; Contact adds slip, anchor, height, and penetration terms.Source: §C.3 · p. 18

D.1 Language Interfaces and Qualitative Whole-Body Example

  • The generation adapter emits duration and typed event intervals, while the editing adapter compiles an attribute, effector, signed change, and interval into a body–time scope and source-relative goal.The interface outputs are schema-validated before consumption by a motion model.Source: §D.1 · p. 19, Table 7 · p. 19
  • Figure 1 qualitatively shows a jumping-jack refinement with a deeper preparatory crouch and higher jump, outside the hand-height benchmark.The quantitative editing benchmark instead starts from held-out G1 sources and measures successive edits from the editor’s preceding output.Source: §D.1 · p. 19

D.2 Markov Editing State and Relative Requests

  • For the supported hand-height requests, the compiler applies signed offsets to the selected arm within the requested interval and protects the remaining coordinates.Offsets are bounded by 0 < |d| ≤ 0.10 m; negated, ambiguous, or unsupported expressions leave the source unchanged.Source: §D.2 · p. 20

D.3 Reference-Conditioned Editor: Architecture and Training

TRACE trains a separate source-conditioned editor for local relative goals, with conditional denoising and reference projection at inference. Its training combines retargeted MotionFix pairs with locally constructed offset examples.

D.3 Reference-Conditioned Editor: Architecture and Training

  • The editor is trained on retargeted MotionFix pairs and additional local examples covering both hands, signed offsets, and temporal windows.The local targets are constructed with source-only bounded inverse kinematics; inference does not use per-request IK optimization or post-hoc smoothing.Source: §D.3 · p. 20
  • The independent editor uses a 40D state, four-frame patches, a source encoder, and separate projections for scope and two-wrist offset patches.Text queries attend to source patches; the editor’s weights and normalization are independent of the main-table generator.Source: §D.3 · p. 20
  • Local training combines edit reconstruction and velocity penalties with wrist-FK and source-offset goal errors.The wrist-FK squared error is normalized by (0.05 m)^2, as are the goal’s vertical and mean horizontal wrist errors.Source: §D.3 · p. 20
  • The final training stage adds a source-motion event-channel adapter and revises only requested hand-height groups by the relative offset.The residual is attenuated during the last quarter of reverse denoising.Source: §D.3 · p. 20

D.4 Automatic Localization and Sequential-Editing Protocols

Across the evaluated four-turn chains, TRACE outperforms two learned G1 adaptations in success and reported quality diagnostics. The protocol evaluates mask localization and successive edits on held-out sources, with shared projection and no failure exclusion.

D.4 Automatic Localization and Sequential-Editing Protocols

  • Mask-localization interventions remove body selection, temporal localization, or arm-chain expansion without retraining or threshold tuning.The interventions open all coordinates in the predicted interval, extend the selected arm over the sequence, or retain only wrist coordinates, respectively.Source: §D.4 · p. 21
  • The four-turn protocol evaluates three fixed patterns of same-arm, alternating-arm, and reversal requests, using each method’s preceding generated motion.All methods receive the same instructions, intervals, compiler, and per-step reference projection; no chain is removed using solver-derived feasibility.Source: §D.4 · p. 21
  • 26.07–35.62% four-turn success for TRACE, versus 4.13–9.10% for TMED-G1 and 1.23–4.47% for SimMotionEdit-G1, using 95% source-cluster intervals.Protected-coordinate error is zero for all methods under shared projection, so success differences concern requested changes.Source: §D.4 · p. 21

D.5 Current-Cohort Editing Quality

  • TRACE improves four-turn success and reported quality diagnostics over the two learned adaptations on the same 199 sources and 597 chains.Each turn uses the method’s actual preceding output, without private targets, feasibility filtering, or test-specific tuning.Source: §D.5 · p. 21, Table 9 · p. 22

E.1 BONES-SEED Data Processing

BONES-SEED is filtered to a predominantly ground-supported motion subset, then organized into leakage-checked splits and motion-derived Plan supervision. The evaluation retains source-derived segmentation, so it is not an unconstrained full-clip duration benchmark.

E.1 BONES-SEED Data Processing

  • The semantic filter leaves 55,459 originals, retaining ordinary locomotion, gestures, squats, dance, and small jumps while excluding difficult acrobatics and externally supported actions.The preceding kinematic and metadata filters reduce 61,292 standing originals to 58,574 before semantic filtering.Source: §E.1 · p. 22
  • The dataset restores supplied mirrors and retains existing captions and temporal annotations, without synthesizing either mirrors or captions.After deduplication, generation uses the first nonempty caption in a fixed order; semantic evaluation uses the corresponding first natural description.Source: §E.1 · p. 22
  • Grouped splitting keeps connected motion families, mirrors, pairs, and normalized exact captions intact; the audit finds zero cross-split overlap on those grouping keys.The resulting 2,557 components are assigned with target proportions 76/12/12; the split is not actor-disjoint and does not guarantee semantic paraphrase separation.Source: §E.1 · p. 22
  • The 120 FPS sources are sampled every sixth frame to produce 20 FPS trajectories, represented as 40D Gravity-Gauged states using training-only normalization statistics.No interpolation, smoothing, or IK is applied during resampling.Source: §E.1 · pp. 22–23
  • Clips are divided into nonoverlapping windows of at most 512 frames, and sparse-event Plan targets are derived from clean G1 motion rather than provider prose phase labels.Test conditioning uses predicted events and durations, never target event labels.Source: §E.1 · p. 23
  • Evaluation reassembles 13,706 eligible test windows into 13,302 full-motion IDs, but source-derived segmentation means predicted durations are not an unconstrained full-clip duration benchmark.Predictions are neither padded nor truncated to ground-truth lengths, and excluded short tails are not filled with ground-truth frames.Source: §E.1 · p. 23

E.2 Training and Evaluation Configuration

The configuration combines a joint LLM event planner with a diffusion generator and a separate source-conditioned editor. It specifies semantic, kinematic, and edit-preservation evaluation procedures, alongside blinded human ratings.

E.2 Training and Evaluation Configuration

  • The joint LLM planner predicts duration and typed support, root, and hand events, while the generator trains on all 86,385 segments with a 50/50 condition mixture.Most predicted training conditions are in-sample; test inputs contain no target events or durations, and invalid Plans use target-free fallback values.Source: §E.2 · p. 23, Table 11 · p. 23

E.3 Inference Procedure

  • At inference, generation uses the main network and text-derived events, whereas editing uses an independent editor conditioned on source features and repeatedly projects protected coordinates toward the reference.If editing parsing or compilation fails, the procedure returns the source motion.Source: §E.3 · p. 24

E.4.1 Semantic and Distributional Metrics

  • Semantic evaluation uses motion-to-text retrieval among 32 caption candidates, paired-caption MM-Dist, full-set FID, and a diversity diagnostic that is not necessarily better when higher.R-Precision averages 20 candidate samplings; FID is computed once per complete set.Source: §E.4.1 · pp. 24–25

E.4.2 Kinematic Metrics

  • Kinematic diagnostics evaluate the common predicted/ground-truth prefix without time warping, and GT-contact masks can penalize valid generated swings during reference support.These metrics do not isolate foot slip under the generated motion’s own contacts; the four errors are normalized separately, with no aggregate score.Source: §E.4.2 · p. 25

E.4.3 Editing and Human Ratings

  • Edit success requires the requested wrist-height profile and preservation thresholds to pass jointly, with every request and parse failure retained in the denominator.The human study separately rates motion Quality and Text match using visible three-point anchors and reports their equal-weight mean as Overall.Source: §E.4.3 · pp. 25–26
Loading 2610.21515v1…