PaperlayerView exact revision on arXiv

Source-linked AI summary

TopX: Text-to-Motion for Unseen Skeletons from the Skeleton File Alone

Tengjiao Sun, Pengcheng Fang, Xiaoyu Zhan, Dongjie Fu, Xiaohao Cai, Hansung Kim

arXiv:2610.34321v2cs.CV

TL;DR

Text-to-motion models generally assume a fixed training skeleton, but new characters may arrive with only a skeleton file and no motion clips. TopX generates motion from the skeleton file, and SkelShift evaluates it across held-out skeleton shifts. On held-out game species, its KIDdesc is about equal to the real–real value.

  • Problem

    Existing text-to-motion models largely assume a fixed training skeleton, while new characters may arrive with only a skeleton file and no clips.

    Source: §1 · p. 1

  • Method

    TopX generates joint rotations and a root trajectory from a skeleton file, while SkelShift evaluates generalisation across held-out skeletons, tree classes and species.

    Source: §1 · pp. 1–2

  • Results

    0.127 against 0.124 KIDdesc on held-out game species, about equal to the real–real value.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    A single model animates benchmark skeletons across five axes of skeleton shift directly from their files.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Text-to-motion models generate motion only for the skeletons they were trained on, while a new character arrives as a skeleton file and nothing else. We study text-to-motion for skeletons unseen in training, without clips of the character, templates, retargeting or fine-tuning. We present TopX, a 73M-parameter flow-matching transformer that generates joint rotations and a root trajectory for any skeleton from its kinematic tree, rest pose and joint names. Motion is normalised by the rest pose and the size of the skeleton, and the rest pose is encoded as a world-frame descriptor that is independent of how the rig's author oriented each joint, while rest conventions are randomised during training. Attention is factorised over the joints and the frames of a motion. Across joints, queries and keys are rotated by HK-RoPE, a heat-kernel rotary encoding of the skeleton graph that is defined for every tree and does not change with the order of its joints or the eigenbasis of its Laplacian; across frames, they are rotated by a rotary encoding of frame order. To evaluate this generalisation, we build SkelShift, which unifies four motion corpora under one codec and holds out whole skeletons, tree classes and species, so that each test bucket isolates one kind of skeleton shift: topology, joint count, body plan, species or rig source. It scores motion quality without a learned evaluator. A single model animates every benchmark skeleton (4--117 joints) directly from its skeleton file and generates up to 240 frames. On held-out game species its motion is about as close to the real clips as two halves of the real clips are to each other (kernel distance 0.127 against 0.124), and text-to-motion retrieval reaches R@1 0.953, compared with 0.929 for real motion. Project page: https://chdtevior.github.io/TopX-page/.

1 Introduction

Text-to-motion models generally assume a fixed skeleton, but new characters may arrive with only a skeleton file. TopX and SkelShift address generation and evaluation across unseen skeletons.

1 Introduction

  • New characters may have only a skeleton file, while existing methods largely build the training skeleton’s joint count, order and rest pose into the model.Retargeting and template-based models cannot represent skeletons beyond their templates, while AnyTop requires motion statistics from the target skeleton.Source: §1 · p. 1
  • TopX animates an unseen skeleton from its file alone, using rest-pose normalization and a world-frame descriptor independent of the rig’s rest-frame conventions.It outputs joint rotations and a root trajectory, without target clips, retargeting or fine-tuning.Source: §1 · p. 1
  • HK-RoPE rotates attention queries and keys using heat-kernel coordinates defined for every tree, invariant to joint relabelling and independent of the Laplacian eigenbasis.This positional encoding is designed to handle unseen tree shapes.Source: §1 · pp. 1–2
  • SkelShift unifies four motion corpora and holds out skeletons, tree classes and species across five axes of skeleton shift.It reports test-tree distance from training trees and scores motion quality without a learned evaluator.Source: §1 · p. 2

2 Related Work

Prior text-to-motion approaches generally fix a skeleton or its topology, while work on arbitrary skeletons and unseen-skeleton evaluation covers distinct parts of the problem. TopX takes any kinematic tree as input, and SkelShift evaluates generalisation along controlled axes.

Text-to-motion

  • Human and animal text-to-motion methods generally fix the kinematic tree or its topology, whereas TopX takes any kinematic tree as input.Animal approaches include fixed templates, shared topologies and a unified 30-joint skeleton; OmniSkel adapts a human model to new bone lengths.Source: §2 · p. 2

Motion on arbitrary skeletons

  • Existing arbitrary-skeleton methods leave gaps: AnyTop lacks text conditioning, Motion2Motion uses target examples, and several text-conditioned methods retain topology or output constraints.TopX instead normalizes by rest pose and skeleton size, uses a convention-independent world-frame rest descriptor and HK-RoPE, and generates up to 240 frames on skeletons of 4 to 117 joints.Source: §2 · p. 2

Evaluating unseen skeletons

  • Prior unseen-skeleton evaluations cover limited settings, while SkelShift tests topology, rare body plans, joint counts, species and held-out rigs with results broken down by axis.It reports distances from test trees to training trees and measures realism, ground contact and text matching on held-out species.Source: §2 · p. 2

Positional encodings on graphs

  • Laplacian eigenvectors can vary by sign or basis within repeated-eigenvalue eigenspaces; HK-RoPE uses heat-kernel signatures as rotary joint coordinates and pairs them with frame-order encoding.The passage situates heat-kernel signatures among diffusion-based node encodings and rotary graph coordinates.Source: §2 · p. 2

3 Method

TopX generates motion for unseen rigs directly from their skeleton files, without target clips, retargeting, or fine-tuning. It combines rest-normalised motion representations and skeleton-aware attention with training and sampling choices designed for varied rigs.

3 Method

  • A shared transformer samples motion from a rig’s kinematic tree, rest pose, joint names, and caption, without using clips from that rig at generation time.It outputs a motion sequence in a common 17-channel representation.Source: §3 · p. 2

3.1 Representation and skeleton inputs

  • Each joint uses 17 shared channels, including rest-relative rotation, position, velocity, and contact; root channels also encode ground track and heading.Velocities are supervision targets, while rotations and the root trajectory drive the rig and can be decoded independently per frame.Source: §3.1 · pp. 2–3
  • Motion is centred on the rest frame and scaled by rig size, while a world-frame rest descriptor conveys pose independently of the author’s joint-frame orientations.The rest frame maps to zero, and the validity mask for an unseen rig follows from its structure.Source: §3.1 · p. 3

3.2 Factorised attention with HK-RoPE

  • Factorised attention processes frames and joints separately, using frame-order rotary encoding and HK-RoPE coordinates that are defined for trees and invariant to joint relabelling and Laplacian eigenbasis.A tree-hop bias also distinguishes ancestor relationships and limb counterparts.Source: §3.2 · p. 4

3.3 Training and sampling

  • Training balances channel-group losses and randomises skeleton structure and rest conventions; sampling integrates the flow path and uses caption-only guidance.Augmentations include inserting or removing joints, merging chain joints, rescaling bones, and randomly rotating rest orientations; generated motion is used without post-processing.Source: §3.3 · pp. 4–5

4 The SkelShift corpus and benchmark

SkelShift standardises four motion corpora and separates test sets by the kind of skeleton shift they represent. The benchmark evaluates generated motion with evaluator-free statistics and reports text retrieval on held-out game species.

4.1 Four corpora under one codec

  • SkelShift combines four corpora under one codec, covering 139,102 training clips on 5,127 rigs and 1,364 distinct trees, with 6 to 142 joints.Clips are encoded at 30 fps, grounded at their lowest point, labelled with a shared contact rule, and quality-checked.Source: §4.1 · p. 5

4.2 Held-out buckets

  • Benchmark buckets hold out clips, rigs, tree classes, joint-count ranges, body plans, species, or rig sources to isolate different skeleton shifts.Held-out units are assigned to development and test splits, and clips are annotated with tree distance, joint count, body plan, and motion reuse.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Protocol and metrics

  • Generation uses five seeds per clip, capped at 240 frames, while evaluator-free quality statistics include kinematic Fréchet and kernel distances, sliding, penetration, floating, and relative speed.KIDdesc is reported with its real–real value; statistics are averaged over seeds and weight rigs equally, except KIDdesc, which compares clip distributions.Source: §4.3 · pp. 5–6
  • R@1 was 0.953 for TopX, compared with 0.929 for real motions, on held-out game species using 439 test clips and caption pools of eight.Shuffled captions scored 0.120; chance level was 0.125.Source: §4.3 · p. 6

5 Experiments

The experiments compare TopX with UniMate on shared training data, evaluate TopX across held-out skeleton-shift buckets, and test its design components through ablations.

5.1 Setup

  • The comparison trains TopX and UniMate on the same 5,023 UniML3D clips, with UniMate guidance tuned separately by FIDkin or FIDdesc.Table 2 reports four UniML3D test buckets, the first 60 frames, and five seeds.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 Skeletons without training clips

  • On held-out game species, TopX reaches FIDdesc 1.19, compared with 3.32 for UniMate and 24.63 for AnyTop; on ID-tree rigs, its FIDdesc is 1.06 against 2.72 and 1.56.The shared-data comparison uses 5,023 UniML3D clips; Table 2 covers the first 60 frames and five seeds.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • KIDdesc is 0.127 versus a real–real value of 0.124 on held-out game species, and 0.664 versus 0.874 on Mixamo’s 65-joint tree.Table 3 reports TopX at full length, up to 240 frames.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Ablations

  • Across the targeted out-of-distribution buckets, the full model has the lowest KIDdesc among ablation arms; rest-convention changes shift its outputs 0.44 versus 0.89 mean bone lengths without augmentation.On unseen topologies, full-model KIDdesc is 1.78, compared with 1.88 using a frame table, 1.87 using a joint table, 1.83 using eigenvectors, and 1.95 without augmentation.Source: §5.3 · p. 8, Table 4 · p. 8

6 Conclusion

TopX animates skeletons unseen in training from their skeleton files alone. On held-out species, its motion is about as close to real clips as two halves of the real clips are to each other.

6 Conclusion

  • TopX animates skeletons unseen in training from their skeleton files alone, and on held-out species its motion is about as close to real clips as two halves of the real clips are to each other.The system uses rest-pose and size normalization, a world-frame rest-pose encoding under randomized conventions, and HK-RoPE for arbitrary skeleton topologies.Source: §6 · p. 8

A Representation and codec

The representation converts clips and skeletons into shared motion channels, normalized using each skeleton’s rest pose and size. Model details describe the per-joint inputs and attention encodings, while the objective calibrates weights across motion target families.

A Representation and codec

  • The codec resamples clips to 30 fps, grounds the lowest joint, smooths and stores the root’s horizontal track, and represents rotations relative to rest orientation.A data-quality check verifies grounding, contact, root track, heading, and agreement between positions and forward kinematics.Source: A Representation and codec · p. 11, Table 5 · p. 11
  • Rest normalization uses the rest pose as the reference and scales position, velocity, and ground-track channels by the skeleton’s bounding-box diagonal and shared gains.The three gains are calibrated once on 148 clips and shared across rigs; constant joint-channel pairs are excluded from loss and input.Source: A Representation and codec · p. 11

B Model details

  • Per-joint inputs combine structural geometry with descriptions based on joint names or, when needed, tree geometry and body plan.The structural vector includes a unit rest offset, normalized bone length, depth, path length, child count, and leaf flag; descriptions are embedded with LLM2Vec.Source: B Model details · p. 11
  • Joint-coordinate heat-kernel features and pairwise tree biases support attention that is equivariant to joint relabelling and independent of eigenvector basis choices.The heat-kernel MLP produces per-head rotation angles; temporal encoding uses frame order, and the proof includes all non-trivial modes without truncating eigenspaces.Source: B Model details · pp. 11–12

C Objective and calibration

  • The objective balances six target families using calibrated weights, with rotation assigned weight 10 and auxiliary losses for FK consistency, world velocity, foot lock, and acceleration.The family weights are calibrated from target shares and group energies; the reported gradient-share check stays within a factor of 1.25 of target.Source: C Objective and calibration · p. 12

D Training details

The reported ablation settings use a 300-epoch training schedule with learning-rate decay spread across those epochs.

D Training details

  • Each ablation arm and the full-model control is trained for 300 epochs, with each arm recalibrated on its own data and model.The control uses the main model’s calibration.Source: D Training details · p. 12

E Benchmark construction

SkelShift constructs controlled test buckets by holding out clips, rigs, and tree classes, then annotates their structural distance from training trees. It also documents caption coverage and release contents.

E Benchmark construction

  • OOD-tree and OOD-plan hold out whole tree classes, while ID-clip and ID-tree provide references that hold out clips or rigs.OOD-tree covers selected biped and quadruped classes; OOD-plan covers rare body plans, with every rig of a held-out class removed from training.Source: E Benchmark construction · p. 12
  • Tree tiers distinguish seen trees (T0), new trees with familiar local patterns (T1), and trees with new local patterns (T2).T2 is subdivided into T2a and T2b at the median pattern distance.Source: E Benchmark construction · p. 12
  • Caption coverage varies by corpus, and 227 OOD-species clips are excluded from retrieval evaluation but retained for motion-quality metrics.Those clips fall outside the recommended subset of rule-based captions.Source: E Benchmark construction · p. 13
  • The benchmark release is specified as per-clip source identifiers, splits, annotations, a pinned UniML3D commit, conversion scripts, and metric code.Source assets retain their own licences.Source: E Benchmark construction · p. 13

F Metric definitions

The section defines evaluator-free motion statistics and the protocols used to compare TopX with UniMate. It also specifies how guidance and comparison settings are selected and scored.

F Metric definitions

  • FIDkin pools three kinetic features across joints into six values per sequence, standardizes them using real-set statistics, and compares distributions with Fréchet distance.The features are computed on world joint positions divided by the rig’s rest diagonal and relative to the first-frame root position.Source: F Metric definitions · p. 13
  • FIDdesc and KIDdesc compare 24-dimensional kinematic descriptors, with KIDdesc using a polynomial kernel and a real–real reference from two halves of real clips.FIDdesc compares fitted Gaussians; KIDdesc uses an unbiased estimator averaged over random subsets.Source: F Metric definitions · p. 13
  • Skate, Slide, and Cov. use named contact joints, while Slip, Pen., Float, and Speed quantify grounded motion, penetration, floating, and relative joint speed.Rigs without a named contact joint or with zero root height are excluded from Skate, Slide, and Cov.; Float excludes captions indicating airborne actions.Source: F Metric definitions · pp. 13–14

G Guidance scale

  • TopX’s guidance scale is selected on dev by lowest mean KIDdesc among 1.5, 2, and 3, with ties resolved in favour of 2.At 50 steps, scale 2 has KIDdesc 0.142, against 0.149 at 1.5 and 0.176 at 3.Source: G Guidance scale · p. 14

H Comparison with UniMate

  • The same-data UniMate comparison scores the first min(60,T) frames and uses paired bootstrap intervals over rigs to assess differences.TopX uses 200 Euler steps at guidance 2, while UniMate’s FIDkin-tuned guidance is 7; a separate comparison tunes UniMate on FIDdesc.Source: H Comparison with UniMate · p. 14

H.1 Released weights

Against released baselines on held-out species, TopX has lower FIDdesc and skating closer to real clips. The FIDdesc ranking is also the same on Truebones test rigs, where both baselines were trained.

H.1 Released weights

  • The released UniMate weights generate at most 71 joints and 60 frames, whereas AnyTop is evaluated without text conditioning.UniMate was trained on UniML3D without a held-out split; the comparison table covers the first 60 frames.Source: §H.1 · pp. 14–15, Table 13 · p. 15
  • On two held-out species, TopX records FIDdesc 1.19 versus UniMate 3.32 and AnyTop 24.63, with Skate 0.208 versus 0.268 and 0.759, and 0.213 for real clips.None of the three methods was trained on these species’ rigs; AnyTop is unconditional and transductive.Source: §H.1 · p. 15, Table 13 · p. 15
  • On Truebones test rigs, FIDdesc ranks TopX, UniMate, and AnyTop at 3.05, 4.62, and 9.85, respectively.Both baselines were trained on these test rigs.Source: §H.1 · p. 15, Table 13 · p. 15

I Ablations per bucket

The ablations test component choices on skeleton-shift buckets, including sensitivity to perturbed rest conventions. The qualitative figures show real clips alongside TopX outputs at matched times across five OOD buckets.

I Ablations per bucket

  • Perturbing test rigs’ joint rest orientations changes generated joint positions by a median 0.44 mean bone lengths with the full model, versus 0.89 without rest-convention augmentation.The reduction is 0.45, with a 95% bootstrap interval of 0.41–0.50 over clips and 0.39–0.51 over rigs; resampling with a different seed changes output by 1.49.Source: I Ablations per bucket · p. 15
  • The full model moves less in each of the seven OOD buckets with probe clips.Table 14 reports KIDdesc uncertainty across the full model and four ablation settings: learned frame and joint tables, eigenvectors with SignNet, and no augmentation.Source: I Ablations per bucket · p. 16, Table 14 · p. 17

J Qualitative figures

  • The figures render generated joint rotations on each asset’s armature, with the asset mesh deformed using its own weights.Rows of a clip share a camera, frames, and ground plane.Source: J Qualitative figures · p. 16
  • Figure 3 pairs real clips above TopX outputs at matching times for five OOD buckets.The Mixamo strobe panel instead places the real clip at left and overlays five poses along its root path.Source: Figure 3 · p. 17

K Text–motion alignment

The section validates a text–motion retrieval evaluator and describes its use to assess held-out-species motion against UniMate.

Evaluator

  • The evaluator pairs frozen DistilBERT caption embeddings with a graph–temporal motion encoder and canonicalizes joint order, making embeddings invariant to relabelling.It is trained with symmetric InfoNCE on 132,368 clips, with 6,949 validation clips held out for model selection.Source: K Text–motion alignment · p. 17

Validity on the training distribution

  • Captions retrieve real held-out clips at R@1 0.92 on AniMo4D animals and 0.93 on game-library additions.Caption pools contain 32 items, chance is 0.031, and species words are masked.Source: K Text–motion alignment · pp. 17–18
  • Shuffling frames removes 79% of the margin above chance, while bone and leaf perturbations and rescaling retain mean cosine of at least 0.988.Across 16 conditions, two training seeds achieve Kendall τ=0.90 in condition ranking.Source: K Text–motion alignment · p. 18

Held-out species

  • Held-out-species scoring covers 439 clips on six rigs at full length; UniMate comparisons also score the first 60 frames under the stated caption and joint-count constraints.The same-data TopX–UniMate pair is compared on 181 clips from three emu rigs because retrained UniMate supports at most 61 joints; capybara rigs have 66 joints.Source: K Text–motion alignment · p. 18

Against UniMate

  • Table 16 reports held-out game-species R@1 within rigs using pools of eight, with shuffled-caption scores and results for all rigs and the emu subset.The table gives means and 95% half-widths over five seeds; chance is 0.125.Source: Table 16 · p. 18
Loading 2610.34321v2…