PaperlayerView exact revision on arXiv

Source-linked AI summary

Retargeting Motions to Diverse Skeletons via Learnable Flattening

Kia-Jüng Yang, Fabian H. Sinz, Paweł A. Pierzchlewicz

arXiv:2609.38578v3cs.CVcs.AI

TL;DR

Cross-structural motion retargeting remains challenging because existing methods struggle with unseen skeletal topologies and transformer attempts have not outperformed specialized methods. This paper uses a Transformer Autoencoder with learnable graph flattening and multiplicative positional encodings for unsupervised retargeting without paired data. It reports lower global joint position error and the highest user-study rankings for motion alignment and physical plausibility.

  • Problem

    Existing methods struggle to reliably retarget motion to unseen skeletal topologies, and recent Transformer-based attempts have not outperformed specialized geometric methods.

    Source: §1 · p. 1

  • Method

    A unified Transformer Autoencoder uses learnable skeletal graph flattening and multiplicative graph positional encodings to learn a topology- and translation-invariant latent space without paired retargeting data.

    Source: §1 · pp. 1–2

  • Results

    Global joint position error falls by 43–47% over existing benchmarks, and the method ranks highest in motion alignment and physical plausibility in a user study (n = 37, p < 0.05).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support learnable graph flattening with multiplicative encodings as a design that makes a single unsupervised Transformer effective for zero-shot retargeting across skeletal topologies.

    Source: §1 · p. 2, §5 · p. 9

    The model assumes a T-pose rest configuration and is designed for human-like skeletons, struggling with many joints from non-standard parts such as capes.

    Source: §5 · p. 10

Abstract

from arXiv · show

Cross-structural motion retargeting aims to transfer motion between different skeletal topologies. Despite recent progress, existing state-of-the-art models struggle with reliability in zero-shot settings, i.e. skeletons with different topologies which were unseen during training, and recent Transformer-based attempts have failed to outperform specialized geometric methods. We bridge this gap with a Transformer Autoencoder that learns a topology- and translation-invariant latent space. Our core contribution is a learnable flattening of skeletal graphs that captures both local dependencies and global structure. Unlike the standard transformer architecture, which adds positional information to token content, we integrate graph-based positional encodings multiplicatively, a design choice that follows directly from our flattening formulation. The resulting model handles diverse skeletal topologies within a single unified architecture and trains in a fully unsupervised manner, requiring no paired retargeting data. Ablation studies show, that the graph encodings, multiplicative formulation, and Transformer backbone is critical for the performance. In zero-shot evaluations, our method reduces global joint position error by $43-47\%$ over current benchmarks. A user study ($n = 37$), including expert animators, further ranks our approach highest in motion alignment and physical plausibility ($p < 0.05$). These results demonstrate that our model design is key to making transformer architectures effective for motion retargeting, outperforming existing approaches.

1 INTRODUCTION

Motion retargeting remains difficult across different skeletal topologies because prior approaches are manual, pair-specific, or tied to fixed structures. This work introduces a unified Transformer Autoencoder whose learnable graph flattening and multiplicative positional encodings support unsupervised zero-shot transfer, with improved benchmark and user-study results.

1 INTRODUCTION

  • Traditional retargeting often requires skilled animators or algorithms specialized for particular skeleton pairs, limiting scalability and generalization.Deep learning methods commonly remain tied to fixed skeletal topologies.Source: §1 · p. 1
  • A Transformer Autoencoder learns a topology- and translation-invariant latent space through learnable flattening of skeletal graphs.The flattening is formulated as learnable tiling and masking, motivating multiplicative graph-based positional encodings.Source: §1 · pp. 1–2
  • End-to-end training is fully unsupervised and cycle-consistent, while data augmentation is reported as critical for zero-shot generalization.The approach requires no paired retargeting data.Source: §1 · p. 2
  • The unified model handles humanoid skeletons with varying joint and edge counts without paired data or textual joint information.It encodes skeletal structure from graph topology and rest-pose geometry.Source: §1 · p. 2
  • Global joint position error falls by 43–47% over existing benchmarks, and the method ranks highest in motion alignment and physical plausibility in a user study (n = 37, p < 0.05).The study included expert animators.Source: §1 · p. 2

2 RELATED WORK

Prior work spans manual or pair-specific retargeting, topology-limited deep representations, and methods that handle varied skeletons with other constraints. The paper positions learnable graph flattening and multiplicative positional encodings as a way to make a single unsupervised Transformer effective across topologies.

2.1 MOTION RETARGETING

  • Traditional retargeting uses space-time optimization or inverse kinematics with extensive manual tuning, while learned approaches have varied constraints across skeleton structures.Villegas et al. (2018) perform poorly across different skeletons; Aberman et al. (2020) require a separate model per skeleton pair; Lee et al. (2023) use proxy ground truth; Liu et al. (2026) rely on textual joint names and do not outperform existing methods.Source: §2.1 · p. 2
  • A training-free patch-based alternative requires explicit bone correspondences and a target motion database at inference time.Chen et al. (2025) present this approach.Source: §2.1 · p. 3
  • The paper presents learnable skeletal graph flattening as the mechanism that makes a single unsupervised Transformer effective for zero-shot retargeting across topologies.The authors contrast this with prior transformer attempts and report outperforming existing methods.Source: §2.1 · p. 3

2.2 DEEP MOTION REPRESENTATIONS

  • Motion representations include fixed-topology cycle-consistent transfer and additive graph integration, whereas the paper's multiplicative positional encoding is reported critical for zero-shot generalization.MotionPuzzle (2022) uses cycle consistency and reconstruction on unpaired data for per-body-part style transfer; Gat et al. (2025) add graph structure to transformer attention maps.Source: §2.2 · p. 3
  • Prior unified motion representations remain limited to fixed humanoid topologies, while SAME (2023) handles varying topologies but requires ground-truth retargeting pairs.The framework requires neither paired data nor fixed topology and learns a skeleton-agnostic pose component alongside a separate root trajectory component.Source: §2.2 · p. 3

3 METHODS

The method replaces joint-count-dependent flattening with learned, fixed-dimensional representations and uses a Transformer autoencoder to reconstruct skeletal motion. Training combines augmentation and cycle consistency without explicit ground truth.

3 METHODS

  • Standard flattening scales with joint count, so the method replaces its fixed operations with learned functions to obtain topology-agnostic latent representations.This addresses the dependence of concatenated representations on the number and structure of joints.Source: §3 · p. 3

3.2 FLATTENING REPRESENTATION LEARNING

  • Learned functions replace fixed flattening operations, producing a fixed-dimensional representation independent of joint count while retaining multiplicative structural gating.The skeleton-derived mask modulates joint features rather than adding positional information to them.Source: §3 · p. 3, §3.2 · p. 4

3.3 MODEL ARCHITECTURE

  • The Transformer autoencoder uses a GraphSAGE-derived rest-pose mask, aggregates masked joint features into a pose token, and decodes them to recover skeletal motion.A separate trajectory token encodes global translation; the decoder reconstructs joint rotations and root trajectory.Source: §3.3 · pp. 4–5

3.4 TRAINING

  • Without explicit ground truth, training combines augmentation consistency and cycle consistency, with translation, bone scaling, and skeleton edits among the augmentations.The loss includes reconstruction, temporal, contact, end-effector, latent-consistency, and adversarial terms.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments compare reconstruction on unseen datasets, latent-space invariance, and Mixamo retargeting against prior methods, including SAME, Aberman et al. (2020), and PALUM. They also evaluate a user study and ablate architectural and training choices.

4.1 RECONSTRUCTION

  • On Bandai-Namco, Joint Position error drops 78% (12.40→2.63 cm) versus SAME, with lower Foot Sliding (0.08→0.05) and Ground Penetration (-0.20→-0.01).On 100Style, SAME's Joint Position and Root Trajectory errors rise to 104.66 and 101.78 cm, while this method records 2.14 and 1.85 cm; direct global-root prediction avoids delta-based drift.Source: §4.1 · p. 6, Table 1 · p. 6

4.2.1 SKELETON INVARIANCE

  • In a shared 2D PCA space, embeddings from different skeletons overlap closely when performing the same action, consistent with skeleton-agnostic motion representations.The projections compare different skeletons performing semantically identical motions.Source: §4.2.1 · p. 6, Figure 4 · p. 7

4.2.2 TRANSLATION INVARIANCE

  • Our pose latent space achieves near-perfect translation invariance (> 0.99 cosine similarity), versus Aberman et al. (2020), whose scores drop below 0.8 with high variance; SAME is 1.0 by definition.Source: §4.2.2 · pp. 6–7, Figure 5 · p. 7

4.2.3 MOTION CLASSIFICATION

  • The pose latent space correctly classifies 59/60 motions, compared with 57/60 reported for SAME, using 128 rather than 32 latent dimensions.The evaluation tests whether semantically similar motions cluster across skeletons.Source: §4.2.3 · p. 7

4.3 RETARGETING

  • Retargeting is evaluated qualitatively and quantitatively across identical-topology Intra and differing-topology Cross settings, using GJP, Jerk, and PART.GJP measures global joint-position error; Jerk reflects motion dynamics, and PART measures root-trajectory shape independent of absolute alignment.Source: §4.3 · p. 7, §4.3.1 · p. 8

4.3.1 QUANTITATIVE MEASUREMENT

  • At least 47% on Intra and 43% on Cross: the method beats the next-best method in GJP on Mixamo despite not being trained there.Its Jerk score of 0.72 is below ground truth (≈1.3), while global-root methods preserve trajectory shape better than delta-based SAME.Source: §4.3.1 · p. 8, Table 2 · p. 8

4.3.2 QUALITATIVE MEASUREMENT

  • In a 37-participant study, our method significantly outperforms SAME and Aberman et al. (2020) in quality, and SAME in alignment (p < 0.05).Participants rated alignment to the source motion and perceptual quality; a perceptual difference from ground truth remains.Source: §4.3.2 · p. 9, Table 3 · p. 9

4.4 ABLATION STUDY

  • Replacing the Transformer with a GAT causes very large Intra/Cross GJP errors (891.96/927.38); additive positional encoding raises GJP by 42%/57%, and removing augmentations raises it to 3.92/2.66.Removing positional encoding also causes failure; the GAT overfits training skeletons, while its large errors are driven mainly by inaccurate global-root prediction.Source: §4.4 · p. 9, Table 4 · p. 9

5 SUMMARY, LIMITATIONS, AND FUTURE WORK

The paper presents a unified unsupervised Transformer Autoencoder for retargeting across arbitrary skeletal topologies and identifies limits in its input assumptions, geometry handling, and supported skeleton types.

Summary

  • The method is evaluated zero-shot and is reported to outperform existing state-of-the-art methods quantitatively and qualitatively across skeletal topologies.Its learnable graph flattening and multiplicative positional encodings are presented as making Transformers effective for this task.Source: §5 · p. 9

Limitations and future work

  • The model assumes a T-pose rest configuration, uses kinematic losses that omit surface geometry, and struggles with non-human-like structures containing many nonstandard joints.These limits can require preprocessing, permit mesh self-penetration or skeleton-mesh misalignment, and motivate mesh-aware constraints and broader skeletal support.Source: §5 · p. 10

Reproducibility

  • The authors report that the implementation, pretrained models, and reproduction instructions are publicly available, alongside architecture, augmentation, data-preparation, and training details.The paper states that generative AI tools were used for readability editing and reference formatting, with AI-assisted edits reviewed by the authors.Source: REPRODUCIBILITY STATEMENT · p. 10, AI USE STATEMENT · p. 10

A ARCHITECTURE DETAILS

The section describes the Transformer Autoencoder's architecture and implementation details.

A ARCHITECTURE DETAILS

A.1 MODEL ARCHITECTURE

The autoencoder pairs Transformer-based encoding and decoding with learned GraphSAGE positional encodings and separate pose and trajectory latents. Its training objective combines reconstruction, temporal, contact, cross-skeleton, latent-consistency, and adversarial losses.

A.1 MODEL ARCHITECTURE

  • The encoder and decoder use Transformer encoders with learned GraphSAGE positional encodings, while separate learnable parameters represent pose and root trajectory.The discriminator also uses a Transformer encoder and a learnable classification parameter.Source: §A.1 · p. 14

A.3.1 RECONSTRUCTION LOSS

  • Reconstruction combines joint-position, root-child position, rotation, and root-trajectory losses to recover motion, including the cycled-back unscaled input.The root-child term targets joints directly connected to the root.Source: §A.3.1 · p. 15

A.3.2 TEMPORAL SMOOTHING

  • Velocity and jerk losses regularize framewise reconstruction by matching reconstructed and ground-truth temporal derivatives.The jerk term compares changes in acceleration across adjacent frames.Source: §A.3.2 · p. 15

A.3.3 CONTACT LOSS

  • Contact-position, contact-velocity, and ground-penetration losses target foot sliding and penetration, using ground-contact labels derived during preprocessing.The contact labels are not required at inference.Source: §A.3.3 · p. 15

A.3.4 END EFFECTOR LOSS

  • The end-effector loss matches path-length-normalized velocities across skeletons, which must share the same number of end effectors.Source: §A.3.4 · p. 16

A.3.5 LATENT CONSISTENCY; A.3.6 ADVERSARIAL LOSS

  • Latent consistency minimizes the distance between pose latents and their reconstructions, while adversarial training uses a discriminator to encourage realistic retargeted motion.The generator update changes autoencoder weights so the discriminator predicts its output is real.Source: §A.3.5 · p. 16, §A.3.6 · p. 16

A.3.7 LOSS FUNCTION WEIGHTING

  • Loss weights balance the objective, with cycle-consistency and augmentation-consistency weights shared for corresponding terms.The positional and rotation losses use λpos and λrot, respectively.Source: §A.3.7 · p. 16

A.4 TECHNICAL DETAILS; A.2 HYPERPARAMETER SETTINGS

  • For 26→26 joints, latency is 2.46 ± 0.02 ms per frame on CPU and 0.12 ± 0.01 ms on an NVIDIA A100 GPU; cost is 1.82 MFLOPs per frame.No scheduler is used for the adversarial discriminator optimizer.Source: §A.2 · p. 14, §A.4 · p. 17

A.5 DATA PREPARATION

Training uses public motion-capture datasets without paired retargeting data or Mixamo sequences. Preparation scales and grounds animations, while dynamic augmentations and held-out skeleton evaluations support the reported zero-shot setting.

A.5 DATA PREPARATION

  • The model trains unsupervised on public motion-capture datasets, requiring neither paired retargeting data nor Mixamo motion sequences.The datasets include LaFAN1, ACCAD, TotalCapture, PFNN, SFU, and CMU.Source: §A.5 · p. 17
  • Preprocessing scales skeletons and motion by the inverse of the largest rest-pose joint distance, then grounds each animation using a per-animation 5th-percentile height threshold.Source: §A.5 · p. 17
  • Training dynamically varies joint lengths, global translations, and joint removal; evaluation uses Bandai-Namco, 100Style, and Mixamo, with Cross and Intra protocols on Mixamo.The Mixamo evaluation excludes object-carrying clips and uses 95 clips per character.Source: §A.5 · pp. 17–18

A.6 TRAINING PROTOCOL

The appendix points to a table for training-protocol specifications and notes that experiments used high-performance GPUs.

A.6 TRAINING PROTOCOL

  • Experiments used high-performance GPUs, with hardware and convergence criteria included in the training-protocol summary.Source: §A.6 · p. 18
  • Training protocol specifications are summarized in Table 8.Source: Table 8 · p. 18

B DATA AUGMENTATION

The augmentation schemes perturb skeletal geometry while preserving kinematic structure, with global translation used to support pose–trajectory disentanglement. The section also motivates a dense, constant-dimensional representation by identifying limitations of a sparse alternative.

B DATA AUGMENTATION

  • The augmentation framework combines rest-pose, bone-scaling, and skeleton perturbations, with global translations; non-humanoid training also uses rest-pose augmentation.Source: B DATA AUGMENTATION · p. 18

B.1 REST POSE AUGMENTATION

  • Rest-pose augmentation applies random joint rotations while maintaining parent-child hierarchy and relative orientations.Source: §B.1 · p. 18

B.2 BONE SCALING AUGMENTATION

  • Bone-scaling augmentation uniformly scales joint offsets, then recalculates joint positions and root trajectory to match the changed geometry.Source: §B.2 · p. 19

B.3 SKELETON AUGMENTATION

  • Skeleton augmentation combines global and per-joint scaling with random joint removal, capped at four removed joints.Source: §B.3 · pp. 19–20

B.4 GLOBAL TRANSLATION AUGMENTATION

  • Global translation shifts joint positions along the horizontal x and z axes and is also applied to associated trajectories.Source: §B.4 · p. 20

B.5 LIMITATIONS OF THE SPARSE REPRESENTATION

  • The sparse representation grows with joint count and leaves some latent dimensions unused, motivating a dense, constant-dimensional alternative.Source: §B.5 · p. 20

C.1 LOSS ABLATION STUDIES

The ablations report that the design choices improve GJP overall, while individual losses have different effects on intra- and cross-retargeting. Cycle consistency has a particularly large effect on cross-retargeting, and contact-position loss trades GJP error against foot sliding.

C.1 LOSS ABLATION STUDIES

  • Removing Lchild increases GJP from 1.45 to 3.21 Intra and 1.28 to 2.51 Cross; removing Lee causes smaller but consistent degradations.Source: §C.1 · pp. 20–21
  • Removing Lcyc. cons. increases Cross GJP from 1.28 to 3.25, while removing Ladv causes smaller but consistent degradations.Source: §C.1 · p. 21
  • Removing Ljerk increases Cross GJP to 2.07, while removing Lvel also degrades performance.Source: §C.1 · p. 21
  • Lcntct vel has a stronger impact than Lcntct pos, while removing Lcntct pos slightly improves Intra but harms Cross performance and decreases foot sliding.Source: §C.1 · p. 21
  • At minimum, the design choices improve GJP by 11% on Intra and 13% on Cross retargeting.Source: §C.1 · p. 21

C.2 TRAJECTORY ERROR ACCUMULATION

The appendix figures show trajectory error across frames, shared latent-space projections for semantically identical motions across skeletons, and visual retargeting comparisons across methods.

C.2 TRAJECTORY ERROR ACCUMULATION

  • The xz root-trajectory error is plotted against increasing frame count for three motions, comparing global-position prediction with delta prediction.Source: Figure 7 · p. 22

C.3 SKELETON INVARIANCE

  • The PCA projection shows a shared latent space across sequences for different skeletons performing semantically identical motions.Source: Figure 8 · p. 23

C.4 RETARGETING

  • Each row shows one motion and each column shows frames 0, 50, and 100, with source, ground-truth, and method retargets color-coded.Source: Figure 9 · p. 24

Related papers

Loading 2609.38578v3…