PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Cross-Skeleton Retargeting Is Non-Identifiable: Structural Limits of Generative Motion Models

Zhiyuan Li, Wenyan Yang, Pekka Marttinen, Joni Pajarinen

arXiv:2609.37297v1cs.CVcs.RO

TL;DR

Cross-skeleton retargeting aims to carry a source clip’s action and motion intention to another body, but sparse heterogeneous data and standard objectives may not identify the source-conditioned map. The paper analyzes two structural routes to this ambiguity and introduces Source-Instance Fidelity to measure whether source-instance relations remain visible. Its conclusion is that action-level success alone does not establish source-conditioned retargeting.

  • Problem

    Sparse heterogeneous motion domains provide limited instance-level correspondence evidence for determining whether a target motion reflects its particular source clip.

    Source: §7 · p. 10

  • Method

    The paper analyzes gauge non-identifiability and conditional-mean degeneration, and introduces Source-Instance Fidelity to test whether outputs differ as their source clips do.

    Source: §7 · p. 10

  • Results

    The analysis identifies two structural mechanisms leaving source-conditioned maps undetermined under standard objectives: relative latent transformations under distribution matching and action-wise average targets under squared-error pairing.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Action recovery alone does not identify the source-conditioned retargeting map; evaluating it requires evidence about source-instance relations.

    Source: §7 · p. 10

    The animal evidence comes from a single dataset.

    Source: §6 · p. 10

Abstract

from arXiv · show

Cross-skeleton motion generation trains generative models to carry action structure and motion intention from one body to another. Yet a target motion that shows the right action has two explanations that the training data cannot tell apart: the model transferred the source clip, or it recovered a typical motion for the requested action. We show that this ambiguity is structural rather than incidental: under standard generative objectives, the source-conditioned retargeting map is non-identifiable in sparse heterogeneous motion domains. Unpaired distribution matching yields gauge non-identifiability: the latent spaces of different skeletons can be transformed relative to one another without changing the training evidence, so different source-conditioned maps fit it equally well. Sparse paired supervision admits the complementary failure mode, \emph{conditional-mean degeneration}: when clips are paired only by action, squared-error training converges to an average target motion that ignores the source clip. To make the missing evidence observable, we introduce Source-Instance Fidelity (SIF), a diagnostic that tests whether outputs differ from one another the way their source clips do, with the target skeleton and action held fixed. Under this diagnostic, methods that succeed at the standard action-level test on animal motion data often sit at the source-blind floor, while the methods that rise above it retain only a partial relational signal. Retargeting therefore needs objectives and evaluations that can identify the source-conditioned map it claims to learn. Project page: https://cross-skeleton-retargeting.netlify.app/.

1 Introduction

Cross-skeleton retargeting must preserve both an action and the source clip’s distinctive motion intention across bodies. The usual retrieval test checks whether the action is recognizable on the target body, not whether the output reflects its particular source clip.

1 Introduction

  • Retargeting should preserve a source clip’s timing, style, and distinguishing details while carrying its action to a body with different proportions, joints, or range of motion.The paper calls these source-specific qualities its motion intention.Source: §1 · p. 1
  • Cross-body motion research has moved from skeletal-structure-aware retargeting toward generative and correspondence-based models for diverse bodies.The cited trajectory includes [Aberman et al., 2020, Lee et al., 2023, Wang et al., 2024, Gat et al., 2025, Chen et al., 2025b].Source: §1 · p. 1
  • Target-motion retrieval AUC measures whether a requested action ranks above other actions on the target body, but not whether the output came from the given source clip.AUC is the probability that a match ranks above a non-match.Source: §1 · p. 1

B. The missing axis

Action-level recognition does not establish that a retargeted motion preserves the source instance. Sparse heterogeneous data and standard training objectives can leave that source-conditioned mapping undetermined; Source-Instance Fidelity tests whether source-instance relations survive retargeting.

B. The missing axis

  • Sparse animal libraries lack many body-action clips and instance-level pairs, so their training evidence does not determine which source-conditioned map is correct.The cited Truebones zoo has heterogeneous skeletons and sparse action coverage.Source: B. The missing axis · p. 2
  • Distribution matching allows relative transformations of body-specific latent spaces without changing training loss, while action-only squared-error pairing converges to an average target motion regardless of source.The paper names these failures gauge non-identifiability and conditional-mean degeneration, respectively.Source: B. The missing axis · p. 2
  • Source-Instance Fidelity tests whether outputs differ as their sources do while target body and action remain fixed; source-ignoring models score near zero.The figure contrasts distinct source-preserving outputs with a shared target-action habit, which standard action-level AUC accepts in either case.Source: B. The missing axis · p. 2, Figure 1 · p. 2
  • Across 1,891 Truebones source-body, target-body, and action combinations, nine of fourteen methods sit at or near the source-blind floor, including methods scoring well on retrieval.ACE [Li et al., 2023] retains only a partial relational signal: its outputs preserve some source ordering but vary little with the sources.Source: B. The missing axis · p. 2
  • Retargeting evaluation must identify the source-conditioned map itself, rather than assess only whether the target action is recognizable.The central open question is what evidence and inductive biases suffice to select that map.Source: B. The missing axis · pp. 2–3

2 Related Work

Related work advances modeling across body structures and sparse motion libraries, while this paper focuses on a distinct question: whether available evidence identifies the source-conditioned map.

2 Related Work

  • Cross-skeleton retargeting has progressed from explicit skeletal structure toward skeleton-agnostic representations, context-conditioned models, and methods for sparse heterogeneous libraries.The cited work includes [Aberman et al., 2020, Lee et al., 2023, Wang et al., 2024] and [Gat et al., 2025, Kim et al., 2025, Li et al., 2023, Chen et al., 2025b].Source: §2 · p. 3
  • Broader motion-generation research treats body identity, morphology, and topology as conditioning variables, whereas this paper asks whether sparse non-instance-level data specify which source intention selects a target motion.The cited broader work includes [Raab et al., 2024, Lee et al., 2025, Egan et al., 2024, Chen et al., 2025a].Source: §2 · p. 3
  • Marginal matching and distributional transport can fit motion distributions without instance correspondences, but retargeting additionally requires relative alignment that lets the target decoder interpret a source instance.The discussion connects identifiability limits to [Locatello et al., 2019, Khemakhem et al., 2020] and transport to [Bortoli et al., 2021, Kim et al., 2024].Source: §2 · p. 3
  • Motion-library retrieval preserves realized target snippets and serves as a diagnostic comparator, rather than claiming to synthesize a new source-conditioned motion.The connection spans classical motion graphs and sparse-correspondence transfer [Kovar et al., 2002, Chen et al., 2025b].Source: §2 · p. 3

3 The Source-Conditioned Map

A retargeting claim concerns a source-conditioned map that preserves action and source-specific motion intention, not merely a plausible target motion. Sparse heterogeneous libraries provide action evidence but little evidence of how same-action source differences should transfer.

3 The Source-Conditioned Map

  • A retargeting map carries each source clip from one skeleton’s motion space to another while preserving its action, timing, and motion intention.Unlike a generator conditioned only on action, the map must change when the source instance changes with target skeleton and action fixed.Source: §3 · p. 3
  • Truebones has 70 skeletons and 616 clips across 90 actions, but only 413 of 6,300 possible skeleton-action cells are occupied, a 6.6% fill rate.The median occupied cell contains one clip, limiting within-action source-instance evidence.Source: §3 · p. 3
  • The sparse library often lacks multiple same-action instances and dense source-target correspondences needed to infer motion intention across bodies.These are the missing observations beyond knowing that a skeleton can perform an action.Source: §3 · pp. 3–4
  • Only 49 source-skeleton/action groups contain at least three clips; pairing them with eligible target skeletons yields 1,891 triples using the same 171 source clips.Those clips span eight actions, with attack and idle making up 94 percent of triples.Source: §3 · p. 4
  • Action labels constrain admissible target actions, but only source-instance evidence constrains how differences among same-action clips should reappear on another body.The section frames identifying the map as distinct from producing recognizable cross-skeleton motion.Source: §3 · p. 4

4 Structural Non-Identifiability

The section identifies two objective-level sources of non-identifiability: marginal matching leaves relative latent alignments unconstrained, while random cell-paired squared-error training favors a source-independent target prototype. Both mechanisms preserve the evidence used by the objective while changing or erasing source-instance dependence.

4 Structural Non-Identifiability

  • Marginal matching leaves relative latent coordinates free: transforming each skeleton’s latent space can preserve the objective while changing its source-to-target map.The transformations preserve per-skeleton latent marginals, and the theorem states that the transformed and original encoders and decoders have equal objective value.Source: §4 · p. 4
  • 34,224 continuous gauge degrees of freedom remain after removing a common orthogonal factor, but observable map ambiguity also requires a decoder that exposes the orbit.The count uses K = 70 skeletons and latent dimension d = 32; it is not a count of distinct realized maps.Source: §4 · pp. 4–5
  • The theorem concerns objective-level identifiability, not a prediction that trained outputs collapse or any method’s position on SIF.The decoder-observability condition is architecture-dependent, and observability of rotation directions does not select the source-target alignment.Source: §4 · p. 5
  • With random target-cell pairing, squared-error regression selects the target cell’s conditional mean, a fixed prototype independent of the specific source clip.The proposition assumes square-integrable targets, measurable predictors, and a nonempty target skeleton-action cell.Source: §4 · p. 5
  • About 2% of oracle within-cell output variance remains as random same-cell support increases from M = 2 to M = 50 pairs per cell.The controlled ladder uses known source-conditioned transport; the oracle variance is measured over held-out source clips.Source: §4 · p. 5
  • The conditional-mean result applies to random cell pairing, not true instance-level pairs, which provide the correspondence sparse data lack.It also assumes a nonempty target cell and Euclidean motion representations for squared-error evaluation.Source: §4 · p. 5
  • Retrieval based on gauge-invariant motion features can characterize the source-blind floor, but it does not select the source-conditioned map.Such rules are invariant to the latent gauge and return an existing target clip rather than a conditional mean.Source: §4 · p. 5

B. Conditional-mean degeneration

The figure contrasts random target-cell pairing with known source-conditioned transport: cell-paired squared-error regression stays near a target prototype rather than preserving source variation.

B. Conditional-mean degeneration

  • From M = 2 to M = 50 pairs per cell, random-pair predictions retain about 2% of oracle within-cell output variance.The oracle variance is Var(T ⋆(xa)); the shaded band is the three-seed standard deviation.Source: Figure 2 · p. 6
  • Panel A illustrates that admissible target clips in a cell yield a squared-error optimum at the cell-mean prototype, not a source-conditioned motion.Panel B uses a controlled ladder with known source-conditioned transport T ⋆.Source: Figure 2 · p. 6
  • Gauge-invariant retrieval can mark the source-blind floor, but it does not identify the retargeting map or resolve source-instance dependence.The comparators avoid learned latent coordinates and the conditional-mean operator by selecting an existing target clip.Source: B. Conditional-mean degeneration · p. 6

5 Source-Instance Fidelity and Empirical Evidence

Source-Instance Fidelity (SIF) tests whether changes among source clips correspond to changes among outputs when target skeleton and action are fixed. Across evaluated methods, most show little source-instance fidelity, while some retain only partial relational signal.

SIF protocol

  • SIF correlates source-side and output-side pairwise motion discrepancies within each source-target-action triple, using aligned clips and both raw and length-controlled scoring.The accompanying output-variation measure is near one when outputs vary as much as sources and zero when they collapse; synthetic known-correspondence and source-blind references calibrate the diagnostic.Source: §5 · pp. 6–7

Empirical results

  • Nine of fourteen methods sat at or near the source-blind floor; Motion2Motion-Direct reached +0.14 raw and +0.13 length-controlled, while MoReFlow stayed near +0.25 under both scorings.ACE-T and ACE-I scored +0.34 and +0.42 raw, but fell to +0.18 and +0.25 with length control; their outputs varied by less than 1% of source variation when length was fixed.Source: §5 · pp. 7–8

Qualitative panorama

  • A fixed Bird-to-KingCobra attack query produced visibly different target motions across methods, illustrating that action-consistent outputs do not specify a unique source-conditioned map.The panorama is qualitative evidence, not a substitute for SIF or a basis for choosing a visually best method.Source: Figure 4 · p. 8, §5 · p. 8

Action-level audit

  • Random same-cluster retrieval matched ANCHOR within ±0.012 on overall and held-out AUC, while exact-action retrieval exceeded it by +0.126 and +0.242 on limited-coverage queries.Exact-action coverage was 221/600 overall and 57/200 held-out; these label-only references have no source-conditioned map.Source: §5 · p. 8, Figure 5 · p. 9, Table 1 · p. 9

Signal controls

  • ACE’s positive SIF signal falls from +0.29 to zero when its source is removed, but removing its adversarial loss leaves SIF unchanged at +0.33.The nearest-neighbor check found 76 to 79 percent of ACE outputs closer to target-skeleton training clips than to source clips, which does not support direct source copying as the explanation.Source: §5 · pp. 8–9

Human-to-robot evaluation

  • On six humanoid robots, the same-action averaging objective collapsed in all 24 evaluations and the unpaired objective in 22, while true-pair models reached 0.77 to 0.99 in all 24.The G1 evaluation used identical test items across training regimes; its table reports raw SIF, output-to-source spread variation, and a realism measure.Source: §5 · pp. 9–10, Table 2 · p. 10

6 Limitations

The evidence has limits in data coverage, paired-motion provenance, and theoretical scope.

Data scope

  • Animal evidence comes from a single dataset, and the enlarged evaluation adds target bodies rather than new source motions.These results therefore do not test broader source-motion coverage.Source: §6 · p. 10

Robot evaluation

  • The robot studies extend evaluation to humanoids, but their paired motions come from retargeting tools rather than animators, and their sparsity is created by the authors.Source: §6 · p. 10

Theoretical and model scope

  • The theorems cover specific training objectives rather than every evaluated method, and several evaluated methods are adaptations of published models to this data.Source: §6 · p. 10

7 Conclusion

The conclusion frames source-conditioned retargeting as non-identifiable under sparse heterogeneous evidence: action-consistent outputs need not preserve source-instance differences. The paper relates this ambiguity to two objective-level mechanisms and argues that evaluation must test source-conditioned transport as well as action recovery.

7 Conclusion

  • In sparse animal data, action-consistent outputs often sit at the source-blind floor; methods above it retain only partial relational signal, while true-pair human-to-robot training recovers the source-instance relation.The conclusion calls for judging progress by action recovery, source-conditioned transport, or both.Source: §7 · p. 10

A Extended Related Work

  • The gauge argument is narrower than a general impossibility result: per-skeleton marginal matching does not select the relative latent alignment required by a source-preserving map.Distributional transport alone likewise leaves the relative-gauge ambiguity unresolved.Source: A Extended Related Work · p. 13

B Data Regime and Method Roster

  • The Truebones regime has mostly empty skeleton-action cells and usually only one clip per occupied cell, making instance-level correspondence rare.The data-regime discussion emphasizes that sparsity limits within-action examples, not merely total library size.Source: B Data Regime and Method Roster · p. 13
  • The evaluated roster distinguishes published generators, supervision-matched comparators, ANCHOR, and source-blind references; AnyTop and Motion2Motion require source-conditioning adaptations.The method placements describe objective structure, not predicted SIF values.Source: B Data Regime and Method Roster · pp. 13–14

C.1 Proof of Theorem 1

  • With 70 skeletons and 32-dimensional latents, the orthogonal subgroup leaves a lower bound of 34,224 relative-gauge degrees of freedom.This is not a count of distinct decoded maps; that count also depends on decoder non-degeneracy.Source: §C.1 · p. 15

C.2 Proof of Proposition 1.1

The proofs characterize conditional-mean degeneration and distinguish it from retrieval, which returns a single library clip. Decoder checks mark a boundary condition for observable gauge ambiguity and show that latent alignment, representation richness, and SIF are distinct.

C.2 Proof of Proposition 1.1

  • Under random same-action target pairing, squared-error training uniquely selects the target-action cell mean, which depends on target skeleton and action but not the specific source clip.The target sample is independent of the source instance once the target skeleton and action cell are fixed.Source: §C.2 · p. 15

C.3 Proof of Corollary 1.1

  • Retrieval is invariant to latent-gauge reparameterization because its source feature is computed directly from motion space, and it returns one existing target clip rather than a cell average.Thus conditional-mean degeneration is not the relevant limiting operation for this retrieval rule.Source: §C.3 · p. 16

D Decoder Non-Degeneracy Boundary

  • The gauge theorem requires decoder observability: a small KL-regularized VAE nearly absorbs the orbit, with a gauge-to-noise ratio of 1.11, whereas the AnyTop sweep exceeds matched-noise divergence in all 72 cells, with a minimum ratio of 1.71.Cross-seed Procrustes alignment and effective-rank checks distinguish latent alignment and representation richness from source-instance fidelity.Source: D Decoder Non-Degeneracy Boundary · p. 16, Figure 6 · p. 16, Table 6 · p. 17

E Source-Instance Fidelity Robustness

The appendix probes SIF's robustness to dependence, sample size, generation noise, and latent-versus-decoded behavior. It also supplies synthetic calibration, describes ANCHOR's comparator role, and documents how action-level AUC can accept label-only behavior.

E Source-Instance Fidelity Robustness

  • The original 49-triple evaluation matches the main pattern, except Motion2Motion-Direct is at the floor there and rises modestly above it on the larger set.The main 1,891-triple evaluation resamples whole source skeletons because source clips recur across triples; treating triples as independent would overstate precision.Source: E Source-Instance Fidelity Robustness · p. 17
  • A single triple’s SIF varies by about 0.6 across three-clip subsets, while averages over random triple sets vary by about 0.12 for 25 triples and 0.02 for 500.The main evaluation's uncertainty intervals resample whole source skeletons because its triples reuse the same 171 source clips.Source: E Source-Instance Fidelity Robustness · p. 18
  • With independent noise, stochastic methods stay within 0.05 of zero on average; shared noise raises AnyTop and source-fed AL-Flow variants to about +0.2 through input differences such as length.All paper results use independent noise; decoded SIF can also decouple from latent SIF, including through collapse after decoding or collapsed variation.Source: E Source-Instance Fidelity Robustness · p. 18

F Synthetic Calibration

  • In the synthetic 2 × 2 calibration, oracle transport lies near one and source-blind references near zero; adding random cell-pairs does not recover oracle within-cell variance.The cell-paired squared-error predictor receives target samples from the correct action and skeleton cell, but not instance-level pairs.Source: F Synthetic Calibration · p. 19

G Deterministic Comparator and Motion Descriptors

  • ANCHOR predicts a coarse source action cluster from motion-space descriptors, then returns an existing target-library clip rather than synthesizing a new motion.Its gauge-invariant role is to mark that action recovery can remain high at the source-blind floor.Source: G Deterministic Comparator and Motion… · pp. 19–20

H Action-Level AUC and Label Recovery

  • The standard contrastive retrieval AUC is an action-level ranking test, and label-only behavior can pass it on the covered queries without identifying a source-conditioned map.Tables 20–22 and Figures 16–17 provide the broader enumeration and AUC-band controls.Source: H Action-Level AUC and Label Recovery · p. 20

I ACE Controls

This appendix section tests source-instance fidelity through controlled ablations, known correspondences, and qualitative comparisons, and documents experimental reproducibility, data and asset use, statistical testing, compute, and ethics-related disclosures.

I ACE Controls

  • With ACE’s source and length inputs, raw SIF is about +0.42, but removing the source and fixing length yields zero; real-source ACE-T retains +0.29, while its adversarial-loss advantage disappears at fixed length.With source removed but length copied, both models score about +0.30; source shuffling scores zero or below. ACE-T’s fixed-length gain over source removal is +0.29 (95% CI [+0.12, +0.47]), with three-seed results of +0.29 ± 0.07.Source: I ACE Controls · pp. 21–22, Table 23 · p. 30, Table 24 · p. 30
  • Stretching ACE outputs to 64 frames creates variation of 0.14 to 0.23 even for identical copies, while most ACE outputs are closer to target-pool training clips than to their corresponding sources.MoReFlow’s variation is unchanged by stretching and exceeds the variation caused by stretching alone, so the paper reports ACE variation raw and with length fixed at generation.Source: I ACE Controls · p. 22, Table 25 · p. 30, Table 26 · p. 30

J Human-to-Robot Studies

  • Two human-to-robot studies provide known pairs and compare unpaired, random same-group squared-error, and true-pair objectives; an ACE-style unpaired adversarial run diverged, while post hoc stabilized seeds did not jointly score well on four measures.The studies use BONES-SEED human-to-Unitree G1 counterparts and LAFAN1 clips retargeted to six robots; the theorem-class objectives do not see the available pairs.Source: J Human-to-Robot Studies · pp. 22–24, Table 30 · p. 32
  • The true human-to-G1 counterpart scores SIF 0.960 without length control and 0.959 with it; across six robots, true-pair training matches the counterpart, while averaging loses variation and unpaired training often collapses.The unpaired objective retains about 40% of variation only on PAL Talos with three clips per group and collapses in every other evaluation; its action AUC is at chance in the six-robot study.Source: J Human-to-Robot Studies · pp. 23–25, Table 27 · p. 30, Table 29 · p. 32

K Qualitative Evidence and Supplementary Videos

  • Qualitative grids hold the source-target-action triple fixed so source-dependent output changes can be distinguished from lockstep behavior; companion videos synchronize playback phase to make these patterns easier to see.The grids label latent collapse, decoder collapse, partial source-conditionality, and collapsed variation across evaluated methods.Source: K Qualitative Evidence and… · pp. 25–26, Figure 19 · p. 34, Figure 20 · p. 34

L Latent Schrödinger Bridge Comparator Scope

  • The evaluated DPG-SB-v3 latent bridge omits decoded-motion bone, contact, and smoothness penalties, so it tests that variant’s position relative to the source-blind floor rather than bridge objectives generally.Source: L Latent Schrödinger Bridge Comparator… · p. 26

M Potential Societal Impacts

  • More reliable motion transfer could enable misleading attributed performances and propagate motion-library biases, so the paper recommends source-instance diagnostics, disclosure of data domains and licenses, and restrained fidelity claims.The proposed benefits include broader access to motion content in animation, education, assistive design, and scientific visualization; the paper contributes analysis and diagnostics, not a deployed generator.Source: M Potential Societal Impacts · pp. 26–27

2. Limitations

  • The paper identifies SIF’s scale invariance and small-sample noise, positive scores without source content, limited data scope, and adapted baselines as limitations.Source: §2 · p. 37
  • The paper says it provides sufficient information to reproduce its main results and specifies the training and test details needed to understand them.Its code is available, but the Truebones dataset cannot be released because of licensing restrictions; BONES-SEED and LAFAN1 are available from their providers under their licenses.Source: §2 · pp. 38–39
  • Section 5 and Appendix E report 95% bootstrap intervals over source skeletons and shuffle-test p-values, with paired comparisons using the same resampling.Source clips recur across triples, motivating resampling whole source skeletons.Source: §2 · p. 39
  • The original experiments used an NVIDIA RTX 4070 Super GPU, while additional controls and robot studies used one NVIDIA H200 GPU per job.Statistical analyses ran on CPUs.Source: §2 · p. 40
  • The paper reports that it conforms to the NeurIPS Code of Ethics, discusses societal impacts in Appendix M, and considers its work to pose no special release risks.It states that it does not involve crowdsourcing or human-subject research, and that LLM use was limited to writing, editing, or formatting.Source: §2 · pp. 40–43
  • The paper credits and respects licenses for its existing datasets and tools, and says new assets are documented alongside them.Truebones Zoo cannot be redistributed; BONES-SEED is used under its academic research license, LAFAN1 under CC BY-NC-ND 4.0, and GMR under MIT.Source: §2 · pp. 41–42
Loading 2609.37297v1…