PaperlayerView exact revision on arXiv

Source-linked AI summary

Skinned Motion Retargeting via Artifact-driven Kinematic Prior Refinement

Seokhyeon Hong, Chaelin Kim, Inseo Jang, Soojin Choi, Junyong Noh

arXiv:2609.06517v1cs.GR

TL;DR

Motion retargeting must transfer motion across characters while accounting for artifacts that can arise on the target mesh after skinning. The paper combines a shared skeleton-agnostic motion prior with posed-geometry artifact cues and geometry-aware decoding. Experiments report improved kinematic accuracy and geometric plausibility across diverse characters.

  • Problem

    Motion retargeting must transfer source motion across characters with different skeletons and body shapes, but joint-plausible motion can still produce target-mesh artifacts after skinning.

    Source: §1 · p. 1

  • Method

    The framework combines a shared skeleton-agnostic motion prior with explicit artifact-driven refinement and joint-aligned target-geometry context.

    Source: §1 · p. 2, §3.3.1 · p. 6

  • Results

    The method improves kinematic retargeting accuracy and geometric plausibility across diverse skeletons, body shapes, and target characters.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The reported improvements extend across fixed and arbitrary skeleton settings and seen and unseen target characters.

    Source: §6 · p. 15

    Artifact observation and correction add computation, and the full pipeline runs only after target-side motion realization.

    Source: §5 · p. 13

Abstract

from arXiv · show

Motion retargeting aims to transfer a source motion to target characters with different skeletal structures, proportions, and body shapes. Although recent neural retargeting methods have improved flexibility across diverse skeletons, target-side geometric artifacts such as self-penetration remain difficult to resolve. Specifically, existing geometry-aware approaches often rely on fixed skeleton templates or implicit geometry-conditioned prediction, requiring a single network to account for target geometry deformation, detect target-side artifacts, and predict the corresponding correction from target geometry alone, which limits their ability to generalize across diverse skeleton structures and body shapes. In this paper, we present a geometry-aware motion retargeting framework that explicitly connects artifacts observed in the posed character geometry to motion refinement while preserving the flexibility of skeleton-agnostic neural retargeting. Our method first learns a motion embedding shared across different skeletons using a transformer-based retargeting autoencoder that transfers motion across arbitrary source--target skeleton pairs. Building on this kinematic motion prior, we introduce an artifact-driven refinement module that observes self-penetration on the posed target mesh and converts it into a corrective cue through a motion-to-vertex Jacobian. We further condition motion decoding on target geometry using skinning weight-based joint-aligned geometry features derived from the rest pose mesh. This design combines explicit target-side artifact reasoning with flexible geometry-aware decoding in a unified framework. Experiments on both fixed and arbitrary skeleton structure settings show that our method improves kinematic retargeting accuracy and reduces geometric artifacts, producing plausible motions across seen and unseen target characters.

1 Introduction

Retargeting must account for target-mesh artifacts that can persist even when motion appears plausible in joint space. The framework separates kinematic transfer, artifact observation, and motion refinement while retaining flexibility across skeletons and meshes.

1 Introduction

  • Joint-space plausibility does not ensure geometric compatibility after skinning, so retargeting must address both kinematic fidelity and target-side artifacts.Self-penetration is one such artifact.Source: §1 · pp. 1–2
  • The framework decomposes geometry-aware retargeting into kinematic transfer, artifact observation, and motion refinement rather than assigning the full task to one implicit model.This decomposition is designed to preserve flexibility across diverse skeleton structures and meshes with arbitrary vertex counts.Source: §1 · p. 2
  • A transformer-based autoencoder learns a motion embedding shared across skeletons from arbitrary source–target motion pairs without predefined skeleton structures.Self-attention between joints is motivated by the need to model full-body coordination as well as local joint relations.Source: §1 · p. 2
  • The refinement module converts artifacts observed on posed target geometry into a first-order corrective cue through a motion-to-vertex Jacobian.The cue estimates a motion-prior correction feed-forward rather than iteratively optimizing variables, and updates the shared embedding rather than per-joint features.Source: §1 · p. 2
  • Skinning weights associate target geometry with skeletal joints to provide joint-aligned geometric context during decoding without fixed templates or shared geometry parameterizations.This context preserves dimensional compatibility across diverse characters.Source: §1 · p. 2

2 Related Work

Prior work spans skeleton-agnostic kinematic retargeting and geometry-aware methods for penetration avoidance and contact preservation. This method augments flexible kinematic retargeting with explicit artifact observations from posed target geometry.

2.1 Kinematic Motion Retargeting

  • Skeleton-agnostic methods learn shared motion representations to retarget across diverse skeleton pairs, and this method adds explicit target-side geometric reasoning to that backbone.Prior approaches include NKN [Villegas et al. 2018], PMnet, SAN [Aberman et al. 2020], and SAME.Source: §2.1 · p. 3

2.2 Geometry-aware Motion Retargeting

  • Geometry-aware retargeting includes optimization-based penetration repair and learned geometry-conditioned approaches such as R2ET [Zhang et al. 2023b] and STaR [Yang et al. 2025].Other work preserves contact through anchors or pair-wise geometric relations.Source: §2.2 · p. 3
  • Unlike approaches that use geometry only as conditioning, this method observes artifacts on posed target geometry and translates them into a corrective cue while injecting joint-aligned geometry.The design retains flexibility across arbitrary skeletal structures and body shapes.Source: §2.2 · p. 3

3 Method

The framework transfers motion across structurally diverse skeletons, then refines the target motion using posed-mesh artifact cues and target-specific geometry. Its representations and training are designed to preserve a shared, skeleton-agnostic motion prior while conditioning the final motion on the target character.

3 Method

  • The framework handles arbitrary target skeleton structures and vertex counts by combining shared motion-embedding correction with target-specific geometry conditioning.The artifact-driven cue updates the shared embedding, which the pretrained decoder realizes for the target character.Source: §3 · p. 3, §3.3 · p. 5

3.1 Data Representation

  • Skeleton, motion, and rest-pose geometry representations encode joint structure, frame-wise motion, and vertex positions, normals, and skinning weights.The motion representation includes joint rotations, positions and velocities, root features, and binary ground-contact labels.Source: §3.1 · pp. 3–4

3.2 Kinematic Motion Retargeting

  • The kinematic retargeter maps motion from a source skeleton to a structurally different target skeleton through a shared motion representation.The design addresses fine-grained joint details, including end-effector orientations, while supporting differences in joint count, hierarchy, and bone lengths.Source: §3.2 · p. 4

3.2.1 Architecture.

  • Global self-attention lets joint tokens and a learnable motion token exchange information across the skeleton without fixed skeletal structures.The target-side decoder conditions the shared motion embedding on target skeleton tokens to produce retargeted motion.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5

3.2.2 Training.

  • Training combines motion reconstruction, temporal smoothness, contact, and contrastive embedding losses to preserve motion details and distinguish different source motions.The contrastive loss pairs embeddings from samples sharing the same underlying source motion as positives and treats other samples as negatives.Source: §3.2.2 · p. 5

3.3 Artifact-driven Geometry-aware Motion Refinement

  • The refinement module observes artifacts on the posed target geometry and translates them into corrective cues for motion refinement rather than relying solely on rest-pose geometry.It predicts an update in the shared motion embedding, keeping correction within the skeleton-agnostic prior.Source: §3.3 · p. 5

3.3.1 Artifact-driven Corrective Cue.

  • For each arm or leg query vertex, the method finds a nearest reference vertex and uses a motion-to-vertex Jacobian to map local penetration displacement into an embedding-space cue.The cue indicates a first-order change to the shared motion embedding that can reduce observed penetration.Source: §3.3.1 · p. 6

3.3.2 Geometry-conditioned Refinement.

  • Rest-pose vertex features are aggregated into joint-aligned geometry context through skinning weights, then used to enrich target skeleton tokens for decoding.This provides pose-independent target shape context, such as body proportions and limb thickness, distinct from posed-geometry artifact cues.Source: §3.3.2 · pp. 6–7

4 Experiments

Across fixed and arbitrary skeleton settings, experiments report stronger kinematic retargeting and reduced geometric artifacts, with ablations and additional tests probing the framework’s components and scope.

4.2.1 Kinematic Motion Retargeting.

  • Our method achieved the best overall kinematic retargeting performance across fixed and arbitrary settings for seen and unseen characters.It consistently improved all metrics over SAME; R2ET-Kin led on JR in fixed settings, while our method had lower RT, JP, and FS and near-best JR.Source: §4.2.1 · p. 8, Table 1 · p. 8

4.2.2 Geometry-aware Motion Retargeting.

  • Our method achieved the best RT, JP, PR, and PD in fixed-structure geometry-aware retargeting, while maintaining competitive kinematic performance.Baselines without explicit geometry conditioning showed larger penetration artifacts; STaR’s penetration gains on seen characters did not generalize to unseen characters.Source: §4.2.2 · pp. 8–9, Table 2 · p. 9

4.2.3 User Study.

  • Our method received the highest user-study scores across semantics preservation, artifact avoidance, and overall quality.Twenty participants rated 50 videos on a five-point Likert scale.Source: §4.2.3 · p. 9, Table 3 · p. 9

4.3 Qualitative Results

  • Qualitative comparisons show preserved fine joint details and more reliable penetration removal in challenging poses, while some baselines lose detail, jitter, or alter motion semantics.The geometry-aware gains also held under strong self-body interactions and large shape variations.Source: §4.3.1 · p. 9, §4.3.2 · pp. 9, 11

4.4.1 Artifact-driven Geometry-aware Refinement Components.

  • The full refinement model achieved the lowest PR and PD on seen and unseen characters; static geometry, joint-aligned features, and Jacobian-guided updates contribute complementary benefits.Static geometry or Jacobian cues alone did not resolve deeper penetrations reliably, whereas their combination improved correction while skinning-based alignment improved the kinematic–geometry balance.Source: §4.4.1 · pp. 11–12, Table 4 · p. 11

4.4.2 Kinematic Motion Embedding.

  • The learnable motion token improved RT and JP reconstruction, while the contrastive embedding objective led JR in all four settings and JP in three.The contrastive variant had only marginal RT differences from the best results; dropping embedding loss yielded the best FS but worse JR and JP.Source: §4.4.2 · p. 12, Table 5 · p. 12, Table 6 · p. 12

4.4.3 Comparison to Optimization.

  • Learned refinement was 95.25× faster in Fixed-SC and 107.85× faster in Fixed-UC than iterative optimization, with comparable geometry correction and more faithful motion semantics.Original and approximately 5K-vertex decimated meshes had comparable results, with minor metric variations.Source: §4.4.3 · p. 13, Table 7 · p. 13, §4.4.4 · p. 13

4.5 Additional Results

  • Additional tests report penetration mitigation on out-of-distribution characters and self-reconstruction, where the method removes penetrations while preserving motion semantics.The out-of-distribution examples include asymmetric skeletons and characters from LaFAN1, SMPL, and RigNet-v1.Source: §4.5 · p. 13

5 Discussion

The discussion identifies computational overhead and residual motion and geometry artifacts as limitations. These constraints are most apparent for complex contacts, exaggerated morphologies, out-of-distribution poses, and corrections based on a local first-order cue.

5 Discussion

  • Penetration detection is the dominant overhead, and its cost increases substantially with mesh vertex count.The full pipeline runs only after target-side motion realization; the neural refinement components remain relatively lightweight.Source: §5 · pp. 13–14
  • Highly exaggerated skeletal morphologies and out-of-distribution poses can leave self-penetration, inaccurate poses, and foot sliding unresolved.In complex multi-contact scenarios, penetration avoidance can adversely affect contact preservation.Source: §5 · p. 14, Figure 13 · p. 15
  • Geometry-aware refinement can increase foot sliding, especially for unseen characters, when artifact reduction conflicts with preserving provisional kinematics.The discussion proposes broader training-data coverage and contact-aware inverse kinematics as future work.Source: §5 · p. 14
  • The corrective signal begins as a local, linear approximation and does not explicitly model coordinated adjustments across body parts.The authors suggest exploring richer formulations beyond first-order sensitivity.Source: §5 · pp. 14–15

6 Conclusion

The conclusion presents a geometry-aware retargeting framework that separates kinematic transfer, target-side artifact observation, and motion refinement. It reports improved kinematic accuracy and geometric plausibility across diverse skeletons, body shapes, and target characters.

6 Conclusion

  • The framework decomposes retargeting into kinematic transfer, target-side deformation, penetration observation, corrective-cue derivation, and geometry-aware decoding.A transformer-based kinematic autoencoder learns a skeleton-agnostic motion embedding as a transferable prior.Source: §6 · p. 15
  • Experiments show improved kinematic retargeting accuracy and geometric plausibility across diverse skeletons, body shapes, and target characters with different skeleton and mesh structures.The refinement framework converts posed target-side artifacts into explicit cues using a motion-to-vertex Jacobian, while skinning weights provide joint-aligned geometric information with dimensional flexibility.Source: §6 · p. 15

A.1 Network Architecture

The kinematic autoencoder uses a three-layer transformer architecture, while the geometry-aware refinement components use two-layer MLPs.

A.1 Network Architecture

  • The encoder and decoder share an architecture of three transformer encoder layers with pre-layer normalization, self-attention, and feed-forward blocks.Residual connections follow the self-attention and feed-forward blocks.Source: §A.1 · p. 17
  • The transformer uses 8 attention heads, a 256-dimensional hidden layer, a 1024-dimensional feed-forward layer, and a 32-dimensional motion embedding.Source: §A.1 · p. 17
  • The geometry-aware refinement components 𝜙corr(·), 𝜙geo(·), and 𝜙enr(·) are two-layer MLPs with GELU activation between the linear layers.Source: §A.1 · p. 17

A.2 Penetration Computation

The appendix reports motion-classification accuracy and describes a part-wise nearest-neighbor penetration detector. Its distance threshold trades some missed penetrations for fewer false positives and cleaner training signals.

A.2 Penetration Computation

  • Penetration detection uses limb-region query vertices and reference vertices from the remaining body parts, with batched nearest-neighbor searches supporting different mesh resolutions.Invalid vertices are masked, selected vertices are packed with batch-wise padding, and an expanded query-side bounding box prunes reference vertices.Source: §A.2 · p. 17
  • A query vertex is considered penetrating when its displacement has a positive projection onto the reference normal, its nearest-neighbor distance is below a threshold, and normal similarity is below a threshold.Source: §A.2 · p. 17
  • Distance filtering can miss long-range penetrations, but suppresses false positives that would create inconsistent correction supervision and destabilize learning.The authors report that repeated detection lets the model address stronger, more reliable penetrations before weaker or previously missed cases.Source: §A.2 · p. 17
  • 0.983 motion classification accuracy for Ours, versus 0.950 for SAME.Source: §A.2 · p. 17

A.3 Training and Evaluation

The training and evaluation setup specifies loss weights and character sets, while the applications section examines the motion embedding in downstream tasks.

A.3 Training and Evaluation

  • Training uses twelve separately weighted losses, with weights ranging from 0.01 to 10.0.The listed weights cover kinematic, contact, geometry, and latent-related terms.Source: §A.3 · p. 17
  • Seven Mixamo characters train the refinement module, four evaluate it quantitatively, and eight more support qualitative results.Additional out-of-distribution characters come from LaFAN1, SMPL, and RigNet-v1 (sample 2977) [Xu et al. 2020].Source: §A.3 · p. 17

B Applications

  • The embedding is examined in classification, arithmetic operations, and motion control.These tasks are presented as tests of the embedding's downstream utility.Source: B Applications · p. 17

B.1 Classification

The motion embedding is evaluated on CMU motion classification, where its classifier outperforms the SAME classifier on the shared validation set.

B.1 Classification

  • A single-layer encoder-decoder block transformer classifies embeddings, trained on 216 CMU clips and evaluated on 60.The split follows the same protocol as SAME.Source: §B.1 · p. 17
  • On the same validation set, a classifier trained on our embedding achieved 0.983 accuracy (59/60), exceeding the classifier trained on SAME, which achieved 0.950 (57/60).Source: §B.1 · p. 17

B.2 Arithmetic Operations

The shared motion embedding supports arithmetic editing by decoding linear combinations into semantically modified motions.

B.2 Arithmetic Operations

  • The embedding places semantically similar motions nearby, supporting arithmetic operations and human-interpretable motion editing.The authors describe the resulting editing as intuitive.Source: §B.2 · p. 18
  • Decoding a linear combination of motion embeddings produces a semantically modified motion, with scalar w controlling modification strength.The described operation removes a semantic component of one motion.Source: §B.2 · p. 18

B.3 Motion Control

Motion matching in the shared embedding supports interactive control across heterogeneous motion categories and character switches.

B.3 Motion Control

  • Motion matching supports smooth transitions among locomotion, crouching, and kicking, plus on-the-fly character switching.The controller uses a database combining heterogeneous motion categories.Source: §B.3 · p. 18
Loading 2609.06517v1…