PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFM: Semantic-Aware Refinement Flow Model for Motion Retargeting

Jingxiang Qu, Lucie Taglienti, Evan Atherton

arXiv:2609.32068v1cs.CVcs.AIcs.GR

TL;DR

Motion retargeting lacks dense paired supervision and a unique target motion, even though results must preserve semantics while meeting target-specific constraints. ReFM uses canonicalization, a pretrained cross-character semantic encoder, and energy-guided progressive refinement; experiments report reduced self-penetration while preserving semantics, including refinement of HumanIK initializations.

  • Problem

    Sparse paired retargeting supervision and the absence of a unique target make it difficult to learn source semantics and define a definitive regression objective.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Method

    ReFM canonicalizes motion, learns a frozen cross-character semantic representation, and progressively refines initialized target motion under explicit motion-energy criteria.

    Source: §1 · p. 2, §4.3 · p. 7

  • Results

    ReFM consistently reduces self-penetration while preserving motion semantics, and further refines motions initialized by HumanIK.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The experiments support progressive refinement as a way to improve different retargeting initializations under semantic, physical, and temporal constraints.

    Source: §6 · p. 11

    The semantic encoder relies on limited, sometimes imperfect cross-character correspondences, and Semsim is not independent of the training objective.

    Source: §6 · p. 11

Abstract

from arXiv · show

Motion retargeting transfers motion across characters with different skeletal structures while preserving semantic intent and physical plausibility. Despite recent progress, two fundamental questions remain: (i) how can reliable source-motion semantics be learned without high-quality paired retargeting data, and (ii) how should retargeting be formulated when no reliable paired motion can serve as a definitive regression objective? Existing methods commonly preserve semantics by constraining predictions toward copied motions. However, such initializations entangle useful articulation cues with artifacts caused by mismatched skeletal proportions and body geometry. Moreover, directly regressing a final motion in one forward pass is restrictive because retargeting is inherently underdetermined, and the desired solution must balance semantic fidelity with target-specific physical and temporal constraints rather than match a unique paired target. Motivated by these limitations, we propose ReFM, a source-mesh-agnostic, energy-guided model that reformulates motion retargeting as progressive refinement. First, an SO(3) canonicalizer removes redundant global-orientation variations. Second, a cross-character semantic encoder, pretrained through contrastive learning, provides a character-invariant representation for both optimization guidance and semantic evaluation. ReFM then progressively refines an initialized target motion through a learned flow guided by semantic consistency, physical plausibility, temporal coherence, and minimal motion modification. The framework is compatible with different initialization strategies, including both direct motion copying and Autodesk HumanIK, an industry-standard full-body inverse-kinematics retargeting system.

1 Introduction

Motion retargeting must preserve semantics across different body structures, but sparse paired supervision and the absence of a unique correct target complicate learning and evaluation. ReFM addresses these challenges with source-mesh-agnostic semantic guidance and progressive, energy-guided refinement.

1 Introduction

  • Sparse cross-character correspondences leave semantic learning without dense supervision, while copied rotations can introduce misalignment, missing contacts, and self-penetration.Because these copied motions can carry geometric artifacts, low-level similarity to them may propagate errors into the result.Source: §1 · pp. 1–2
  • Retargeting is underdetermined: satisfactory targets must jointly meet semantic, geometric, temporal, and physical constraints rather than match a unique paired motion.A one-step prediction offers no mechanism to reject detrimental changes or continue improving an initialization.Source: §1 · p. 2
  • ReFM combines canonicalization, cross-character semantic modeling, and energy-guided progressive refinement without requiring the source character mesh.It is designed to improve both direct-copy and industry-standard initializations.Source: §1 · p. 2

2 Related Work

Prior work addresses retargeting through geometry-aware methods that preserve contacts and interactions, and skin-agnostic methods that work without source-mesh information. ReFM follows the latter setting while combining canonicalization, semantic representation learning, and progressive refinement.

Skinned Motion Retargeting

Skin-Agnostic Motion Retargeting

  • Skin-agnostic methods address pipelines where source motion is available without its original mesh or skinning information.Examples include NKN [Villegas et al., 2018], SAN [Aberman et al., 2020], SAME, and PAN [Hu et al., 2024].Source: §2 · p. 3
  • ReFM uses the source-mesh-agnostic setting while explicitly canonicalizing motion, learning cross-character semantics, and progressively refining copied motion.R2ET and M-R2ET also combine semantic losses with target-shape-aware geometric correction [Zhang et al., 2023, 2024b].Source: §2 · p. 3

3 Preliminaries

Skin-agnostic retargeting seeks an admissible target motion that preserves source semantics while meeting target structural and geometric requirements, rather than regressing to a unique paired target. Group canonicalization removes nuisance variations before learning.

3.1 Skin-Agnostic Motion Retargeting

  • Skin-agnostic retargeting maps source motion, source and target skeletons, and the target mesh to an admissible target motion, without requiring the source mesh.The admissible set preserves source semantics while satisfying target structure and geometry.Source: §3.1 · p. 3

3.2 Group Canonicalization

  • ReFM treats yaw changes in global facing direction as nuisance transformations, while pitch and roll may retain semantic information.Canonicalization maps motions differing only in facing direction to a shared space while preserving pitch, roll, and parent-relative articulation.Source: §3.2 · pp. 3–4
  • The framework overview places SO(3) canonicalization before semantic encoding and progressively applies local corrections to reduce a joint energy.That energy includes semantic deviation, self-penetration, and temporal inconsistency.Source: Figure 1 · p. 4

4 Methodology

ReFM operates without source-mesh geometry, first canonicalizing motion and then learning cross-character semantics to guide refinement. Its learned flow iteratively updates an initialized target motion under explicit energy criteria rather than regressing a unique paired target.

4 Methodology

  • ReFM requires source motion, source and target skeletons, and the target mesh, but not source-mesh geometry.Its goal is a target motion that preserves semantics while remaining temporally coherent and physically plausible.Source: §4 · p. 4

4.1 SO(3) Canonicalizer

  • The parameter-free SO(3) canonicalizer removes redundant yaw while preserving pitch, roll, and parent-relative joint rotations.It applies the same first-frame yaw correction across the sequence and is exactly invertible.Source: §4.1 · p. 5

4.2 Cross-Character Semantic Encoder

  • The semantic encoder learns transferable motion representations from same-identity motions across characters as positives and different identities as negatives.It is pretrained with a supervised-contrastive variant, then frozen; its embedding distance defines semantic energy and similarity is used as a consistency metric.Source: §4.2 · p. 6

4.3 Energy-Guided Refinement Flow

  • Five energy terms jointly guide semantic fidelity, physical plausibility, temporal coherence, and minimal modification from the initialization.The terms penalize semantic discrepancy, initialization deviation, target self-penetration, temporal inconsistency, and excessive trajectory deformation.Source: §4.3 · p. 7
  • The refinement flow predicts per-frame, per-joint velocities conditioned on source semantics, both skeletons, and target-mesh shape features.A spatio-temporal graph transformer uses skeletal-hierarchy and temporal attention.Source: §4.3 · p. 7
  • Because no unique ground-truth refinement trajectory exists, ReFM derives local training directions from the differentiable motion energy.The negative energy gradient is projected onto the valid rotation tangent space and normalized; the learned field approximates that descent direction.Source: §4.3 · p. 7
  • At inference, ReFM accepts candidate updates only when they lower explicit motion energy and stops when no tested step improves it.The resulting refinement field describes iterative dynamics, not a probability distribution.Source: §4.3 · p. 8

5 Experiments

Experiments evaluate ReFM quantitatively on Mixamo and qualitatively on ScanRet, comparing retargeting methods, testing the semantic encoder, and ablating its components.

5.1 Comparison with Existing Retargeting Methods

  • ReFM-Copy lowers Pen from 0.140 to 0.117 (~16%) with Semsim 0.998, while ReFM-HumanIK lowers Pen from 0.141 to 0.117 (~17%) with Semsim 0.996.Curv and MSE increase relative to the corresponding initializations, so the metrics reflect a trade-off rather than uniform improvement.Source: Table 1 · p. 8, §5.1 · p. 9
  • Ground-truth motions can themselves show self-penetration; ReFM-HumanIK reduces the Body Jab Cross hand–upper-body interference and Dancing arm–leg intersection while retaining characteristic poses.The qualitative examples support target-aware geometric correction rather than regression toward potentially imperfect paired motions.Source: Figure 4 · p. 9, §5.1 · pp. 9–10

5.2 Effectiveness of Semantic Encoder

  • On held-out motion groups, cross-character positives score cosine similarity 0.9969 ± 0.0202 versus 0.0859 ± 0.2911 for different-motion negatives, with compact intra-motion clusters.These results indicate that the pretrained encoder captures motion-dependent semantics while remaining largely invariant to character identity.Source: §5.2 · p. 10
  • Despite near-ceiling Semsim for naive copy and HumanIK, participants rank both below ReFM-HumanIK and R2ET, so Semsim is not a measure of human semantic judgment.The authors do not claim a clip-level correlation between participant rankings and Semsim.Source: §5.2 · p. 10

5.3 Ablation Study

  • Removing canonicalization raises Pen to 0.130 under copy and 0.131 under HumanIK; one-step prediction yields Pen 0.137 versus 0.117 and 0.136 versus 0.117, respectively.Removing the semantic component jointly removes semantic conditioning and Esem, lowers Semsim, and increases penetration; progressive refinement primarily improves geometric correction, not all similarity metrics.Source: §5.3 · p. 10, Table 2 · p. 11

6 Conclusion and limitations

ReFM combines SO(3) canonicalization, cross-character semantic representation learning, and energy-guided progressive refinement. The authors report reduced self-penetration with preserved motion semantics, while noting limits in the semantic encoder and yaw-only canonicalization.

6 Conclusion and limitations

  • ReFM progressively refines different initializations under semantic, physical, and temporal constraints, and experiments report reduced self-penetration while preserving motion semantics.The framework can further refine motions initialized by HumanIK.Source: §6 · p. 11
  • The semantic encoder uses limited, sometimes imperfect cross-character correspondences, and Semsim reuses the frozen encoder defining Esem, so the metric is not fully independent of training.The authors propose larger, more diverse motion corpora and joint interpretation with human evaluation and physical metrics.Source: §6 · p. 11
  • The current canonicalizer handles only the yaw subgroup of SO(3); tasks with additional ground, contact, or environment constraints may require redefining the task-relevant transformation group.The proposed principle would canonicalize nuisance transformations while preserving constraint-relevant components.Source: §6 · p. 11

A Evaluation of the Cross-Character Semantic Encoder

The encoder is evaluated on motion identities held out from pretraining, testing cross-character matching and discrimination between distinct executions. Its results support an operational representation of motion semantics, while not directly testing broader semantic equivalence.

Evaluation of the Cross-Character Semantic Encoder

  • Evaluation partitions Mixamo and ScanRet clips by motion identity, ensuring no validation identity appears during encoder pretraining.A motion identity is one underlying motion performed by multiple characters; an action category may contain distinct identities.Source: A Evaluation of the Cross-Character… · p. 14
  • Positive pairs match the same identity across characters; negatives use different identities, including distinct executions within the same action category.Because separately performed but semantically equivalent clips are not positives, this protocol tests cross-character matching and fine-grained discrimination, not broader semantic equivalence.Source: A Evaluation of the Cross-Character… · p. 14
  • Same-identity motions performed by different characters achieve cosine similarity 0.9969 ± 0.0202, while different identities have substantially lower similarity and form separated motion groups.The held-out results support using motion identity as an operational representation of semantics for refinement guidance and semantic-consistency evaluation.Source: A Evaluation of the Cross-Character… · p. 14, Table 3 · p. 14
  • The encoder is a representation of motion meaning, not a predictor of target motions, and ReFM receives no paired target-motion supervision from the held-out evaluation set.The held-out identities include fine-grained negatives such as different kicking motions within the same category.Source: A Evaluation of the Cross-Character… · pp. 14–15

B Implementation Details

Implementation combines a frozen semantic encoder with an energy-guided refinement field. Training approximates local energy-descent directions, while inference accepts only energy-reducing updates and constrains quaternion edits to selected joints.

Implementation Details

  • Additional engineering details, including data loading, optimization, numerical stabilization, and checkpoint management, are provided in the open-source project.Source: B Implementation Details · p. 15
  • Quaternion gradients are projected onto the unit-quaternion tangent space, then editable joints are restricted to the core22_limbs mask and normalized after updates.The mask permits edits to major arm and leg chains while keeping remaining joints fixed to the initialization.Source: B Implementation Details · p. 15
  • Training uses local energy-descent directions as supervision because paired progressive-refinement trajectories are unavailable, rather than regressing a unique final target motion.Four intermediate refinement states are generated using η = 0.05, and the target direction is normalized per sample.Source: B Implementation Details · pp. 15–16
  • The refinement field uses six spatio-temporal graph Transformer blocks, while a frozen, four-layer graph Transformer semantic encoder produces 128-dimensional clip embeddings.The refinement field has hidden dimension 320, four attention heads, and MLP dimension 768; a frozen shape encoder supplies target-mesh conditioning.Source: B Implementation Details · p. 16
  • The refinement field is trained for 40 epochs with Adam at learning rate 10^-3, using an effective batch size of 32 across eight NVIDIA T4 GPUs.Mixamo training uses 2,175 clips and takes roughly 14 GPU-hours per device; ScanRet uses identical optimization settings.Source: B Implementation Details · p. 16
  • The energy combines penetration and initialization preservation with temporal smoothness, trajectory curvature, and semantic preservation.The one-sided curvature penalty suppresses only curvature introduced beyond the initial motion; the semantic encoder is frozen and yaw-canonicalized.Source: B Implementation Details · p. 16
  • At inference, candidate step sizes are evaluated using the complete motion energy, and refinement stops if no nonzero update reduces it.The candidate set is {0, 0.02, 0.05, 0.1}, with at most four refinement steps; including zero guarantees non-increasing accepted energy.Source: B Implementation Details · pp. 16–17

C Sensitivity to the Semantic Energy Weight

The semantic-energy weight is varied for ReFM-HumanIK while other energy weights and training configurations remain fixed. Across the tested range, semantic similarity and physical and temporal metrics remain stable.

Sensitivity to the Semantic Energy Weight

D Experimental Settings

Experiments use Mixamo and ScanRet with a shared skeleton topology and separate motion-retargeting and semantic-encoder splits. Evaluation reports semantic consistency, physical plausibility, temporal curvature, and ground-truth position error.

Experimental Settings

  • Mixamo and ScanRet jointly contain 113 characters and approximately 11,973 motion sequences, converted to a shared 65-joint topology.Sequences are cropped or padded to 60 frames; semantic-encoder training and validation are partitioned by motion identity.Source: D Experimental Settings · p. 17
  • Mixamo evaluation uses 222 paired cases from a character-motion-instance-disjoint split, including three characters never animated during training; ScanRet examines real-human motions.The test data is also excluded from semantic-encoder pretraining.Source: D Experimental Settings · p. 17
  • Retargeting quality is assessed with Semsim, penetration rate, trajectory curvature, and joint-position error to ground truth.Because the frozen encoder also defines semantic energy, Semsim primarily measures retained semantic consistency while other motion aspects improve, rather than serving as fully independent evaluation.Source: D Experimental Settings · p. 18

E Introduction of Evaluation Metrics

The evaluation pairs semantic and geometric plausibility metrics as primary measures with auxiliary motion-correspondence and temporal-jitter diagnostics. Frame-level comparisons show how ReFM’s localized refinement addresses geometric conflicts while preserving transferred poses.

E Introduction of Evaluation Metrics

  • Lower penetration rate and higher semantic similarity are the primary indicators, while MSE and curvature are complementary diagnostics.Metrics are computed on a unified 65-joint representation after forward kinematics or skinning as required.Source: E Introduction of Evaluation Metrics · pp. 18, 20
  • Semantic similarity compares canonicalized motions in a character-invariant embedding space, without requiring a unique target-motion reference.A larger cosine similarity indicates stronger preservation of source-action semantics.Source: E Introduction of Evaluation Metrics · p. 18
  • The penetration rate counts geometric violations on the posed target mesh, so a lower rate indicates better physical plausibility.The metric classifies target-mesh vertices as penetrating based on their signed position relative to relevant body regions.Source: E Introduction of Evaluation Metrics · pp. 18–19
  • Trajectory curvature is an auxiliary acceleration-based jitter indicator, not strict differential-geometric curvature, and depends on motion speed, frame rate, and spatial scale.An artificially low value can result from an excessively static motion, so curvature should be interpreted with semantic similarity and qualitative motion speed.Source: E Introduction of Evaluation Metrics · p. 19
  • MSE measures agreement with recorded target performance but is auxiliary because valid retargetings need not match one reference, which may itself contain errors.The implemented quantity is an averaged height-normalized Euclidean joint error over 65 joints.Source: E Introduction of Evaluation Metrics · p. 19

F.1 Frame-Level Qualitative Comparisons

  • Body Jab Cross comparisons show ReFM mitigating local penetration artifacts while largely preserving the transferred pose and overall motion semantics.Full-frame views show characteristic body configurations; zoomed views reveal local conflicts at frame 1 and improved geometric compatibility at frame 21.Source: §F.1 · pp. 21–22, Figure 5 · p. 21, Figure 6 · p. 21
  • For Dribble, ReFM selectively adjusts the arm–torso interaction to reduce penetration while retaining the characteristic pose.The refinement focuses changes on regions where the initialized result violates geometric constraints rather than reconstructing the complete motion.Source: §F.1 · p. 23, Figure 9 · p. 23, Figure 10 · p. 23
  • In Dancing, ReFM-HumanIK produces cleaner hand–leg separation than several baselines while remaining close to the source-derived pose.The views contrast visible intersections in some baselines with other methods’ larger local arm changes; overall crouched posture is broadly retained.Source: §F.1 · pp. 24–25, Figure 11 · p. 24, Figure 12 · p. 24

F.2 Results on ScanRet Dataset

On ScanRet, where direct copying and HumanIK already yield limited self-penetration, ReFM generally accepts few or no refinement steps. Its two initialization variants preserve characteristic poses and temporal progression while applying target-aware corrections when needed.

F.2 Results on ScanRet Dataset

  • ScanRet’s relatively modest geometric variation and low initial penetration mean it generally needs substantially less correction than Mixamo.Table 5 reports penetration before and after refinement, accepted steps, and the proportion of sequences needing zero steps; Mixamo’s penetration rate is approximately 0.140.Source: §F.2 · p. 25, Table 5 · p. 25
  • ReFM’s energy-monitored inference accepts an update only when it lowers motion energy, allowing many ScanRet sequences to stop at zero steps.This preserves an already satisfactory initialization rather than forcing unnecessary changes.Source: §F.2 · p. 25, Table 5 · p. 25
  • ReFM-Copy and ReFM-HumanIK both preserve characteristic poses and temporal progression across ScanRet sequences despite their different initializations.The visualizations show consistent major body configurations, limb articulation, and overall motion trajectories over time.Source: §F.2 · p. 25

F.3 Inference-Time Analysis

The analysis compares learned refinement with direct energy optimization, tests penetration improvements across initializations and target familiarity, and adds a user study of perceived motion quality, penetration, and semantic preservation.

F.3 Inference-Time Analysis

  • Figure 13 compares ground-truth animations with ReFM-Copy and ReFM-HumanIK across uniformly sampled frames, focusing on semantic preservation and temporal consistency.Self-penetration is relatively infrequent in these ScanRet examples.Source: Figure 13 · p. 26
  • Against direct optimization, ReFM-Copy lowers penetration from 0.131 to 0.117 and raises semantic similarity from 0.994 to 0.998, while MSE changes from 0.085 to 0.086.Average inference time falls from 4.3 s to 0.52 s per motion, an approximately 8.3× speedup; the learned flow predicts correction directions instead of repeatedly differentiating the full energy.Source: §F.3 · p. 26, Table 6 · p. 27

F.4 Statistical Reliability of the Penetration Improvement

  • ReFM-Copy reduces mean penetration by approximately 0.023 and ReFM-HumanIK by 0.024, relative reductions of 16% and 17%, respectively.Both cluster-bootstrap confidence intervals lie above zero, and positive median reductions indicate gains were not confined to a few large-improvement sequences.Source: §F.4 · p. 27, Table 7 · p. 27

F.5 Generalization to Unseen Characters and Motions

  • On unseen target characters, ReFM-Copy reduces penetration from 0.090 to 0.080 and ReFM-HumanIK from 0.091 to 0.080; ReFM-Copy semantic similarity is 0.998 on both seen and unseen targets.The evaluation is predominantly on unseen motions (218 of 222 pairs), but penetration comparisons across target groups are limited because their ground-truth penetration levels differ substantially.Source: §F.5 · pp. 27–28, Table 8 · p. 28

F.6 User Study

  • ReFM-HumanIK receives the lowest mean rank across all three user-study criteria, with an overall average rank of 2.13.Against HumanIK initialization, semantic-preservation rank improves from 3.30 to 2.02, self-penetration rank from 2.66 to 2.35, and overall motion-quality rank from 2.39 to 2.03.Source: Table 9 · p. 28, §F.6 · p. 29
Loading 2609.32068v1…