Source-linked AI summary
Neural Kinematic Networks for Unsupervised Motion Retargetting
Ruben Villegas, Jimei Yang, Duygu Ceylan, Honglak Lee
TL;DR
Motion retargetting must transfer movement between characters with different skeleton proportions without requiring expensive paired training sequences. The paper combines recurrent networks with a Forward Kinematics layer and adversarial cycle consistency to learn unsupervised, online retargetting, reporting state-of-the-art results on varied Mixamo motions and characters. It also demonstrates transfer from monocular human videos using an off-the-shelf 3D pose estimator.
Problem
Motion retargetting must adapt motions between characters with different bone lengths and proportions, while paired cross-skeleton training sequences are expensive to collect.
Method
A Neural Kinematic Network uses two RNNs, a Forward Kinematics layer, and adversarial cycle consistency to discover target rotations without ground-truth rotations or paired motions.
Results
The method achieves state-of-the-art results across varied Mixamo motions and characters and demonstrates retargetting from monocular human videos using an off-the-shelf 3D pose estimator.
Takeaways & Limitations
Forward kinematics helps produce skeleton-independent target rotations, cycle consistency prevents regression to input end-effector positions, and adversarial training supports realistic motion.
Takeaways & Limitations
The method assumes a fixed number of joints, an environment without physical constraints such as gravity, and 3D joint-coordinate input.
Abstract
from arXiv · showhide
We propose a recurrent neural network architecture with a Forward Kinematics layer and cycle consistency based adversarial training objective for unsupervised motion retargetting. Our network captures the high-level properties of an input motion by the forward kinematics layer, and adapts them to a target character with different skeleton bone lengths (e.g., shorter, longer arms etc.). Collecting paired motion training sequences from different characters is expensive. Instead, our network utilizes cycle consistency to learn to solve the Inverse Kinematics problem in an unsupervised manner. Our method works online, i.e., it adapts the motion sequence on-the-fly as new frames are received. In our experiments, we use the Mixamo animation data to test our method for a variety of motions and characters and achieve state-of-the-art results. We also demonstrate motion retargetting from monocular human videos to 3D characters using an off-the-shelf 3D pose estimator.
1. Introduction
Motion retargetting must adapt movement between characters with different kinematic structures while preserving natural, realistic motion. The paper proposes a Neural Kinematic Network using forward kinematics, recurrent networks, and unsupervised adversarial cycle consistency.
- Motivation: Motion retargetting transfers movement from a source character to a target character with different bone lengths and proportions.The target has the same topology but a different kinematic structure.
- Motivation: Skeletal differences require disentangling skeleton-independent motion features and adapting them to the target in one shot.The intended process avoids post-processing optimization and hand-tuning.
- Motivation: Retargetted motion must remain natural and realistic-looking, a long-standing challenge in animation.
- Approach: The proposed architecture combines two RNNs with an analytic Forward Kinematics layer to encode motion features and decode target joint rotations.The Forward Kinematics layer takes joint rotations and a target skeleton T-pose to render the resulting motion.
- Contributions: The Neural Kinematic Network discovers target joint rotations, solving the Inverse Kinematics problem without ground-truth rotations during training.
- Contributions: A sequence-level adversarial cycle consistency objective enables unsupervised training without paired motions from different skeletons.
2. Related work
Prior retargetting methods use optimization, per-frame inverse kinematics, or neural motion synthesis, but neural joint-coordinate generation requires additional post-processing. The paper instead uses a forward kinematics layer to disentangle articulated-skeleton rotations.
- Optimization-based methods: Earlier retargetting methods formulated whole-sequence spacetime optimization or combined per-frame inverse kinematics with smoothing curves.
- Optimization-based methods: Other approaches added dynamics constraints or performed online retargetting through sequential per-frame inverse-kinematics updates.
- Learning-based methods: Neural motion models have generated human motions, but direct xyz-coordinate generation requires further post-processing for motion retargetting.
- Differentiable representations: The paper relates its approach to differentiable rendering and physics work by using a differentiable forward kinematics layer to disentangle hierarchical skeleton rotations.
- Differentiable representations: Figure 2 depicts forward kinematics starting from a T-pose skeleton and rotating bones to achieve the desired output configuration.
3. Background
Forward kinematics computes joint positions from rotations and initial skeleton positions through recursive traversal of the skeleton. Inverse kinematics reverses this mapping, but is inherently ill-posed and may require iterative optimization.
- Forward kinematics: Forward kinematics recursively rotates a skeleton tree from the root joint to leaf joints to compute 3D joint positions.
- Forward kinematics: The forward-kinematics relation uses each joint’s rotation and offset from its parent to update its 3D position.The input T-pose provides the initial joint positions used by the relation.
- Inverse kinematics: Inverse kinematics computes joint rotations that place specified joints at desired target locations from initial positions.
- Inverse kinematics: Inverse kinematics is ill-posed because a target joint configuration may correspond to multiple rotations or none.Classical solutions often use numerical or analytical inverse-Jacobian optimization for the nonlinear forward-kinematics function.
4. Method
The method combines a neural kinematic network for skeleton-conditioned motion synthesis with adversarial cycle-consistency training for unsupervised motion retargetting. Its Forward Kinematics layer maps predicted rotations onto a target skeleton, while recurrent encoding and decoding support online synthesis.
- Neural kinematic network: The architecture uses two RNNs and an analytic Forward Kinematics layer to synthesize motion for characters with different skeleton structures.The encoder extracts motion features, and the decoder generates target-skeleton joint rotations and global motion parameters.
- Forward kinematics layer: The Forward Kinematics layer applies predicted joint rotations to a target skeleton and renders the resulting joint locations.This maps rotations to actual positions and encourages learning skeleton-independent motion features.
- Online motion synthesis: The recurrent synthesis module conditions each current motion step on previous steps through encoder and decoder hidden representations.The decoder outputs unit-quaternion rotations for the condition skeleton, along with estimated local and global motion.
- Adversarial cycle training: Training retargets a motion from skeleton A to B and back to A using adversarial and cycle-consistency objectives without paired cross-skeleton motions.The adversarial discriminator learns motion distributions for target skeletons, while cycle consistency penalizes differences between the original and reconstructed motions.
- Training objectives: The full objective combines adversarial, cycle-consistency, joint-twist, and velocity-smoothing losses.The twist term constrains rotation around a bone’s own axis, while smoothing constrains changes in global motion across consecutive timesteps.
5. Experiments
Experiments evaluate online retargetting across Mixamo characters, unseen-motion and unseen-character scenarios, neural baselines, qualitative examples, and human-video inputs. The proposed kinematic and consistency objectives improve motion quality while addressing skeleton-specific artifacts.
- Dataset and protocol: The Mixamo evaluation uses approximately 2400 motion sequences across 71 characters, with 1646 training sequences from seven characters.
- Dataset and protocol: Testing covers four combinations of whether the input motion and target character were seen during training.
- Baselines: Baselines include RNN and MLP models without the Forward Kinematics layer, both trained with adversarial cycle consistency, plus direct quaternion-and-velocity copying.
- Quantitative and ablation results: The Forward Kinematics layer with an autoencoder objective outperforms all standard neural-network baselines, while those baselines can ignore target bone lengths and produce stretching or shrinking.
- Quantitative and ablation results: Cycle consistency improves preservation of high-level motion properties by discouraging direct end-effector matching that can cause implausible limb stretching between characters of different heights.
- Quantitative and ablation results: The full adversarial cycle consistency objective performs best by combining cycle consistency with realism, producing motions indistinguishable from realistic target-character motions.
- Qualitative comparisons: The copying baseline respects target bone lengths and avoids global-motion drift but can produce foot floating for substantially different skeletons and does not scale to differing limits or topologies.
- Human-video retargetting: Human-video experiments retarget estimated 3D poses from Human 3.6M into Mixamo characters, using an off-the-shelf pose estimator and additional root-displacement information.
6. Conclusion and Future Work
The paper concludes that its neural kinematic network combines forward kinematics, cycle consistency, adversarial realism, and bone-twist constraints for unsupervised retargetting. It identifies fixed joint counts, physically unconstrained environments, and required 3D input as current boundaries.
- Conclusion: The network transfers xyz-coordinate motion from animations, motion capture, or monocular-video pose estimates to humanoid characters without the skeleton deformations seen in baselines.
- Future work: The method currently retargets a fixed number of joints, assumes no environmental physical constraints such as gravity, and requires 3D joint coordinates as input.
- Future work: Future work targets variable-joint handling, physics simulation for physically plausible motion, and end-to-end monocular-video input with view-invariant features.
A. Quantitative Evaluation per Motion Retargetting Scenario, and Analysis
The evaluation compares online retargetting across seen and unseen motions and characters, then analyzes performance by movement magnitude. Full adversarial cycle training improves generalization, especially to new target skeletons and higher-movement sequences.
- Scenario evaluation: The evaluation covers known or new input motions retargetted to characters seen or unseen during training.Tables 2 and 3 separate these four motion–character scenarios.
- Scenario evaluation: 12.9% improvement occurs for known motions retargetted to new characters, compared with 5.8% for known characters.These gains are reported for full adversarial cycle training versus cycle training alone.
- Scenario evaluation: 23.3% improvement occurs for new motions retargetted to new characters, while new motions retargetted to known characters show similar performance.The results indicate stronger generalization to unseen skeletons under full adversarial cycle training.
- Movement analysis: The movement analysis bins test examples by average ground-truth movement and compares mean square error against the copy-quaternions-and-velocities baseline.Figure 7 uses normalized average joint-location variance on the horizontal axis and MSE on the vertical axis.
- Movement analysis: When average joint-location variance exceeds 5, the proposed method substantially outperforms the baseline; at 5 or below, the baseline marginally performs better.The result supports modeling relationships between the input motion and target character rather than simply copying motion values.
B. Denoising 3D Pose Estimation by Motion Retargetting
The model is applied to estimated 3D poses to reduce temporal noise through motion retargetting. It denoises hand trajectories well and generally improves feet trajectories, though feet height can sometimes differ from the original estimate.
- Denoising procedure: The model retargets 3D poses estimated from Human3.6M videos back into the input motion skeleton for denoising.This application addresses pose estimators that process frames independently and ignore temporal correlations.
- Denoising results: Figure 8 presents end-effector trajectories for five Human3.6M test examples, with each row representing one example.The figure focuses on hand and feet trajectory denoising.
- Denoising results: Hand end-effectors are denoised well without training on such data, while feet denoising is good but can miss the original estimated feet height.The method also removes much of the shaking in the initially estimated 3D pose after retargetting.
C. Demo Video and Qualitative Motion Retargetting Evaluation
The paper provides a demo video for qualitative motion-retargetting results and directs readers to additional videos for more examples.
- Qualitative evaluation: The results demo video supplements the paper’s qualitative motion-retargetting evaluation, with additional videos available through the linked resources.The demo video contains audio.
D. Data collection process
The data collection process builds Mixamo training and testing sets from character-specific animation pages and evaluates predefined retargetting scenarios.
- Data collection: Training data comprise 1656 unique motion sequences distributed across seven characters.The testing set contains 68 unique sequences of at least four seconds, yielding 173 non-overlapping four-second clips, or 185 including final clips.
- Data collection: Table 4 records data collection for each character and animation page on the Mixamo website.
- Testing scenarios: Testing evaluates online retargetting under known-motion/known-character, known-motion/new-character, new-motion/known-character, and new-motion/new-character scenarios.These scenarios correspond to the quantitative evaluations summarized in Tables 5 and 6.
E. Architecture and training details
The implementation uses recurrent networks for motion processing and a convolutional discriminator for adversarial training.
- The RNN architectures use 2-layer Gated Recurrent Units with 512-dimensional hidden states.
- The discriminator is a 5-layer 1D fully convolutional network with size-4 kernels and stride-2 convolutions.
- Discriminator layers 1–4 use leakyReLU activations with a 0.2 leak and dropout with 0.7 keep probability.