PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Aware Networks for Deep Motion Retargeting

Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, Baoquan Chen

arXiv:2005.05732v1cs.CVcs.GRcs.LG

TL;DR

The paper addresses motion retargeting between homeomorphic skeletons with different structures and without paired motion examples. It introduces skeleton-aware representations and differentiable operators that map motions through a shared primal-skeleton latent space, enabling intra- and cross-structural decoding. The authors report effective motion retargeting and motion processing, with quantitative evaluation on synthetic paired motions, while noting limitations for very different T-poses and unseen object-interaction tasks.

  • Problem

    Motion retargeting across motion-capture domains is needed when skeleton structures, joint counts, lengths, and proportions differ, but existing methods do not automatically handle structural differences without manual correspondences.

  • Method

    The framework uses a deep static-dynamic motion representation and differentiable skeleton-aware convolution, pooling, and unpooling to encode homeomorphic skeleton motions into a shared primal-skeleton latent space for decoding.

  • Results

    The method performs unpaired retargeting between topologically equivalent skeletons with different joint counts and is quantitatively evaluated on a synthetic dataset containing paired motions across skeletons.

  • Takeaways & Limitations

    A common structured latent representation supports motion retargeting across homeomorphic skeleton domains and also serves broader motion-processing tasks such as denoising.

  • Takeaways & Limitations

    Retargeting is not effective for homeomorphic skeletons with very different motions and T-poses, and unseen complex object interactions are not retargeted properly.

Abstract

from arXiv · show

We introduce a novel deep learning framework for data-driven motion retargeting between skeletons, which may have different structure, yet corresponding to homeomorphic graphs. Importantly, our approach learns how to retarget without requiring any explicit pairing between the motions in the training set. We leverage the fact that different homeomorphic skeletons may be reduced to a common primal skeleton by a sequence of edge merging operations, which we refer to as skeletal pooling. Thus, our main technical contribution is the introduction of novel differentiable convolution, pooling, and unpooling operators. These operators are skeleton-aware, meaning that they explicitly account for the skeleton's hierarchical structure and joint adjacency, and together they serve to transform the original motion into a collection of deep temporal features associated with the joints of the primal skeleton. In other words, our operators form the building blocks of a new deep motion processing framework that embeds the motion into a common latent space, shared by a collection of homeomorphic skeletons. Thus, retargeting can be achieved simply by encoding to, and decoding from this latent space. Our experiments show the effectiveness of our framework for motion retargeting, as well as motion processing in general, compared to existing approaches. Our approach is also quantitatively evaluated on a synthetic dataset that contains pairs of motions applied to different skeletons. To the best of our knowledge, our method is the first to perform retargeting between skeletons with differently sampled kinematic chains, without any paired examples.

1 INTRODUCTION

Motion retargeting is needed across motion-capture setups whose skeletons differ in structure, joints, lengths, and proportions, but existing CNN operators do not naturally fit hierarchical, irregular skeletons. The paper introduces skeleton-aware processing and unpaired retargeting for topologically equivalent skeletons, with quantitative evaluation and motion-denoising results.

  • Motivation: Different motion-capture setups produce skeletons with varying structures, joint counts, bone lengths, and proportions, making cross-setup motion retargeting necessary.The paper also links cross-setup retargeting to combining datasets for setup-agnostic motion-processing models.
  • Motivation: Irregular connectivity and hierarchical organization make standard image-CNN convolution and pooling poorly suited to articulated-skeleton motion.The paper argues that skeleton structure and joint adjacency should be explicitly represented during motion analysis and synthesis.
  • Contributions: The framework introduces differentiable skeleton-aware convolution, pooling, and unpooling operators that learn local joint correlations in shallow layers and body-part correlations in deeper layers.Its shared latent representation is jointly learned by encoder-decoder pairs for each skeletal domain.
  • Research gap: Existing approaches cannot automatically retarget between skeletons differing in structure or joint count, so correspondences are manually specified and errors corrected by animators.The described correction process involves manipulating key frames, which the paper characterizes as tedious.
  • Contributions: The proposed architecture performs unpaired retargeting between topologically equivalent skeletons that may have different numbers of joints.The authors also demonstrate motion denoising and quantitatively evaluate retargeting on a synthetic dataset containing paired motions across skeletons.

2 RELATED WORK

Prior motion-processing methods use optimization, recurrent models, CNNs, graph networks, or hierarchical RNNs, but commonly rely on constraints, correspondences, or representations that do not fully encode skeleton structure. The paper positions its approach against these limitations in data-driven and unpaired motion retargeting.

  • Motion retargeting: Classical retargeting methods use spacetime optimization or framewise inverse kinematics with smoothing to satisfy motion constraints.These approaches formulate retargeting around hand-crafted kinematic constraints and simplifying assumptions.
  • Motion retargeting: Data-driven methods improve flexibility but include latent-transfer, autoencoder, recurrent, adversarial, and pose-movement-disentanglement approaches for motion retargeting.Unpaired methods use mechanisms such as forward kinematics, cycle consistency, adversarial training, or disentangled representations.
  • Cross-character retargeting: Retargeting to non-humanoid characters can require target-style human motion, key-pose matching, or manually defined body-part correspondences.These requirements constrain transfer across substantially different character structures.
  • Related transfer problems: Earlier mesh-transfer methods require multiple correspondences, whereas unpaired shape-space methods avoid pairing but address a different problem from temporal motion retargeting.The distinction is between mapping shape spaces and processing temporal animation sequences.
  • Neural motion processing: CNN-based motion representations may use global joint support, thereby ignoring joint connectivity and hierarchical skeleton structure.Joint-position representations may also require inverse kinematics, while joint rotations alone do not fully capture 3D pose without skeleton information.
  • Neural motion processing: Graph-convolution and hierarchical-RNN approaches process spatiotemporal skeleton data, but the cited hierarchical RNN uses a hand-crafted hierarchy and learned fully connected node merging.The related-work discussion contrasts these designs with explicit skeleton convolution and pooling.

3 OVERVIEW

The framework reduces homeomorphic skeletons to a common primal skeleton, encodes their motions into a shared latent space, and decodes them for intra- or cross-structural retargeting. Its deep representation preserves static and dynamic motion information while skeleton-aware operators model structure from local joints to global body parts.

  • Primal skeleton: Homeomorphic skeleton graphs are reduced to a common minimal graph by eliminating degree-two nodes along linear branches; this reduced graph is the primal skeleton.Pooling implements the reduction by removing degree-two nodes and merging adjacent edges.
  • Shared latent space: Motions from different homeomorphic skeletons are encoded into a deep representation independent of original structure and bone proportions, enabling retargeting without paired training data.The common latent space supports data-driven cross-domain translation.
  • Retargeting architecture: The shared latent representation can be decoded with a same-domain decoder for intra-structural transfer or another-domain decoder for cross-structural transfer.Target skeletons may differ in bone lengths or skeletal structure depending on the decoder domain.
  • Deep motion representation: The deep motion representation models each armature with dynamic rotation features and static offset features, preserving this static-dynamic organization through processing.The representation is designed for temporal motion sequences on articulated-skeleton graphs.
  • Deep skeletal operators: Skeleton-aware differentiable operators account for hierarchy and joint adjacency, learning local joint correlations in shallow layers and higher-level body-part correlations in deeper layers.Interleaved skeletal convolution and pooling produce temporal features associated with primal-skeleton joints.
  • Framework organization: The framework's motion representation and skeleton-aware operators support an architecture and loss functions for data-driven cross-structural motion retargeting.The paper describes the representation and operators separately from the retargeting architecture and losses.

4 SKELETON-AWARE DEEP MOTION PROCESSING

The framework represents articulated motion with static and dynamic armature features on a skeleton graph, then processes them using structure-aware convolution, pooling, and unpooling. These operators preserve skeletal topology while building lower-resolution deep features and restoring armature resolution.

  • Motion representation: Motion is represented by static skeleton offsets, dynamic joint rotations, and separately stored global root translations and orientations.The static component is time-independent, while the dynamic component is a temporal sequence of joint rotations.
  • Motion representation: Skeletons are modeled as tree graphs whose edges are armatures, with adjacency defined by kinematic-chain distance.Each adjacency list identifies edges within a specified tree distance from a centered armature.
  • Skeletal convolution: The dynamic and static branches use skeleton-aware convolutions, while dynamic convolutions additionally operate across contiguous temporal neighborhoods.Temporal weights are shared along time but not across armatures, allowing body-part-specific features.
  • Skeletal convolution: Dynamic convolution combines tiled static features with dynamic features, whereas the static branch uses only static inputs to keep its representation motion-independent.Both branches share the same connectivity map, maintaining dimensional and semantic consistency.
  • Skeletal convolution: Root convolution includes nearby armatures and end-effectors, injecting global root information into deeper dynamic features.This support reflects low-level correlations between global root motion and end-effector motion.
  • Topology-preserving skeletal pooling: Topology-preserving pooling merges adjacent edges along degree-2 branches, producing a homeomorphic skeleton with fewer armatures.Pooling regions are disjoint consecutive edge sequences, and the dynamic branch also downsamples temporally.
  • Topology-preserving skeletal pooling: Unpooling restores merged armatures by copying recorded feature activations, increasing skeletal resolution without adding information.The dynamic branch additionally upsamples along the temporal axis.
  • Evaluation: In denoising comparisons, skeleton-aware operators outperform Holden et al. [2016] for both Gaussian-noise and random-zero corruption.The reported improvements include smaller local joint-position errors, better global positions, and greater stability.

5 CROSS-STRUCTURAL MOTION RETARGETING

The paper formulates retargeting as unpaired translation between homeomorphic skeleton domains and uses domain-specific encoders and decoders around a shared, skeleton-independent latent representation. At test time, target static features are combined with source dynamic features to generate retargeted motion.

  • 5.1 Problem setting: Motion retargeting is formulated as unpaired cross-domain translation between skeletons with shared structure but potentially different bone lengths and proportions.The formulation assumes a homeomorphism between the source and target skeletal structures and no explicit cross-domain motion pairs.
  • 5.1 Problem setting: The goal is to map source skeleton offsets and joint rotations into rotations describing the motion on a target skeleton.The learned mapping is denoted G_A→B, with a symmetric reverse mapping learned concurrently.
  • 5.2 Network architecture: Each motion domain has a dynamic encoder, static encoder, decoder, and discriminator.The dynamic and static encoders separately process motion and skeletal properties before decoding domain-specific outputs.
  • 5.2 Network architecture: At test time, the target decoder combines source dynamic features with target static features to produce retargeted motion.This works because the learned deep encoding is independent of original skeletal properties and associated with a common primal skeleton.
  • 5.2 Network architecture: The architecture uses encoders and decoders for each domain while training information flow in both translation directions.Figure 9 presents the A → B direction, with the reverse direction obtained symmetrically.

Cross-structural motion translation is achieved by feeding the output of EQ

The described framework trains motion translation with end-effector and latent-consistency losses, then performs retargeting at test time. The latent-consistency term compares representations before and after translation.

  • Training applies an end-effectors loss to the translated motion.
  • A latent consistency loss compares the original latent representation with that of the translated motion.
  • Retargeting is performed at test time using the translated motion.

is performed by using DB to combine motion encoded by EQ

Training combines reconstruction, latent-consistency, adversarial, and end-effector losses to preserve motion content and target-domain plausibility. End-effector constraints and automatic inverse-kinematics cleanup address contact-related artifacts such as foot sliding.

  • Training losses: The reconstruction loss is applied within a motion domain to joint rotations and normalized joint positions computed through forward kinematics.The position term helps prevent error accumulation along the kinematic chain.
  • Training losses: Latent consistency encourages retargeted motion to retain the original clip’s dynamic features in the shared primal-skeleton representation.The common latent space is enabled by pooling features of consecutive armatures.
  • Training losses: Because retargeted motions lack ground truth in unpaired data, an adversarial discriminator evaluates whether decoded rotations look plausible for the target skeleton.Real examples come from the target motion domain, while translated outputs serve as fake examples.
  • Training losses: The end-effectors loss requires matching normalized end-effector velocities across source and retargeted skeletons.Normalization accounts for different character scales, and the constraint helps mitigate artifacts such as foot sliding.
  • Foot-contact cleanup: Automatic inverse-kinematics cleanup extracts foot-contact labels and fixes feet to their average contact-time positions.The paper reports the effects of the end-effector loss and IK cleanup in supplementary video.
  • Training losses: The full training objective combines reconstruction, latent-consistency, adversarial, and end-effector losses.The stated weights are λ_ltc = 1, λ_adv = 0.25, and λ_ee = 2.

6 EXPERIMENTS AND EVALUATIONS

The experiments evaluate retargeting across same- and cross-structural skeletons, compare against established and naive baselines, and ablate major framework components. Results show advantages for the proposed skeleton-aware framework, while highlighting setting-dependent roles for adversarial and end-effector losses.

  • Dataset and setup: The dataset contains 2,400 motion sequences from 29 characters, with randomly assigned performers preventing motion pairs and 64-frame training windows.Groups A and B contain skeletons with similar structure but different body proportions, enabling quantitative evaluation against ground truth.
  • Intra-structural retargeting: The method outperforms NKN, PMnet, and naive CycleGAN adaptations in intra-structural retargeting, according to Table 2.The comparison uses unseen characters from group A for testing after training on 20 characters.
  • Cross-structural retargeting: Qualitative and quantitative comparisons show better stability, global positioning, and local joint accuracy than alternative cross-structural retargeting methods.The cross-structural comparison includes a naive CycleGAN adaptation and a cross-structural NKN variant.
  • Cross-structural retargeting: The lower cross-structural error than intra-structural error is attributed to group B having fewer characters and smaller body-proportion differences.The single test character in group B was therefore closer to characters seen during training.
  • Special characters: Special-structure experiments show retargeting for homeomorphic skeletons with asymmetric limbs and extra arm bones, including challenging clapping motions.A separate model is trained for each unique skeletal structure because no other skeletons share its domain.
  • Ablation study: Skeleton-aware operators outperform conventional operators because structured latent features encode different kinematic chains into distinct parts of the shared representation.Relevant target-chain decoders can then decode these features for chains with different numbers of bones.
  • Ablation study: Adversarial loss improves cross-structural retargeting but can reduce intra-structural performance, where copying rotations and constraining end-effectors may suffice.The ablation identifies adversarial loss as necessary when joint correspondence between domains is not well defined.
  • Ablation study: End-effector loss mitigates foot-sliding artifacts by converting large contact-position errors into less noticeable high-frequency motions near fixed contact points.A subsequent automated IK cleanup produces cleaner zero-velocity slots during contact periods and outputs more correlated with ground truth.

7 DISCUSSION AND FUTURE WORK

The framework transfers animations between homeomorphic skeletons through a shared latent space, but its scope is limited when skeletons have non-homeomorphic topology, very different T-poses, or unseen object interactions.

  • 7 DISCUSSION AND FUTURE WORK: Networks encode animations of homeomorphic skeletons into a common latent space and decode them to transfer motion between skeletons.The shared encoding space is attributed to the primal skeleton's topological proximity, geometric features, and weakly decoupled motion and shape properties.
  • 7 DISCUSSION AND FUTURE WORK: Retargeting between non-homeomorphic skeletons remains future work because a reduced primal skeleton cannot yet encapsulate the required information.
  • 7 DISCUSSION AND FUTURE WORK: Learned primal-skeleton features are needed because joint-independent animation representations and bone-length encoding are too complex to specify by design.
  • 7 DISCUSSION AND FUTURE WORK: The method can retarget to asymmetric characters and to characters with three bones per arm, whereas nearest-neighbor rotation copying produces implausible fine motions such as clapping.
  • 7 DISCUSSION AND FUTURE WORK: Very different T-poses hinder retargeting because joint rotations are interpreted relative to the T-pose, and unseen object-interaction tasks are not retargeted properly.

A NETWORK ARCHITECTURES

The appendix specifies skeleton-aware and skeleton-unaware network architectures, including temporal processing, skeletal pooling and unpooling, and patch-GAN discriminators.

  • A NETWORK ARCHITECTURES: The skeleton-aware architecture uses convolution, leaky ReLU, average skeletal pooling, skeletal unpooling, and temporal linear upsampling layers.Convolution layers use reflected padding; k denotes kernel width and s denotes stride.
  • A NETWORK ARCHITECTURES: The architecture reports input and output channels per joint and input and output joint counts in its rightmost column.
  • A NETWORK ARCHITECTURES: The regular skeleton-unaware network uses one-dimensional temporal convolution, LeakyReLU, and temporal linear upsampling.It matches the skeleton-aware network's total channel count, defined as number of joints multiplied by channels per joint.
  • A NETWORK ARCHITECTURES: Discriminators CA and CB are patch-GAN discriminators sharing the encoder architectures of EA and EB, but using sigmoid as the final activation.
Loading 2005.05732v1…