Source-linked AI summary
Unpaired Motion Style Transfer from Video to Animation
Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen
TL;DR
Motion style transfer is constrained by paired-data requirements, scarce examples, and styles unavailable during training. This paper learns disentangled content and style representations from unpaired motions, aligns 3D and 2D style embeddings, and applies video-derived styles to 3D motion. It reports state-of-the-art-comparable performance overall and stronger results for unseen or video-extracted styles, while differing-body-proportion characters may require prior retargeting.
Problem
Motion style-transfer methods commonly require paired motions with matching content and styles seen during training, despite the scarcity and collection difficulty of such data.
Method
A generative network disentangles content and style, decodes content through temporal convolutions, applies style through temporally invariant AdaIN, and shares a style embedding across 3D and 2D joint positions.
Results
The method matches state-of-the-art results with unpaired training and outperforms other methods for previously unseen styles and styles extracted from ordinary videos.
Takeaways & Limitations
The framework supports style transfer from scarce or unseen examples and enables direct transfer of styles from ordinary videos to 3D animations.
Takeaways & Limitations
Characters with body proportions different from those seen during training may produce implausible results and may require motion retargeting before style transfer.
Abstract
from arXiv · showhide
Transferring the motion style from one animation clip to another, while preserving the motion content of the latter, has been a long-standing problem in character animation. Most existing data-driven approaches are supervised and rely on paired data, where motions with the same content are performed in different styles. In addition, these approaches are limited to transfer of styles that were seen during training. In this paper, we present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring motion styles not observed during training. Furthermore, our framework is able to extract motion styles directly from videos, bypassing 3D reconstruction, and apply them to the 3D input motion. Our style transfer network encodes motions into two latent codes, for content and for style, each of which plays a different role in the decoding (synthesis) process. While the content code is decoded into the output motion by several temporal convolutional layers, the style code modifies deep features via temporally invariant adaptive instance normalization (AdaIN). Moreover, while the content code is encoded from 3D joint rotations, we learn a common embedding for style from either 3D or 2D joint positions, enabling style extraction from videos. Our results are comparable to the state-of-the-art, despite not requiring paired training data, and outperform other methods when transferring previously unseen styles. To our knowledge, we are the first to demonstrate style transfer directly from videos to 3D animations - an ability which enables one to extend the set of style examples far beyond motions captured by MoCap systems.
1 INTRODUCTION
The paper addresses practical barriers in motion style transfer by learning from unpaired motions, supporting scarce and unseen styles, and extracting style directly from video. Its disentangled generative framework combines 3D motion content with 3D- or video-derived style, achieving state-of-the-art-comparable results and stronger performance on unseen or video-extracted styles.
- Motivation: Motion style transfer is motivated by the difficulty of capturing diverse stylized motions and the practicality of modifying existing motions into new styles.Videos offer a broader and cheaper source of motion styles than motion-capture data.
- Challenges: Paired, registered training data are difficult to collect at scale, while uncommon styles often have only a few available examples.Paired capture requires identical actions performed in different styles, which is tedious and unscalable.
- Method: The framework disentangles content and style into latent codes, recombines them with a temporal-convolutional generator, and trains using content consistency.Instance normalization and restricted content-code dimensionality discourage copying the content input directly.
- Method: A shared style embedding maps 3D motions and their 2D projections into the same style space, enabling video-based extraction, style interpolation, and style-distance measurement.The content input uses 3D joint rotations, whereas the style input uses joint positions.
- Contributions: The approach requires no paired training data, transfers styles unseen during training from as little as one clip, and extracts styles directly from ordinary videos.These capabilities are stated as the paper's main contributions.
- Results: The results match state-of-the-art motion style transfer using unpaired clips and outperform other methods for unseen styles and styles extracted from ordinary videos.The comparison is reported at the paper level without a specific numerical metric in the supplied passages.
2 RELATED WORK
Related work spans handcrafted and data-driven motion style transfer, image-inspired deep methods, and video motion extraction. The paper distinguishes its approach by using temporally invariant AdaIN for arbitrary motion styles and by extracting style, rather than motion, directly from video without 3D reconstruction.
- 2.1 Image Style Transfer: Image style-transfer research introduced normalization-based manipulation of deep-feature statistics, including AdaIN for applying various target styles.The paper adapts this idea to motion using temporally invariant AdaIN parameters.
- 2.1 Image Style Transfer: The paper claims first use of temporally invariant AdaIN for motion processing, enabling manipulation of a motion sequence into arbitrary styles.This mechanism is presented as the connection between image style transfer and the proposed motion framework.
- 2.2 Motion style transfer: Earlier motion style-transfer methods used handcrafted frequency- or time-domain features, physics-based optimization, or spectral differences between motions.These approaches target style manipulation in 3D motion sequences.
- 2.2 Motion style transfer: Data-driven methods infer style from examples but commonly rely on paired or registered motions, limiting applicability when corresponding performances are unavailable.Examples include pairwise correspondence, database retrieval, and neural networks trained on paired examples.
- 2.2 Motion style transfer: Prior few-shot motion transfer was limited to locomotion, whereas this approach extracts and transfers styles regardless of motion content.The comparison is made specifically with Mason et al.'s locomotion method.
- 2.3 Motion from Videos: Video-motion methods typically extract 2D or 3D motion, while this paper extracts style from video-captured human motion and bypasses 3D reconstruction.The paper positions this as a distinct contribution from motion reconstruction or pose-estimation work.
3 MOTION STYLE TRANSFER FRAMEWORK
The framework decomposes motion into content and style codes, recombines them through convolutional decoding and AdaIN, and supports style extraction from 3D motions or videos. It uses unpaired style-labeled clips and addresses global-velocity and foot-contact issues with test-time heuristics.
- Motion Translator: The network disentangles temporal content from temporally invariant style, decoding content with temporal convolutions and modifying deep-feature means and variances with AdaIN.The affine style transformation preserves temporal signal shape, supporting preservation of motion content.
- Joint 2D-3D Style Embedding: Two style encoders map 3D joint positions and their 2D projections into a common style embedding for extracting styles from videos.Training uses projected 2D motions and 3D motions to encourage the same style code.
- Motion Translator: The translator combines a content motion and a style motion to produce the content performed in the target style.The output is defined as G(ms|nt)=F(EC(ms)|ES(nt)).
- Motion Representation: Content is represented by 3D joint rotations, while style is represented by joint positions to support both 3D and video-based inputs.The output is also represented by joint rotations for standard character-animation files.
- Global Motion Handling: Global root positions are copied from the content input, while global velocity is adjusted by dynamic time warping using the input velocity ratio.This heuristic addresses style-correlated velocity when the style and content motions differ.
- Foot Contacts: Foot-contact labels from the content input guide inverse kinematics corrections, but the approach assumes contact timing belongs to content rather than style.The paper notes this assumption can fail for motions such as a zombie dragging a foot.
4 DISCUSSION
The discussion relates the framework to frequency-domain and convolutional style transfer, interpreting AdaIN and instance normalization as style-dependent modulation of temporal convolutions. Its modular source-style removal and target-style injection support unpaired learning and unseen-style transfer.
- Relation to Prior Work: The framework generalizes Yumer and Mitra’s frequency-domain style transfer while replacing paired processing with learning for unseen styles.Both methods share analysis and style-transfer building blocks, but the proposed framework is unpaired.
- Adaptive Normalization: A temporal convolution followed by IN or AdaIN is equivalent to a convolution whose effective weights and bias depend on style-related parameters.The effective kernel is βk(τ), with effective bias βb(τ)+γ.
- Modular Transfer: In the proposed architecture, IN removes source-style attributes and AdaIN injects target-style attributes through temporally invariant feature-channel transformations.The source and target styles control separate stages rather than one source-target-specific convolution.
5 EXPERIMENTS AND EVALUATION
Experiments show that the framework transfers styles from diverse sources, separates content and style in latent spaces, and handles unseen styles and video inputs. User studies report strong content preservation, style transfer, and realism, while style generalization improves with more training classes.
- Style Transfer Results: The framework transfers styles from 3D characters, 2D motion projections, and real videos, including a previously unseen style provided by a 3-second clip.Outputs joint rotations directly, avoiding additional processing such as inverse kinematics.
- Style Code: Style parameters cluster by style label, indicating that similarly styled inputs manipulate motion content in similar ways.Triplet loss is not required for clustering, although it can tighten the clusters.
- Video-to-3D Style Transfer: Style codes extracted from videos mostly fall into the same clusters as codes extracted from 3D motions.This supports a shared style representation across the two input modalities.
- Unseen Styles: With 16 style classes, the network generalizes an excluded heavy style better, whereas training with fewer styles produces outputs resembling visually similar seen styles.The unseen-style experiment excludes old from dataset A and heavy from dataset B.
- Content Code: Content codes show little correlation with style labels, while walking content is represented largely by a single phase parameter and preserves 97.4% of variation after PCA.The phase interpretation is linked to similar velocities, discarded global motion, and fixed temporal windows.
- Comparison: User studies found the method more successful for content preservation and style transfer with both 3D and video style inputs, while 75% of its outputs were judged realistic.The comparison also attributes competing-method difficulties to content-style dependence and mismatched input content.
6 CONCLUSIONS AND FUTURE WORK
The framework transfers motion style through an asymmetric network that implicitly disentangles style and content, including for previously unseen styles. The authors identify limitations involving dissimilar test inputs, locomotion-heavy training data, differing body proportions, and deterministic outputs.
- The network transfers motion style between sequences without paired data or explicit definitions of motion content and style.It implicitly disentangles style and content and recombines previously unseen styles with a given content.
- Style is controlled through temporally invariant instance-normalization transformations, while content is represented by deep convolutional layers.
- The method supports style interpolation, including transfers from depressed to proud and from neutral to old.
- If test-time content and style motions lack commonalities or the target style departs too far from training styles, the network may not infer which properties to transfer.
- Different body proportions from those seen during training may produce implausible results, requiring motion retargeting before style transfer.End-to-end transfer across unseen skeletons remains future work.
- A given content-style input pair currently yields a deterministic output, limiting variation and temporal prolongation without repetitions or discontinuities.The authors propose injecting noise and segmenting sequences for future extensions.