Source-linked AI summary
Motion Representations for Articulated Animation
Aliaksandr Siarohin, Oliver J. Woodford, Jian Ren, Menglei Chai, Sergey Tulyakov
TL;DR
Articulated animation needs motion representations that capture complete object parts, their shapes, and poses without labelled training data. The paper introduces unsupervised PCA-based regions, explicit background-motion modeling, and shape–pose disentanglement, outperforming prior methods and reaching 96.6% user preference on its most challenging benchmark. Generalization to out-of-domain inanimate objects remains a significant challenge.
Problem
Existing unsupervised methods struggle to represent and animate articulated parts because keypoints omit complete part shape and pose, while prior supervised methods require labelled object data.
Method
The method detects and tracks unsupervised regions, measures motion through principal axes, models global affine background motion, and disentangles object shape from pose.
Results
96.6% of independent raters preferred the method to the state of the art on the new TED-talks benchmark, with stronger results particularly for articulated objects.
Takeaways & Limitations
Region-based motion representations improve object-part stability, reconstruction accuracy, perceived quality, and scalability to more regions across datasets and tasks.
Takeaways & Limitations
Generalization remains a significant challenge for broadly practical articulated animation of out-of-domain inanimate objects.
Abstract
from arXiv · showhide
We propose novel motion representations for animating articulated objects consisting of distinct parts. In a completely unsupervised manner, our method identifies object parts, tracks them in a driving video, and infers their motions by considering their principal axes. In contrast to the previous keypoint-based works, our method extracts meaningful and consistent regions, describing locations, shape, and pose. The regions correspond to semantically relevant and distinct object parts, that are more easily detected in frames of the driving video. To force decoupling of foreground from background, we model non-object related global motion with an additional affine transformation. To facilitate animation and prevent the leakage of the shape of the driving object, we disentangle shape and pose of objects in the region space. Our model can animate a variety of objects, surpassing previous methods by a large margin on existing benchmarks. We present a challenging new benchmark with high-resolution videos and show that the improvement is particularly pronounced when articulated objects are considered, reaching 96.6% user preference vs. the state of the art.
1. Introduction
Unsupervised animation methods still struggle to represent and animate articulated object parts under realistic motion. This work addresses those gaps with region-based motion, explicit background-motion modeling, and shape–pose disentanglement, achieving stronger animation quality and 96.6% user preference on a challenging benchmark.
- Motivation: Unsupervised motion-transfer methods avoid labelled pose and shape data but leave object-part representation and driving-video animation unresolved.The open problems concern representing articulated or non-rigid parts, including shape and pose, and animating those parts from driving-video motion.
- Motivation: Boundary keypoints make frame-to-frame correspondence difficult and fail to encode semantically meaningful parts, shapes, and poses.Static-background assumptions can also leak camera motion into keypoints, while absolute motion transfer can transfer the driving object's shape and reduce source-identity fidelity.
- Method: The proposed PCA-based region representation measures first-order motion from regions rather than regressing it, improving stability, convergence, and motion segmentation.The representation empirically captures the shape of underlying object parts and is inspired by Hu moments.
- Method: A global affine transformation models background or camera motion, while shape and pose are disentangled in unsupervised region space.The framework uses no labels and is optimized with reconstruction losses.
- Results: 96.6% of independent raters preferred the method to the state of the art on the new TED-talks benchmark.The method also outperformed previous unsupervised animation methods across talking faces, taichi videos, and animated pixel art.
- Results: The method improves fidelity particularly for articulated objects and scales to more unsupervised regions for more detailed motion.The new benchmark was collected from TED-talk speakers to make articulated-object evaluation more challenging.
2. Related work
Related work spans supervised methods that use object knowledge and unsupervised methods that avoid labelled shape or landmark data. Recent unsupervised approaches enable broader animation but commonly rely on incomplete keypoint or single-texture representations.
- Supervised methods: Supervised animation methods use landmarks, semantic segmentations, or parametric 3D models and therefore target object categories with substantial labelled data.Examples include faces and human bodies, where prior shape and pose knowledge can be learned.
- Unsupervised methods: Unsupervised methods remove the need for labelled object shape or landmark data and can generate motion from videos without domain knowledge.Video-generation methods may condition on action labels, while playable video generation allows action selection at each timestamp.
- Unsupervised methods: Prior unsupervised animation work represents objects with a single RGB texture or keypoints, with keypoints often located on object edges.The paper identifies incomplete representation of object parts, shape, and pose as the central limitation for articulated animation.
- Motion descriptors: Hu moments have been used to construct motion descriptors from temporal image differences aggregated into motion history images.Blob statistics then discriminate between different actions.
3. Method
The method represents articulated-object motion with PCA-derived regions, separately models background motion, and disentangles shape from pose for animation.
- Method overview: The model extends FOMM with PCA-based motion estimation, background motion representation, and animation via disentanglement.These components replace regression-based motion estimation, explicitly model non-object-related motion, and separate shape from pose.
- PCA-based motion estimation: A heatmap-based region predictor estimates K part regions, whose translations, rotations, and x/y scales define motion through a common reference frame.Translations use softargmax, while PCA of each heatmap estimates principal axes and scale-related motion.
- PCA-based motion estimation: The PCA representation captures meaningful object parts such as articulated limbs, but omits shear and therefore has five rather than six affine degrees of freedom.The reference frame is the whitened heatmap coordinate frame, with zero mean and identity covariance.
- Background and image generation: A separate affine background transformation prevents camera or background motion from consuming foreground motion capacity, while the flow network combines region and background transformations.The resulting dense flow warps source features, and a confidence map guides inpainting of newly introduced regions.
- Shape-pose disentanglement: Shape and pose encoders support animation that retains source shape instead of directly transferring driving-frame shape.This addresses limitations of standard and relative animation for articulated objects.
4. Evaluation
The evaluation spans reconstruction, animation, ablation, and segmentation experiments across datasets of faces, bodies, and cartoons. Results show the proposed representations are especially effective for articulated objects, improving metrics, animation quality, scalability, and foreground segmentation.
- Evaluation setup: Experiments evaluate reconstruction and animation on faces, human bodies, and animated cartoons, using L1, AKD, MKR, and AED metrics.Comparisons include FOMM on reconstruction and animation, with X2Face and Monkey-Net included for video reconstruction.
- Animation quality: Animation comparisons show clearer improvements over FOMM for articulated bodies, especially in rendering limbs in the driving pose.A user study found comparable performance on non-articulated faces, while FOMM often failed to detect and animate articulated body parts such as hands.
- Ablations: The PCA-based representation produces unsupervised regions aligned with articulated object parts, whereas regression-based representations localize individual edges.Without PCA, motion can leak into the background branch, degrading keypoint quality and AKD/AED scores.
- Ablations: PCA-based representations scale well as the number of regions increases, while the regression-based representation fails when the number of keypoints reaches 20.Background-motion modeling significantly lowers L1 error and improves region segmentation quality, but does not improve pose-only AKD and MKR.
- Part segmentation: The method achieves 0.81 foreground segmentation IoU on TaiChiHD, compared with 0.77 for MSCS and 0.55 for SCOPS.This capability is evaluated despite the method not being optimized specifically for co-part segmentation.
5. Conclusion
The paper concludes that PCA-based region motion representations and background-motion estimation improve unsupervised articulated animation, while generalization to out-of-domain inanimate objects remains challenging.
- PCA-based regions improve motion representation by capturing object-part shape and pose, while background estimation decouples foreground and background motion.
- The method improves region distribution and stability, reconstruction accuracy, user-perceived quality, and scalability to more regions across datasets and tasks.
- The TED-talks dataset provides a more challenging benchmark for future improvements in unsupervised articulated animation.
- Generalization remains a significant challenge for broadly practical articulated animation of inanimate objects.
A. Toy Experiment Details
The toy experiment evaluates angle prediction on randomly rotated, randomly colored rectangles using three network designs and training-set sizes.
- The rotated rectangles dataset contains images rotated randomly from 0° to 90°, with labels indicating rotation angle and varied random colors.
- Naive directly predicts the angle, Regression-based regresses it per pixel and pools using heatmap weights, while PCA-based predicts heatmaps and applies PCA.
- All networks minimize L1 loss on rotation angle for 100 epochs using Adam at learning rate 10^-4 and batch size 8.
- Training-set sizes range from 32 to 1024, with results evaluated on a separate fixed test set of 128 images.
B. Co-part segmentation details
The co-part segmentation procedure assigns pixels to the strongest predicted region response, identifies background through a response threshold, and supports qualitative comparisons and cloth swapping.
- Each pixel z is assigned to the part with the maximum response of M_k(z).
- Pixels are assigned to background when the summed region response is below 0.001.
- The method produces more meaningful co-part segmentations than SCOPS and separates foreground objects from backgrounds more accurately than MSCS.
- The produced segmentations enable copying cloth from a source person onto a person in the driving video.
- Figure 9 compares input images and segmentation masks or overlays from the method, SCOPS, and MSCS.
C. Implementation details
The implementation follows a FOMM-like multi-network architecture, using U-Nets for region and flow prediction, a separate background-motion encoder, and a shape-pose disentanglement network.
- The implementation largely follows FOMM’s architecture design and operates motion-related predictors at one-quarter of the original resolution.
- Region and pixel-wise flow predictors use U-Nets with five encoder and five decoder blocks, while the background-motion predictor uses only the encoder.
- The image generator uses a Johnson architecture with down-sampling blocks, residual blocks, up-sampling blocks, and warped confidence-weighted skip connections.
- The shape-pose disentanglement network uses two identical encoders and one decoder, trained on 1M source-driving pairs.
D. Background movement
Background modeling separates non-object global motion from the object, while animation sets estimated background motion to zero for articulated objects. The framework nevertheless supports controlled camera movement during visualization or animation.
- Background modeling primarily frees network capacity to handle the object.
- For articulated-object animation, estimated background motion is set to zero because it is usually unnecessary.
- The framework can control camera motion by keeping the background still, translating it left or right, or rotating it counterclockwise.
E. TED-talks dataset creation
The TED-talks dataset was constructed by filtering licensed YouTube videos for visible presenters, presentation activity, resolution, and temporal continuity. Videos were split into stable-camera chunks, cropped around presenters, and standardized to 384 × 384 pixels.
- The dataset began with 3,035 YouTube videos shared under the CC BY–NC–ND 4.0 International license.The license permits non-commercial use.
- Selection retained videos showing the presenter’s upper body for at least 64 frames and a bounding-box height of at least 384 pixels.Static videos and videos in which the person was not presenting were manually removed.
- Videos were split into chunks without significant camera changes and with limited presenter displacement from the starting position.Chunks shorter than 64 frames or lacking sufficient resolution were removed.
- Each retained chunk was cropped to a square around the presenter, scaled consistently, and downscaled to 384 × 384 pixels.