Source-linked AI summary
Thin-Plate Spline Motion Model for Image Animation
Jian Zhao, Hui Zhang
TL;DR
Unsupervised motion transfer remains challenged by large pose gaps and realistic restoration of regions missing after warping. The paper introduces TPS-based optical-flow estimation with multi-resolution occlusion masks and auxiliary losses, achieving state-of-the-art performance on most benchmarks with visible motion-metric improvements, while extreme identity mismatch remains difficult.
Problem
Unsupervised motion-transfer methods struggle with large pose gaps and inadequate inpainting of missing regions, limiting realistic animation.
Method
The framework combines multiple TPS transformations with affine background motion, warps source features using estimated optical flow, applies multi-resolution occlusion masks, and uses auxiliary losses.
Results
The method achieves state-of-the-art performance on most benchmarks, including visible motion-related improvements and 15.5% AKD and 28.0% MKR improvements on TaiChiHD.
Takeaways & Limitations
The framework supports unsupervised animation across talking faces, human bodies, and pixel animations while improving motion estimation and image quality.
Takeaways & Limitations
The approach does not perform well when an extreme identity mismatch occurs.
Abstract
from arXiv · showhide
Image animation brings life to the static object in the source image according to the driving video. Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge. However, it remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images. In this paper, a new end-to-end unsupervised motion transfer framework is proposed to overcome such issue. Firstly, we propose thin-plate spline motion estimation to produce a more flexible optical flow, which warps the feature maps of the source image to the feature domain of the driving image. Secondly, in order to restore the missing regions more realistically, we leverage multi-resolution occlusion masks to achieve more effective feature fusion. Finally, additional auxiliary loss functions are designed to ensure that there is a clear division of labor in the network modules, encouraging the network to generate high-quality images. Our method can animate a variety of objects, including talking faces, human bodies, and pixel animations. Experiments demonstrate that our method performs better on most benchmarks than the state of the art with visible improvements in pose-related metrics.
1. Introduction
Image animation transfers motion from a driving video to a static source object, but unsupervised methods struggle with large pose gaps and realistic restoration of missing regions. The proposed framework addresses these challenges with TPS motion estimation, multi-resolution occlusion masks, and auxiliary training losses.
- Image animation transfers an object's motion from a driving video to a static source image and supports applications including video conferencing, movie effects, and entertainment.
- Labeled motion-transfer methods are restricted to specific object categories and cannot generalize to objects without labeled data.Obtaining labeled data or pretrained keypoint extractors is costly.
- Unsupervised methods avoid object-specific prior knowledge by learning motion representations from pairs of video frames reconstructed with reconstruction losses.
- Large pose gaps expose insufficient motion flexibility, causing discrepancies between warped source features and the driving-image feature domain and increasing occluded regions.
- TPS motion estimation replaces locally affine motion with flexible nonlinear transformations, combining multiple TPS transformations with an affine background transformation to estimate optical flow.Early-training dropout encourages each TPS transformation to contribute to the flow.
- The framework predicts multi-resolution occlusion masks for feature layers and uses auxiliary losses to improve feature fusion and image generation.The stated contributions include end-to-end unsupervised transfer and evaluation across talking faces, human bodies, and pixel animations.
2. Related Work
Prior motion-transfer work either relies on object-specific supervision or uses unsupervised representations with limitations in complex motion and missing-region restoration. This paper relates TPS-based motion estimation to multi-scale feature fusion for more realistic unsupervised animation.
- Motion transfer: Supervised motion-transfer methods use landmarks, 3D models, domain labels, or FACS, but cannot readily handle new objects without labeled data, such as pixel animations.
- Motion transfer: Unsupervised methods avoid object-specific prior knowledge by learning identity and motion representations or directly estimating optical flow from video frames.
- Multi-scale feature fusion: Different network layers emphasize different feature levels: lower-resolution maps capture overall patterns, while larger-resolution maps emphasize detailed textures.
- Multi-scale feature fusion: The proposed approach estimates multi-resolution occlusion masks for feature maps at different scales, enabling more complete feature fusion during inpainting.
3. Method
The framework estimates flexible motion with multiple thin-plate spline transformations and an affine background transformation, then warps and inpaints source features. Multi-resolution occlusion masks, auxiliary losses, and a dedicated testing mode support reconstruction across pose and identity gaps.
- TPS Motion Estimation: The model combines K TPS transformations with one affine background transformation to approximate source-to-driving motion.The Dense Motion Network predicts contribution maps that combine these transformations into optical flow.
- TPS Motion Estimation: TPS transformations represent complex nonlinear motions, while the affine background transformation models camera-induced background motion.Each TPS transformation is generated from N=5 keypoint pairs in this paper.
- TPS Motion Estimation: Dropout on TPS transformations during early training prevents excessive reliance on a few transformations and improves network robustness.Dropout is removed after several epochs once each TPS transformation contributes to the estimated optical flow.
- Multi-scale Feature Fusion: The Dense Motion Network predicts multi-resolution occlusion masks, and the Inpainting Network uses them to fuse warped source features at corresponding scales.Warped encoder features are masked and passed to the decoder through skip connections before upsampling and residual processing.
- Training Losses: Auxiliary warp and equivariance losses constrain optical-flow estimation and keypoint detection alongside multiresolution reconstruction loss.The warp loss encourages warped source features to approach the driving image’s feature domain.
- Testing Stage: At testing, the framework animates a source image using a driving video and adopts animation via disentanglement for motion transfer.This mode uses an additional trained network to predict the motion applied to the source image.
4. Experiments
Experiments evaluate the method on talking faces, human bodies, and pixel animations using video reconstruction and image animation tasks. The method improves motion-related performance and temporal continuity, while ablations show complementary effects from TPS estimation, dropout, multi-resolution masks, and auxiliary losses.
- 4.1. Benchmarks: Experiments cover VoxCeleb, TaiChiHD, TED-talks, and MGif, spanning talking faces, human bodies, and pixel animations.Video reconstruction uses the first video frame as the source and evaluates generated frames with L1, AKD, and related metrics.
- 4.2. Comparison: 15.5% for AKD and 28.0% for MKR are the TaiChiHD improvements in motion-related metrics over prior methods.The method reaches state-of-the-art results on VoxCeleb, TaiChiHD, and MGif, and improves AKD over MRAA on TED-talks while being slightly worse on L1 and AED.
- 4.2. Comparison: The method produces better temporal continuity than MRAA in reconstructed videos.The paper attributes MRAA's pixel jitter to large keypoint changes between adjacent frames, whereas TPS transformations generated from multiple keypoints increase motion-estimation robustness.
- 4.2. Comparison: Qualitative image-animation results are better for human bodies, faces, and pixel animations, although preserving clothes and faces is slightly poor.MRAA produces ghosts on VoxCeleb, while neither method works well when source and driving identities differ greatly in size.
- 4.2. Comparison: Most users preferred the proposed videos for authenticity, and the method performed much better than MRAA on temporal continuity.The questionnaire compared 20 randomly selected video pairs from TaiChiHD, TED-talks, and VoxCeleb.
- 4.3. Ablations: TPS improves AKD and MKR, dropout improves motion estimation but can worsen AED, and multi-resolution masks improve L1 and AED.Adding Lbg and Lwarp reduces AKD and gives the full method a better balance across the four metrics; similar motion-description dimensions also favor the method over MRAA on motion-related metrics.
5. Discussion and Conclusion
The framework combines TPS motion estimation, multi-resolution occlusion masks, and auxiliary losses to improve unsupervised motion transfer. It achieves state-of-the-art performance on most benchmarks but remains challenged by extreme identity mismatch.
- TPS motion estimation produces optical flow that warps source feature maps into the driving image’s feature domain.
- Multi-resolution occlusion masks enable more effective feature fusion during restoration of missing regions.
- Additional auxiliary loss functions support a clear division of labor among network modules.
- The method achieves state-of-the-art performance on most benchmarks with visible improvements in motion-related metrics.
- Extreme identity mismatch remains a setting where the approach does not perform well.