Source-linked AI summary
Animating Arbitrary Objects via Deep Motion Transfer
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe
TL;DR
Animating a static object image from a driving video requires separating appearance from motion and recombining them without object-specific models. Monkey-Net uses self-supervised keypoints, dense motion estimation, and motion-aware image synthesis, achieving high-quality arbitrary-object animation and outperforming prior unsupervised methods. Its relative-motion transfer assumes rough alignment between source and driving objects.
Problem
Image animation must decouple and recombine motion and content information, while many prior approaches use object-specific representations or restricted domains.
Method
Monkey-Net learns motion-specific keypoints self-supervised, predicts dense optical flow with coarse and residual components, and deforms encoded appearance features to synthesize frames.
Results
The method produces high-quality videos across three public datasets and outperforms previous unsupervised image-animation and image-to-video translation methods.
Takeaways & Limitations
Motion-specific keypoints provide a compact but rich motion representation for animating arbitrary objects and adapting the method to image-to-video translation.
Takeaways & Limitations
Relative-motion transfer requires the driving-video object to be roughly aligned with the source-image object; otherwise estimated keypoint positions may be physically impossible.
Abstract
from arXiv · showhide
This paper introduces a novel deep learning framework for image animation. Given an input image with a target object and a driving video sequence depicting a moving object, our framework generates a video in which the target object is animated according to the driving sequence. This is achieved through a deep architecture that decouples appearance and motion information. Our framework consists of three main modules: (i) a Keypoint Detector unsupervisely trained to extract object keypoints, (ii) a Dense Motion prediction network for generating dense heatmaps from sparse keypoints, in order to better encode motion information and (iii) a Motion Transfer Network, which uses the motion heatmaps and appearance information extracted from the input image to synthesize the output frames. We demonstrate the effectiveness of our method on several benchmark datasets, spanning a wide variety of object appearances, and show that our approach outperforms state-of-the-art image animation and video generation methods. Our source code is publicly available.
1. Introduction
The paper targets motion-driven animation of arbitrary objects by combining source-image appearance with motion extracted from a driving video. Monkey-Net addresses limitations of object-specific and category-specific approaches through unsupervised motion keypoints and achieves strong benchmark performance.
- Motivation: Motion-driven image animation combines appearance from a source image with motion patterns extracted from a driving video.The task requires representing the object, extracting driving motion, and mapping that motion onto the object representation.
- Proposed Approach: Figure 1 illustrates animation of arbitrary target objects according to motion from a driving video.The figure presents the core source-image and driving-video transfer setting.
- Limitations of Prior Work: Existing approaches often rely on object-specific representations or learn transformations for a single object category or domain.Examples include 3D morphable models for faces and data-driven methods trained on restricted categories or domains.
- Proposed Approach: Monkey-Net animates arbitrary objects using self-supervised object representations and sparse motion-specific keypoints describing relative pixel movements.The framework transfers relevant motion patterns while avoiding unrelated information from the driving video.
- Results: Experiments on three public datasets show high-quality animation across varied objects and outperform state-of-the-art image-to-video translation methods.The evaluated datasets are Tai-Chi, BAIR robot pushing, and UvA-NEMO Smile.
2. Related work
Prior video-generation methods synthesize sequences using spatio-temporal or recurrent architectures, while object-animation methods commonly depend on domain-specific representations. This work instead decouples motion and content to animate static images of arbitrary objects.
- Deep Video Generation: Early deep video-generation methods used spatio-temporal networks to synthesize multiple frames simultaneously, but typically produced poor visual quality.VGAN and TGAN are cited as examples of this approach.
- Deep Video Generation: More recent methods used recurrent neural networks with adversarial training and conditional information such as labels or static images.CMM-Net combines conditional LSTM and VAE components, while MoCoGAN uses recurrent adversarial learning.
- Future Frame Prediction: Future-frame prediction methods synthesize coherent continuations by predicting pixels directly or learning transformations between frames.The related approaches include direct pixel prediction and learned pixel transformations.
- Object Animation: Image animation is more challenging because it must decouple, model, and recombine motion and content information across domains.This distinguishes image animation from related deep video-generation tasks.
- Object Animation: Traditional object-animation approaches require explicit representations and are designed for specific domains such as faces or human silhouettes.Their domain specificity limits the range of objects they can animate.
- Object Animation: The proposed self-supervised deep network animates static images of arbitrary objects without requiring an object-specific representation.The method is presented as a broader alternative to approaches restricted to particular domains.
3. Monkey-Net
Monkey-Net animates a source image using motion from a driving video by extracting sparse keypoints, converting them into dense motion, and recombining motion with appearance. Its modules are trained self-supervised and include object-agnostic deformation and residual-flow mechanisms.
- 3. Monkey-Net: Monkey-Net uses a Keypoint Detector, Dense Motion prediction network, and Motion Transfer network to animate a source object from driving-video motion.The detector extracts sparse keypoints, the motion network produces motion heatmaps, and the transfer network combines these with source appearance.
- 3. Monkey-Net: Training requires only videos of objects and no keypoint annotations, enabling self-supervised learning and test-time animation from a static image plus a motion descriptor.Frames are generated one at a time using the source image and the corresponding driving-video frame.
- 3.2. Unsupervised Keypoint Detection: The keypoint detector learns heatmaps and Gaussian location distributions whose covariances encode landmark orientation, while heatmaps remain compatible with convolutional networks.The bottlenecked Gaussian representation encourages indirect landmark learning; heatmap encoding supplies inputs to the generator and motion networks.
- 3.3. Generator Network with Deformation Module: The generator encodes source appearance with a U-Net and uses optical-flow warping to align encoder features with the target frame for object-agnostic deformation.A bilinear sampler implements the warping operation differentiably, while heatmap differences are provided to the decoder to facilitate training.
- 3.4. From Sparse Keypoints to Dense Optical Flow: The motion network predicts masks for keypoint-associated rigid parts, a static-background mask, and residual motion, combining coarse part motion with non-rigid refinement.The final flow is F = Fcoarse + Fresidual.
- 3.6. Generation Procedure: At test time, relative keypoint motion transfers driving-video motion to source keypoints, but requires roughly aligned source and driving objects to avoid physically impossible positions.The transferred keypoints are encoded as heatmaps and supplied to the dense-motion and generator networks.
4. Experiments
The experiments evaluate video reconstruction, image-to-video translation, and image animation across three datasets using quantitative metrics, qualitative comparisons, ablations, and user studies. The method outperforms prior approaches across the reported evaluations, while ablations identify the contributions of dense motion components and covariance encoding.
- Evaluation Protocol: Experiments cover video reconstruction, image-to-video translation, and image animation on Tai-Chi, BAIR robot pushing, and UvA-NEMO datasets.The evaluation uses several metrics, qualitative comparisons, ablations, and user studies.
- Ablation Study: The full model outperforms the no-deformation baseline across all metrics in the Tai-Chi video reconstruction ablation.The result supports deforming feature maps according to estimated motion.
- Ablation Study: Removing coarse or residual flow components reduces performance, indicating that neither component alone accurately estimates dense motion.The paper attributes these weaknesses to non-rigid motion and large-motion cases, respectively.
- Ablation Study: Replacing covariance matrices with static diagonal matrices lowers performance on Tai-Chi videos with complex and large motion.The ablation therefore supports encoding richer motion information.
- Comparison with Previous Works: The method outperforms X2Face on all metrics and evaluated datasets for video reconstruction, using 10 landmarks represented by a 50-dimensional motion representation.The paper describes this representation as compact but rich because it encodes motion with motion-specific keypoints.
- Comparison with Previous Works: The method clearly outperforms MoCoGAN, SV2P, and CMM-Net across all reported metrics for image-to-video translation.CMM-Net is evaluated only on Nemo, while SV2P results are reported on BAIR as in its original study.
- User Evaluation: User studies preferred the method over X2Face in more than 80% of comparisons across all datasets, with higher preference on datasets containing larger motion patterns.For image animation, users assessed 50 video pairs with annotations from 10 users per video.
5. Conclusion
The paper presents a self-supervised, motion-specific keypoint approach for animating arbitrary objects and reports improved performance over prior unsupervised image-animation methods. It also extends to image-to-video translation with little adaptation.
- The proposed method uses motion-specific keypoints learned through self-supervision to animate images of arbitrary objects from driving-video motion.
- Experiments using automatic metrics and human judgments show that the method outperforms previous work on unsupervised image animation.
- With little adaptation, the framework also performs image-to-video translation.
- Future work includes handling multiple objects and investigating alternative strategies for motion embedding.
A. Implementation details
The implementation uses common U-Net architectures across modules, sharp softmax-based keypoint heatmaps, artifact-removing residual blocks, dataset-specific training schedules, and a GRU for image-to-video translation.
- All networks use the same U-Net architecture with convolutional, batch-normalization, ReLU, pooling, and five-block encoder structures.
- The keypoint detector uses softmax temperature 0.1 to produce sharper heatmaps and avoid keypoints collapsing to the image center.
- The generator adds four residual blocks to remove possible warping artifacts produced by the dense-motion network.
- Training lasts 250, 500, and 10 epochs for Tai-Chi, Nemo, and BAIR, respectively, using Adam with staged learning rates.
- For image-to-video translation, a single-layer GRU with 1024 hidden units predicts keypoint sequences using L1 minimization.
B. MGif dataset
MGif is a 1,000-video cartoon-animal dataset designed to test animation under substantial appearance variation and motion diversity.
- MGif contains 1,000 moving-GIF videos, with 900 used for training and 100 for evaluation.
- Each video has 128 × 128 resolution and contains between 5 and 100 frames.
- The dataset is challenging because it combines high appearance variation with diverse motion.
- Experiments use absolute driving-video keypoint locations rather than relative keypoint motion.
C. Additional experimental results
Additional experiments examine the pose-alignment assumption and report qualitative results, including failure cases when source and driving poses are mismatched.
- The additional-results section includes visual motivation for the alignment assumption, completed ablations, qualitative image-to-video and animation results, and keypoint visualizations.
- The method assumes that source-image and first driving-frame objects have similar poses to avoid meaningless motion transfer.
- When poses are misaligned, mouth movements can produce disappearing or implausible mouths in generated videos.
C.2. Additional ablation study
The ablation evaluates how the number of keypoints affects reconstruction quality and selects 10 keypoints as a trade-off between accuracy and computational cost.
- Increasing the number of keypoints lowers reconstruction error on the Tai-Chi dataset.The evaluation uses L1 and AKD metrics.
- 10 keypoints are used in all experiments because they balance reconstruction quality against memory and computational overhead.
C.3. Image-to-Video translation
The image-to-video translation evaluation compares the method with established competitors using user studies and qualitative examples. The method is preferred on average and produces more realistic, detailed, and temporally coherent videos across datasets.
- User Study: Users preferred the proposed method over competing methods on average, especially for Tai-Chi and Bair sequences with larger motion.The study compares generated videos from 50 randomly selected references using pairwise judgments of animation realism.
- Qualitative results: Additional qualitative results further support realistic video generation from source images and driving sequences.
- Qualitative results: On Nemo, the method preserves person identity and generates realistic smiles, while MoCoGAN and CMM-Net produce more artifacts.CMM-Net preserves identity better than MoCoGAN but fails to generate realistic eyes and teeth.
- Qualitative results: On Tai-Chi, the method generates realistic human bodies with richer details than MoCoGAN, whose videos obscure some body parts.
- Qualitative results: On Bair, the method generates a sharp robotic arm with detailed, temporally coherent backgrounds, unlike blurred or artifact-laden competitor outputs.SV2P produces blurred robotic videos, while MoCoGAN generates more details but temporally incoherent backgrounds.
C.4. Image animation
The image-animation comparison with X2Face spans faces, human bodies, robotic arms, and animals. The proposed method transfers motion more realistically while preserving or adapting relevant appearance details.
- Nemo: On Nemo, the method generates more realistic smiles than X2Face on most samples and transfers relative motion without inheriting the driver’s facial appearance.X2Face can inherit large cheeks from the driving video, whereas the proposed method transfers relative motion.
- Tai-Chi: On Tai-Chi, X2Face cannot handle the driving motion and instead warps the source body as a single blob.
- Bair: On Bair, X2Face often makes the robotic arm indistinguishable, while the proposed model generates a realistic arm following the driving motion.
- MGif: On MGif, the model transfers motion despite substantial source–driver appearance differences, correctly animating legs, rigid animal parts, and a fox tail.The examples include motion transfer between different animals, such as a fox and a cheetah.
C.5. Keypoint visualization
The paper presents qualitative keypoint visualizations and image-animation examples across Nemo, Tai-Chi, Bair, and MGif. The visualizations show dataset-specific semantic consistency, while the examples document the evaluated animation settings.
- Keypoint visualization: On Nemo, learned keypoints are semantically consistent, with recurring points corresponding to the nose and chin.
- Keypoint visualization: On Tai-Chi, keypoints also show semantic consistency, including recurring chin and left-side arm locations.The left-side arm corresponds to the right arm in frontal views and the left arm in back views.
- Image-animation examples: Qualitative image-animation results are reported for Tai-Chi, Nemo, Bair, and MGif.The figures cover standard and additional comparisons, including X2Face on Nemo, Tai-Chi, and Bair.
- Image-animation examples: Additional qualitative results are shown for MGif image animation.
- Keypoint visualization: The keypoint visualizations are presented separately for Nemo, MGif, Tai-Chi, and Bair.