Source-linked AI summary
First Order Motion Model for Image Animation
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe
TL;DR
The paper addresses image animation without object-specific priors or annotations by learning motion and appearance separately from videos of the same object category. It uses keypoints, local affine transformations, dense motion, and occlusion-aware generation to animate a source image from a driving video. The method clearly outperforms prior approaches across the reported benchmarks, while relative motion transfer assumes roughly similar initial poses.
Problem
Image animation must synthesize a video by combining source-image appearance with driving-video motion without relying on object-specific priors or costly annotations.
Method
The method learns sparse keypoints and local affine transformations, converts them into dense motion and occlusion maps, and uses a generator to render the target image.
Results
The approach clearly outperforms state-of-the-art methods on all reported benchmarks and improves every metric across four datasets in the video-reconstruction comparison.
Takeaways & Limitations
A richer first-order motion representation and explicit occlusion modeling support more realistic animation of complex object motions.
Takeaways & Limitations
Relative motion transfer assumes that the source and driving objects have similar initial poses; without rough alignment, it can produce physically impossible keypoint locations.
Abstract
from arXiv · showhide
Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.
1 Introduction
Image animation synthesizes videos by combining a source image’s appearance with motion from a driving video, but many existing methods depend on object-specific priors or costly annotations. The paper proposes a self-supervised first-order motion model using keypoints, local affine transformations, and occlusion modeling.
- Motivation: Image animation combines source-image appearance with motion patterns derived from a driving video.Applications include movie production, photography, and e-commerce.
- Limitations of existing methods: Traditional methods often require strong object-specific priors such as 3D models, while deep generative approaches commonly depend on annotated pretrained representations.These limitations motivate a model-free, self-supervised approach.
- Contributions: The first-order motion model represents complex motion with self-learned keypoints and local affine transformations.This addresses Monkey-Net’s zeroth-order neighborhood model, which performs poorly under large pose changes.
- Contributions: An occlusion-aware generator estimates invisible regions and combines source appearance with driving-video motion to render animated frames.The model also extends the equivariance loss to improve local affine-transformation estimation.
2 Related work
Prior video-generation work includes conditional synthesis and frame prediction, while image-animation methods have often relied on domain-specific object models. The paper instead uses warping driven by externally supplied motion and a self-supervised sparse-keypoint representation without an explicit reference pose.
- Video Generation: Conditional video-generation methods synthesize videos from inputs such as noise, labels, static images, or initial frames.The paper is related because it also uses a warping formulation to generate video sequences.
- Video Generation: In image animation, spatial deformations come from the driving video rather than being predicted as in future-frame generation.This distinguishes the task from conditional prediction of future frames.
- Image Animation: Traditional image-animation systems target specific domains and commonly require strong priors such as 3D face models.Such models may be unavailable in many applications.
- Image Animation: Model-free alternatives include dense motion with canonical reference poses and self-supervised sparse keypoint trajectories.The proposed formulation avoids requiring an explicit reference pose and uses a richer motion representation than Monkey-Net.
3 Method
The method separates motion estimation from image generation, using self-supervised keypoints and local affine transformations to derive dense motion between driving and source frames. It then combines warped source features with occlusion-aware inpainting to render the animated image.
- Training reconstructs videos by combining a single source frame with a learned latent representation of motion from videos in the same object category.
- The motion estimation module predicts backward optical flow from the driving frame to the source frame through an abstract reference frame.Backward flow supports differentiable back-warping, while the abstract reference frame enables independent processing of source and driving frames.
- A self-supervised keypoint encoder-decoder provides a compact motion bottleneck, with keypoint locations and local Jacobians describing first-order transformations around each keypoint.The keypoint predictor outputs confidence heatmaps and additional channels used to estimate local transformation matrices.
- A dense motion network combines local transformations, transformed source images, background information, and keypoint heatmaps to estimate the final dense motion field.The part-based model uses K+1 masks, including a background mask for non-moving regions.
- The generator back-warps source features and uses an occlusion mask to suppress unrecoverable regions before inpainting and rendering the output image.The occlusion map identifies feature regions that should be inferred rather than reconstructed by warping.
- Relative motion transfer preserves global object geometry but assumes the source and driving objects have similar initial poses.Without rough alignment, transferred keypoint locations may become physically impossible for the object.
4 Experiments
Experiments evaluate the method on four object datasets using video reconstruction, qualitative comparisons, and a paired user study. The full model improves reconstruction metrics and produces better animation, especially for complex human-body motion.
- The method is trained and tested on four datasets containing varied object categories, rendering higher-resolution videos than Monkey-Net in all experiments.
- Video reconstruction combines each frame’s sparse motion representation with the first frame, while image-animation quality is additionally evaluated through a user study.The protocol uses K=10 keypoints and reports L1, AKD, and other metrics.
- Adding pyramid loss, occlusion modeling, and local affine transformations progressively improves ablation results, while unconstrained jacobians produce unstable motion estimates.The full model’s MKR is 2.75 times smaller than the Baseline model’s MKR.
- On Tai-Chi-HD, the Full model detects keypoints on both arms and reconstructs complex poses more accurately than the Baseline model.The Baseline lacks arm keypoints and fails as pose differences from the initial pose increase.
- The full model consistently improves every metric across four datasets and outperforms X2Face and Monkey-Net, particularly on the highly non-rigid Tai-Chi-HD dataset.Compared with X2Face, it uses a smaller motion embedding; its advantage over similarly sized Monkey-Net representations is clearest on Tai-Chi-HD.
- Qualitative comparisons show that X2Face and Monkey-Net warp the human body as a blob, whereas the proposed method independently animates body parts and is preferred by users.The paired user study uses 10 AMT workers per question across 50 input pairs.
5 Conclusions
The paper presents an image-animation approach using learned keypoints, local affine transformations, and explicit occlusion modeling. Across quantitative and qualitative benchmarks, it clearly outperforms the state of the art.
- The approach represents motion with keypoint displacements and local affine transformations, while a generator combines source appearance with driving-video motion.
- Explicit occlusion modeling indicates which image regions the generator should inpaint during target motion.
- The method clearly outperforms state-of-the-art approaches across all evaluated benchmarks.
A Detailed Derivations
The derivation approximates transformations near learned keypoints using first-order Taylor expansion and composition rules. It also formalizes equivariance constraints and relative-motion transfer between source and driving frames.
- First-order Taylor expansion approximates TS←D near keypoint zk using zeroth- and first-order terms.
- TS←D is decomposed into transformations through an intermediate reference frame.
- Equivariance constraints follow by matching corresponding Taylor coefficients on the two sides of the transformation relation.
- Relative motion is transferred by shifting coordinates to the driving keypoint, applying the driving transformation, and shifting them back.
B.1 Architecture details
The model uses compact U-Net-based motion networks and a Johnson-style generator, trained with Adam under dataset-specific iteration schedules.
- The keypoint detector and dense motion predictor operate at 64 × 64 resolution instead of 256 × 256 to reduce memory and computation.
- The motion networks use five convolutional encoder blocks and five upsampling decoder blocks, while the generator uses two downsampling blocks, six residual blocks, and two upsampling blocks.
- Training runs for approximately 100k iterations on Tai-Chi-HD and VoxCeleb and 40k on Nemo and Bair.
B.2 Equivariance loss implementation
Equivariance transformations are implemented with randomly sampled thin plate splines whose deformation and affine components use separate Gaussian parameter scales.
- The equivariance transformation TX←Y is implemented using randomly sampled thin plate splines.
- Spline parameters have zero mean, with variance 0.005 for deformation and 0.05 for affine components.
- The deformation component uses a uniform 5 × 5 grid.
C.1 Image Animation
Qualitative evaluations across faces, human bodies, and robotic arms show stronger animation quality than competing methods, while learned keypoints and occlusion masks provide interpretable motion and visibility cues.
- Image Animation: On VoxCeleb, X2Face produces unrealistic sequences, while Monkey-Net misses specific facial expressions and eye movements.
- Image Animation: On Nemo, X2Face cannot inpaint unseen parts such as teeth, whereas the proposed approach better reproduces teeth and closed eyes than Monkey-Net.
- Image Animation: On Tai-Chi-HD, competing methods warp bodies unrealistically, while the proposed model preserves the background and moves body parts independently, despite minor artifacts.
- Image Animation: On Bair, X2Face fails to transfer motion, and the proposed approach slightly outperforms Monkey-Net by better preserving the robot arm.
- Keypoints: Learned keypoints are often semantically consistent, corresponding to body parts or facial landmarks across datasets.
- Occlusion masks: Occlusion masks identify regions absent from the source and needing inpainting, while low-motion sequences produce nearly white masks.