Source-linked AI summary
Latent Image Animator: Learning to Animate Images via Latent Space Navigation
Yaohui Wang, Di Yang, Francois Bremond, Antitza Dantcheva
TL;DR
Image animation methods often depend on explicit structure representations, which add modules and can struggle with appearance variation. LIA instead uses a self-supervised autoencoder to navigate latent space with learned orthogonal motion directions, and the paper reports state-of-the-art performance across its benchmarks. Its supported scope remains constrained by occlusions and difficult small articulated motions.
Problem
Structure-guided image-animation methods require additional representation modules and may fail when source and driving images have large appearance variation.
Method
LIA is a self-supervised autoencoder that linearly navigates source latent codes using combinations of learned orthogonal motion directions, then decodes dense flow for warping.
Results
LIA outperforms state-of-the-art algorithms across VoxCeleb, TaichiHD, and TED-talk benchmarks in quantitative and qualitative evaluations.
Takeaways & Limitations
The learned motion dictionary is interpretable, with directions corresponding to basic visual transformations, while explicit structure representations are unnecessary.
Takeaways & Limitations
Body occlusion limits motion transfer, especially for occluded limbs in Taichi videos and small, articulated, or occluded hands in TED-talks.
Abstract
from arXiv · showhide
Due to the remarkable progress of deep generative models, animating images has become increasingly efficient, whereas associated results have become increasingly realistic. Current animation-approaches commonly exploit structure representation extracted from driving videos. Such structure representation is instrumental in transferring motion from driving videos to still images. However, such approaches fail in case the source image and driving video encompass large appearance variation. Moreover, the extraction of structure information requires additional modules that endow the animation-model with increased complexity. Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation. LIA is streamlined to animate images by linear navigation in the latent space. Specifically, motion in generated video is constructed by linear displacement of codes in the latent space. Towards this, we learn a set of orthogonal motion directions simultaneously, and use their linear combination, in order to represent any displacement in the latent space. Extensive quantitative and qualitative analysis suggests that our model systematically and significantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t. generated quality.
1 INTRODUCTION
LIA addresses the complexity and appearance-variation limitations of structure-guided image animation by navigating learned motion directions directly in latent space. Its single encoder-generator architecture also jointly handles motion and appearance, while evaluations report improved performance across multiple datasets and unseen identities.
- Motivation: Existing self-supervised animation methods predict dense optical flow using explicit keypoints or regions learned by additional networks.These approaches avoid domain knowledge and labeled ground truth but require explicit structure representations as motion guidance.
- Contribution: LIA eliminates explicit structure representations by linearly navigating a source latent code along a learned path to a target code.Linear Motion Decomposition represents each path as a combination of learned orthogonal motion directions and associated magnitudes.
- Contribution: A learned orthogonal basis describes the motion space, with each direction corresponding to a basic visual transformation.The approach is motivated by latent-space directions that induce transformations such as zooming and rotation in generated images.
- Contribution: LIA disentangles motion and appearance within one encoder-generator architecture instead of using separate networks for disentangled features.The authors state that this reduces model complexity and simplifies training.
- Evaluation: LIA is evaluated on VoxCeleb, TaichiHD, and TED-talk, with additional one-shot tests on unseen FFHQ and GermanPublicTV datasets.The paper reports improved facial-structure preservation over state-of-the-art methods in the unseen-dataset setting.
2 RELATED WORK
Prior image-animation methods commonly rely on structural labels or intermediate representations, while LIA transfers motion by navigating a learned latent space. The pipeline encodes images, combines learned motion directions, decodes a flow field, and warps the source image.
- Video generation: Video-generation work has explored mapping noise to videos and combining image generators with recurrent networks for high-resolution synthesis.Earlier deep probabilistic approaches reported results mainly on low-resolution toy datasets.
- Latent-space approach: LIA replaces explicit structure guidance with latent-space navigation using learned motion directions and associated magnitudes.Its two-step pipeline encodes a source image, navigates its latent code, decodes a dense flow field, and warps the source image.
- Latent space editing: Latent-space editing methods discover semantic directions for manipulating attributes, colors, rotation, and zooming in generated or inverted real images.Both supervised and unsupervised approaches have been proposed.
- Image animation: Earlier image-animation approaches used human keypoints, facial landmarks, or other strong structural labels as motion guidance.These methods mapped structural representations to videos in image-to-image translation settings.
- Latent-space approach: The LIA overview describes an encoder E and generator G trained to reconstruct a driving frame from a source frame sampled from the same video.Linear Motion Decomposition combines a driving-derived magnitude vector with an orthogonal motion dictionary.
3 METHOD
LIA trains a self-supervised autoencoder to animate source images by navigating latent space and decoding motion into flow fields for warping. Its pipeline combines orthogonal motion decomposition, appearance-aware encoding, multi-scale flow and mask prediction, refinement, and reconstruction losses for video inference.
- Training: During training, LIA encodes source–driving image pairs and learns to reconstruct the driving image from the source image.The model uses reconstruction, perceptual, and adversarial losses.
- Latent code driven image animation: The encoder navigates the source code to a target code, and the generator decodes that code into a dense flow field used to warp the source image.The generator includes a flow-field generator and a refinement network.
- Latent motion representation: LIA represents motion as a latent path formed by a linear combination of learned orthogonal motion directions and their magnitudes.The directions are orthogonalized with Gram-Schmidt during each forward pass, while magnitudes are predicted from the driving-image code.
- Latent code driven image animation: Multi-scale flow fields and masks warp source features while identifying regions that require inpainting because of occlusions.Residual modules produce two flow channels and one mask channel, and refinement combines inpainted features with upsampled images.
- Inference: At inference, relative transfer removes the driving sequence’s initial-motion component so motion can be applied to a source image from another identity.The method preserves the source pose while transferring driving-video motion, but exact pose replication requires similar relative poses in the source and first driving frame.
4 EXPERIMENTS
LIA is evaluated across three datasets through qualitative, quantitative, user-study, ablation, and latent-code analyses. Results show strong reconstruction and motion-transfer quality, while further analyses examine the motion dictionary, canonical reference images, and semantic directions.
- Experimental setup: LIA is evaluated on VoxCeleb, TaichiHD, and TED-talk using reconstruction, cross-video motion transfer, user preference, ablation, and latent-code analyses.The setup uses 256 × 256 frames, four comparison methods, and metrics covering reconstruction faithfulness, video quality, and semantic consistency.
- Qualitative results: LIA transfers motion on articulated bodies without explicit structure representations and avoids visible face-shape deformations when transferring from VoxCeleb to unseen FFHQ images.The qualitative comparison highlights large pose variations between source and driving images.
- Comparison with state-of-the-art methods: LIA outperforms competing approaches on all foreground-reconstruction metrics across the three evaluated datasets.Same-identity reconstruction uses the first frame as the source and the remaining frames as driving images.
- Comparison with state-of-the-art methods: LIA achieves the best video FID among the compared approaches, indicating the best generated video quality in the reported cross-video evaluation.Table 2 covers inner- and cross-dataset tasks on VoxCeleb and GermanPublicTV.
- Comparison with state-of-the-art methods: Twenty human raters preferred LIA’s generated videos to those from FOMM or MRAA across all three datasets, matching the quantitative evaluation.The user study displayed paired videos for subjective realism judgments.
- Ablation study: Removing the motion dictionary causes failures in high-quality image generation, while 20 learned directions produce the best reconstruction results.The dictionary ablation is consistent across all datasets, and five dictionary sizes are tested.
5 CONCLUSIONS
LIA is a self-supervised autoencoder that animates still images through latent-space navigation, eliminating explicit structure representations. Its motion dictionary is interpretable, and evaluations report performance exceeding state-of-the-art algorithms across benchmarks.
- LIA formulates motion transfer from driving videos to source images as learning linear transformations in latent space.
- The approach animates still images while eliminating the necessity of explicit structure representations.
- LIA’s motion dictionary is interpretable and contains directions pertaining to basic visual transformations.
- Both quantitative and qualitative evaluations showed that LIA outperforms state-of-the-art algorithms on all benchmarks.
ETHIC STATEMENT
The framework is intended for transferring motion onto still images and is presented as applicable to several creative, educational, and computer-vision uses. It focuses on modeling motion distribution rather than directly modeling appearance.
- The approach aims to synthesize high-quality videos by transferring motion onto still images.
- The framework is described for movie production, video games, online education, and synthetic-data generation for computer-vision tasks.
- The framework mainly focuses on modeling motion distribution rather than directly modeling appearance.
A DETAILS OF MODEL ARCHITECTURE
The architecture section details the encoder, its ResBlocks, appearance features, and the generator components used to produce and refine motion-related outputs.
- The encoder E uses ResBlocks that downsample the spatial size of input feature maps.
- Feature maps ranging from 8 × 8 to 256 × 256 are used as appearance features xenc_i.
- The generator G consists of a flow-field generator Gf and a refinement network Gr.
- Figure 7 presents details of the encoder E and its ResBlock architecture.
B EXPERIMENTS
This section introduces the datasets and evaluation metrics used in the experiments.
- The section introduces the details of the datasets used in the experiments.
- The section introduces the evaluation metrics used in the experiments.
- The datasets and metrics are presented as part of the paper’s experimental evaluation.
B.1 DATASETS
The evaluation uses VoxCeleb, TaiChiHD, and TED-talk datasets alongside reconstruction, perceptual, keypoint, identity, and video-quality metrics. Results with full MRAA show competitive reconstruction and keypoint performance, including better AKD and MKR than MRAA on TaiChiHD, though background content limits reconstruction comparisons.
- Datasets: VoxCeleb contains 17,928 training videos and 495 test videos, cropped to 256×256 resolution.
- Datasets: TaiChiHD contains 1,096 training videos and 115 testing videos of full-body Tai Chi actions at 256×256 resolution.
- Datasets: TED-talk contains 1,124 training videos and 130 testing videos, resized to 256×256 resolution after cropping the main subjects.
- Evaluation metrics: The evaluation combines L1, LPIPS, AKD, MKR, AED, and video FID to assess reconstruction, perceptual similarity, keypoints, identity, and video quality.Video FID compares real and generated video distributions using spatio-temporal features from the first 100 frames.
- Comparison with full MRAA: On TaiChiHD, LIA outperforms MRAA in AKD and MKR despite not explicitly predicting keypoints.Reconstruction scores are not fully fair for subject quality because backgrounds strongly contribute on TaiChiHD and TED-talk.
B.5 QUALITATIVE RESULTS ON EFFECTIVENESS OF USING MOTION DICTIONARY
The motion dictionary improves disentanglement during motion transfer, preventing driving-video appearance from leaking into generated videos.
- Without the motion dictionary, appearance information is undesirably transferred from driving videos to generated videos.The comparison transfers motion from VoxCeleb to GermanPublicTV.
B.6 LIMITATIONS
LIA remains limited in human-body motion transfer when occlusion or small, articulated hands make motion difficult to represent.
- Arm-leg occlusion in Taichi videos can prevent successful motion transfer.
- Hand motion in TED-talk videos remains challenging because hands are small, articulated, and sometimes occluded by bodies.
- Disentanglement of appearance and motion is much better when Dm is used.