Source-linked AI summary
LEO: Generative Latent Image Animator for Human Video Synthesis
Yaohui Wang, Xin Ma, Xinyuan Chen, Cunjian Chen, Antitza Dantcheva, Bo Dai, Yu Qiao
TL;DR
Human video synthesis still faces challenges in maintaining spatio-temporal coherence, and earlier approaches have struggled to strictly separate appearance from motion. LEO uses flow maps, an image animator, and a Latent Motion Diffusion Model to generate videos; it improves coherence over previous methods and supports long-form synthesis and content-preserving editing.
Problem
Human video synthesis with rich global and local deformations remains challenging, while prior methods can leave appearance and motion incompletely disentangled.
Source: §1 · pp. 1–2Method
LEO combines a flow-based image animator that maps motion codes to flow maps with LMDM, which synthesizes motion-code sequences.
Source: §1 · p. 2Results
LEO significantly improves spatio-temporal coherency over previous methods, including in synthesized videos of 512 frames.
Source: §1 · p. 3Takeaways & Limitations
LEO supports infinite-length human video synthesis through autoregressive LMDM application and content-preserving video editing with an off-the-shelf image editor.
Source: §8 · p. 12LEO still has a large performance gap compared with large-scale video diffusion models.
Source: §7 · p. 12
Abstract
from arXiv · showhide
Spatio-temporal coherency is a major challenge in synthesizing high quality videos, particularly in synthesizing human videos that contain rich global and local deformations. To resolve this challenge, previous approaches have resorted to different features in the generation process aimed at representing appearance and motion. However, in the absence of strict mechanisms to guarantee such disentanglement, a separation of motion from appearance has remained challenging, resulting in spatial distortions and temporal jittering that break the spatio-temporal coherency. Motivated by this, we here propose LEO, a novel framework for human video synthesis, placing emphasis on spatio-temporal coherency. Our key idea is to represent motion as a sequence of flow maps in the generation process, which inherently isolate motion from appearance. We implement this idea via a flow-based image animator and a Latent Motion Diffusion Model (LMDM). The former bridges a space of motion codes with the space of flow maps, and synthesizes video frames in a warp-and-inpaint manner. LMDM learns to capture motion prior in the training data by synthesizing sequences of motion codes. Extensive quantitative and qualitative analysis suggests that LEO significantly improves coherent synthesis of human videos over previous methods on the datasets TaichiHD, FaceForensics and CelebV-HQ. In addition, the effective disentanglement of appearance and motion in LEO allows for two additional tasks, namely infinite-length human video synthesis, as well as content-preserving video editing.
1 Introduction
Human video synthesis remains challenging when motion and appearance are not strictly disentangled, with prior methods exhibiting spatial and temporal artifacts. LEO represents motion with flow maps and combines a flow-based image animator with a Latent Motion Diffusion Model to synthesize coherent videos.
1 Introduction
- LEO supports unconditional generation, single-image conditional generation, and video editing from a starting image.The model is trained on TaichiHD, FaceForensics, and CelebV-HQ.Source: Figure 1 · p. 2
- Prior approaches to appearance-motion disentanglement can still produce spatial distortions and interframe semantic jitter, even in 16-frame videos.The authors argue these artifacts stem from incomplete disentanglement without strong constraints.Source: §1 · p. 2
- LEO represents motion as a sequence of flow maps, which inherently isolate motion from appearance.The framework combines a flow-based image animator with LMDM.Source: §1 · p. 2
- The image animator generates flow maps from motion codes and synthesizes frames in a warp-and-inpaint manner.An initial frame can be provided for conditional generation or generated for unconditional generation.Source: §1 · pp. 2–3
- LEO trains the image animator first, then trains LMDM to synthesize motion-code sequences and capture the training data's motion prior.Linear Motion Condition models residuals relative to a starting motion code so longer videos can be formed by concatenating residual sequences.Source: §1 · p. 3
- LEO significantly improves spatio-temporal coherency over previous methods, including in synthesized videos of 512 frames.The evaluation covers TaichiHD, FaceForensics, and CelebV-HQ.Source: §1 · p. 3
2 Related Works
Prior video-generation work has used GANs, VAEs, discrete spatiotemporal codebooks, and diffusion models, alongside attempts to separate appearance and motion. LEO uses dense flow maps for motion modeling in its single-image conditional video-generation pipeline.
2 Related Works
- Unconditional video generation aims to learn the full training-dataset distribution; most previous works were built upon GANs [27], while VAE-based approaches were proposed but showed results only on toy datasets.Source: §2 · p. 3
- Earlier approaches attempted appearance-motion disentanglement through training strategies or architectures, but strong constraints remained difficult to obtain.Examples include MoCoGANHD, DIGAN, StyleGAN-V, MoStGAN-V, VideoGPT [19], and TATS.Source: §2 · p. 3
- Conditional video-generation methods have used semantic maps, keypoints, motion labels, 3DMM, optical flow, and text as guidance.LEO also supports single-image conditioning, using dense flow maps for motion modeling in an image-animation pipeline.Source: §2 · p. 4
3 Method
LEO first learns motion-only codes and maps them to flow fields, then trains a diffusion model to generate motion-code sequences. Those codes animate a starting image through warping and inpainting, with options for conditional or unconditional video generation and autoregressive long videos.
3 Method
- A two-phase pipeline trains an image animator for motion codes and flow maps, then an LMDM to learn a motion prior and generate frame-by-frame videos through warp-and-inpaint.The animator warps the starting frame and inpaints holes before refining the image.Source: §3 · p. 4
3.1 Learning Latent Motion Codes
- LIA encodes source and driving images into motion codes and maps them to flow fields, while self-supervised reconstruction yields codes tied to motion rather than appearance.The training objective reconstructs the driving image.Source: §3.1 · p. 5
3.2 Leaning a Motion Prior
- A temporal diffusion model learns sequence correlations, while cLMDM conditions generation on the first motion code and noises only the motion differences.The first code is retained as a condition, then combined with the generated differences to form the motion sequence.Source: §3.2 · p. 5
- LMDM combines cLMDM with a frame-wise diffusion model for conditional and unconditional motion generation, and autoregressive reuse of the last code supports infinite-length sequences.The last code from a generated segment becomes the first code of the next segment.Source: §3.2 · pp. 5–6
3.3 Learning Starting Frames
- A starting image can come from real or generated images, or from a conditional DDPM conditioned on the starting motion code for unconditional video synthesis.When an image is supplied, LEO predicts future motion from it.Source: §3.3 · pp. 6–7
4 Experiments
LEO is evaluated on three human-video datasets using visual comparisons, video-quality and identity metrics, and user studies. Results report better or competitive short-video generation, stronger long-video results, and qualitative gains in human-structure preservation.
4 Experiments
- Experiments cover TaichiHD, FaceForensics, and CelebV-HQ, using FVD, KVD, ACD, and user ratings of video quality and coherency.The user study involved 20 raters; the evaluation also compares conditional and unconditional short- and long-term generation.Source: §4 · pp. 7–8, §4.2 · p. 8
- Training uses two phases: an image animator learns motion codes and flow-map reconstruction, then LMDM learns sequences of those codes.The evaluation includes both 128×128 and 256×256 TaichiHD, and 256×256 FaceForensics and CelebV-HQ.Source: §4 · pp. 7–8
4.1 Qualitative Evaluation
- Qualitative comparisons show better appearance and motion, complete human structures on TaichiHD, and preserved facial structures on FaceForensics during large motion.For 512-frame TaichiHD videos, LEO preserves human structure and diverse actions, whereas TATS begins to crash around 50 frames and then fades; LEO was trained on 64-frame sequences.Source: §4.1 · p. 8
4.2 Quantitative evaluation
- On short videos, LEO systematically outperforms other methods for conditional generation and achieves better or competitive unconditional results on FVD, KVD, and ACD.The user study found that nearly all users rated LEO's generated results higher than other approaches for quality and coherency.Source: §4.2 · p. 8, Table 1 · p. 9, Table 3 · p. 9
- LEO outperforms other methods on 128-frame generation, while its short-video results are better or competitive across the reported metrics.The long-video comparison covers FVD, KVD, and ACD; unconditional short-video results are reported across datasets and resolutions.Source: §4.2 · pp. 8–9, Table 2 · p. 9, Table 1 · p. 9
- Conditioning on real starting images achieves the lowest FVD, KVD, and ACD values among the reported short-video settings.The authors state that starting-image quality is pertinent to output video quality.Source: §4.2 · p. 9, Table 1 · p. 9
5 Ablation Study
The ablation study tests Linear Motion Condition (LMC) in LMDM on TaichiHD and FaceForensics and reports improved generated-video quality with LMC.
5 Ablation Study
- Using LMC significantly improves generated-video quality on TaichiHD and FaceForensics, with the lowest FVD on both datasets.The authors identify LMC as an effective mechanism for involving α1 in LMDM.Source: §5 · p. 9
6 Additional Analysis
Additional analyses examine appearance–motion disentanglement, video editing, and long-term generation. They report diverse motion from shared content, appearance edits from the starting image, and long sequences with methods to address repeated motion.
6 Additional Analysis
- Different motion codes produce different motions for the same subject, while different noise sequences yield diverse motion sequences.The authors interpret this as evidence that appearance and motion are disentangled and that the model does not overfit the training dataset.Source: §6 · pp. 9–10, Figure 6 · p. 10
- Editing the starting image with ControlNet changes appearance across the video while preserving its original temporal consistency.The described editing setup modifies the starting image in a one-shot manner.Source: §6 · p. 10, Figure 7 · p. 10
- Autoregressive generation produces more than 1000 FaceForensics frames, while TaichiHD's limited motion patterns lead to repetition that Transition DM is reported to prevent.Transition DM generates motion between the last code of the original sequence and a new code; repeated motion was assessed by human raters.Source: §6 · p. 10, Figure 8 · p. 11, Table 5 · p. 11
- Long-sequence inference takes only a few seconds for LMDM, and the image animator uses one-step inference.The authors attribute efficiency to LMDM's small scale and its focus on 1-D motion codes.Source: §6 · p. 11
7 Limitations
LEO’s current limitations concern geometry and occlusion, its human-centric scope, and the gap to large-scale video diffusion models. Broader use would require further architectural and data scaling, with substantial computational cost.
7 Limitations
- LEO currently performs better on human-centric videos because its pretrained image animator focuses on talking heads and human bodies.The authors report only a small-scale UCF101 generalizability experiment.Source: §7 · p. 11
- Because LEO predicts 2D flow maps, it handles human-body occlusion poorly, especially on Taichi; NeRF or Tri-plane integration is proposed as a possible remedy.Source: §7 · p. 11
- Extending LEO to settings such as text-to-video would require redesigning both LIA and LMDM, training on larger curated datasets, and substantial computational resources.Source: §7 · p. 12
- 1356.2 FVD on UCF101; LEO is competitive with GAN-based methods but has a large performance gap compared with large-scale video diffusion models.The authors describe the UCF101 evaluation as a small-scale generalizability experiment.Source: §7 · pp. 11–12
- Both the image animator and latent motion diffusion model rely on convolutional networks; transformer architectures remain unexplored.The authors identify novel architecture design and larger-scale training as future work.Source: §7 · p. 12
8 Conclusions
LEO combines a Latent Image Animator (LIA) and Latent Motion Diffusion Model (LMDM) in a two-phase training strategy to disentangle appearance and motion. The authors report photo-realistic long human videos and demonstrate infinite-length synthesis and content-preserving editing.
8 Conclusions
- LEO combines a Latent Image Animator (LIA) and Latent Motion Diffusion Model (LMDM), using two-phase training to disentangle appearance and motion.Source: §8 · p. 12
- The authors report that LEO produces photo-realistic long human videos and supports infinite-length synthesis and content-preserving video editing.Infinite-length synthesis applies LMDM autoregressively; editing employs an off-the-shelf image editor such as ControlNet.Source: §8 · p. 12