LaMD: Latent Motion Diffusion for Image-Conditional Video Generation

1 Introduction

• Pixel-space generation is difficult to train efficiently at video scale, while image-latent generation may lack temporal coherence and video-latent generation is challenging to model.Direct pixel generation also risks allocating capacity to spatial high-frequency details that are less perceptually noticeable.Source: §1 · pp. 1–2
• Figure 1 distinguishes training-only operations with dashed lines from operations used in both training and sampling with solid lines.The figure compares video generation in different latent spaces, including the proposed latent motion paradigm.Source: Figure 1 · p. 2, §1 · p. 2
• The proposed latent motion paradigm separates motion from appearance, then generates motion in a compact latent space to improve expressiveness and reduce modeling complexity.The paper states that the compressed representation eliminates the need for a 3D-CNN structure in generative models, decreasing sampling time.Source: §1 · p. 2
• LaMD comprises MCD-VAE, which decomposes motion from appearance and reconstructs video, and DMG, which denoises motion variables in continuous latent space under image and optional text conditions.DMG’s computational complexity is described as comparable to image diffusion models.Source: §1 · p. 2
• LaMD achieves competitive performance across BAIR, Landscape, NATOPS, MUG and CATER-GEN, spanning stochastic dynamics to highly controllable movements, while substantially reducing sampling time.The reported tasks include stochastic image-to-video, class-guided image-to-video and text-image-to-video generation; sampling speed is comparable to image diffusion models.Source: §1 · pp. 2–3
• The framework produces coherent, realistic motion under an appearance constraint, with optional text conditioning increasing motion control.The contribution also introduces latent motion generation as a paradigm focused on motion expressiveness and perceptive video quality.Source: §1 · p. 2

Pixel Space Video Generation

• Pixel-space methods model video frames directly or generate global motion representations, but video-data complexity makes high-resolution generation challenging and can leave models spending capacity on imperceptible information.Some methods add structural or action clues for controllability, while diffusion models also face slow sampling.Source: §2 · p. 3

Latent Space Video Generation

• Latent-space approaches use autoencoders to compress video, generating either framewise image latents or spatio-temporal video latents, each with distinct coherence and modeling trade-offs.Image autoencoders support high-quality frames but may break temporal dependencies; 3D-based video autoencoders preserve spatio-temporal structure.Source: §2 · p. 3
• Image-conditional latent video models can generate motion-rich videos, but typically require considerable model complexity and slow sampling.Some leverage extensive video datasets or off-the-shelf text-to-video models.Source: §2 · pp. 3–4

Related Work

• Unlike LFDM (Ni et al, 2023) and Shi et al (2024), which use 3D-UNet diffusion for latent flows or optical flows, LaMD uses a compact latent motion representation with a 2D-UNet.The paper attributes increased motion expressiveness and sampling speed to this design.Source: §2 · p. 4

3 Method

• The two-stage pipeline trains MCD-VAE on video-only data, then trains DMG on motion latents; at sampling, DMG generates motion latents that the decoder uses with image content.Motion generation can also be guided by optional conditions.Source: §3 · p. 4

3.1 Motion-Content Decomposed Video Autoencoder

• Separate encoders preserve image content while compressing motion, whose spatial and temporal redundancy supports a compact latent representation.The motion encoder removes the temporal dimension by transferring motion information into channels.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 Decomposition of motion and content

• An information bottleneck limits the motion branch so it retains temporal information while excluding content already carried by the image branch.A KL penalty with adjustable β controls the motion-branch capacity.Source: §3.1.1 · pp. 4–5

3.1.2 Video reconstruction

• The fusion decoder combines content features with video features through concatenation and 1×1 convolutions, then uses temporal interpolation to restore the video’s temporal dimension.Training also uses L1 and LPIPS reconstruction losses, a KL penalty, and an adversarial objective.Source: §3.1.2 · p. 5

3.2 Diffusion-based Motion Generator

• DMG learns motion distributions through a diffusion denoising process, using a 2D-UNet to predict noise and conditioning on image content plus optional class labels or text.The reverse process starts from Gaussian noise and progressively denoises motion latents.Source: §3.2 · p. 6
• LaMD has the fastest sampling among video diffusion paradigms, with computational cost comparable to image diffusion models.Its 2D-UNet-based motion diffusion is at least L·r_s^2 less complex than pixel-space diffusion.Source: Figure 4 · p. 6, §3.2 · p. 6

• Evaluation covers stochastic image-to-video, class-guided image-to-video, and text-image-to-video generation.Source: §4 · p. 6
• Experiments span five benchmark datasets across robotics, landscape, human body or face, and synthetic scenarios.Source: §4 · p. 7
• The benchmarks encompass varied stochastic dynamics and highly controllable motion.Source: §4 · p. 7

• BAIR and Landscape evaluate stochastic image-to-video generation, NATOPS and MUG evaluate class-guided generation, and CATER-GEN evaluates text-image-to-video generation.Source: §4.1 · p. 7
• BAIR videos use 16 frames at 64 × 64 resolution, while Landscape videos use 32 frames at 128 × 128 resolution.BAIR generation is conditioned on the first image; Landscape settings follow prior work for fair comparison.Source: §4.1 · p. 7
• NATOPS testing holds out 10 of 20 subjects, MUG testing holds out 26 of 52 subjects, and CATER-GEN-v2 provides 6,000 test video-text pairs.Source: §4.1 · p. 7
• Evaluation uses FID and LPIPS for image-level quality and FVD for video-level quality, with cFVD and sFVD assessing class alignment and subject consistency.Both mean and standard variance are reported for cFVD and sFVD.Source: §4.1 · pp. 7–8
• Both MCD-VAE and DMG are trained without external data, and DMG sampling uses 200 timesteps.Sampling time is measured on a single NVIDIA GeForce RTX 3090 GPU.Source: §4.1 · p. 7, Table 2 · p. 8
• For stochastic generation, additional metrics assess motion smoothness, dynamic degree, temporal flickering, and background consistency.The metrics use motion priors, optical-flow strength, frame differences, and CLIP feature similarity, respectively.Source: §4.1 · p. 7

4.2 Reconstruction Ability of MCD-VAE

• The evaluation examines both decomposition and reconstruction because MCD-VAE is designed to compress motion, fuse it with content, and reconstruct pixels.Source: §4.2 · p. 9

4.2.1 Reconstruction Capability

• MCD-VAE retains fine-grained movement under compression and matches VQGAN on frame-level metrics while outperforming it on a video-level metric.The reported example uses spatial compression rs = 4 and temporal compression rt = 32; reconstruction quality is treated as an upper bound for generation.Source: §4.2.1 · p. 9

4.2.2 Decomposition Capability

• Cross-video transfer combines one video's motion with another video's content; the synthesized video follows the former's movement while preserving the latter's appearance.The authors use this qualitative test because decomposition efficacy is difficult to measure quantitatively.Source: §4.2.2 · p. 9

4.3.1 Stochastic image-to-video generation

• Generated BAIR and Landscape videos preserve the input image's appearance, show reasonable cloud deformation and long-range consistency, and provide realistic, diverse motion across samples.Source: §4.3.1 · p. 9
• LaMD significantly outperforms state-of-the-art methods in overall video quality on BAIR and Landscape, as measured by FVD.Source: §4.3.1 · p. 9, Table 6 · p. 11, Table 7 · p. 11
• On Landscape, LaMD exceeds all baselines in motion smoothness and temporal flickering; versus cINN, it produces larger motions with only marginally lower background consistency.DynamiCrafter has higher dynamic degree, while LaMD performs better in motion smoothness and temporal consistency.Source: §4.3.1 · pp. 9–10, Table 5 · p. 10

4.3.2 Class-guided image-to-video generation

• On NATOPS and MUG, LaMD outperforms latent image/video baselines and is competitive with LFDM; it also reports lower cFVD and sFVD variance and faster sampling.Source: §4.3.2 · pp. 10–11

4.3.3 Text-image-to-video generation

• For text-image-to-video generation, DMG combines text embeddings with image content features so latent motion can condition on both inputs.Source: §4.3.3 · p. 11
• On CATER-GEN, LaMD outperforms existing methods on LPIPS, FID, FVD, action precision, and referring-expression precision, while generating diverse videos with coherent text-specified motion.The authors note that standard metrics have difficulty assessing fine-grained movement precision.Source: §4.3.3 · p. 11

4.4.2 Ablation Study of Motion Capacity

• Removing latent motion’s temporal dimension preserves reconstruction performance and permits 2D diffusion, reducing complexity and bringing inference burden near image diffusion.The comparison is made at the same capacity.Source: §4.4.2 · p. 12, Table 12 · p. 14

4.4.1 Sampling Time

• 26.6s vs 10.7s for 128-frame videos: LaMD samples faster than LVDM, while its speed is comparable to image sampling with LDM.LaMD’s sampling time increases only slightly with longer videos or higher resolution; LFDM’s time increases significantly with video length.Source: §4.4.1 · pp. 11–12, Table 11 · p. 14

4.4.2 Ablation Study of Motion Capacity

• In a channel-size ablation with β unchanged, MCD-VAE achieves satisfactory reconstruction at d = 1; performance improves as channels increase, then slightly attenuates, and the authors settle on d = 3.The attenuation is attributed to redundancy and potential overfitting.Source: §4.4.2 · pp. 12, 14

4.4.3 Trade-off analysis

• On Landscape, smaller latent spaces weaken MCD-VAE reconstruction but ease generator training and improve LaMD generation, revealing a reconstruction–generation trade-off.The paper reports that a similar trade-off also exists in other two-stage paradigms.Source: §4.4.3 · p. 14, Table 13 · p. 15

4.4.4 Visualization of sampling process

• At t=999, random-motion samples show chaotic movement with maintained backgrounds; recursive denoising progressively yields orderly, natural motion by t=0.The figure caption identifies t=999 as the random-latent-motion state and t=0 as the final generated videos.Source: Figure 10 · p. 14, §4.4.4 · p. 15

5 Conclusion and Discussion

• LaMD decomposes and compresses video motion, then reconstructs videos through motion-content fusion and progressively generates motion in continuous latent space.The framework comprises MCD-VAE and the diffusion-based motion generator DMG.Source: §5 · p. 15
• Experiments report high-quality videos spanning stochastic dynamics to highly controllable movements, with faster sampling than other video diffusion models.Source: §5 · p. 15
• LaMD is currently image-conditional and evaluated on close-domain datasets; its motion decomposition is primarily effective for short videos without major content changes.For long videos with new objects, latent motion may entangle partial appearance features from those emerging elements.Source: §5 · p. 15