PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, Martin Renqiang Min

arXiv:2303.13744v1cs.CV

TL;DR

Conditional image-to-video generation must jointly preserve spatial appearance and generate condition-consistent temporal dynamics. LFDM addresses this by generating temporally coherent latent flow to warp the input image, using a two-stage training strategy, and achieves state-of-the-art performance across multiple datasets. Its scope is currently limited by single-subject, class-label conditioning and slow 1000-step DDPM sampling.

  • Problem

    Conditional image-to-video methods often struggle to preserve spatial details while maintaining temporal coherence when directly synthesizing video frames from an image and condition.

  • Method

    LFDM generates condition-guided latent optical-flow sequences that warp the given image in latent space, trained through separate latent-flow auto-encoder and conditional diffusion stages.

  • Results

    LFDM achieves state-of-the-art performance across multiple datasets and consistently outperforms prior methods on facial-expression, human-action, and gesture videos.

  • Takeaways & Limitations

    LFDM’s disentangled training process enables adaptation to new domains by finetuning the image decoder.

  • Takeaways & Limitations

    Current experiments are limited to single moving subjects and class-label conditions, while 1000-step DDPM sampling is slower than GAN sampling.

Abstract

from arXiv · show

Conditional image-to-video (cI2V) generation aims to synthesize a new plausible video starting from an image (e.g., a person's face) and a condition (e.g., an action class label like smile). The key challenge of the cI2V task lies in the simultaneous generation of realistic spatial appearance and temporal dynamics corresponding to the given image and condition. In this paper, we propose an approach for cI2V using novel latent flow diffusion models (LFDM) that synthesize an optical flow sequence in the latent space based on the given condition to warp the given image. Compared to previous direct-synthesis-based works, our proposed LFDM can better synthesize spatial details and temporal motion by fully utilizing the spatial content of the given image and warping it in the latent space according to the generated temporally-coherent flow. The training of LFDM consists of two separate stages: (1) an unsupervised learning stage to train a latent flow auto-encoder for spatial content generation, including a flow predictor to estimate latent flow between pairs of video frames, and (2) a conditional learning stage to train a 3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike previous DMs operating in pixel space or latent feature space that couples spatial and temporal information, the DM in our LFDM only needs to learn a low-dimensional latent flow space for motion generation, thus being more computationally efficient. We conduct comprehensive experiments on multiple datasets, where LFDM consistently outperforms prior arts. Furthermore, we show that LFDM can be easily adapted to new domains by simply finetuning the image decoder. Our code is available at https://github.com/nihaomiao/CVPR23_LFDM.

1. Introduction

LFDM addresses conditional image-to-video generation by synthesizing temporally coherent latent optical flow that warps the given image, separating spatial content from temporal dynamics through two-stage training.

  • Conditional image-to-video generation must preserve the given image’s spatial details while producing temporally coherent motion under a condition.
  • LFDM generates a condition-guided latent optical-flow sequence and warps the given image in latent space to produce new videos.The generated flow is temporally coherent and reuses the given image’s spatial content.
  • LFDM training uses two stages: an unsupervised latent flow auto-encoder followed by a conditional 3D U-Net diffusion model for temporal latent-flow generation.The stages separately address spatial content generation and temporal dynamics.
  • During inference, LFDM generates latent flow and occlusion-map sequences, then warps the original image frame-by-frame instead of previous synthesized frames.Warping the original image avoids artifact accumulation.
  • The disentangled training process enables adaptation to new domains by finetuning the image decoder.
  • LFDM consistently outperforms previous state-of-the-art methods across facial-expression, human-action, and gesture video datasets.

2. Related Work

Related image-to-video methods differ in their motion guidance and synthesis strategy, while diffusion-based video generation provides a relevant foundation for LFDM.

  • Image-to-video generation includes stochastic methods using only the given image and conditional methods using additional motion cues.
  • Pose-guided and pose-to-video methods can struggle to produce fine-grained details, while interactive methods require users to specify motion manually.
  • Some methods estimate optical flow from motion-direction inputs, but complex motions remain difficult to generate without detailed guidance.
  • Prior stochastic flow-based methods generate flow with GANs or VAEs and condition only on the given image, unlike LFDM’s condition-guided diffusion flow generation.
  • Video diffusion models adapt image diffusion architectures by replacing 2D U-Nets with 3D U-Nets for video generation.

3. Our Method

LFDM formulates conditional image-to-video generation as temporally coherent latent-flow synthesis that warps a starting image, separating spatial-content reconstruction from motion generation. Its two-stage framework uses an unsupervised latent flow auto-encoder and a conditional diffusion model operating in a lower-dimensional latent flow space.

  • Problem: cI2V aims to generate a video matching a starting image and class condition, with the synthesized conditional distribution matching real videos given those inputs.The paper considers class labels as the input condition.
  • Framework: LFDM separates spatial-content generation from temporal dynamics through two training stages.Stage one trains a latent flow auto-encoder; stage two trains a diffusion model for temporally coherent latent-flow sequences.
  • Stage One: Latent Flow Auto-Encoder: In stage one, an unsupervised latent flow auto-encoder estimates flow and occlusion between video frames, warps a reference latent map, and decodes it to reconstruct the driving frame.The reconstruction objective uses a perceptual loss between the output and driving frame, requiring only unlabeled video frames.
  • Stage Two: Diffusion Model: In stage two, a 3D-UNet diffusion model learns temporally coherent latent flow and occlusion sequences conditioned on the starting image and class embedding.Training derives flow and occlusion sequences with the stage-one flow predictor, concatenates them, and applies the DDPM forward process before conditional denoising.
  • Stage Two: Diffusion Model: The diffusion model operates in a K × Hz × Wz × 3 latent flow space containing motion and shape features, which can be lower-dimensional than RGB pixel space.The paper states that this space can be easier to model than latent feature spaces coupling spatial and temporal information.
  • Inference: During inference, the diffusion model generates flow and occlusion sequences, warps the starting latent map, and decodes the resulting latent maps into video frames without the flow predictor.The image encoder and decoder remain part of the inference path, while the stage-one flow predictor is used only during training.

4. Experiments

Experiments evaluate LFDM across facial-expression, human-action, and gesture datasets using FVD and qualitative comparisons. LFDM consistently outperforms baselines, better preserves details and motion, generalizes to unseen images, and reduces sampling cost relative to VDM.

  • Datasets and Metrics: Experiments use MUG, MHAD, and NATOPS datasets covering facial expressions, human actions, and body-and-hand gestures.Videos are resized to 128 × 128, and training clips contain 40 temporally sorted frames.
  • Datasets and Metrics: FVD evaluates visual quality, temporal coherence, and sample diversity using I3D features from real and generated videos.
  • Conditional Generation: LFDM consistently outperforms ImaGINator, VDM, and LDM for cI2V generation at both 64 × 64 and 128 × 128 resolutions.Qualitatively, ImaGINator and LDM miss fine-grained details, while VDM can produce nearly static sequences.
  • Result Analysis: LFDM requires about 0.9GB and 36s to generate one 128 × 128 video, versus VDM’s 2.5GB and 112.5s for a 64 × 64 video.The comparison uses batch size 10, 1000-step DDPM sampling, and one NVIDIA A100 GPU.
  • Result Analysis: LFDM shows the smallest training-testing performance gap on MUG, indicating stronger performance when conditioned on unseen images than the baselines.The authors attribute this to the warp-based design and two-stage disentangled training framework.
  • Stochastic Generation: LFDM also outperforms the baselines on MUG stochastic image-to-video generation without a condition label.This setting removes the condition input during inference and retrains ImaGINator without that input.
  • Sampling Strategies: Sampling ablations find that 10-step DDIM performs better than 100-step DDIM while sampling faster, whereas guidance scale 1.1 slightly improves FVD but doubles inference time.The default setting uses guidance scale g = 1.

5. Conclusion and Discussion

The paper concludes that LFDM generates videos by warping given images with class-conditioned latent flow and achieves state-of-the-art performance across multiple datasets. Its main scope boundaries are single-subject videos, class-label conditioning, and slower 1000-step DDPM sampling than GAN models.

  • Conclusion: LFDM generates videos by warping given images with class-conditioned flow sequences in the latent space.
  • Conclusion: Comprehensive experiments show that LFDM achieves state-of-the-art performance on multiple datasets.
  • Limitations: Current LFDM experiments are limited to videos containing a single moving subject.The authors plan to extend the method to multi-subject flow generation.
  • Limitations: LFDM is conditioned on class labels rather than natural text descriptions.Text-to-flow generation is left for future work.
  • Limitations: LFDM samples more slowly than GAN models when using 1000-step DDPM.The authors plan to investigate faster sampling methods to reduce generation time.

A1. Potential Negative Social Impact

The paper acknowledges that conditional image-to-video models can be used for unethical purposes, including creating celebrity videos for fake-news spreading, and restricts model usage to research.

  • Conditional image-to-video models can be misused to create celebrity videos for spreading fake news.
  • The authors restrict usage of their models to research purposes.

A2. Additional Experiments

Additional experiments examine architecture choices, occlusion maps, and inference time for LFDM. Results favor the default six-block decoder and simpler diffusion network, while occlusion maps improve reconstruction and DDPM sampling trades speed for FVD.

  • A2. Additional Experiments: The experiments use the MUG dataset and generate videos at 128 × 128 resolution.
  • A2.1. Additional Ablation Study on Network Architecture: A deeper image decoder slightly improves self-reconstruction L1 error but worsens video FVD, so the six-residual-block decoder remains default.The comparison increases decoder depth from 6 to 10 residual blocks while keeping other modules unchanged.
  • A2.1. Additional Ablation Study on Network Architecture: Adding a layer to the stage-two diffusion network decreases performance, so the simpler channel multipliers (1, 2, 4, 8) remain default.The default denoising network uses a 3D U-Net with four down-sampling and four up-sampling blocks.
  • A2.2. Additional Ablation Study on Occlusion Map: Without occlusion maps, MUG self-reconstruction L1 error increases from 0.418 to 0.450.Occlusion maps indicate regions that should be masked and repainted by the decoder.
  • A2.3. Inference Time Comparison: Table A3 compares the average time required to generate one video across methods on one NVIDIA A100 GPU using batch size 10.The comparison notes that VDM uses 200-step DDIM, whereas LDM and LFDM use 1000-step DDPM.

A3. More Discussion about Future Work

The discussion identifies changing backgrounds, broader motion-category generalization, and faster diffusion sampling as future directions for LFDM.

  • Future work will extend LFDM to generate videos with changing backgrounds or contexts.The proposed direction first models foreground motion with LFDM, then synthesizes changing backgrounds conditioned on generated foreground frames.
  • More labeled training videos and continual or incremental learning are planned to improve generalization to diverse motions and categories.
  • Ten-step DDIM provides acceptable generation quality with faster sampling, but better hyperparameter settings remain to be explored.The authors identify diffusion sampling steps as one target for future hyperparameter optimization.

A4. Information about Attached Videos

The attached videos demonstrate LFDM across facial expressions, human actions, gestures, new-domain adaptation, sampling strategies, and comparisons with other models.

  • Attached examples: The supplementary clips include seven generated MP4 examples based on unseen testing images.
  • Attached examples: The MUG clip displays all seven expressions of one subject, while MHAD clips cover 26 actions and NATOPS covers 24 gestures.The sit-to-stand action is excluded from MHAD because the subject in the input image is standing.
  • Access: The supplementary videos are also available through the paper’s GitHub repository.
  • New-domain adaptation: The new-domain clip compares pretrained and decoder-finetuned LFDM on four expressions across four FaceForensics subjects.The other modules, including the stage-two diffusion model, remain unchanged during finetuning.
  • Sampling comparison: Ten-step DDIM generates visually acceptable videos at 0.3s per video versus 36s for DDPM-1000, but FVD is 50.18 versus 32.09.DDPM-1000 remains the default because its FVD score is better.
  • Model comparison: The SOTA comparison shows LFDM alongside ImaGINator, VDM, and LDM on subjects from MUG, MHAD, and NATOPS.LFDM and LDM results use 128×128 resolution, whereas ImaGINator and VDM use 64 × 64.
Loading 2303.13744v1…