PaperlayerView exact revision on arXiv

Source-linked AI summary

DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, Ying Shan

arXiv:2310.12190v2cs.CV

TL;DR

Existing image-animation methods are limited by assumptions about motion or object categories, making open-domain animation difficult. DynamiCrafter injects image information into pretrained text-to-video diffusion models through a dual-stream mechanism, producing natural, coherent motion with stronger conformity to the input image than existing methods.

  • Problem

    Traditional image-animation methods focus on stochastic motions or specific object categories, limiting their applicability to open-domain images.

  • Method

    DynamiCrafter animates images by leveraging pretrained text-to-video diffusion priors with a dual-stream image-injection mechanism and dedicated training.

  • Results

    The proposed method demonstrates notable superiority over existing competitors, producing natural motion, coherent frames, and visual conformity to the input image.

  • Takeaways & Limitations

    The framework supports open-domain image animation and extends to storytelling, looping-video generation, and generative frame interpolation.

  • Takeaways & Limitations

    The method may struggle when input images cannot be semantically understood, and its text-based motion control cannot yet generate specific motions precisely.

Abstract

from arXiv · show

Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

1. Introduction

The paper targets open-domain image animation by leveraging text-to-video diffusion priors while addressing the need for both visual context understanding and detail preservation. It introduces image-conditioning techniques and reports superiority over existing competitors, with performance comparable to some commercial demos.

  • Motivation: Traditional image-animation methods focus on stochastic or oscillating motions and specific object categories, limiting their applicability to general visual content.These approaches impose strong assumptions on the motion or object domain.
  • Motivation: The paper leverages text-to-video generative models to animate still images by governing video diffusion with a conditional image.The motivation comes from T2V models’ ability to create diverse and vivid videos from textual prompts.
  • Challenge: Image animation requires visual context understanding for creating dynamics while preserving the input image’s details.Existing multimodal controllable video diffusion models made preliminary attempts to provide image-based visual guidance.
  • Method: The proposed context-learning network uses a pretrained CLIP image encoder and learnable query transformer to project images into a text-aligned representation space.The resulting context features enter through cross-attention layers and combine with text-conditioned features through gated fusion.
  • Results: The proposed method significantly outperforms contemporary competitors and achieves performance comparable to commercial demos including Gen-2 and PikaLabs.The paper also analyzes visual injection streams, text-prompt utility, and dynamics control.
  • Contributions: The method introduces a dual-stream image-injection paradigm and pioneers text-based motion control for open-domain image animation.The paper presents these designs as contributions enabled by analysis of the conditional space of text-to-video diffusion models.

2. Related Work

Prior image-animation methods largely target specific objects, curated domains, or reference-driven motion, while video diffusion models mainly rely on text guidance. This motivates a generic approach for animating diverse open-domain images with direct visual conditioning.

  • Image Animation: Early physical-simulation methods model object-specific motion, limiting generalizability across categories.Reference-based methods instead transfer motion or appearance from signals such as videos to guide synthesis.
  • Image Animation: Existing approaches primarily animate curated stochastic or oscillating motions in restricted categories such as fluid, natural scenes, hair, portraits, and bodies.These constraints make open-domain animation challenging because of the overwhelming complexity and diversity of possible content and styles.
  • Video Diffusion Models: Video diffusion models extend diffusion-based generation to video, including low-resolution spacetime-factorized modeling, cascaded high-definition synthesis, and cost-reduced training.These developments follow the strong generative performance of diffusion models in text-to-image generation.
  • Video Diffusion Models: Text-to-video diffusion models accept text as their sole semantic guidance, which can be vague and may not accurately reflect user intention.Although structure, pose, and Canny-edge controls have received increasing attention, visual conditions such as RGB images remain underexplored.

3. Method

DynamiCrafter treats still-image animation as image-conditioned video generation requiring both natural motion and strong visual conformity. Its dual-stream injection paradigm combines text-aligned image context with visual detail guidance, supported by staged training and layer-adaptive fusion.

  • Dual-stream image injection: The method uses a dual-stream conditional image injection paradigm to provide image context for dynamics synthesis while preserving visual details in generated videos.The two streams are text-aligned context representation and visual detail guidance.
  • Text-aligned context representation: A query transformer projects CLIP image tokens into a text-aligned context representation that the denoising U-Net can interpret compatibly.The representation is computed as Fctx = P(Fvis), where P is a lightweight learnable query transformer.
  • Text-aligned context representation: Layer-adaptive tanh gating fuses text- and image-conditioned features, with λ learned separately across U-Net layers to control image-condition strength.Intermediate U-Net layers are associated more with object shapes or poses, while the two-end layers are more linked to appearance.
  • Observations and analysis of λ: Increasing λ suppresses cross-frame movements, whereas decreasing λ makes preserving object shape more difficult, revealing a motion–shape trade-off in image conditioning.The learned coefficients also show that image information has greater impact on the two-end layers than on intermediate layers.
  • Visual detail guidance (VDG): Additional visual detail guidance supplements the context representation because CLIP’s visual-language alignment objective can leave minor input-image discrepancies in generated videos.The method therefore provides the video model with additional visual information to enhance conformity.
  • Training strategy: Training proceeds in three stages: learning P with a lightweight T2I model, adapting P with the T2V model’s spatial layers, and jointly fine-tuning with VDG.The input image is concatenated with per-frame noise during joint fine-tuning, while temporal layers remain fixed to preserve pretrained temporal priors.

4. Experiment

Experiments evaluate video quality, temporal coherence, and conformity to input images using standard distances and a new perceptual metric. DynamiCrafter outperforms prior open-source methods broadly, while ablations show that dual-stream conditioning and staged training support coherent, natural, and image-conforming animation.

  • Metrics and datasets: The evaluation reports FVD, KVD, and Perceptual Input Conformity (PIC) on zero-shot generation using UCF-101 and MSR-VTT.Error metrics are evaluated at 256 × 256 resolution with 16 frames.
  • Quantitative comparison: DynamiCrafter significantly outperforms VideoComposer and I2VGen-XL on all evaluation metrics except KVD on UCF-101.The paper attributes this performance to its dual-stream image injection design for exploiting the video diffusion prior.
  • Qualitative comparison: DynamiCrafter produces temporally coherent videos that adhere to input images across varied content and styles, including comparisons with PikaLabs and Gen-2.The commercial-product results accessed on Nov. 1, 2023 may differ from later versions because of rapid product iterations.
  • User study: In a 49-participant user study, DynamiCrafter is significantly superior to other open-source methods, with commercial-level temporal coherence and input conformity and better motion quality.Participants selected the best motion quality and temporal coherence results and identified outputs with good conformity to the input image.
  • Dual-stream image injection: Removing context conditioning causes significant performance declines and temporal inconsistencies, while removing visual detail guidance reduces the method’s ability to preserve the full image extent.The full method leverages the video diffusion prior to produce natural motion, coherent frames, and conformity to the input image.
  • Training paradigm: The two-stage training strategy improves conditioning over one-stage training, while full-model fine-tuning destabilizes the model and can collapse, disrupting the temporal prior.After learning compatible context conditioning, the method jointly fine-tunes the context network and spatial layers with image concatenation; random selection avoids mapping the image only to the first frame.

5. Discussions on Motion Control using Text

Text is introduced as a complementary control signal for tailoring image-animation dynamics to user preferences, while decoupled training addresses motion descriptions being overshadowed by scene content. A filtered and re-annotated WebVid10M dataset enables this training, and DynamiCrafterDCP is validated on ambiguous human-action cases with varied motion prompts.

  • Motivation for text control: Text can guide dynamic content toward user preferences because images may support multiple plausible dynamics, but dataset captions often emphasize scene descriptions over motion.For image animation, scene content is already present in the image, motivating purer dynamic wording.
  • Dataset construction: The authors construct a filtered and re-annotated WebVid10M dataset with captions containing purer dynamic wording and motion categories.An example caption is “Man doing push-ups.”, with “human” given as a category.
  • Validation: DynamiCrafterDCP is trained on the constructed dataset and evaluated on 40 image-prompt cases involving human figures with ambiguous potential actions.The testing cases use prompts describing various motions.

6. Applications

DynamiCrafter extends beyond single-image animation to support storytelling videos from scripted image shots and looping video generation with minor modifications.

  • Storytelling with shots: DynamiCrafter supports storytelling by animating ChatGPT- and DALL-E 3-generated story shots using their corresponding scripts.The resulting storytelling videos are illustrated in Figure 10 (top).
  • Looping video generation: With minor modifications, DynamiCrafter can also generate looping videos.

7. Conclusion … A.2. Hyper-parameters

DynamiCrafter animates open-domain images using pre-trained video diffusion priors, a dual-stream image injection mechanism, and a dedicated training paradigm. The supplementary implementation details describe its VideoCrafter foundation, query-transformer conditioning, spatial dual-attention integration, and hyper-parameter choices.

  • 7. Conclusion: DynamiCrafter animates open-domain images by leveraging pre-trained video diffusion priors with a dual-stream image injection mechanism and dedicated training paradigm.The study reports that the approach is effective and superior to existing methods.
  • 7. Conclusion: The framework also explores text-based dynamic control for image animation using a constructed dataset.
  • A. Implementation Details: DynamiCrafter is built upon VideoCrafter, a latent VDM-based text-to-video generation model.The authors recommend VideoCrafter [8] for further details of the text-to-video backbone.
  • A. Implementation Details: Its video-diffusion-prior approach can theoretically extend to other text-to-video diffusion models using cross-attention text conditioning.
  • A.1. Network Architecture: The query transformer learns context representations whose frame-wise outputs are used by the spatial dual-attn transformer in the denoising U-Net.The architecture includes an FPS embedding layer and a query transformer for context representation learning.
  • A.1. Network Architecture: Cross-attention projects context queries into queries while the inputs provide keys and values, followed by a Feed-forward projection.
  • A.2. Hyper-parameters: The denoising U-Net input tensor has twice the channel count of z_t because visual detail guidance is concatenated channel-wise.

A.3. Training … D. Details of Constructed Dataset

The supplementary details describe the model’s efficient fine-tuning setup, evaluation metrics and baselines, and controlled user-study protocol. They also report strong generalizability to images outside the training domain, including anime and paintings.

  • A.3. Training: The model adds input channels for concatenated conditional image and noisy latents, initializes new-channel weights to zero, and fine-tunes using eight NVIDIA V100 GPUs.Existing video-diffusion weights are initialized from pretrained checkpoints.
  • A.3. Training: DynamiCrafter demonstrates strong generalizability when animating images outside its domain, including anime and paintings.
  • B.1. Dataset and metric: FVD and KVD evaluate synthesized-video quality and temporal coherence through feature-level similarity to real videos using Fréchet-distance and kernel-based methods.The metrics compare 2048 model samples with samples from evaluation datasets.
  • B.2. Baselines: The evaluation compares DynamiCrafter with VideoComposer, I2VGen-XL, PikaLabs, and Gen-2 as available open-source and commercial baselines.For I2VGen-XL, the image-to-video first stage is used because its refinement stage primarily involves text-to-video generation.
  • C. User Study: The user study uses 20 Internet-sourced image cases spanning varied content and styles, with corresponding captions and results generated through official code or online demos.Methods are shuffled and use the same input still image, with text prompts when applicable.
  • C. User Study: The study standardizes outputs to FPS=8 and two seconds at method-specific resolutions while concealing lower watermarks, reducing engineering-related comparison effects.I2VGen-XL uses 256×448 resolution, while other methods use 256×256.
  • C. User Study: Participants evaluate 20 video comparisons three times on motion quality and related questions, while responses completed within three minutes are filtered out.The expected study duration is 5–10 minutes.

D.1. Dataset construction details … E.1. FPS Control

The supplementary sections describe a GPT4-assisted dataset-construction pipeline, its scale and human validation, DynamiCrafterDCP training, and FPS-conditioned motion control. The resulting dataset contains around 2.6 million caption-video pairs, while FPS conditioning enables varying motion magnitudes.

  • D.1. Dataset construction details: The dataset pipeline filters large camera movements, poor caption-video alignment, and Graphics/CGI content before using GPT4 to annotate dynamic confidence, wording, and source category.GPT4 is configured with temperature=0.2 and frequency penalty=0.
  • D.1. Dataset construction details: GPT4 outputs a 0–100 dynamic confidence score, compact subject-action wording, and one of six source categories: human, animal, nature, machine, others, or none.The category none is used when the dynamic wording is none.
  • D.2. Statistics of the dataset: The constructed dataset contains around 2.6 million caption-video pairs, excludes low-proportion class combinations, and will release its annotations publicly for research on motions and dynamics.Excluded combinations include animal&human, human&machine, and animal&machine.
  • D.3. Human validation on the dataset: Human evaluation of 1K randomly sampled responses assesses dynamic-scene judgments and GPT4-generated wording and categories, with dynamic confidence=40 identified as a sweet spot for human alignment.The validation uses an accuracy-threshold curve based on human judgments.
  • D.4. DynamiCrafterDCP: DynamiCrafterDCP is initialized from DynamiCrafter’s 60K-iteration checkpoint and trained for another 40K iterations using human category data and dynamic wording as text prompts.The baseline DynamiCrafter is trained for 100K iterations, with all other settings held identical for fair comparison.
  • D.4. DynamiCrafterDCP: DynamiCrafterDCP is evaluated with CLIP-SIM because CLIP provides open-domain text-image representations that associate image dynamics with appropriate dynamic wording.This evaluation rationale is stated for comparison with the baseline model.
  • E.1. FPS Control: Training with dynamic FPS values of 5–30 enables image animations with varying motion magnitudes, illustrated through low-FPS and high-FPS outputs.The model is conditioned on FPS during generation.

E.2. Multi-condition Classifier Free Guidance

DynamiCrafter uses multi-condition classifier-free guidance during DDIM inference, allowing separate image and text guidance scales to control adherence to each condition. Setting s_img = s_txt = 7.5 generally balances image conformity and text-prompt reflection, while lowering either scale prioritizes the other condition.

  • Guidance mechanism: Separate guidance scales s_img and s_txt trade off the influence of the input image and text prompt during DDIM inference.The scales control how strongly generated samples correspond to the input image versus the text prompt.
  • Balanced setting: s_img = s_txt = 7.5 generally produces animations that adhere to the input image while reflecting the text prompt.This setting works well in most cases and is illustrated in Figure 14(top).
  • Condition trade-offs: Reducing s_txt makes animations tend to ignore the text condition, such as “dancing.”This behavior is shown in Figure 14(middle).
  • Condition trade-offs: Reducing s_img can weaken conformity to the input image while preserving stronger reflection of the text prompt.This behavior is shown in Figure 14(bottom).
  • User control: Multi-condition classifier guidance provides greater flexibility for user requirements.Users can adjust the two scales to change the relative influence of image and text controls.

F. Limitations

DynamiCrafter may struggle with semantically unclear input images and cannot reliably generate specific motions because its text-motion dataset lacks precise motion descriptions. Its low-resolution, short-duration pretrained model also inherits flickering and face distortions, limiting quality, video length, and practical readiness for products.

  • Semantic understanding: Semantically unclear input images can prevent the model from producing convincing videos.The approach may struggle when the input image condition cannot be semantically understood.
  • Motion control: The method cannot reliably generate specific motions because its text-motion dataset lacks precise motion descriptions.The dataset was constructed to improve motion control with text but still lacks sufficiently precise motion descriptions.
  • Model quality: Low-resolution, short-duration pretraining causes inherited flickering artifacts in high-frequency regions and human face distortions.These issues are technically caused by frame-wise VAE decoding.
  • Practical applications: Resulting limitations in resolution, fidelity, and video length may restrict practical applications and leave the method unready for products.The limitation is contrasted with commercial products such as PikaLabs and Gen-2.

G. More Qualitative Results

Additional qualitative comparisons show baseline-specific failures, while DynamiCrafter produces image animations across diverse internet and benchmark inputs. The supplementary video results include showcases, baseline comparisons, text-based motion control, applications, other controls, ablations, and limitations.

  • More qualitative comparisons: VideoComposer struggles with frame coherence and is misled by text prompts, while I2VGen-XL loses local image details and PikaLabs produces still or weakly dynamic videos.These observations are reported for additional comparisons in Figure 17.
  • Gallery of our results: The method’s gallery presents additional image-animation results from Internet images, DAVIS, and JourneyDB.The results appear in Figures 18 and 19.
Loading 2310.12190v2…