PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamVideo: Composing Your Dream Videos with Customized Subject and Motion

Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, Hongming Shan

arXiv:2312.04433v1cs.CV

TL;DR

Customized video generation must control both subject appearance and motion, while existing methods often optimize only one aspect. DreamVideo separates subject and motion learning, and experiments report stronger customization performance than state-of-the-art methods.

  • Problem

    Customized video generation must control spatial subject content and temporal motion together, while existing methods are limited in optimizing both aspects.

    Source: §1 · p. 2

  • Method

    DreamVideo separates customization into subject learning with textual inversion and an identity adapter, and motion learning with a motion adapter trained on target videos.

    Source: §1 · p. 2

  • Results

    Extensive evaluations report that DreamVideo surpasses state-of-the-art methods, and users preferred it across text alignment, subject fidelity, motion fidelity, and temporal consistency.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    Experiments support DreamVideo’s efficiency and flexibility in joint customization and individual customization of subjects and motions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.

1. Introduction

DreamVideo addresses customized video generation by learning subject identity and motion separately, then combining them to generate videos with a specified subject and motion. The paper reports evaluation across 20 subjects and 30 motion patterns, with results surpassing existing state-of-the-art methods.

Introduction

  • Customized video generation is challenging because it must control diverse spatial content and intricate temporal dynamics simultaneously.The introduction contrasts this difficulty with the progress of customized image generation.Source: §1 · p. 2
  • Prior methods optimize either spatial subject identity or temporal motion, while generalized animation methods can lack motion diversity.The paper argues that modeling both subject and motion is necessary for video customization.Source: §1 · p. 2
  • DreamVideo learns subject identity and target motion separately from a few images and videos, then combines two lightweight adapters to customize videos.Subject learning uses Textual Inversion for a coarse identity and an identity adapter for fine appearance; motion learning trains a motion adapter on target videos.Source: §1 · p. 2
  • DreamVideo incorporates image features into its motion adapter to prevent appearance learning and focus motion training on motion patterns.The separation is intended to reduce optimization complexity and increase customization flexibility.Source: §1 · p. 2
  • The evaluation covers 20 customized subjects and 30 motion patterns, and the paper reports qualitative and quantitative results surpassing state-of-the-art methods.The authors describe the approach as the first attempt, to their knowledge, to customize both subject identity and motion.Source: §1 · p. 2

2. Related Work

Video generation has advanced through diffusion models and controllable conditioning, while customized video generation remains less developed than image customization. DreamVideo addresses this gap by targeting both subject and motion customization with lightweight adapters.

Text-to-video generation

  • Video generation has progressed from GANs and autoregressive transformers to diffusion models, including latent-space and conditionally controlled approaches.Make-A-Video [56] uses image-diffusion priors without paired text-video data, while Video Diffusion Models [29] and ImagenVideo [28] model videos in pixel space.Source: §2 · p. 2

Customized generation

  • Image customization methods learn subjects from a few images, but customized video remains underexplored: Dreamix [47] risks overfitting and weak motion, while [83] customizes motion but not subjects.Textual Inversion [17] learns a subject embedding without model fine-tuning, whereas DreamBooth [52] fully fine-tunes an image diffusion model.Source: §2 · pp. 2–3

Parameter-efficient fine-tuning

  • Parameter-efficient fine-tuning has been adopted for video generation and editing, and DreamVideo explores lightweight adapters for customized video generation.Prior PEFT work draws on NLP [32] [38] and vision [3] [77] [78], with video applications [48] [74].Source: §2 · p. 3

3. Methodology

DreamVideo separates customized video generation into independently learned subject and motion adapters that can be combined at inference. Its adapter placement follows different parameter roles in subject and motion learning.

3.1. Preliminary: Video Diffusion Models

  • VDMs extend image diffusion to video by predicting noise in text-conditioned noisy video latents with a 3D UNet containing spatiotemporal convolution and spatial and temporal transformers.The noise-corrupted latent is formed from the ground-truth video latent and Gaussian noise using diffusion hyperparameters.Source: §3.1 · pp. 3–4

3.2. DreamVideo

  • For subject learning from 3–5 images, DreamVideo first learns a coarse pseudo-word identity with Textual Inversion [17], then freezes it and trains a lightweight bottleneck identity adapter for appearance details.The two-step strategy freezes the video diffusion model during textual inversion and optimizes only the identity adapter in the second step.Source: §3.2 · p. 4
  • The motion adapter learns patterns from a class of videos, multiple videos, or one video, while CLIP image-embedding appearance guidance helps disentangle subject appearance from motion.A randomly selected training-video frame is encoded and its embedding is broadcast across frames; the adapter adds a condition linear layer to the temporal hidden state.Source: §3.2 · p. 4

3.3. Model Analysis, Training and Inference

  • Cross-attention layers are crucial for subject appearance learning, whereas all parameters contribute similarly to motion learning, motivating identity adapters in cross-attention and motion adapters across temporal-transformer layers.The reported pattern held across various images and videos.Source: Figure 4 · p. 4, §3.3 · p. 5
  • DreamVideo trains the identity and motion adapters independently with the pretrained model frozen, then combines them at inference using a user-provided image as appearance guidance; either adapter can also be used alone.Choosing different training images for appearance guidance has only a marginal impact on generated results.Source: §3.3 · p. 5

4. Experiment

DreamVideo combines subject and motion customization across joint and individual-generation tasks, and ablations show that its textual identity, motion adapter, and appearance guidance each contribute to performance.

4.1. Experimental Setup

  • Experiments use 20 customized subjects, 30 motion patterns, and 42 prompts, evaluating subject similarity, text alignment, and temporal consistency.Generation uses 50-step DDIM sampling to produce 32-frame videos at 8 fps.Source: §4.1 · pp. 5–6

4.2. Results

  • DreamVideo outperforms other methods on CLIP-T, CLIP-I, and DINO-I, while AnimateDiff’s highest Temporal Consistency accompanies small motions.DreamVideo’s Temporal Consistency is comparable to Dreamix’s while requiring fewer parameters; qualitative comparisons also show more harmonious combinations of subject identity and motion.Source: Table 1 · p. 7, §4.2 · p. 7
  • DreamVideo outperforms alternatives on CLIP-T and DINO-I for subject customization, with CLIP-I and Temporal Consistency comparable to Dreamix.It preserves precise subject appearance while following varied text contexts; Textual Inversion struggles to reconstruct appearance, and Dreamix can overfit, produce small motions, or omit prompted objects.Source: Figure 6 · p. 6, Table 2 · p. 7, §4.2 · pp. 7–8
  • DreamVideo achieves the highest CLIP-T and Temporal Consistency for motion customization compared with baselines.Qualitatively, it models target motions while avoiding training-video appearance coupling and generating temporally consistent, diverse videos.Source: Table 3 · p. 7, Figure 7 · p. 7, §4.2 · p. 8
  • Users most preferred DreamVideo across Text Alignment, Subject Fidelity, Motion Fidelity, and Temporal Consistency in evaluations of customized subject-motion combinations.Five annotators rated 50 groups of videos, with each group covering 5 motion patterns and 10 subjects.Source: Table 4 · p. 8, §4.2 · p. 8

4.3. Ablation Studies

  • Removing textual identity or appearance guidance slightly decreases all metrics; subject-only customization improves CLIP-I and DINO-I, while the motion adapter increases CLIP-T and Temporal Consistency.Without textual identity, appearance details may be lost; without the motion adapter, the desired motion may not emerge, and without appearance guidance, subject identity and background may be corrupted.Source: Table 5 · p. 8, §4.3 · pp. 8–9, Figure 8 · p. 8

5. Conclusion · Appendix

DreamVideo separates customized video generation into subject and motion learning, using lightweight adaptation to combine a subject with a motion. Experiments support its efficiency and flexibility for joint and individual customization, while the method remains limited to one subject and one motion at a time.

5. Conclusion

  • DreamVideo decouples video customization into subject learning and motion learning to enhance flexibility.The method combines textual inversion and identity-adapter tuning for subjects with an appearance-guided motion adapter for motions.Source: §5 · p. 9
  • Textual inversion and identity-adapter tuning model the desired subject’s appearance.These are the components used for subject learning.Source: §5 · p. 9
  • An appearance-guided motion adapter is trained to learn a target motion.Source: §5 · p. 9
  • 20 subjects and 30 motion patterns support qualitative and quantitative experiments demonstrating efficiency and flexibility in joint and individual customization.Source: §5 · p. 9
  • DreamVideo efficiently combines a single subject and a single motion, but fails to generate videos with multiple subjects and multiple motions.Source: §5 · p. 9
  • Possible solutions to the multiple-subject, multiple-motion limitation include a fusion module or a general customized video model.The passage points to Appendix D for further analysis and discussion but does not describe its contents.Source: §5 · p. 9

A. Experimental Details … A.3. Video Customization

The appendix establishes shared evaluation settings and describes the training and inference procedures for DreamVideo and baseline methods across subject, motion, and combined video customization.

A. Experimental Details

  • All methods use fine-tuned ModelScopeT2V as the base video model except AnimateDiff [21] and Tune-A-Video [70], which use Stable Diffusion v1-51.ModelScopeT2V was fine-tuned for 30k iterations on about 30,000 internal text-video pairs; inference uses 50 DDIM steps, classifier-free guidance scale 9.0, and 32-frame videos at 256 × 256 and 8 fps.Source: A. Experimental Details · p. 14

A.1. Subject Customization

  • DreamVideo learns the subject with ∼3000 textual-identity iterations and ∼800 identity-adapter iterations, taking ∼12 minutes for adapter training on one A100 GPU.The respective learning rates are 1.0 × 10−4 and 1.0 × 10−5; the adapter hidden dimension is half the input dimension, and subject-learning batch size is 4.Source: §A.1 · p. 14
  • Textual Inversion [17], Dreamix [47], and Custom Diffusion are reproduced on the video model using their described subject-learning procedures.Textual Inversion optimizes a pseudo-word embedding for 3000 iterations; Dreamix trains spatial UNet parameters while freezing temporal ones; Custom Diffusion jointly optimizes cross-attention key/value matrices and the token embedding for 250 iterations.Source: §A.1 · p. 14

A.2. Motion Customization

  • DreamVideo trains its motion adapter for ∼1000 iterations, taking ∼15 minutes for one video and ∼30 minutes for multiple videos on one A100 GPU.The motion-adapter learning rate is 1.0 × 10−5 and its hidden dimension is half the input dimension; batch size is 1 for single-video training and 2 for multiple videos.Source: §A.2 · p. 14

A.3. Video Customization

  • For video customization, DreamVideo combines the trained identity and motion adapters at inference with no additional training, and different appearance-guidance images have marginal impact.The method randomly selects one training image as appearance guidance.Source: §A.3 · p. 14
  • AnimateDiff [21] uses a pre-trained motion module fine-tuned for 50 iterations, while ModelScopeT2V [64] combines learned spatial and temporal UNet parameters at inference.The authors report that training AnimateDiff's motion module from scratch may make generated videos unstable and temporally inconsistent; ModelScopeT2V subject and motion settings follow the procedures in A.1 and A.2.Source: §A.3 · pp. 14–15
  • The LoRA video-customization baseline uses rank 32, applying LoRA to cross-attention key/value matrices for subjects and all attention layers for motion.Spatial and temporal LoRAs are merged into their corresponding layers at inference.Source: §A.3 · p. 15

B. More Results … B.3. Motion Customization

The supplemental results report that DreamVideo preserves subject identity while modeling motion, outperforms Custom Diffusion, and generalizes across additional subjects and motion examples.

B. More Results

  • Across the supplemental experiments, DreamVideo is presented as superior in video, subject, and motion customization results.The experiments include comparisons with baselines and additional customization examples.Source: B. More Results · p. 15, §B.1 · p. 15, §B.2 · p. 15

B.1. Video Customization

  • DreamVideo preserves both subject identity and motion patterns, while AnimateDiff produces little motion and other methods struggle to fuse identity with motion.The supplemental baseline comparisons are shown in Fig. A1.Source: §B.1 · p. 15

B.2. Subject Customization

  • DreamVideo outperforms Custom Diffusion and retains subject identity across diverse contextual descriptions with fewer parameters.The paper reports qualitative comparisons in Fig. A2 and quantitative comparisons in Tab. A1.Source: §B.2 · p. 15
  • Customization results for additional subjects further demonstrate DreamVideo’s favorable generalization.These results are shown in Fig. A3.Source: §B.2 · p. 15

B.3. Motion Customization

  • DreamVideo outperforms baselines on multiple- and single-training-video motion customization while ignoring training-video appearance when modeling motion patterns.The qualitative comparisons are shown in Fig. A4.Source: §B.3 · p. 15
  • Additional motion-customization results in Fig. A5 provide further evidence of DreamVideo’s robustness.Source: §B.3 · p. 15

B.4. User Study

The user study evaluates customized subject and motion generation through participant comparisons on text alignment, fidelity, and temporal consistency. DreamVideo consistently outperforms other methods across all motion-customization metrics.

B.4. User Study

  • Subject customization evaluates 120 videos from 15 subjects, using 8 text prompts per subject and 3–5 reference images to assess text alignment, subject fidelity, and temporal consistency.Participants compare videos from two anonymous methods on which better matches the text, resembles the target subject, and is smoother with less flicker.Source: §B.4 · p. 15
  • DreamVideo consistently outperforms other methods on all motion-customization metrics in the human evaluation.The evaluation compares text alignment, motion fidelity, and temporal consistency.Source: Table A3 · p. 16, §B.4 · p. 16
  • Motion customization evaluates 120 videos across 20 motion patterns and 6 text prompts, measuring text alignment, motion fidelity, and temporal consistency.Participants compare whether each video better matches the text, reproduces target-video motion, and is smoother with less flicker.Source: §B.4 · p. 16

C. More Ablation Studies … C.3. Effects of Different Adapters

The ablations favor adapters over LoRA for video customization and parallel adapters on all layers for motion customization. Additional qualitative results show that videos generated without appearance guidance may include noise, artifacts, backgrounds, or other subject-unrelated information.

C.1. More Qualitative Results

  • Without appearance guidance, generated videos may learn noise, artifacts, backgrounds, and other subject-unrelated information from training videos.Source: §C.1 · p. 16
  • Figure A6 provides further qualitative checks of each method component, with conclusions consistent with the main paper.Source: §C.1 · p. 16

C.2. Effects of Parameters in Adapter and LoRA

  • For a parameter-count comparison with LoRA, the adapter hidden dimension is set to 32 without textual identity or appearance guidance.The comparison uses the DreamBooth paradigm.Source: §C.2 · p. 16
  • The adapter alleviates fusion conflicts and enables a more harmonious combination when merging spatial and temporal components.Source: §C.2 · p. 16
  • The adapter outperforms LoRA in video customization tasks in the quantitative comparison reported in Tab. A4.Source: §C.2 · p. 16

C.3. Effects of Different Adapters

  • Among four adapter and parameter-layer combinations, parallel adapters on all layers achieve the best motion-customization performance.The authors therefore use parallel adapters uniformly in their approach.Source: §C.3 · p. 16

D. Social Impact and Discussions · LoRA · Generated

DreamVideo makes video customization accessible through low-cost subject and motion adaptation, but remains constrained by base-model capabilities, fine-motion matching, and difficult multi-object combinations. The supplementary material also presents qualitative results and an Adapter–LoRA comparison.

D. Social Impact and Discussions

  • Low-cost fine-tuning of a few images or videos lets users flexibly combine customized subjects and motions or customize either individually.The paper notes that training large-scale video diffusion models is unaffordable for most individuals.Source: D. Social Impact and Discussions · p. 16
  • Generative risks include fake-data generation, for which the paper identifies reliable video-forgery detection as a possible solution.Source: D. Social Impact and Discussions · p. 16
  • The method inherits base-model limits, may miss frame-by-frame correspondence for fine motion, and still struggles with combinations such as a cat and horse.The wolf-riding-a-bicycle prompt illustrates the base-model limitation; the paper suggests video editing methods [8] [9] [70] for fine-motion issues and decoupling subject attention maps for multi-object confusion.Source: D. Social Impact and Discussions · pp. 16–17
  • The appendix includes qualitative comparisons and examples for joint subject-motion generation, subject customization, motion customization, and component ablations.Source: Figure A1 · p. 18, Figure A2 · p. 18, Figure A3 · p. 19

LoRA

  • Figure A7 qualitatively compares Adapter and LoRA using the same hidden dimension (rank) and a comparable number of parameters.Source: Figure A7 · p. 21

Generated

  • Figure A8 presents failure cases for subject customization, fine single-video motion customization, and video customization involving difficult multi-object combinations.The examples include a base-model failure, rough rather than frame-by-frame motion matching, and confusion between multiple objects.Source: D. Social Impact and Discussions · p. 17, Figure A8 · p. 21

Related papers

Loading 2312.04433v1…