PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionBooth: Motion-Aware Customized Text-to-Video Generation

Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, Kai Chen

arXiv:2406.17758v3cs.CV

TL;DR

MotionBooth targets customized video generation that must learn a subject while preserving video-generation capability and motion. It fine-tunes a T2V model with subject-focused losses and adds training-free subject and camera controls. The authors report that it outperforms existing video-generation models and can control both motions.

  • Problem

    Fine-tuning a T2V model on few subject images can overfit and impair video generation, while existing approaches may offer little motion or require motion-specific tuning.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    MotionBooth fine-tunes a T2V model with subject region, video preservation, and subject token cross-attention losses, then controls subject and camera motion during inference without training.

    Source: §1 · p. 2

  • Results

    The authors report that MotionBooth outperforms existing state-of-the-art video generation models and demonstrates effective motion control across different base T2V models without further tuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MotionBooth generates videos from customized subjects with controllable subject and camera motions.

    Source: §5 · p. 10

    MotionBooth can confuse a subject with other objects and may fail to depict motions unlikely for that subject.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of a specific object, we efficiently fine-tune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss to integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce a novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Our project page is at https://jianzongwu.github.io/projects/motionbooth

1 Introduction

MotionBooth addresses the challenge of learning a customized subject while retaining a text-to-video model’s motion and scene-generation capabilities. It combines loss-augmented subject fine-tuning with training-free subject and camera motion control.

1 Introduction

  • Few-image subject learning can shift the base model’s distribution and degrade video generation, while some prior methods require motion-specific fine-tuning and adapters.VideoBooth can produce clips with minimal or missing motion, and other methods can restrict motion types.Source: §1 · p. 2
  • MotionBooth accepts combinations of a subject, subject motion, and camera motion as inputs and aims to generate diverse videos at quality on par with pretrained T2V models.Source: §1 · p. 2
  • Subject region and video preservation losses support subject fidelity and video quality, while subject token cross-attention loss connects the customized subject to motion signals.At inference, cross-attention manipulation controls subject motion and a latent shift module governs camera movement without further tuning.Source: §1 · p. 2
  • Experiments demonstrate that MotionBooth outperforms existing state-of-the-art video generation models.Source: §1 · p. 2

2 Related Work

Related work spans text-to-video modeling, subject customization, and explicit motion control. MotionBooth’s pipeline combines subject fine-tuning with training-free controls for camera and subject motion.

2 Related Work

  • Recent T2V approaches include diffusion-based models, Transformer backbones, and autoregressive video generation [51] [54] [36] [14] [29].The field builds on text-to-image generation while adding motion and time [44] [41] [19] [21] [37] [48] [60] [62] [46] [18] [2] [68] [59].Source: §2 · pp. 2–3
  • MotionBooth fine-tunes the T2V model with subject region, video preservation, and subject token cross-attention losses, then uses latent shifts for camera motion and cross-attention maps for subject motion.Source: Figure 2 · p. 3
  • Customized generation learns subjects from a few images, but video customization can overfit and miss motion; DreamVideo learns motion types from video data but is limited to predefined types [23] [8] [10] [43] [47] [39] [57].Source: §2 · p. 3
  • Camera-control methods often use dedicated modules and large-scale training data, whereas MotionBooth proposes training-free camera control; object-motion methods manipulate inference-time attention [13] [53] [17] [66] [56] [63] [31] [32] [24] [4] [69] [27] [22].Source: §2 · p. 3

3 Method

MotionBooth fine-tunes a text-to-video model on a few subject images, then controls subject and camera motion during inference. It combines losses to preserve subject appearance and video capability with training-free motion controls.

3.1 Overview

  • MotionBooth takes a customized subject, a bounding-box sequence for subject positions, and horizontal and vertical camera-movement ratios as inputs.Subject customization typically uses 3–5 images; the model is fine-tuned during training, while motion control occurs at inference.Source: §3.1 · pp. 3–4

3.2 Subject Learning

  • Subject learning combines masked subject-region reconstruction, video preservation using common captioned videos, and STCA loss linking subject tokens to their spatial attention maps.The region loss is intended to prevent background overfitting; video data helps preserve video-generation knowledge, while STCA guides attention to the subject mask.Source: §3.2 · pp. 4–5

3.3 Subject Motion Control

  • At inference, cross-attention editing uses the requested boxes to amplify subject-token attention inside target regions and suppress it outside them.Amplification is stronger in earlier denoising steps and absent in later steps; suppression outside the boxes persists throughout generation.Source: §3.3 · pp. 5–6

3.4 Camera Movement Control

  • Camera motion shifts the noised latent, then fills newly exposed areas with horizontally or vertically sampled tokens, excluding subject tokens.The shift is spread across timestamps after approximate subject layout is fixed but before video details are completed.Source: §3.4 · pp. 6–7

4 Experiments

Experiments compare MotionBooth with customization and camera-control baselines, evaluate its training components and human preference, and report limitations involving multiple objects and implausible motions.

4.1 Experimental Setup

  • The evaluation uses 26 customized objects and 40 text-object motion pairs plus 40 text-camera motion pairs, with Zeroscope and LaVie as base models.Comparisons include DreamBooth, CustomVideo, DreamVideo, AnimateDiff, and CameraCtrl; metrics cover subject fidelity, temporal consistency, camera-motion fidelity, and video quality.Source: §4.1 · pp. 7–8

4.2 Main Results

  • MotionBooth outperforms all baselines on motion-aware customized video generation with both Zeroscope and LaVie.The comparison is reported in Table 1.Source: §4.2 · p. 8, Table 1 · p. 8
  • Against CameraCtrl, MotionBooth improves flow error, CLIP-T, and T-Cons. on both Zeroscope and LaVie.The reported improvements are 0.617, 0.015, and 0.009 on Zeroscope, and 0.511, 0.004, and 0.024 on LaVie, respectively.Source: §4.2 · p. 8, Table 2 · p. 8
  • In qualitative customized-video comparisons, MotionBooth excels in subject-motion alignment, prompt alignment, and overall video quality.DreamBooth and CustomVideo produce vague backgrounds, while CustomVideo and DreamVideo struggle to capture subject appearance.Source: §4.2 · pp. 8–9, Figure 6 · p. 9
  • For camera control, MotionBooth follows user-defined movements while maintaining temporal consistency and video quality.AnimateDiff does not support user-defined directions, while CameraCtrl produces subpar aesthetics and flash movements.Source: §4.2 · p. 9, Figure 7 · p. 9

4.3 Ablation Studies

  • Removing the subject region loss, STCA loss, or video preservation loss lowers metrics; removing the region loss decreases R-DINO by 0.256.Using class-specific rather than randomly sampled common preservation videos also yields worse results, attributed to restricted scene and background variety.Source: §4.3 · pp. 9–10, Table 3 · p. 9

4.4 Human Preference Study

  • In a study of 30 video groups rated by 7 colleagues, MotionBooth was most preferred across models and evaluation aspects, especially for subject appearance alignment.The criteria were subject-motion alignment, camera-movement alignment, subject appearance alignment, and temporal consistency.Source: §4.4 · p. 10, Figure 8 · p. 10

4.5 Limitations and Future Work

  • MotionBooth can confuse multiple-object scenes by merging the subject with other objects, and may misrepresent motions unlikely for the subject.The authors identify subject separation and motion understanding as areas for improvement.Source: §4.5 · p. 10

5 Conclusion

The conclusion presents MotionBooth as a framework for customized videos with controllable subject and camera motion. It highlights subject-focused training, video preservation, and training-free motion control.

5 Conclusion

  • MotionBooth fine-tunes a text-to-video diffusion model for specific subjects using subject region loss and video preservation data.An STCA loss connects subject tokens with the cross-attention map.Source: §5 · p. 10
  • Training-free inference technologies control both subject and camera motion in generated videos.The conclusion characterizes the experiments as demonstrating effectiveness and generalization ability.Source: §5 · p. 10
  • The paper concludes that MotionBooth generates vivid videos with given subjects and controllable subject and camera motions.Source: §5 · p. 10

A Appendix

The appendix lists supplementary implementation details, method discussions, comparisons, ablations, social impacts, and qualitative results, and notes a separate video demo.

A Appendix

  • The supplementary material covers implementation details, method discussions, more baseline comparisons, ablations, social impacts, and additional qualitative results.Source: A Appendix · p. 14
  • The appendix overview identifies six supplementary sections, from implementation details through additional qualitative results.Source: A Appendix · p. 14
  • A separate supplementary file presents the video demo in video format.Source: A Appendix · p. 14

A.1 Implementation Details

The appendix reports model-specific inference hyperparameters, a 26-subject evaluation set, the human-study interface, and latent-shift pseudocode.

A.1 Implementation Details

  • LaVie and Zeroscope use different values for τ, σ1, and σ2, while both use α = 10.0.LaVie uses τ = 0.7T, σ1 = 0.8T, and σ2 = 0.6T; Zeroscope uses τ = 0.9T, σ1 = 0.9T, and σ2 = 0.7T.Source: §A.1 · p. 14
  • The evaluation set contains 26 subjects spanning pets, plushie toys, cartoons, and vehicles.Source: §A.1 · p. 14
  • The appendix shows the human-study interface and latent-shift pseudocode, noting that latent shift controls camera movement training-free at minimal cost.Source: §A.1 · p. 14

A.2 Discussions

MotionBooth generally outperforms the compared alternatives, while discussion experiments probe camera and subject control limits, flexibility, and efficiency. Its camera motion degrades at extreme speeds, though it handles large foreground objects.

A.2 Discussions

  • MotionBooth masks spatial cross-attention outside the subject box; compared methods scale spatial and temporal maps, train motion layers, or tune latents at test time.The text also contrasts its latent shifting with Text2Video-Zero’s shifting and mirroring strategy.Source: §A.2 · pp. 14–16
  • MotionBooth generally outperforms the compared alternatives, although some methods were excluded because their code was unavailable.The comparison is reported in Table 2 and Table 4b.Source: §A.2 · p. 16
  • Changing the subject bounding box size produces zoom-in or zoom-out effects, and zoom-in can be combined with camera motion.The examples are shown in Fig. 12a and Fig. 12d.Source: §A.2 · p. 16
  • At triple the initial camera speed, control degrades to a downward tiling effect, despite success at half-width movement and doubled speed.The tested vectors were [−0.5, 0.45] and [−1.5, 1.35]; the method handled camera movements spanning the video width but weakened at extreme speeds.Source: §A.2 · p. 16
  • Latent shifts support rightward panning even when a large candy house nearly fills the frame, moving foreground and background together.The authors attribute this robustness to shifting both foreground and background elements.Source: §A.2 · p. 16
  • Subject control adds 0.3 seconds over the naive pipeline, while camera plus subject control takes 21.5 seconds per video.For LaVie, the naive pipeline takes 15.0 seconds; subject-only control takes 15.3 seconds, and camera-only control takes 20.6 seconds.Source: §A.2 · pp. 15–16

A.3 Comparison with More Baselines

The section compares MotionBooth with additional subject-motion baselines and text guidance for camera control. The reported comparisons favor MotionBooth overall and describe its camera control as more precise and stable than text prompts alone.

A.3 Comparison with More Baselines

  • Text prompts such as “camera pan right” convey simple directions but lack speed and distance specificity, yielding sub-optimal results versus latent shifting.The authors characterize their approach as more precise and stable than text prompts alone.Source: §A.3 · p. 17
  • MotionBooth generally outperforms TrailBlazer [34], Directed Diffusion, and Motion-Zero [4] in the subject-motion comparison.The comparison is presented in Table 4b; Table 4 is captioned as a comparison with more baselines.Source: Table 4 · p. 16, §A.3 · p. 17

A.4 Ablation Studies

The ablations examine motion-control strengths, preservation data, training-image masking, and latent filling. They report trade-offs between control and quality, and favor straight-sample filling for preserving video fidelity without limiting camera flexibility.

A.4 Ablation Studies

  • Increasing subject-control strength and duration strengthens alignment, but excessive control can make subjects unnaturally square.With weak control, the subject may miss the target position or only partly align; the study varies α of S and the maximum manipulation timestamp τ.Source: §A.4 · p. 17
  • Middle values of σ1 and σ2 provide the best camera control; early shifts leave movement incomplete, while late shifts degrade quality and add artifacts.The comparison is shown in Fig. 13b.Source: §A.4 · p. 17
  • Using 100 to 900 preservation videos causes little change in quantitative scores, suggesting video data matters more than its quantity within that range.The authors conclude the number of preservation videos can be flexible.Source: §A.4 · p. 17
  • Combined subject and camera control slightly lowers some Zeroscope metrics but improves others, a trade-off the authors consider acceptable.R-CLIP drops by 0.1 and T-Cons by 0.01, while R-DINO and CLIP-T improve slightly; Fig. 12g shows qualitative combined control.Source: §A.4 · p. 18
  • Masking training images directly impairs subject learning, whereas masking the diffusion loss better preserves learned representations.The authors attribute the decline to the unnatural distribution created by masked images; quantitative results are reported in Table 6b.Source: §A.4 · p. 18, Table 6 · p. 18
  • Straight-sample latent filling maintained video quality without significant artifacts or reduced camera flexibility, unlike the tested alternatives.Random Init and Random Sample produced artifacts; Loop improved quality but limited movement diversity, while Reflect did not work satisfactorily in this T2V setting.Source: §A.4 · p. 18

A.5 Social Impacts

MotionBooth is presented as a resource-light tool for artists and other creators, while the paper also identifies misuse and bias risks. The authors propose ethical guidelines, usage restrictions, and transparency as mitigation strategies.

A.5 Social Impacts

  • MotionBooth is described as enabling artists, filmmakers, and content creators to make customized videos without extensive resources or professional equipment.The paper frames precise subject and camera control as opening new avenues for visual-content creation.Source: §A.5 · p. 18
  • Realistic customized videos could be misused for deepfakes, disinformation, privacy violations, or reputational harm, and biased training data may reinforce stereotypes or exclude groups.The paper highlights political manipulation and social media as particularly significant contexts for misuse.Source: §A.5 · p. 19
  • The authors propose ethical guidelines, use restrictions, and transparency about generated content to mitigate misuse.Source: §A.5 · p. 19

A.6 More Qualitative Results

The appendix points to additional qualitative results for MotionBooth and identifies a figure containing latent-shift algorithm pseudo-code.

A.6 More Qualitative Results

  • Figure 15 contains additional qualitative results for MotionBooth.The text directs readers to Figure 15 for these results.Source: §A.6 · p. 19, Figure 15 · p. 21
  • Figure 14 presents pseudo-code for the latent shift algorithm.Source: Figure 14 · p. 20
Loading 2406.17758v3…