Source-linked AI summary
VideoBooth: Diffusion-based Video Generation with Image Prompts
Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, Ziwei Liu
TL;DR
Text prompts alone do not reliably specify customized subject appearances in generated videos. VideoBooth addresses this with coarse-to-fine image-prompt embeddings and attention injection, and the paper reports state-of-the-art image alignment with feed-forward, tuning-free inference.
Problem
Text prompts alone cannot fully capture the desired visual characteristics of customized video subjects.
Method
VideoBooth combines coarse image-encoder embeddings with fine multi-scale image prompts injected into cross-frame attention.
Results
VideoBooth achieves state-of-the-art image alignment performance compared with baseline methods and supports tuning-free inference.
Takeaways & Limitations
A single feed-forward VideoBooth model generates consistent videos containing subjects specified by image prompts.
Takeaways & Limitations
Coarse visual embeddings are universal and high-level, so they omit some visual details.
Abstract
from arXiv · showhide
Text-driven video generation witnesses rapid progress. However, merely using text prompts is not enough to depict the desired subject appearance that accurately aligns with users' intents, especially for customized content creation. In this paper, we study the task of video generation with image prompts, which provide more accurate and direct content control beyond the text prompts. Specifically, we propose a feed-forward framework VideoBooth, with two dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine manner. Coarse visual embeddings from image encoder provide high-level encodings of image prompts, while fine visual embeddings from the proposed attention injection module provide multi-scale and detailed encoding of image prompts. These two complementary embeddings can faithfully capture the desired appearance. 2) In the attention injection module at fine level, multi-scale image prompts are fed into different cross-frame attention layers as additional keys and values. This extra spatial information refines the details in the first frame and then it is propagated to the remaining frames, which maintains temporal consistency. Extensive experiments demonstrate that VideoBooth achieves state-of-the-art performance in generating customized high-quality videos with subjects specified in image prompts. Notably, VideoBooth is a generalizable framework where a single model works for a wide range of image prompts with feed-forward pass.
1. Introduction
VideoBooth addresses the difficulty of generating videos whose subjects match image-prompt appearances by combining coarse semantic and fine spatial embeddings. Its feed-forward design uses multi-scale attention injection to refine details and maintain temporal consistency.
- 1. Introduction: Image prompts provide more direct appearance control than text alone, which struggles to enumerate and accurately capture all desired subject attributes.The paper describes image prompts as complementary to text prompts because they enrich details that are difficult to depict textually.
- 1. Introduction: VideoBooth tackles video generation with image prompts while preserving both subject appearance and dynamic movement.The task requires accurate appearance capture and natural object motion, making direct adaptation of image-prompt methods to video insufficient.
- 1. Introduction: The framework combines coarse image-encoder embeddings with fine multi-scale attention injection to capture high-level appearance and detailed spatial information.The image encoder maps image features into text-embedding space, while attention injection supplies layer-specific detail.
- 1. Introduction: Multi-scale image prompts enter cross-frame attention as additional keys and values, refining the first frame before propagating appearance to later frames.This design uses spatial information at different granularities to improve detail and temporal consistency.
- 1. Introduction: VideoBooth uses a dedicated dataset and generates customized videos with image-specified subjects in a tuning-free inference pass.The paper reports that generated videos better preserve image-prompt visual attributes and that inference requires no tuning.
2. Related Work
Prior work on personalized image generation includes encoder-based methods that map reference images into embeddings for feed-forward generation. Other approaches fine-tune model parameters or combine encoding with fine-tuning.
- 2. Related Work: Text-to-video research has progressed from VQVAE-based explorations to diffusion models with temporal attention and related architectures.The paper situates VideoBooth within the development of text-to-video generation methods.
- 2. Related Work: Encoder-based methods learn mappings that embed reference images, enabling personalized generation through a feed-forward pass.ELITE uses global and local mapping networks, while another method adds cross-attention for image-feature embedding.
- 2. Related Work: Fine-tuning-based methods and hybrid approaches address image-conditioned generation but may require reference-image optimization or combine encoding with fine-tuning.The related work distinguishes parameter fine-tuning from encoder-based approaches and notes methods that combine both categories.
3. VideoBooth
VideoBooth combines coarse image embeddings with fine multi-scale attention injection to generate videos that preserve subject appearance and temporal consistency.
- 3. VideoBooth: VideoBooth combines coarse visual embeddings from a CLIP image encoder with fine multi-scale visual embeddings injected into cross-frame attention.The coarse embedding is mapped into text-embedding space, while attention injection supplies spatial detail at multiple resolutions.
- 3.2. Coarse Visual Embeddings via Image Encoder: Image and text prompts jointly condition generation, with image features replacing the target subject's text-token embeddings.The image prompt specifies visual characteristics, while the text prompt provides complementary information; the target subject tokens are replaced by the image embedding.
- 3.3. Fine Visual Embeddings via Attention Injection: Multi-scale image latents are appended as keys and values in cross-frame attention to refine synthesized details while preserving spatial information.Latent representations from different U-Net stages are supplied to attention layers with corresponding resolutions.
- 3.3. Fine Visual Embeddings via Attention Injection: Clean image latents are noised according to the diffusion process before being appended to attention, matching the intermediate video latents.The procedure addresses the domain discrepancy between clean image latents and noisy intermediate sampling states.
- 3.3. Fine Visual Embeddings via Attention Injection: The attention injection updates the first frame with image-prompt keys and values, then propagates the updated values to remaining frames.This staged update consistently transfers visual cues from the image prompt across the generated video.
4. VideoBooth Dataset
The VideoBooth dataset augments text-video data with segmented subject images, and its benchmark evaluates generation from paired image and text prompts.
- 4. VideoBooth Dataset: The dataset adds image prompts by segmenting target subjects from each video's first frame using Grounded-SAM.Noun chunks from original text prompts provide Grounded-SAM's word prompts, followed by filtering of unsuitable object sizes.
- 4. VideoBooth Dataset: The dataset is established specifically to support video generation conditioned on image and text prompts.It begins from WebVid, which supplies a text prompt for each video, and adds segmented subject images.
- 4. VideoBooth Dataset: The benchmark contains 650 non-overlapping test pairs, each providing one image prompt and one text prompt.The pairs are selected from the remaining WebVid-10M data rather than the training set.
5. Experiments
VideoBooth achieves strong image-prompt fidelity and competitive text alignment, with user preferences favoring it across evaluation dimensions. Ablations show that coarse and fine embeddings, together with coarse-to-fine training, are necessary for detailed and temporally consistent videos.
- Quantitative Comparisons: VideoBooth achieves the best image alignment and comparable text alignment performance against baseline methods.Table 1 reports the overall quantitative comparison.
- User Study: VideoBooth receives the highest user preference ratios across image alignment, text alignment, and overall quality.The user study included 25 participants evaluating twelve groups of videos.
- Qualitative Comparisons: VideoBooth preserves subject details better than baselines across dog, cat, and car video examples.Baselines miss, distort, or alter prompted appearances, while VideoBooth retains the specified subjects and details.
- Ablation Study: The full model outperforms ablations because coarse embeddings capture overall appearance while fine embeddings refine visual details.The ablation table identifies the full model as having the best scores.
- Ablation Study: Using fine embeddings alone or unified training causes later-frame distortions because the image encoder lacks sufficient coarse-level capability.The ablations indicate that attention injection can overfit the prompt without coarse-to-fine training.
6. Discussion
VideoBooth generates videos from image and text prompts by combining coarse image embeddings with fine attention-injected embeddings. The modules are trained coarse-to-fine to produce consistent videos containing the desired subjects.
- Discussion: VideoBooth combines coarse image-encoder embeddings with fine attention-injected embeddings to generate videos containing prompted subjects.The image encoder supplies coarse guidance, while attention injection refines it.
- Discussion: The framework uses coarse-to-fine training so the two embedding modules cooperate during generation.The paper describes the coarse embeddings as guidance for attention-injection refinement.
A. Comparison methods
The comparison methods adapt established personalized-generation approaches to video generation, including token optimization, weight optimization, and encoder-based image embedding.
- Comparison Methods: Textual Inversion optimizes a subject token, DreamBooth optimizes both model weights and a subject token, and ELITE uses an encoder for fast customized generation.These methods are adapted or retrained for the same pretrained video model setting.
B. More Discussions on Ablation Study
Additional ablations show that coarse-to-fine design is especially important when videos contain larger motions. Unified training can preserve appearance under small motion but may misalign motion or distort appearance under larger motion.
- More Discussions on Ablation Study: With fine embeddings alone, failed first-frame encoding propagates random or distorted subject appearances through later frames.Without coarse embeddings, later frames rely on propagation from the first frame.
- More Discussions on Ablation Study: Unified training can maintain consistent appearance for small or static motion, but larger motions expose appearance distortion and text-motion misalignment.The paper reports this behavior in the additional ablation example.
C. WaterMark Removal Module
The watermark-removal module is added before the final convolution and trained with watermark-free video data to improve visual quality.
- C. WaterMark Removal Module: Six added blocks before the last conv out layer form the watermark-removal module.The blocks can be regarded as a small UNet.
- C. WaterMark Removal Module: The module output is added as a residual to the original features.
- C. WaterMark Removal Module: The added module and conv out layer are finetuned on video data without watermarks.The finetuning uses Vimeo data.