PaperlayerView exact revision on arXiv

Source-linked AI summary

CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects

Zhao Wang, Aoxue Li, Lingting Zhu, Yong Guo, Qi Dou, Zhenguo Li

arXiv:2401.09962v3cs.CV

TL;DR

Multi-subject video customization must ensure that subjects co-occur while retaining their identities; combining multi-subject T2I customization with I2V models also has inaccurate customization and motion limitations. CustomVideo trains on concatenated multi-subject images and uses mask-guided attention control to disentangle subjects, with class-image prior preservation in its training objective. CustomVideo exceeds DreamVideo by 11.99% in CLIP Image Alignment and 23.39% in DINO Image Alignment.

  • Problem

    Multi-subject video customization must ensure that subjects co-occur while retaining their identities; combining multi-subject T2I customization with I2V models also has inaccurate customization and motion limitations.

    Source: §I · p. 1, B. CustomVideo with Multiple Subjects · p. 4

  • Method

    CustomVideo trains on concatenated multi-subject images and uses mask-guided attention control to disentangle subjects, with class-image prior preservation in its training objective.

    Source: §I · p. 1, C. Model Training and Inference · p. 4

  • Results

    CustomVideo exceeds DreamVideo by 11.99% in CLIP Image Alignment and 23.39% in DINO Image Alignment.

    Source: B. Main Results · p. 7

  • Takeaways & Limitations

    CustomVideo is presented as a strong baseline for multi-subject video customization and requires subject images rather than subject videos for training.

    Source: §V · p. 12

    The method depends on base-model capability and struggles with too many subjects whose spatial positions conflict; predefined spatial bounding boxes are proposed for that case.

    Source: D. Limitations · pp. 11–12

Abstract

from arXiv · show

Customized text-to-video generation aims to generate high-quality videos guided by text prompts and subject references. Current approaches for personalizing text-to-video generation suffer from tackling multiple subjects, which is a more challenging and practical scenario. In this work, our aim is to promote multi-subject guided text-to-video customization. We propose CustomVideo, a novel framework that can generate identity-preserving videos with the guidance of multiple subjects. To be specific, firstly, we encourage the co-occurrence of multiple subjects via composing them in a single image. Further, upon a basic text-to-video diffusion model, we design a simple yet effective attention control strategy to disentangle different subjects in the latent space of diffusion model. Moreover, to help the model focus on the specific area of the object, we segment the object from given reference images and provide a corresponding object mask for attention learning. Also, we collect a multi-subject text-to-video generation dataset as a comprehensive benchmark. Extensive qualitative, quantitative, and user study results demonstrate the superiority of our method compared to previous state-of-the-art approaches. The project page is https://kyfafyd.wang/projects/customvideo.

I. INTRODUCTION

Multi-subject text-to-video customization must keep subjects co-occurring while preserving their identities. CustomVideo combines co-occurrence training and attention control, and the introduction reports stronger alignment than prior methods.

I. INTRODUCTION

  • Existing multi-subject video methods struggle to guarantee co-occurrence and disentangle similar identities, while combining T2I and I2V adds inaccurate customization and weak motion.The introduction identifies co-occurrence and identity preservation as central challenges.Source: §I · p. 1
  • CustomVideo combines multi-subject co-occurrence training with mask-guided attention control to disentangle subjects and focus on their corresponding image regions.The masks come from segmentation or human annotation; the attention design uses positive emphasis on the subject area and negative values outside it.Source: §I · p. 1
  • CustomVideo collects a benchmark dataset covering diverse subject categories and pairs.The dataset is presented as a comprehensive benchmark for multi-subject text-to-video generation.Source: §I · p. 1
  • 11.99% CLIP Image Alignment and 23.39% DINO Image Alignment better than the previous best method.The introduction also reports consistent outperformance in qualitative and human-preference evaluation.Source: §I · p. 1

II. RELATED WORK

Prior work spans diffusion-based text-to-video generation and subject customization, but existing video methods have limitations for multi-subject generation. CustomVideo proposes co-occurrence and mask-guided attention control to address this setting.

a) Text-to-Video Generation

  • Text-to-video work progressed from GAN and VQ-VAE approaches to diffusion models, with later methods adding temporal modeling, controllability, layouts, and event structures.Examples include temporal attention fine-tuning and guidance from depth maps or generated layouts.Source: §II · pp. 1–2

b) Subject-driven Customization

  • Video personalization methods largely target single objects; VideoDreamer lacks guaranteed co-occurrence and disentanglement, while multi-subject T2I+I2V lacks co-occurrence training and subject-specific motion priors.The section also describes static or disjointed motion in the car-and-barn example for the combined approach.Source: §II · p. 2

II. RELATED WORK

  • CustomVideo proposes a co-occurrence and attention mechanism that uses masks to disentangle multiple subjects while preserving their co-occurrence in generated videos.The framework overview also describes class-specific prior preservation and fine-tuning cross-attention key and value weights alongside a learnable token for each subject.Source: §II · p. 3, Figure 2 · p. 3

A. Preliminary: Text-to-Video

Video diffusion models generate videos through iterative denoising, learning to predict noise under text conditioning. CustomVideo uses a pre-trained text-to-video model as its base.

A. Preliminary: Text-to-Video

  • Video diffusion models iteratively denoise sampled Gaussian noise, modeling temporal dependencies through a reverse process resembling a fixed-length Markov chain.At each timestep, the model predicts added noise given a text condition.Source: A. Preliminary: Text-to-Video · p. 3
  • The training objective is a reconstruction loss comparing model-predicted noise with sampled noise, conditioned on the video latent and text.The noisy latent is formed by adding noise to the ground-truth latent using a diffusion hyperparameter.Source: A. Preliminary: Text-to-Video · p. 3
  • CustomVideo uses the zeroscope text-to-video model, built on a 3D U-Net with spatial and temporal modeling, as its base model.The input video latent is indexed by batch, video length, height, width, and latent dimension.Source: A. Preliminary: Text-to-Video · p. 3

B. CustomVideo with Multiple Subjects

CustomVideo trains on concatenated subject images to encourage co-occurrence, then uses mask-guided attention control to separate subject identities. Training also removes image backgrounds and requires only prompts with learned subject tokens at inference.

B. CustomVideo with Multiple Subjects

  • Each subject receives a learnable word token, and training updates those tokens plus cross-attention key and value weights; inference uses prompts containing the learned tokens.This is described as parameter-efficient fine-tuning.Source: B. CustomVideo with Multiple Subjects · p. 3
  • Training only on concatenated multi-subject images is sufficient for co-occurrence, whereas mixing single-subject and concatenated images can confuse the model; background removal helps it learn subject characteristics.Background removal may be performed manually or with an automatic tool such as SAM.Source: B. CustomVideo with Multiple Subjects · pp. 3–4
  • Mask-guided positive attention aligns each subject token’s cross-attention map with its ground-truth subject region, directing attention toward the correct area.The mask marks the subject region with 1 and the remaining area with 0.Source: B. CustomVideo with Multiple Subjects · p. 4
  • Negative guidance assigns a small negative value outside the subject mask to reduce irrelevant-area influence and improve generated-subject fidelity.The section illustrates unwanted color transfer to a dog from a cat when negative attention is omitted.Source: B. CustomVideo with Multiple Subjects · p. 4

C. Model Training and Inference

CustomVideo trains on subject and class images rather than subject videos, using concatenation and attention control to learn multiple identities. At inference, a text prompt with learned subject tokens is sufficient, and the method produces diverse, identity-faithful videos with smooth motion.

C. Model Training and Inference

  • CustomVideo fine-tunes cross-attention key and value weights using subject images extended into single-frame videos, avoiding the need for subject videos.Concatenated class images are also used for class-specific prior preservation, with reconstruction loss as supervision.Source: C. Model Training and Inference · p. 4
  • Training concatenates different subjects without overlap, allowing attention control to locate their regions and learn their features.The combined objective weights attention control and prior preservation with hyperparameters α and β.Source: C. Model Training and Inference · p. 4
  • At inference, a text prompt containing the learned subject tokens generates the video; ground-truth subject masks are not required.Source: C. Model Training and Inference · p. 4
  • CustomStudio spans 13 object categories, and Table I presents subject pairs constructed from those categories.Source: Figure 3 · p. 5, Table I · p. 5
  • Image-based fine-tuning produces videos with high diversity, subject fidelity, and motion smoothness.Source: C. Model Training and Inference · p. 5

A. Experimental Setup

The experiments evaluate CustomVideo on CustomStudio using paired subjects and varied prompts, with comparisons to personalization and video-generation baselines. The paper reports that CustomVideo outperforms prior methods across all four metrics and shows better subject fidelity in qualitative comparisons.

A. Experimental Setup

  • CustomStudio contains 63 individual objects and 68 meaningful pairs across 13 categories, with 4–21 reference images per subject.Source: A. Experimental Setup · p. 5
  • Inference uses prompt templates for videos, and Table III gives 20 independent prompts used to generate class images for prior preservation.The implementation trains for 500 steps and generates 24-frame videos at 576 × 320 resolution and 8 fps.Source: Table II · p. 5, Table III · p. 5, A. Experimental Setup · p. 5
  • The evaluation uses 25 types of subject pairs, with 10 prompts per pair spanning different contexts and actions.Source: A. Experimental Setup · p. 5
  • Qualitative comparisons show better subject fidelity for CustomVideo than the five comparison methods.Figure 4 presents the given subjects, method-generated frames, and inference prompt.Source: Figure 4 · p. 6
  • Training fine-tunes key and value weights in cross-attention layers while leaving the temporal transformer responsible for motion fixed.The trained weights and tokens can be loaded into a high-resolution model without additional training computation.Source: A. Experimental Setup · p. 6
  • The comparison set includes image-customization methods, VideoDreamer, single-subject DreamVideo, and DisenDiff paired with SVD-XT.Source: A. Experimental Setup · p. 6
  • CustomVideo consistently outperforms DreamBooth, CustomDiffusion, VideoDreamer, DisenDiff + SVD-XT, and DreamVideo on all four evaluation metrics.Source: Table IV · p. 7
  • The four metrics measure text alignment, image alignment, DINO image similarity, and temporal consistency.The specified encoders include CLIP ViT-B/32 for the two CLIP metrics and DINO ViT-S/16 for DINO image alignment.Source: A. Experimental Setup · p. 7

B. Main Results

CustomVideo is reported to outperform prior approaches in quantitative, qualitative, and human-preference evaluations. Its results emphasize subject fidelity, co-occurrence, and video consistency.

Qualitative Results

  • CustomVideo ensures multiple subjects co-occur and are disentangled, while outperforming comparison methods on challenging car and cat–dog scenes.The comparisons report identity and color failures for other methods, while CustomVideo captures structural car details and handles similar subjects.Source: B. Main Results · p. 7

Quantitative Results

  • 11.99% better CLIP Image Alignment, 23.39% better DINO Image Alignment, 6.46% better CLIP Textual Alignment, and 6.16% better Temporal Consistency than DreamVideo.The evaluation generated 2,720 videos per method using 10 prompts and four random seeds for each subject pair.Source: B. Main Results · p. 7

Human Preference Study

  • Human raters preferred CustomVideo over five comparison methods for textual alignment, subject-image alignment, and overall quality.The study collected 1,500 answers from 25 independent raters.Source: B. Main Results · p. 7, Figure 7 · p. 10

C. Ablation Studies

The ablations examine co-occurrence training, attention control, cross-attention map levels, and attention-control parameters. They identify choices associated with more consistent subject identity and alignment.

Component Analysis

  • Removing subject concatenation causes a significant performance drop and lets one subject dominate, while mixing single and concatenated subjects produces unstable generation.The component analysis also reports a performance drop without background removal.Source: C. Ablation Studies · p. 10, Table V · p. 10
  • Positive attention guidance improves CLIP Image Alignment by 11.45%, while positive and negative guidance together support subject fidelity and temporal consistency.Positive guidance preserves a subject’s characteristics; negative guidance weakens other subjects’ influence on that subject.Source: C. Ablation Studies · p. 10

Cross-attention Maps

  • With attention control, learnable tokens align better to the correct cat and dog regions than without control, where the cat token aligns to the dog.The cross-attention comparison is shown in Fig. 9.Source: C. Ablation Studies · p. 10, Figure 9 · p. 11

Layers of Cross-attention Maps

  • Cross-attention maps from ℓ3 work better than maps from ℓ1, ℓ4, or the other tested middle level, so ℓ3 is used by default.The paper associates large maps with loss of subtle features and small maps with inaccurate optimization against downsampled masks.Source: C. Ablation Studies · p. 11, Table VI · p. 11

Effect of Hyper-parameters in Attention Control

  • The best tested attention-loss weight is α=0.2, and even a slight negative value in the guidance mask can improve generation quality.The paper links negative guidance to suppressing the influence of other subjects on a specific subject.Source: C. Ablation Studies · p. 11

D. Limitations

CustomVideo’s performance is bounded by the base model and by scenes with many subjects whose spatial positions conflict. The paper suggests predefined spatial bounding boxes as a possible remedy for the latter case.

Limitations

  • The method may fail when the base model cannot generate the needed content, such as small faces in a global view.The failure examples also include vivid facial content in global views.Source: D. Limitations · p. 11, Figure 11 · p. 12
  • The method cannot handle too many subjects when their spatial positions conflict; predefined bounding boxes are proposed as an inference-time remedy.The failure example customizes generation with six different subjects.Source: D. Limitations · p. 12, Figure 11 · p. 12

V. CONCLUSIONS

The conclusion presents CustomVideo as a multi-subject text-to-video framework using co-occurrence and attention control to disentangle subjects and preserve identity.

Conclusions

  • CustomVideo combines co-occurrence and attention control to disentangle similar subjects and preserve their identities.The conclusion describes the framework as a multi-subject text-to-video generation method.Source: §V · p. 12
  • Its training process requires subject images rather than subject videos, and inference uses a user-provided text prompt.Source: §V · p. 12
  • The authors collect CustomStudio for evaluation and report quantitative, qualitative, and user-study results indicating effectiveness and outperformance of state-of-the-art methods.Source: §V · p. 12
Loading 2401.09962v3…