PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-subject Open-set Personalization in Video Generation

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Yuwei Fang, Kwot Sin Lee, Ivan Skorokhodov, Kfir Aberman, Jun-Yan Zhu, Ming-Hsuan Yang, Sergey Tulyakov

arXiv:2501.06187v2cs.CV

TL;DR

Video personalization is constrained by narrow domains, limited subject support, and costly optimization, while suitable training data and evaluation remain difficult. Video Alchemist combines bound image-word conditioning in a latent Diffusion Transformer with augmented data construction and a subject-fidelity benchmark. The authors report improved quantitative and qualitative performance over existing personalization methods, while noting remaining overfitting and segmentation burdens.

  • Problem

    Existing video personalization methods often support limited domains or single subjects, require costly optimization, and lack a benchmark for multi-subject subject-fidelity evaluation.

  • Method

    Video Alchemist uses a latent Diffusion Transformer that binds each reference image to its entity word, with augmented video-derived data and the MSRVTT-Personalization benchmark.

  • Results

    Video Alchemist outperforms existing personalization methods in quantitative and qualitative evaluations.

  • Takeaways & Limitations

    The model supports multi-subject, open-set personalization across foreground objects and backgrounds without test-time optimization.

  • Takeaways & Limitations

    The method can retain undesirable reference-image properties such as similar facial expressions or postures, and segmented-image inputs may require additional user effort when localization fails.

Abstract

from arXiv · show

Video personalization methods allow us to synthesize videos with specific concepts such as people, pets, and places. However, existing methods often focus on limited domains, require time-consuming optimization per subject, or support only a single subject. We present Video Alchemist $-$ a video model with built-in multi-subject, open-set personalization capabilities for both foreground objects and background, eliminating the need for time-consuming test-time optimization. Our model is built on a new Diffusion Transformer module that fuses each conditional reference image and its corresponding subject-level text prompt with cross-attention layers. Developing such a large model presents two main challenges: dataset and evaluation. First, as paired datasets of reference images and videos are extremely hard to collect, we sample selected video frames as reference images and synthesize a clip of the target video. However, while models can easily denoise training videos given reference frames, they fail to generalize to new contexts. To mitigate this issue, we design a new automatic data construction pipeline with extensive image augmentations. Second, evaluating open-set video personalization is a challenge in itself. To address this, we introduce a personalization benchmark that focuses on accurate subject fidelity and supports diverse personalization scenarios. Finally, our extensive experiments show that our method significantly outperforms existing personalization methods in both quantitative and qualitative evaluations.

1. Introduction

Video Alchemist addresses the limited scope and costly optimization of prior video personalization by supporting multiple open-set entities across foregrounds and backgrounds without test-time fine-tuning. It also introduces data construction and evaluation strategies for training and assessing this setting.

  • Existing methods are limited to faces, single subjects, or foreground objects, and many require costly test-time optimization.
  • Video Alchemist supports multiple open-set entities, including foreground objects and backgrounds, without fine-tuning for new concepts.Its personalization modules combine video text conditioning with reference-image embeddings and subject-level word descriptions.
  • Reference frames and target clips from the same video can produce a copy-and-paste effect because non-identity factors remain correlated across frames.
  • MSRVTT-Personalization evaluates diverse conditioning modes using subject fidelity for each object segment rather than only image-level similarity.
  • The paper contributes a large-scale curated training dataset, anti-overfitting techniques, and a benchmark with varied conditioning modes and subject-fidelity measurement.

2. Related Work

Prior work spans diffusion video generation and personalized image or video synthesis, but personalization methods commonly remain constrained by optimization, subject count, domain, or foreground-only conditioning. Video Alchemist instead targets multiple open-set entities across foreground and background within an end-to-end video model.

  • Diffusion Video Models: Diffusion video models generate realistic, temporally coherent videos using architectures including cascaded upsamplers, latent image generators, and transformer-based designs.
  • Personalized Image Generation: Personalized image methods adapt models through weight optimization, learned text embeddings, or cross-attention composition, but optimization-based approaches require fine-tuning or subject-specific optimization.
  • Personalized Image Generation: Encoder-based methods reduce test-time fine-tuning by converting reference images into conditioning tokens or adapter representations.
  • Personalized Video Generation: Personalized video methods include optimization-based adapters and optimization-free mechanisms, but most focus on faces, single subjects, or restricted domains.
  • Personalized Video Generation: Video Alchemist supports multiple open-set entities across foreground objects and backgrounds through an end-to-end method.

3. Methodology

The methodology constructs segmented subject and background references from videos, then conditions a latent Diffusion Transformer on text and bound image-word concepts. Data augmentation addresses reference-image overfitting, while separate cross-attention layers support personalization conditioning.

  • Dataset Collection: The dataset pipeline retrieves subject, object, and background entities from captions and filters ambiguous videos, including plural subject descriptions.
  • Dataset Collection: Three frames sampled at the 5%, 50%, and 95% video percentiles are detected and segmented to capture varied poses and lighting.Face crops are additionally extracted for human references.
  • Dataset Collection: A clean background image is created from the middle frame by removing foreground entities, dilating masks, and applying inpainting.
  • Method Overview: The training objective denoises videos conditioned on a text prompt, reference images, and corresponding entity words.
  • Video Personalization Model: The latent Diffusion Transformer compresses videos into latent 1-D tokens, adds Gaussian noise, and learns denoising with a rectified-flow formulation.
  • Video Personalization Model: DiT blocks combine self-attention with separate text and personalization cross-attention layers to integrate video, text, and image conditioning.
  • Video Personalization Model: Without image-word binding, the model can apply a reference image to the wrong subject, such as placing a human face on a dog.
  • Video Personalization Model: Personalization embeddings concatenate per-reference representations whose components fuse each image with its corresponding entity word.The image tokens come from a shared frozen image encoder, while word tokens are aligned and projected with the image tokens.

4. Experiments

The experiments introduce MSRVTT-Personalization, compare Video Alchemist with existing methods, and analyze its design choices. Video Alchemist achieves stronger personalization fidelity and dynamics, while ablations expose trade-offs among encoders, concept binding, and augmentation.

  • MSRVTT-Personalization Benchmark: MSRVTT-Personalization evaluates face, single-subject, multi-subject, foreground, and background conditioning scenarios.The benchmark is designed to measure personalization across diverse conditioning modes and subject settings.
  • MSRVTT-Personalization Benchmark: The benchmark constructs 2,130 clips from MSR-VTT using shot splitting, captioning, annotation, and manual quality selection.One clip is selected from each long video to increase test diversity.
  • Quantitative Evaluation: Video Alchemist achieves 23.2% higher subject similarity than VideoBooth and 11.3% higher face similarity than IP-Adapter, while also obtaining the best text alignment and greatest video dynamics.The comparison covers open-set and face-specific personalization methods on MSRVTT-Personalization.
  • Quantitative Evaluation: More reference images improve subject or face fidelity and background conditioning improves similarity to the ground-truth video, but additional references can worsen textual alignment.The authors attribute the alignment degradation potentially to reduced flexibility from more reference images.
  • Qualitative and Human Evaluation: Qualitative comparisons report more photorealistic videos with higher target-subject fidelity than existing models, and human evaluation finds higher visual quality and subject fidelity.Human fidelity scores positively correlate with automated subject- and face-similarity scores.
  • Ablation Study: DINOv2 improves subject similarity while CLIP improves text similarity, showing a trade-off between subject representation and text-image alignment.The authors hypothesize that DINOv2 captures distinctive object features, whereas CLIP emphasizes prompt-describable details.
  • Ablation Study: Binding each image to its entity word prevents conditions from being assigned to the wrong subject, which otherwise causes missing subjects and lower subject similarity.The ablation describes a reference face being placed on a dog without this binding mechanism.
  • Ablation Study: Data augmentation reduces copy-and-paste behavior and improves text alignment and dynamics, although removing it can increase subject similarity while reproducing reference expressions.Without augmentation, a prompt describing a smiling woman may produce the reference expression without the requested smile.

5. Conclusion

Video Alchemist enables multi-subject, open-set personalization of foreground objects and backgrounds without test-time optimization. Its dataset curation, augmentation, and benchmark support improved evaluation, while experiments show stronger quantitative and qualitative performance than existing methods.

  • Video Alchemist supports multi-subject and open-set personalization for both foreground objects and background without test-time optimization.
  • The model integrates conditional images with subject-level prompts through cross-attention layers.
  • Dataset curation and data augmentation reduce overfitting to undesirable properties of reference images.
  • The introduced benchmark evaluates personalization models across varied conditioning scenarios.
  • Experiments show that the method outperforms existing methods in quantitative and qualitative measures.

A. Details of Training Datasets and Augmentations

The training data are curated from large video datasets, with language-model-based entity extraction and filtering to identify suitable personalization subjects. The resulting dataset contains 37.8M videos and is characterized through sampled entity-word statistics.

  • The personalization dataset is built from Panda-70M and internal video-caption datasets totaling 86.8M videos before filtering.
  • An LLM retrieves entity words from captions using the prompt template shown in Figure 7.
  • Entity-word processing removes invalid outputs, caption-mismatched terms, and samples without subject entities.
  • Entity words are classified as subjects, objects, or backgrounds, with visually separable entities distinguished from background terms.
  • The curated training dataset comprises 37.8M videos, and a word cloud summarizes entity words from 10k randomly sampled videos.

A.3. Data Augmentation and Conditional Images Sampling

Training augmentations reduce reliance on undesirable reference-image properties, while conditional-subject sampling prevents overfitting to the number of reference images. The framework also uses a DiT backbone with image encoders for localized reference representations.

  • Data augmentation reduces copy-and-paste behavior but trades higher subject similarity for lower dynamic degree and text similarity.Without augmentation, the generated subject can replicate the reference expression instead of following the prompt.
  • Conditional training samples one to three subjects and one to three reference images per subject, with replacement.
  • Background conditioning is randomly included with probability 50%.
  • The latent diffusion framework uses CogVideoX-5B as autoencoder, DiT as video backbone, and CLIP and DINOv2 as image encoders.
  • Patch-token image embeddings retain localized reference properties and produce higher fidelity than class-token embeddings.

B.2. Model Training

Model training proceeds in two stages, first training the video backbone and then introducing and fine-tuning personalization cross-attention. Inference combines text and image conditioning with separate guidance scales and resolution-dependent denoising.

  • Training uses 60k initial steps without personalization cross-attention, followed by 40k steps that introduce and fine-tune the new cross-attention layer.
  • The model is trained on videos with varying resolutions and lengths and supports generation up to 12 seconds at 256px × 144px.
  • Training uses 256 A100 GPUs in stage I and 64 GPUs in stage II.
  • Large image-conditioning guidance can replicate reference images directly, reducing natural motion and appearance variation.
  • The guidance formulation combines predictions under image and text conditioning with separate text and image scales.
  • Sampling uses sT = 8 and sI = 3, with 256, 128, and 64 denoising steps for 256px × 144px, 512px × 288px, and 1024px × 576px videos, respectively.

C.1. Additional Results of Multi-subject Open-set Personalization

Video Alchemist demonstrates multi-subject and open-set personalization across varied subjects, prompts, and backgrounds. Additional results also test changing reference images and compare personalization quality across object categories.

  • Multi-subject and open-set personalization: Generated samples condition on one to three subjects or backgrounds by incrementally adding reference images.Synthetic videos without reference images are also shown to illustrate image-conditioning effects.
  • Background personalization: The same prompt supports varied reference backgrounds, including bridges, deserts, sea ice, and the Moon’s surface.The examples are presented as evidence of personalization across diverse background settings.
  • Reference-image variation: Changing one subject’s reference image while keeping other conditional inputs fixed produces samples for people and dogs.Figures 13 and 14 specifically vary the person or dog reference image under otherwise unchanged conditions.
  • Qualitative comparisons: Qualitative comparisons evaluate conditional subjects including dogs, cats, cars, and dinosaur toys against existing personalization models.The comparisons extend prior horse and woman evaluations to additional subject categories.

D. Limitations

The paper identifies limitations involving residual overfitting, reliance on segmented image inputs, occasional unrealistic multi-subject composition, and video-quality evaluation.

  • Model Overfitting: Residual overfitting can produce facial expressions or postures resembling those in reference images.The paper notes that this issue also appears in reconstruction-based methods such as IP-Adapter and remains open for future work.
  • Taking Image Segments as Inputs: Personalization requires segmented images, so inaccurate localization may impose additional user effort.The authors plan to train with segmented images pasted onto random backgrounds to reduce this burden.
  • Unnatural Composition for Multi-subject Conditioning: Multi-subject conditioning can sporadically produce unrealistic relative compositions and scales between subjects.The paper relates this behavior to the minority of multi-subject videos in the training dataset.
  • Unsupported Measure on Video Quality: MSRVTT-Personalization does not assess visual quality, requiring alternative evaluations such as user studies.The limitation is distinct from measuring subject personalization fidelity.
Loading 2501.06187v2…