PaperlayerView exact revision on arXiv

Source-linked AI summary

ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning

Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, Kun Gai

arXiv:2501.04698v2cs.CV

TL;DR

MCVC requires preserving multiple identities and obtaining suitable video-entity data, challenges that existing customization approaches do not adequately address. ConceptMaster learns decoupled concept embeddings, injects them through standalone cross-attention, and constructs large-scale data for training and evaluation. The framework is reported to maintain fidelity and decoupling across diverse multi-concept scenarios without test-time parameter tuning.

  • Problem

    MCVC lacks reliable identity decoupling for multiple concepts and sufficient high-quality video-entity pairs for training.

  • Method

    ConceptMaster learns decoupled multi-concept embeddings and injects them through a standalone cross-attention layer in diffusion transformers, supported by a dedicated data pipeline.

  • Results

    ConceptMaster is reported to maintain concept fidelity and identity decoupling, including for highly similar concepts, while generating high-quality customized videos without additional parameter tuning.

  • Takeaways & Limitations

    The framework supports personalized multi-concept video generation across six composition scenes, evaluated for fidelity, identity decoupling, and video quality.

Abstract

from arXiv · show

Text-to-video generation has made remarkable advancements through diffusion models. However, Multi-Concept Video Customization (MCVC) remains a significant challenge. We identify two key challenges for this task: 1) the identity decoupling issue, where directly adopting existing customization methods inevitably mix identity attributes when handling multiple concepts simultaneously, and 2) the scarcity of high-quality video-entity pairs, which is crucial for training a model that can well represent and decouple various customized concepts in video generation. To address these challenges, we introduce ConceptMaster, a novel framework that effectively addresses the identity decoupling issues while maintaining concept fidelity in video customization. Specifically, we propose to learn decoupled multi-concept embeddings and inject them into diffusion models in a standalone manner, which effectively guarantees the quality of customized videos with multiple identities, even for highly similar visual concepts. To overcome the scarcity of high-quality MCVC data, we establish a data construction pipeline, which enables collection of high-quality multi-concept video-entity data pairs across diverse scenarios. A multi-concept video evaluation set is further devised to comprehensively validate our method from three dimensions, including concept fidelity, identity decoupling ability, and video generation quality, across six different concept composition scenarios. Extensive experiments demonstrate that ConceptMaster significantly outperforms previous methods for video customization tasks, showing great potential to generate personalized and semantically accurate content for video diffusion models.

1. Introduction

Multi-Concept Video Customization remains difficult because existing methods mix identities and suitable video-entity data are scarce. ConceptMaster addresses these challenges with decoupled embeddings, standalone injection, and a dedicated data pipeline.

  • Tuning-based customization is time-consuming and often requires manually collecting multiple reference samples, limiting practical use.
  • MCVC must preserve each concept’s identity while expressing its semantic behavior, but existing approaches can conflate attributes, especially for visually similar concepts.
  • ConceptMaster learns decoupled multi-concept embeddings and injects them through a standalone cross-attention layer without affecting original textual cross-attention.
  • The standalone injection strategy is designed to maintain high-fidelity customized videos with multiple identities, including highly similar visual concepts, without additional inference-time parameter tuning.
  • The data pipeline collects over 1.3 million video-entity pairs across diverse conceptual domains and supports evaluation across six multi-concept composition scenes.The evaluation covers concept fidelity, identity decoupling, and video generation quality.

2. Related Work

Text-to-video diffusion models have progressed from UNet-based latent diffusion systems to transformer-based architectures with stronger generation capabilities. Video customization methods remain divided between costly tuning-based approaches and feed-forward pretraining-based methods, with limited attention to multi-concept video customization.

  • Transformer-based diffusion models, including DiT and SORA, scale model parameters and generate realistic, long-range, physically consistent videos.
  • Customization methods use reference images to retain user-provided identities and are primarily categorized as tuning-based or pretraining-based approaches.
  • Tuning-based methods optimize embeddings or diffusion-model weights for new references, but consume time and may require manually collected training samples.
  • Pretrained multi-concept video customization has received limited attention, while preliminary methods predominantly focus on single-concept scenarios.
  • ConceptMaster targets MCVC in a feed-forward manner using a transformer-based latent diffusion model with Flow Matching to regress diffusion velocity.

4. ConceptMaster

ConceptMaster targets MCVC by preserving each concept’s identity while aligning generated videos with captions. It combines decoupled visual-text representations, standalone injection into diffusion transformers, and a multi-concept data pipeline.

  • 4.1. Multi-Concept Video Customization: MCVC generates videos from a caption, multiple concept images, and corresponding labels while preserving each concept’s identity and semantic behavior.The task incorporates all image-defined visual concepts into the generated video while aligning them with the caption.
  • 4.2. Decoupling and Injecting Concept Embeddings: ConceptMaster extracts visual tokens with CLIP and a learnable Q-Former, then aligns each visual representation with its concept label through DAM.The framework binds visual representations with corresponding text embeddings before forming composite multi-concept representations.
  • 4.2. Decoupling and Injecting Concept Embeddings: The MC-Injector concatenates multi-modal concept embeddings and injects them through an additional standalone cross-attention layer in each diffusion-transformer block.The layer follows the original text cross-attention and uses composite concept features to learn concepts without interfering with the original textual pathway.
  • 4.3. MC-Oriented Video Data Construction: The data pipeline addresses multi-concept extraction by filtering unsuitable videos, extracting captions and labels, and performing fine-grained identity information extraction.It includes scene-transition and quality filtering, caption-based noun extraction, text-guided detection, and mask-based refinement.
  • 4.3. MC-Oriented Video Data Construction: The constructed data is supplemented with single-concept image and video datasets to enhance concept and human representation.The sampling ratio among the built data, BLIP-Diffusion, and CelebV is 8:1:1.

5. Experiments

Experiments evaluate ConceptMaster across concept fidelity, identity decoupling, semantic alignment, and video quality using qualitative, quantitative, and ablation studies. Results support standalone multi-concept embedding injection and the Q-Former/DAM design.

  • Comparing with other methods: ConceptMaster shows superior qualitative concept fidelity, identity decoupling, and caption semantic consistency compared with several MCVC methods.
  • Experimental Setup: The evaluation set measures concept fidelity, identity decoupling, and video generation quality across six multi-concept composition scenarios.It contains 210 manually curated samples spanning persons, livings, and stuffs while reducing information leakage.
  • Comparing with other methods: ConceptMaster maintains multiple concept representations while generating high-quality text-aligned videos, outperforming a naive image-customization-plus-I2V solution.The naive solution is limited by the decoupling and representation abilities of both component models and by I2V instruction following.
  • Multi-Concept Embeddings Injection Manner: Standalone injection of decoupled multi-concept embeddings is preferred because joint visual-text or whole-image integration struggles with fidelity, identity decoupling, and video dynamics.Joint visual-text integration also influences the original text cross-attention layer and significantly reduces dynamic degree.
  • Ablation Study: Replacing the Q-Former with an MLP prevents videos from capturing reference appearances and substantially lowers quantitative metrics.
  • Ablation Study: The DAM module is the optimal tested design, while paired textual descriptions improve concept uniqueness and alignment with the diffusion model space.Cross-attention preserves consistent visual appearances better than self-attention, which produces more artifacts.

6. Conclusion

The conclusion presents ConceptMaster as a framework for multi-identity video customization that addresses identity decoupling and concept fidelity. It combines standalone decoupled embedding injection, a data pipeline, and a six-scenario evaluation set.

  • ConceptMaster addresses identity decoupling while maintaining concept fidelity when customizing multiple identities in videos.
  • The framework learns decoupled multi-concept embeddings and injects them into diffusion models in a standalone manner, including for highly similar visual concepts.
  • Its data construction pipeline collects precise multi-concept video-entity data across diverse concepts, while the testing set evaluates three dimensions across six composition scenarios.
  • Extensive experiments demonstrate that ConceptMaster significantly outperforms previous approaches for personalized, semantically accurate multi-concept video generation.

Supplementary Material

The supplementary materials provide implementation, evaluation, comparison, injection, ablation, and qualitative-result details for ConceptMaster and its experimental setting.

  • The supplement documents the text-to-video diffusion transformer models and ConceptMaster implementation details.
  • It discusses the data collection pipeline relative to Grounded-SAM and provides additional details about the multi-concept evaluation set.
  • It includes comparison-method implementations, further discussion of multi-concept embedding injection, ablation studies, and qualitative demonstrations.

1. Introduction of our text-to-video diffusion transformer models

The foundational generator is a transformer-based latent diffusion model for text-to-video generation. A 3D-VAE first maps videos from pixel space into latent space, where the video diffusion transformer operates.

  • The model uses a transformer-based latent diffusion architecture as the foundational text-to-video generator.
  • A 3D-VAE transforms videos from pixel space into latent space before transformer-based video diffusion processing.

2. Implementation Details of ConceptMaster

ConceptMaster combines a CLIP image encoder, Q-Former, DAM module, and transformer-based video generation components to process reference images and train video customization models.

  • Implementation Details of ConceptMaster: CLIP extracts visual features from reference images before the Q-Former and DAM modules process them.The Q-Former consists of stacked cross-attention and FFN layers, followed by an additional cross-attention layer for DAM.
  • Implementation Details of ConceptMaster: ConceptMaster injects visual conditioning through a DAM module while preserving the diffusion transformer’s video-generation backbone.The implementation uses proprietary transformer-based text-to-video diffusion models.
  • Implementation Details of ConceptMaster: Training jointly optimizes most transformer parameters while excluding the 3D spatiotemporal layer, Q-Former, and DAM module.The spatiotemporal self-attention layer is retained while other transformer parameters are fine-tuned to enhance video dynamics.
  • Implementation Details of ConceptMaster: Training batches use padded videos with effective attention masks, and each training video segment contains 77 frames.The padding strategy standardizes video height and width within each batch.

3. Discussions on Comparison between Our Data Collection Pipeline and Grounded-SAM

The data collection pipeline is designed to improve extraction of multiple concepts and their identities from videos, addressing weaknesses in Grounded-SAM-based collection.

  • Pipeline Motivation: Grounded-SAM is considered insufficient because Grounding-DINO often struggles to distinguish visually similar concepts.The limitation is attributed to Grounding-DINO’s reliance on the CLIP text encoder.
  • Pipeline Design: The proposed pipeline has two stages: fast elimination of unsuitable videos and fine-grained identity information extraction.These stages target both video filtering and detailed concept extraction.
  • Pipeline Evaluation: 52% is the success rate reported for Grounded-SAM on the evaluated samples, substantially below the proposed pipeline’s success rate.Success means that extracted concept information contains no errors; any error counts as failure.
  • Pipeline Evaluation: Manual evaluation by 20 experienced workers was used to assess the constructed evaluation samples from Panda-2M.The study randomly sampled 2000 videos from Panda-2M for success-rate analysis.

5. Comparison Methods Implementation

ConceptMaster is compared with two-stage combinations of multi-concept image customization and image-to-video generation, alongside analyses of its embedding-injection components.

  • Comparison Methods Implementation: ConceptMaster achieves better generation results and greater practicality than the compared two-stage solutions.The comparison combines open-sourced image customization methods with I2VGen-XL for naive MCVC baselines.
  • Multi-Concept Embeddings Injection: Previous whole-condition or aggregated-visual-embedding injection methods struggle with identity decoupling and concept fidelity, especially for similar concepts.The discussed alternatives include BLIP-Diffusion and IP-Adapter.
  • Multi-Concept Embeddings Injection: Standalone injection of multi-concept embeddings is presented as ConceptMaster’s key design choice for handling multiple concepts.The comparison covers three multi-concept embedding injection methods and includes additional quantitative results in Figure 10.
  • Q-Former and DAM Analysis: Removing DAM causes concepts to lose fidelity because text-label representation is insufficiently represented and differentiated.The passage also links DAM removal to poorer alignment with the original diffusion space.

8. More Qualitative Results Demonstration

Additional qualitative results show ConceptMaster generating concept-consistent customized videos across diverse multi-concept compositions, with supplementary comparisons and module visualizations.

  • More Qualitative Results Demonstration: ConceptMaster produces high-quality, concept-consistent videos from multiple reference images across six diverse concept-composition scenarios.The scenarios include combinations of persons, livings, and stuffs, including multiple entities of the same category.
  • More Qualitative Results Demonstration: Figure 9 compares ConceptMaster with naively combining multi-concept image customization and image-to-video generation models.This comparison evaluates the two-stage alternative against the end-to-end customization approach.
  • More Qualitative Results Demonstration: Figures 10 and 11 provide further qualitative comparisons of injection methods and the Q-Former and DAM modules.Figures 12 and 13 present additional qualitative results on diverse scenarios.
Loading 2501.04698v2…