PaperlayerView exact revision on arXiv

Source-linked AI summary

Identity-Preserving Text-to-Video Generation by Frequency Decomposition

Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyuan Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, Li Yuan

arXiv:2411.17440v3cs.CVcs.MM

TL;DR

Identity-preserving text-to-video generation lacks a broadly applicable tuning-free solution, especially for DiT-based models. ConsisID decomposes facial identity into low- and high-frequency signals and injects them at frequency-appropriate locations with hierarchical training. The resulting model generates high-quality, editable, identity-preserving videos and outperforms existing identity-preserving T2V models, although common metrics distinguish methods only weakly.

  • Problem

    Existing IPT2V methods face high case-by-case finetuning overhead, while available tuning-free approaches have limited generation scope and poor identity preservation.

  • Method

    ConsisID uses global and local facial extractors to inject low- and high-frequency identity signals into specific DiT locations, supported by hierarchical consistency training.

  • Results

    ConsisID generates high-quality, editable, identity-preserving videos and outperforms current state-of-the-art identity-preserving T2V models.

  • Takeaways & Limitations

    Frequency-aware control provides a tuning-free solution for identity-preserving generation with DiT-based video models.

  • Takeaways & Limitations

    Common metrics such as CLIPScore and FID show minimal differences from previous methods and may not accurately assess identity-preservation capabilities.

Abstract

from arXiv · show

Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity. It is an important task in video generation but remains an open problem for generative models. This paper pushes the technical frontier of IPT2V in two directions that have not been resolved in literature: (1) A tuning-free pipeline without tedious case-by-case finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V model to keep human identity consistent in the generated video. Inspired by prior findings in frequency analysis of diffusion transformers, it employs identity-control signals in the frequency domain, where facial features can be decomposed into low-frequency global features and high-frequency intrinsic features. First, from a low-frequency perspective, we introduce a global facial extractor, which encodes reference images and facial key points into a latent space, generating features enriched with low-frequency information. These features are then integrated into shallow layers of the network to alleviate training challenges associated with DiT. Second, from a high-frequency perspective, we design a local facial extractor to capture high-frequency details and inject them into transformer blocks, enhancing the model's ability to preserve fine-grained features. We propose a hierarchical training strategy to leverage frequency information for identity preservation, transforming a vanilla pre-trained video generation model into an IPT2V model. Extensive experiments demonstrate that our frequency-aware heuristic scheme provides an optimal control solution for DiT-based models. Thanks to this scheme, our ConsisID generates high-quality, identity-preserving videos, making strides towards more effective IPT2V. Code: https://github.com/PKU-YuanGroup/ConsisID.

1. Introduction

ConsisID addresses tuning overhead and DiT-specific limitations in identity-preserving text-to-video generation by decomposing facial identity into low- and high-frequency signals. Its frequency-aware control scheme and hierarchical training strategy target consistent, editable, high-quality videos.

  • Existing IPT2V methods often require case-by-case finetuning, while the tuning-free ID-Animator is limited to talking-head videos and has poor identity preservation.
  • DiT-based video models are harder to train and weaker at perceiving facial details than U-Net-based approaches.
  • ConsisID decomposes identity into low-frequency facial shape and high-frequency identity markers to compensate for DiT limitations.
  • The method concatenates low-frequency reference and key-point features with noise latents, injects high-frequency features into transformer vision tokens, and uses hierarchical training.
  • ConsisID is a tuning-free DiT-based IPT2V model designed to preserve identity while generating high-quality, editable videos.

2. Related Work

Prior identity-preserving generation methods include tuning-based approaches that adapt models for each new identity and tuning-free approaches that inject identity features without inference-time finetuning. ConsisID extends this direction to tuning-free DiT-based video generation.

  • Tuning-based identity-preserving methods finetune pretrained models for each new person during inference.
  • ConsisID’s overview places low-frequency facial information in shallow layers and high-frequency information in attention-block vision tokens.
  • Tuning-free methods avoid finetuning parameters for newly introduced identities during inference by injecting identity features into pretrained models.

3. Methodology

ConsisID combines latent diffusion with frequency-specific identity control in a DiT-based video generator. Global facial features support low-frequency processing, local facial features supply high-frequency identity detail, and consistency training improves adaptation and generalization.

  • 3.1. Preliminaries: ConsisID uses latent diffusion, replacing pixel-space inputs with encoded video latents while conditioning denoising on text and identity signals.
  • 3.2. Identity Signal Injection: The method injects low-frequency facial information into shallow network layers and high-frequency information into attention-block vision tokens.
  • 3.2.1. Low-frequency View: Global Facial Extractor: The global facial extractor converts reference images and facial key points into low-frequency features used with the input noise latent.
  • 3.2.2. High-frequency View: Local Facial Extractor: The local facial extractor combines facial-recognition and CLIP features through a Q-Former to capture intrinsic identity and semantic information.
  • 3.2.3. Consistency Training Strategy: Consistency training uses randomly selected facial references, coarse-to-fine learning, facial-mask supervision, and cross-face references with noise to improve generalization.

4. Experiments

Experiments evaluate ConsisID against tuning-free and other video-generation baselines, then analyze its components and frequency-aware signal injection. The results support strong identity preservation, prompt-aligned editing, and the value of combining frequency signals at suitable DiT locations.

  • Qualitative Analysis: ConsisID preserves reference identity while generating realistic videos that follow prompts for actions, backgrounds, and other content.Qualitative cases show distortions or reduced quality when low-frequency information is missing, whereas ConsisID remains aligned with the reference identity and prompt.
  • Quantitative Analysis: ConsisID outperforms state-of-the-art methods across five metrics in quantitative evaluation.The comparison reports higher identity-preservation and text-relevance performance, while FID is treated only as a reference because of limited alignment with human perception.
  • User Study: 103 valid questionnaires show significant human-evaluation superiority over ID-Animator across identity preservation, visual quality, text alignment, and motion amplitude.Participants separately judged which method performed better on each of the four dimensions.
  • Identity Signal Injection: Only the configuration combining high- and low-frequency face signals at a suitable DiT location performs best.Injecting only high-frequency signals increases training difficulty and can prevent convergence, while low-frequency-only control limits expression and motion control.
  • Ablation Studies: Removing LFE, GFE, CFT, DML, or DCL significantly reduces identity preservation, text relevance, or visual quality.The ablations associate CFT with avoiding competing frequency objectives, DML with reducing background interference, and DCL with generalization to unseen faces.
  • Frequency Analysis: Fourier-spectrum analysis verifies that the proposed decomposition separates low- and high-frequency identity information in generated videos.The spectrum center represents low frequencies, surrounding regions represent high frequencies, and larger relative log amplitudes indicate more frequency information.

5. Conclusion

The paper presents ConsisID as a unified frequency-decomposition framework for consistent faces in DiT-based video generation. Extensive experiments report high-quality, editable, identity-preserving videos and performance beyond existing identity-preserving text-to-video methods.

  • Conclusion: ConsisID integrates frequency-decomposed identity control into DiT-based text-to-video models for consistent facial identity.The framework is described as generating high-quality, editable videos while preserving identity.
  • Conclusion: Extensive experiments show ConsisID outperforms current state-of-the-art identity-preserving text-to-video models.The paper characterizes its frequency-aware heuristic DiT control scheme as an effective solution for IPT2V generation.

1. ConsisID Dataset

The ConsisID dataset pipeline constructs identity-preserving human-centric video data with diverse content beyond cropped face regions. It filters, tracks, segments, and captions people across video clips for training.

  • Dataset Motivation: The dataset targets human-centric videos with broader content than face-focused identity-preserving datasets.The paper describes the dataset as more flexible and diverse, retaining relevant actions, backgrounds, and other video content.
  • Data Curation: Video clips are filtered using face, head, and person bounding boxes together with facial keypoints.YOLO-Box extracts category-specific boxes, while YOLO-Pose detects facial keypoints for selecting suitable clips.
  • Identity Verification: Tracking assigns unique identifiers to individuals so each person can be processed consistently through the video.The pipeline uses frame-by-frame bounding-box information to determine the number of individuals and assign identifiers.
  • Dataset Statistics: The dataset contains diverse categories, durations, and caption lengths, with most videos recorded at 1080P resolution.These statistics are summarized in the dataset overview figure.
  • Segmentation and Captioning: SAM2 generates and refines masks for each person, while time-aware annotation produces captions for the clips.Masks cover face, head, and person categories, and Qwen2-VL-72B is used to create high-quality video captions.

2. Additional Experimental Results

ConsisID is evaluated against closed-source, tuning-based, and image-to-video baselines, with results emphasizing identity preservation, visual quality, and scalability.

  • Comparison with Closed-source Method: ConsisID consistently outperforms Vidu 1.5 across all four automatic metrics and produces fewer artifacts with stronger identity preservation.Vidu 1.5 retains some high-frequency facial characteristics but fails to maintain intrinsic identity features consistently.
  • Comparison with Tuning-based Methods: ConsisID outperforms tuning-based methods despite requiring shorter inference time on a single Nvidia H100.The compared methods tune parameters for each new identity before inference.
  • Comparison with I2V Methods: End-to-end ConsisID preserves identity throughout videos more consistently than the evaluated image-to-video foundation models.OpenSora-Plan achieves higher fidelity partly because of lower motion amplitude, but does not align with the real video.

2.4. Fine-grained Ablation Study

The ablations show that semantic, facial, masking, and generalization components each affect distinct aspects of ConsisID, while inference quality varies non-linearly with step count.

  • Ablation Components: Removing CLIP reduces semantic information for video editing, while removing FaceExtractor lowers facial consistency.The effects are reflected by declines in CLIPScore and FaceSim, respectively.
  • Ablation Components: Noise ζ and Dynamic Cross Loss improve generalization, but excessive Dynamic Cross Loss may prevent convergence.The ablation evaluates these components on a subset of the ConsisID-Dataset.
  • Ablation Components: Dynamic Mask Loss improves identity consistency by focusing on facial features, but excessive reliance can reduce background generation.The complete model integrating all components yields optimal performance.
  • Inference Steps: Quality peaks at t = 50 and then declines as inference steps increase.Beyond t = 125, low-frequency facial information is lost; beyond t = 150, visual clarity progressively deteriorates.
  • Inference Steps: The inference-step trend is attributed to early denoising stages emphasizing low-frequency structure and later stages emphasizing high-frequency details.The observed behavior is non-linear rather than a continuous quality improvement with more steps.

2.6. Increasing Inference Speed

ConsisID can be accelerated with limited quality degradation and post-processed to produce smoother videos, while also demonstrating zero-shot generalization to stylized content.

  • Inference Acceleration: ConsisID integrates with training-free inference acceleration strategies with minimal visual-quality degradation.TeaCache provides a speed-quality trade-off controlled by rel_l1_thresh.
  • Inference Acceleration: ConsisID requires about 44 GB of GPU memory to decode 49 frames at 720x480, compared with 42 GB for the baseline.The inference time is almost identical to the baseline despite an additional 0.5B parameters.
  • Higher FPS Videos: At 8 fps, ConsisID generates coherent videos, but frame interpolation is used to reach 16 fps because 8 fps is below the stated smoothness threshold.The interpolated videos retain high clarity, indicating sufficient coherence in the original frames.
  • Style Transfer Applications: ConsisID preserves animated-character identity in stylized prompts without additional training.The paper presents this as zero-shot generalization beyond realistic customized videos.
  • Generalization: ConsisID controls expressions, attire, actions, age, backgrounds, and camera angles while matching faces to reference identities.The results cover different people and text prompts.

3. Additional Experimental Details

Additional experiments clarify the selected DiT injection scheme, evaluation caveats, implementation setting, and comparison-model configurations.

  • Identity Signal Injection: ConsisID injects high-frequency information between Attention and FFN modules while placing low-frequency signals and facial key points in shallow layers.This scheme is reported to achieve the optimal result among the visualized alternatives.
  • Metric Validation: Automatic metrics align only partially with human perception, and FID can rank a lower-quality model above better outputs.The authors therefore advise cautious interpretation of quantitative results.
  • Implementation Details: Experiments use Nvidia H100 and A100 GPUs with official implementations based on PyTorch.The reported hardware includes 40 H100 GPUs and 8 A100 GPUs.
  • Compared Models: Vidu 1.5 is the only closed-source model described as supporting tuning-free IPT2V, while ID-Animator is the sole open-source model described as supporting it.The comparison uses each model’s official settings where available.
  • Compared Models: The study compares tuning-based and image-to-video models using specified reference-image, prompt, resolution, and training configurations.These settings differ across DreamVideo, Magic-Me, and the foundational image-to-video models.

4. Additional Statement

ConsisID’s supporting findings indicate that frequency decomposition improves DiT-based identity preservation, while the model generalizes to image generation and realistic video synthesis. The paper also identifies evaluation and misuse concerns that constrain interpretation and deployment.

  • Supporting Findings: Low-frequency features stabilize diffusion-transformer training, while high-frequency features improve facial-detail handling when the signals are decoupled.The paper links omitted low-frequency features to training instability and reports improvements from separating high- and low-frequency signals.
  • Generalization: ConsisID can generate high-quality identity-preserving images despite being trained exclusively on video data.This is achieved by setting the frame parameter to 1 or extracting a video’s first frame as an image.
  • Ethics: The technique may be misused to produce deceptive content for fraudulent activities.This is identified as a potential negative impact of generating realistic human videos.
  • Limitations: Existing CLIPScore and FID measurements show minimal differences between identity-preservation methods, limiting their ability to reflect human-perceived quality.The paper calls for metrics better aligned with human perception.
  • Qualitative Results: ConsisID generates realistic human videos while preserving input identity and allowing text-based control.The reported showcases emphasize realistic outputs, identity matching, and control through text prompts.
Loading 2411.17440v3…