Source-linked AI summary
ID-Animator: Zero-Shot Identity-Preserving Human Video Generation
Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Jie Zhang
TL;DR
Personalized human-video generation faces case-by-case fine-tuning costs, limited suitable datasets, and challenges preserving identity from reference images. ID-Animator combines a face adapter, an ID-oriented data pipeline, and random-reference training. The authors report stable videos with superior ID fidelity compared with previous models.
Problem
Personalized human-video generation faces costly case-by-case fine-tuning, a scarcity of suitable human-video data, and the challenge of reducing ID-irrelevant reference-image features.
Source: §1 · p. 2, §3.3 · p. 5Method
ID-Animator uses a lightweight face adapter with learnable facial latent queries, an ID-oriented caption-and-face-pool pipeline, and random-reference training with an ID-preserving objective.
Source: §1 · p. 3Results
The authors report that ID-Animator generates stable videos with superior ID fidelity compared with previous models.
Source: §5 · p. 14Takeaways & Limitations
The framework is reported to complete training within one day on a single A100 GPU and then perform zero-shot inference.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Generating high-fidelity human video with specified identities has attracted significant attention in the content generation community. However, existing techniques struggle to strike a balance between training efficiency and identity preservation, either requiring tedious case-by-case fine-tuning or usually missing identity details in the video generation process. In this study, we present \textbf{ID-Animator}, a zero-shot human-video generation approach that can perform personalized video generation given a single reference facial image without further training. ID-Animator inherits existing diffusion-based video generation backbones with a face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To facilitate the extraction of identity information in video generation, we introduce an ID-oriented dataset construction pipeline that incorporates unified human attributes and action captioning techniques from a constructed facial image pool. Based on this pipeline, a random reference training strategy is further devised to precisely capture the ID-relevant embeddings with an ID-preserving loss, thus improving the fidelity and generalization capacity of our model for ID-specific video generation. Extensive experiments demonstrate the superiority of ID-Animator to generate personalized human videos over previous models. Moreover, our method is highly compatible with popular pre-trained T2V models like animatediff and various community backbone models, showing high extendability in real-world applications for video generation where identity preservation is highly desired. Our codes and checkpoints are released at https://github.com/ID-Animator/ID-Animator.
1 Introduction
ID-Animator addresses the costs, data gaps, and identity loss that challenge personalized human-video generation. It combines a reference-face adapter with an ID-oriented data pipeline and random-reference training to generate identity-specific videos without per-identity fine-tuning.
1 Introduction
- Existing approaches face costly case-by-case fine-tuning, scarce human-video datasets, and harmful ID-irrelevant features in reference images.These issues constrain training efficiency, dataset quality, and identity preservation.Source: §1 · p. 2
- The authors report zero-shot ID-specific video generation and integration with existing community models, with robust generalization and identity preservation.The framework is also described as lightweight.Source: §1 · p. 3
- ID-Animator uses a lightweight face adapter and learnable facial latent queries to generate identity-specific videos from a reference face without further fine-tuning.The framework uses pre-trained video diffusion models and is described as a zero-shot approach.Source: §1 · pp. 2–3
- The ID-oriented data pipeline builds on publicly available sources, using unified captions for human attributes and actions and a facial image pool.The pipeline is intended to address the lack of suitable training data for personalized human-video generation.Source: §1 · pp. 2–3
- Randomly sampled reference faces and an ID-preserving objective are used to reduce ID-irrelevant content and focus the adapter on identity-relevant features.The authors state this improves identity fidelity and generalization.Source: §1 · p. 3
2 Related Work
Prior video-generation work spans image-to-video and text-to-video diffusion methods, while personalized generation includes both per-identity fine-tuning and training-free image approaches. Subject-driven video methods remain constrained by training-subject coverage or per-identity training.
2.1 Video Generation
- Video-generation methods include image-to-video and text-to-video approaches, with leading systems generally adapting pre-trained text-to-image models using temporal mixing layers.The passage names Animate Diffusion [12], Dynamicrafter [38], Modelscope [33], AnimateAnything [6], and Stable Video [1].Source: §2.1 · p. 3
2.2 ID-preserving Image Generation
- Personalized image generation includes methods requiring per-identity fine-tuning, such as DreamBooth [31], textual inversion [8], and LoRA [17], as well as training-free reference-image methods.Face0 [32], PhotoMaker [24], IP-Adapter [43], and InstantID [34] are described as training-free approaches using reference-image features in different ways.Source: §2.2 · pp. 3–4
2.3 Subject-driven Video Generation
- VideoBooth [22] requires subjects to appear in its training data, while MagicMe [28] requires separate training for each identity.The paper distinguishes ID-Animator as applicable to any human image without retraining during inference.Source: §2.3 · p. 4
3 Method
ID-Animator combines a dataset reconstruction pipeline, a face-adapter framework, and random-reference training to generate ID-specific human videos from a reference image. Its design separates identity cues from video content while retaining the pretrained video model's generative capacity.
3.1 Overview
- The method has three components: dataset reconstruction, the ID-Animator framework, and random-reference training.Together, they target high-fidelity ID-specific human video generation from a reference ID image.Source: §3.1 · p. 4, Figure 2 · p. 4
3.2 ID-Animator
- A lightweight face adapter adds image cross-attention to a compatible text-to-video backbone while leaving temporal attention unchanged.The adapter refines reference-image CLIP features with query-based encoding; its trainable projection weights are initialized from IP-Adapter.Source: §3.2 · p. 5
3.3 ID-Oriented Human Dataset Reconstruction
- The dataset pipeline rewrites captions to combine human attributes and actions, and constructs a pool of facial reference images.Attribute captions describe appearance and context; action captions capture video-wide motion, and a unified caption combines both. The face pool uses facial regions from five randomly selected frames, discarding frames with multiple faces.Source: §3.3 · pp. 5–6
3.4 Random Reference Training Methods For Diminishing ID-Irrelevant Features
- Random-reference training conditions each video on a weakly correlated face-pool image rather than the same image used as its reconstruction target.Random sampling is intended to reduce reliance on identity-invariant features such as background and viewing angle, while encouraging identity consistency across different angles and attire.Source: §3.4 · pp. 6–7
- An ID-preserving reward compares the generated face with the reference face, and the total objective balances this loss with diffusion loss.A face detector extracts the generated face region, a face encoder computes similarity, and λ balances the identity-preserving and diffusion losses.Source: §3.4 · p. 7
4 Experiment
Experiments use AnimateDiff as the video-generation backbone and train only the face adapter in a two-stage process.
4.1 Implementation details
- Training uses AnimateDiff, with only the face-adapter parameters updated while the pretrained text-to-video model remains frozen.The experiments use one NVIDIA A100 GPU (80GB), batch size 2, and a learning rate of 5e-5 for the trainable adapter.Source: §4.1 · p. 7
- The first stage trains on single video frames for one epoch using identity and diffusion losses with λ set to 1.Null-text embeddings replace the updated text embedding with 20% probability for classifier-free guidance.Source: §4.1 · p. 7
- The second stage continues training for eight epochs using diffusion loss alone on 16-frame videos.Training videos are clipped to 16 frames, center-cropped, and resized to 512x512 pixels.Source: §4.1 · p. 7
4.2 Dataset and Evaluation Metrics
Evaluation uses a 13k-video training set and an unseen-image test set, with metrics covering video quality, motion, and identity preservation.
4.2 Dataset and Evaluation Metrics
- After filtering videos containing multiple faces, the training set contains 13k videos from a 15k-video CelebV-text subset.The identity-oriented dataset is constructed on this filtered foundation.Source: §4.2 · p. 7
- Table 1 presents quantitative comparisons between state-of-the-art methods, marking best results in bold and second-best results with underlining.The caption gives the comparison's scope and visual key but does not provide metric values.Source: Table 1 · p. 8
4.3 Quantitative Comparison
ID-Animator outperforms prior methods across four reported video-generation metrics and also surpasses AnimateDiff on human-related prompts. The comparison highlights trade-offs in facial similarity, structure, and motion quality.
4.3 Quantitative Comparison
- ID-Animator surpasses state-of-the-art methods on CLIP-I, Dover Score, Motion Score, and Dynamic degree.The passage reports superiority across all four metrics but provides no numerical values.Source: §4.3 · p. 8
- Applying IP Adapter Face ID Portrait to video models improves facial feature similarity but reduces facial structural similarity and video-motion quality.The reported declines include Dover Score, Motion Score, and Dynamic degree.Source: §4.3 · p. 8
- ID-Animator also surpasses AnimateDiff on human-related prompts.Source: §4.3 · p. 8
- Figure 4 compares the methods on three ordinary individuals.Source: Figure 4 · p. 8
4.4 Qualitative Comparison
On three unseen ordinary individuals, ID-Animator better preserves reference identity than the compared methods. The qualitative examples also illustrate changes to appearance, context, actions, age, and gender.
4.4 Qualitative Comparison
- For three unseen individuals—men, women, and children—ID-Animator better preserves the reference identity than the compared methods.Source: §4.4 · p. 8
- IP-Adapter-Plus-Face produces some facial deformation, while IP-Adapter-FaceID-Portrait lacks facial structural information.The passage associates the latter shortcoming with diminished similarity to the reference.Source: §4.4 · p. 8
- Figure 5 shows recontextualization of hair, clothing, background, actions, age, and gender.Source: Figure 5 · p. 9
4.5 Applications
The applications examples show ID-Animator changing contextual attributes while preserving identity, mixing identities, using ControlNet conditions, and working with community-model weights.
4.5 Applications
- The applications include recontextualization, age or gender alteration, identity mixing, and integration with ControlNet or community models.Source: §4.5 · p. 9
4.5.1 Recontextualization
- Given a reference image, the model changes character features, hair, clothing, backgrounds, and actions through text-guided generation.Figure 5 examples also include changes to age and gender.Source: §4.5.1 · p. 9
4.5.2 Identity Mixing
- Blending embeddings from two identities in varying proportions combines features from both identities in the generated video.The experiment is presented as evidence of the face adapter's ability to learn facial representations.Source: §4.5.2 · pp. 9–10
4.5.3 Combination with ControlNet
- With SparseControlNet, single-frame controls fuse with the face reference, while multiple control images guide the generated sequence.Source: §4.5.3 · p. 10
4.5.4 Inference with Community Models
- ID-Animator works with Lyriel and Raemuxi community-model weights despite not having been trained on them, with reliable facial preservation and motion generation.Source: §4.5.4 · p. 10
4.6 Ablation Experiment
Ablations report declines when the ID-preserving objective, random reference training, or unified human-video captions are removed. The caption ablation particularly affects motion and video quality.
On the Effectiveness of ID Preserving Loss
- Removing the ID-preserving objective lowers facial feature similarity and CLIP-I, and degrades Motion Score and Dynamic degree.The authors interpret these declines as validating the objective's effectiveness.Source: §4.6 · p. 11
On the Effectiveness of Random Reference Training
- Replacing random reference training with a fixed first-frame face reference decreases video quality and CLIP-I.The authors attribute inferior results to confusion from irrelevant reference-image semantics and neglect of identity-relevant embeddings.Source: §4.6 · p. 11
On the Effectiveness of Unified Captioning on Human Videos
- Using original CelebV captions instead of unified captions decreases Motion and Dover scores and yields poorer video quality.The passage links the decline to limited caption diversity and difficulty generating motion-rich videos from human-style text.Source: §4.6 · p. 11
4.7 Prompt List for Evaluation
The section describes a 50-prompt evaluation protocol and visual evidence for recontextualization, identity mixing, and component ablations.
4.7 Prompt List for Evaluation
- Evaluation uses 50 GPT-4-rewritten prompts across four categories and 50 identities, yielding 2,500 videos per method.Table 2 presents prompts rewritten to align with human language style.Source: §4.7 · p. 12, Table 2 · p. 12
- ID-Animator manipulates facial attributes through text and mixes identities from multiple facial inputs in generated videos.The authors present these demonstrations as evidence of generalization and extendability.Source: §4.7 · p. 12, Figure 9 · p. 13
- IP-Adapter Plus Face produces excessively small, blurry faces, while IP-Adapter FaceID portrait lacks facial structural information.The reported visual comparison says both outputs diverge from a recognizable real-person appearance.Source: §4.7 · p. 12, Figure 10 · p. 13
- Visual ablations associate removing ID loss with lower facial similarity and removing random-reference training with nearly static outputs.The figure also compares the ablations with IP-Adapter variants.Source: §4.7 · p. 13, Figure 10 · p. 13
5 Conclusion
ID-Animator combines an ID-oriented dataset and random-reference training with an ID-preservation loss to generate ID-specific human videos. The authors report stable videos and superior identity fidelity compared with previous models.
5 Conclusion
- ID-Animator drives text-to-video models to generate ID-specific human videos from identity images, using a dataset built from public resources.The dataset pipeline incorporates unified caption generation and face-pool construction.Source: §5 · p. 13
- Random-reference training minimizes ID-irrelevant reference content, while an ID-preservation loss directs the adapter toward ID-related features.The training pipeline uses the constructed ID-oriented dataset.Source: §5 · p. 13
- The authors report stable generated videos with superior identity fidelity compared with previous models.The conclusion attributes this result to extensive experiments.Source: §5 · p. 14