MagicMirror: ID-Preserved Video Generation in Video Diffusion Transformers

Limitations of prior work

• Existing methods often produce limited motion, low-quality videos, temporal artifacts, or unstable long sequences while preserving identity.These limitations arise across inflated-UNet methods, DiT-based approaches, and two-stage image-to-video pipelines.Source: §1 · p. 1, Figure 2 · p. 2

Motivation and contribution

• MagicMirror is a single-stage framework designed to generate high-quality videos with strong identity consistency and dynamic natural facial motions.It aims to bridge personalized identity generation and high-quality video synthesis without case-specific fine-tuning.Source: §1 · pp. 1–2

Technical challenges

• The dual-branch facial extractor preserves high-level identity features and reference-specific structure for identity conditioning in Video DiT models.The design addresses the incompatibility between full-attention DiT architectures and conventional cross-attention conditioning.Source: §1 · p. 2
• Conditioned Adaptive Normalization incorporates identity conditions as a distribution prior, complemented by learnable cross-attention for attention and feature-distribution guidance.The module targets the spatial-fidelity trade-off introduced by video foundation models optimized for text-video alignment.Source: §1 · p. 2

Training strategy

• A synthetic data pipeline and progressive two-stage training first learn identity representations from images, then temporal consistency from video.The strategy addresses the scarcity of high-quality identity-consistent image-video pairs.Source: §1 · p. 2

Evaluation

• Extensive evaluations report high-quality videos with dynamic content and strong facial consistency on a human-centric test set without case-specific fine-tuning.The evaluation compares MagicMirror with competitive identity-preserving video-generation methods using general metrics and visual assessments.Source: §1 · p. 2

Video Generation via Diffusion Models

• Diffusion Transformers such as DiT, Latte, SORA, Open-Sora, and CogVideoX have advanced video generation, but identity-preserving video generation remains relatively unexplored.Current research has focused primarily on image-to-video efficiency and motion control.Source: §2 · p. 3

ID-Preserving Generation

• Identity-preserving image generation includes tuning-based methods that fine-tune models from one or more reference images.Textual Inversion [14] and DreamBooth [50] are cited examples.Source: §2 · p. 3
• Tuning-free image methods use additional conditioning and trainable parameters to reduce computational overhead while maintaining identity fidelity.IP-Adapter [65], InstantID, and PhotoMaker [5] [19] [24] [34] [44] [56] are cited examples.Source: §2 · p. 3

ID-Preserving Video Generation

• Identity-preserving video generation must synthesize realistic facial movements from static references while maintaining identity consistency.This adds temporal and motion challenges beyond identity-preserving image generation.Source: §2 · p. 3
• Magic-Me [39] requires per-identity optimization, whereas ID-Animator [20] is tuning-free but remains constrained by base-model video quality, duration, and prompt adherence.ConsisID [69] uses a DiT architecture but faces limitations in video smoothness and facial dynamics.Source: §2 · p. 3

3. MagicMirror

• MagicMirror extracts identity and structural facial conditions from reference images and injects them into a DiT-based video generator.The identity branch supports consistency across frames, while the structural branch captures facial detail and natural motion.Source: §3.2 · pp. 3–4, §3 · p. 3

3.2. Dual-Branch Facial Feature Extraction

• The identity branch uses dense CLIP ViT features, ArcFace-derived identity queries, and a structural perceiver to produce complementary facial embeddings.Identity embeddings are fused into text features at identity-relevant tokens, while structural embeddings directly condition generation.Source: §3.2 · pp. 3–4

3.3. Conditioned Adaptive Normalization

• The lightweight adapter adds facial modality normalization to mm-DiT full attention, using CAN to predict identity-conditioned distribution shifts for text and video features.Facial features receive dedicated adaptive normalization, while CAN modulates text and video distributions through a trainable conditional MLP.Source: §3.3 · pp. 4–5
• The adapter combines joint self-attention with reinitialized trainable cross-attention key-value projections to aggregate identity features.Both attention paths use the complete aggregated representation as their query input.Source: §3.3 · p. 5

3.4. Data and Training

• MagicMirror uses two-stage training: synthetic identity-paired image pre-training followed by video post-training on Pexels, Mixkit, and web videos.The image pipeline combines LAION-Face, SFHQ, and FFHQ-derived synthetic pairs to improve identity diversity and facial-motion diversity.Source: §3.4 · p. 5
• The training objective combines general denoising with an identity-aware cosine loss, applying face-region denoising to 50% of random samples.The identity term compares facial query features with decoded denoised latents and is weighted by λ.Source: §3.4 · p. 5

4.1. Implementation Details

• The training pipeline uses self-referenced and filtered synthetic identity pairs, followed by video-data training and caption-based evaluation against ID-preserved and I2V baselines.Synthetic pairs are filtered using facial-embedding similarity, and the evaluation includes 50 identities with varied prompts.Source: §4.1 · pp. 5–6, Figure 5 · p. 6

4.2. Quantitative Evaluation

• MagicMirror achieves superior facial similarity to I2V approaches while remaining competitive with ID-Animator and ConsisID across text alignment, video quality, and dynamic performance.The evaluation uses VBench and EvalCrafter metrics, optical-flow smoothness, Average Similarity, and facial-motion measures.Source: §4.2 · p. 6, Table 1 · p. 6
• MagicMirror and ConsisID maintain identity preservation throughout videos, unlike standard I2V models that exhibit significant temporal similarity decay.DynamiCrafter shows improved stability through random reference selection, but the passage distinguishes it from the consistently preserving methods.Source: §4.2 · pp. 6–7

4.3. Qualitative Evaluation

• Qualitative comparisons show stronger cross-frame identity consistency, dynamic range, text alignment, and natural motion than conventional CogVideoX, ID-Animator, and ConsisID.The comparisons are presented with captions and reference identity images for each case.Source: §4.3 · p. 7, Figure 6 · p. 7
• In a 173-participant user study, MagicMirror consistently outperforms baseline methods across most evaluated dimensions, while human judgments reveal a gap from ConsisID’s designed video-quality and identity metrics.Participants rated motion dynamics, text-motion alignment, video quality, and identity consistency on a 1–10 scale.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4. Ablation Studies

• The dual-branch facial embedding improves motion over single-branch variants, while removing CAN reduces performance from 0.911 to 0.886.Single-branch variants achieve 0.865–0.869 identity similarity, and CAN is described as important for distribution alignment and facial-condition injection.Source: §4.4 · p. 8, Table 3 · p. 8
• The two-stage strategy combines image pre-training for robust identity preservation with video post-training for temporal consistency and dynamic facial motion.The implementation uses 30K image-pretraining iterations followed by 5K video-finetuning iterations.Source: §4.1 · pp. 5–6, §4.4 · p. 8

5. Conclusion

• MagicMirror is a zero-shot framework that combines dual facial embeddings and CAN within DiT architectures for identity-preserving video generation.The conclusion reports robust identity preservation from a single reference image and strong performance across benchmarks and human evaluations.Source: §5 · p. 8
• The principal limitation is incomplete support for multiple identities and fine-grained attributes beyond facial features, including clothing.The authors identify these capabilities as necessary for practical customized video applications.Source: §5 · p. 8

Appendix

• The supplementary materials include additional ablations, data-processing details, architectural specifications, implementation information, and qualitative results.Figure 8 documents filtering criteria, prompt examples, processing specifications, and data-exclusion rules.Source: Appendix · p. 9, Figure 8 · p. 10

A.1. Training Data Preparation

• The training corpus is built from filtered LAION-Face data, SFHQ, FFHQ-based synthetic identity pairs, and Pexels-400K and Mixkit video data.The appendix describes face detection, captioning, motion filtering, PhotoMakerV2 synthesis, and quality filtering in the preparation pipeline.Source: §A.1 · p. 9

A.2. Test Data Preparation

• The evaluation set contains 50 identities across seven demographic categories, with one to four reference images and four prompts assigned per identity.Prompts focus on human-centric actions derived from VBench, with custom prompts added for the baby category.Source: §A.2 · p. 9

A.3. Comparisons

• Comparisons use enhanced prompts for ID-Animator and ConsisID, while I2V baselines receive PhotoMakerV2 reference images and, in the main protocol, concise video-generation prompts.The appendix states that long prompts can bias I2V models toward text alignment over reference-image fidelity, degrading identity preservation and video quality.Source: §A.3 · pp. 9–11

• Face-specific evaluation measures identity preservation through facial embedding similarity and temporal stability through frame-wise similarity decay.Source: §A.4 · p. 11
• The proposed facial dynamics metric uses five RetinaFace landmarks to assess motion relative to the reference and maximized inter-frame motion.FMref uses aspect-ratio-normalized frames to reduce positional bias, while FMinter preserves translational movements from original frames.Source: §A.4 · p. 11, Figure 10 · p. 11
• Success-rate analysis evaluates face recognition, identity checks, motion, and text alignment across 200 videos generated from 50 prompts and four seeds.Source: §A.4 · p. 12, Table 5 · p. 12

A.5. Implementation Details

• The implementation combines a two-token PhotoMakerV2 identity embedding with 32 facial-structure query tokens, then projects both modalities before diffusion input.Source: §A.5 · p. 12
• CAN predicts facial modulation factors from projected identity embeddings, time embeddings, and video shifts using MLPs initialized to zero for stable training.Source: §A.5 · p. 12
• Residual modulation prediction avoids the abnormal video quality produced by directly predicting CAN data distributions.Source: §A.5 · p. 12

B.1. Advantages of CAN

• CAN accelerates image-stage convergence and improves identity-information capture, including simple appearance attributes such as hairstyle.Source: §B.1 · p. 12, Figure 12 · p. 12
• t-SNE analysis shows modality-specific modulation distributions across Transformer layers, with face-conditioned shifts away from baseline distributions.Source: Figure 13 · p. 13
• MagicMirror adds minimal computational overhead, requiring only slight increases in GPU memory and inference time relative to ConsisID [69] and CogVideoX [64].The comparison measures computation for generating a 49-frame 480P video on one A800 GPU.Source: §B.1 · pp. 12–13, Table 6 · p. 13

• Face-conditioned residual modulation shifts baseline distributions in modality-aware ways, and these shifts empirically accelerate identity-conditioned convergence.Source: §B.2 · p. 13
• Fine-tuning only normalization layers on CelebV-Text [66] versus the Pexels collection substantially changes generated-video spatial fidelity.The result supports aligning modality distributions during training and validates the quality of the curated video dataset.Source: §B.2 · p. 13, Figure 14 · p. 13
• 0.71 to 0.84: VBench [25] dynamic motion score after using twice the frame rate and training only modulation layers on the Pexels dataset.Source: §B.2 · p. 13

• Combining image pre-training with video training produces dynamic, high-fidelity ID-preserving videos without the color-shift artifacts seen after image-only training.Image pre-training prioritizes facial consistency, whereas the combined stages align identity representation and color distribution.Source: §B.3 · p. 13

B.4. Limitation Analysis

• MagicMirror struggles with multi-person scenarios, fine-grained feature preservation, and temporal consistency in generated videos.Reported failures include incomplete transfer of facial details or accessories and motion artifacts caused by the base model.Source: §B.4 · pp. 13–14
• Figure 16 highlights incomplete reference-detail transfer and temporal motion artifacts as representative failure cases.Source: Figure 16 · p. 14

C.1. Additional Applications

• MagicMirror generates identity-preserved outputs across artistic styles and visual effects.This extends the framework beyond realistic customized video generation by leveraging CogVideoX’s generative capabilities.Source: §C.1 · p. 14
• The framework can generate temporally consistent multi-shot sequences with coherent characters and style descriptions.Source: §C.1 · p. 14

C.2. Image Generation Results

• The image-pretrained stage achieves superior facial identity fidelity compared with video-finetuned variants.The passage attributes this difference primarily to video-training constraints such as limited batch sizes and dataset scope.Source: §C.2 · pp. 14–15

C.3. Video Generation Results

• MagicMirror’s one-stage approach handles occluded initial frames, dynamic range, and facial-rotation temporal consistency better than I2V.Source: §C.3 · p. 15
• Additional video results compare MagicMirror across human-face scales and highlight advantages in video generation.Source: §C.3 · p. 15