Lynx: Towards High-Fidelity Personalized Video Generation

1 Introduction

• The field’s move from image generation to text-to-video and controllable video tasks has brought personalized video generation into focus as a way to preserve subject identity.Prior image-domain personalization uses fine-tuning or lightweight identity and reference-feature conditioning, while recent work extends identity preservation to video.Source: §1 · p. 2
• Two adapters condition a video foundation model on a single face image: ID-adapter converts facial embeddings into identity tokens, while Ref-adapter integrates VAE features through a frozen pathway.The ID tokens use a Perceiver Resampler; reference features are fused through cross-attention across the backbone’s DiT blocks.Source: §1 · p. 2
• Lynx outperforms recent personalized video methods in identity preservation while delivering stronger prompt alignment and superior overall video quality.The evaluation uses 40 subjects and 20 unbiased prompts, producing 800 test cases.Source: §1 · p. 2

Video Foundation Models

• Video foundation models shifted from U-Net-based latent diffusion with temporal modules toward DiT and MMDiT architectures for spatio-temporal modeling and temporal consistency.Wan2.1 and other named models are examples of current transformer-based video foundation models trained with large data and substantial computation.Source: §2 · pp. 2, 4

Identity-Preserving Content Creation

• Image-domain identity preservation ranges from subject-specific fine-tuning to tuning-free lightweight ID-injection modules, including IP-Adapter and InstantID.InstantID adds a ControlNet module for input decoupling and finer-grained control.Source: §2 · p. 4
• Video personalization methods use frequency decomposition, visual-text fusion, multimodal conditions, or concatenated reference inputs, while balancing identity resemblance and editability remains challenging.The cited methods include ConsistID, ConceptMaster [21], HunyuanCustom [20], SkyReels-A2 [11], VACE [22], and Phantom [31].Source: §2 · p. 4

3.1 Model Architecture

• Rather than restructuring or fine-tuning the full base model, Lynx adds ID-adapter and Ref-adapter modules to inject identity features.The base model is Wan2.1, whose DiT blocks apply spatio-temporal self-attention followed by text-conditioned cross-attention.Source: §3.1 · p. 4
• ID-adapter maps a 512-dimensional face feature vector into 16 tokens of dimension 5120, concatenates 16 register tokens, and cross-attends them with visual tokens.A Perceiver Resampler produces the identity-token sequence, which is then added back to the main branch.Source: §3.1 · p. 5
• Ref-adapter processes the reference image through a frozen base-model copy at zero noise with the prompt “image of a face,” then cross-attends its layerwise features into generation.This pathway captures reference spatial details across all layers.Source: §3.1 · p. 5

3.2.1 Spatio-Temporal Frame Pack

• Frame packing concatenates patchified images and videos into one sequence, with attention masks separating samples and 3D-RoPE applied independently per video.The approach targets flexible batching across heterogeneous resolutions and video durations.Source: §3.2.1 · p. 5

3.2.2 Progressive Training

• Training progresses from image pretraining to video training: pretrained Resampler initialization yields recognizable resemblance after 10k iterations, and the first stage runs for 40k iterations.Image-only training tends to produce static videos; a 60k-iteration video stage restores temporal dynamics while retaining identity conditioning.Source: §3.2.2 · pp. 5–6

4 Data Pipeline

• Reliable person–video pairing is the central data challenge; direct face crops can overfit expression and lighting, while multi-scene data are scarce.The pipeline draws on single images, single videos, and multi-scene image and video collections.Source: §4 · p. 6
• X-Nemo edits source faces to match target expressions, while LBM relights faces and replaces backgrounds to broaden expression and lighting variation.The two augmentation strategies are illustrated in Figure 4.Source: §4 · p. 6
• After augmentation, face-recognition verification filters low-resemblance pairs; the pipeline constructs 50.2M pairs and uses weighted sampling to balance pair types.The total comprises 21.5M single-scene, 7.7M multi-scene, and 21.0M augmented single-scene pairs.Source: §4 · p. 6

5.1 Benchmark and Metrics

• The benchmark covers 40 subjects and 20 unbiased prompts, producing 800 test videos.Subjects include celebrity photos, AI-synthesized portraits, and licensed photos spanning diverse demographic groups.Source: §5.1 · pp. 6–7
• Evaluation measures face resemblance with three independent recognizers and assesses prompt alignment, aesthetics, motion naturalness, and overall video quality with Gemini.Face resemblance uses cosine similarity; Gemini scores generated videos with task-specific prompts.Source: §5.1 · pp. 7–8

5.2 Qualitative Results

• Qualitative comparisons show Lynx maintaining identity, natural motion, coherent visual details, and scene integration where baselines exhibit identity drift, unrealistic actions, or copied backgrounds and lighting.Examples include poor resemblance, unrealistic behavior, and background or lighting copy-paste effects.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3 Quantitative Results

• Lynx leads all three identity evaluators and scores highest on three of four Gemini metrics, while VACE leads motion naturalness.The Gemini metrics where Lynx leads include prompt alignment, aesthetics, and overall video quality.Source: Table 1 · p. 8, Table 2 · p. 8, §5.3 · pp. 8–9

• Lynx combines ArcFace-derived identity tokens with dense VAE features from a frozen reference pathway to balance identity fidelity, motion naturalness, and visual coherence.The conclusion describes evaluation on 40 subjects and 20 unbiased prompts, totaling 800 test cases.Source: §6 · p. 9