PaperlayerView exact revision on arXiv

Source-linked AI summary

Expression-Diverse References for Identity-Preserving Video Generation

Tianwen Fu, Wenbin Teng, Gonglin Chen, Junyi Ouyang, Haolin Xiong, Yajie Zhao

arXiv:2610.11023v1cs.CV

TL;DR

A single portrait captures only one facial configuration, leaving identity under unseen expressions underdetermined and complicating evaluation. This paper uses expression-diverse galleries for identity evaluation and video generation, and reports higher identity similarity than evaluated baselines across expression-intensity regimes, with the largest improvements for extreme expressions.

  • Problem

    A single portrait captures only one facial configuration, while expression changes make identity similarity to a neutral reference harder to interpret in generation and evaluation.

    Source: §1 · p. 1

  • Method

    The paper pairs an expression-stratified, gallery-based identity evaluation protocol with Stand-In conditioning on real or synthesized expression-diverse references.

    Source: §1 · p. 2

  • Results

    Under extreme expressions, CurricularFace similarity is 74.80 with real galleries and 68.36 with synthetic galleries, versus 60.95 for the strongest baseline; both galleries outperform all evaluated baselines across all three regimes.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The findings highlight the importance of expression diversity in identity-preserving video generation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Identity-preserving video generation aims to maintain a subject's identity while synthesizing realistic videos. Yet a single reference portrait captures the subject's appearance under only one facial configuration. As expressions change, facial appearance can vary in highly identity-specific ways, leaving the subject's appearance under unseen expressions underdetermined by the reference alone. This expression-dependent variation also complicates evaluation: similarity to a neutral reference may decrease under strong expressions even for real images of the same person. We investigate this limitation from both generation and evaluation perspectives. First, we quantify how face-recognition similarity varies with expression intensity using controlled photographs and MEAD videos. We then construct a compact yet expressive reference gallery that captures diverse expression-dependent facial configurations. Matching against this gallery provides a more robust measure of identity similarity under expressive motion. To further expose performance degradation with expression intensity, we report identity similarity separately for mild, intense, and extreme expressions. For generation, we extend Stand-In to condition on our expression-diverse reference sets and develop a data-curation pipeline that extracts consistent yet diverse face crops from training videos. In practical settings where only a single portrait is available, we construct the reference set by synthesizing additional expressions with a pretrained facial reenactment model. On our controlled benchmark, both real and synthesized reference sets outperform the evaluated baselines in identity similarity across all three expression-intensity regimes, with the largest improvements for extreme expressions.

1 INTRODUCTION

A single portrait underdetermines how a person’s face changes with expression, weakening identity-preserving generation and neutral-reference evaluation. Expression-diverse galleries provide a shared basis for conditioning and evaluation.

1 INTRODUCTION

  • Across 70 non-neutral photographs of 10 subjects, CurricularFace similarity to a neutral reference decreased with expression intensity (Spearman ρ = −0.701), with a similar trend on MEAD.This shows that neutral-reference similarity can mistake expression variation for identity error, including in real images.Source: §1 · pp. 1–2
  • For evaluation, the authors use a standardized gallery of eight facial configurations per subject and score each generated face by its maximum embedding cosine similarity to that gallery.They also report identity similarity and the fraction of generated frames separately for mild, intense, and extreme expressions.Source: §1 · p. 2
  • For generation, the authors extend Stand-In (Xue et al., 2026) to condition on multiple references and curate consistent, diverse face crops from OpenHumanVid (Li et al., 2024).The model is a 315M-parameter LoRA adapter trained for 25k steps at batch size 16, with the number of conditioning references varied during training.Source: §1 · p. 2
  • When only a neutral portrait is available, LivePortrait (Guo et al., 2024) synthesizes expressive references using driving expressions from a subject outside the evaluation set.Synthetic galleries score below real galleries but still exceed all evaluated baselines; both gallery types outperform baselines in every expression regime.Source: §1 · pp. 2–3
  • 74.80 (real) and 68.36 (synthetic) vs. 60.95 for the strongest baseline on extreme-expression CurricularFace similarity; both galleries beat all baselines across three regimes.Across both recognizers and all evaluated baselines, the real-gallery model wins on at least 97.9% of matched subject–prompt pairs, and at least 96.1% when its peak expression intensity is greater.Source: §1 · p. 2

2 RELATED WORK

Prior work offers strong video-generation backbones and reference-conditioned identity methods, but existing identity evaluation typically uses a single reference, while facial reenactment is a distinct, explicitly driven task.

Video Generation Models

  • General-purpose video models provide strong visual priors, but text prompts lack precise identity control and image-to-video generation generally requires a complete initial frame.Examples include CogVideoX (Yang et al., 2024), Wan2.2 (Wan Team, 2025), LTX-2 (HaCohen et al., 2026), and MiniMax H3.Source: §2 · p. 3

Identity-Preserving Video Generation

  • Identity-preserving generators commonly condition on one reference image, using different feature-injection strategies.ConsisID (Yuan et al., 2025b) separates global and local face features; Concat-ID (Zhong et al., 2025) concatenates reference features with video latents; Stand-In (Xue et al., 2026) uses a conditional image branch; Phantom (Liu et al., 2025) learns joint text–image injection.Source: §2 · p. 3

Benchmarking Video Generation Models

  • VBench measures temporal subject consistency, while OpenS2V-Eval FaceSim compares detected faces with a single input face and can penalize genuine expression changes.The paper’s benchmark instead matches frames to the nearest member of a standardized expression gallery and reports results by expression intensity.Source: §2 · p. 3

Facial Reenactment and Related Tasks

  • Facial reenactment animates a portrait using a driving video’s pose and expressions, unlike the paper’s text-driven generation setting.LivePortrait (Guo et al., 2024) warps source-image features using motion from implicit 3D keypoints; GaussianAvatars (Qian et al., 2024) rigs optimized 3D Gaussians, while OmniFace transfers identity to an existing target video.Source: §2 · p. 3

3 EVALUATING IDENTITY FIDELITY AT HIGH EXPRESSION INTENSITY

Face-recognition similarity falls as expression intensity rises, making a single neutral reference less reliable for evaluating expressive faces. The section introduces a diverse reference gallery and nearest-gallery scoring to assess identity fidelity across expression intensities.

3 EVALUATING IDENTITY FIDELITY AT HIGH EXPRESSION INTENSITY

  • The study collects frontal, flat-lit portraits of 10 subjects across eight expressions, including neutral, to form an expression-diverse gallery.Each non-neutral image is compared with its subject’s neutral image using ArcFace (Deng et al., 2019) and CurricularFace (Huang et al., 2020); expression intensity is estimated with SMIRK (Retsinas et al., 2024).Source: §3 · p. 3, Figure 2 · p. 4
  • ArcFace and CurricularFace similarity decrease as expression intensity increases on both the collected photographs and MEAD videos.The section reports Pearson’s r and Spearman’s ρ to quantify this trend; supplied passages do not include their values.Source: §3 · p. 4, Figure 3 · p. 4
  • Nearest-gallery identity similarity is the maximum recognizer similarity between a generated frame and any of the subject’s eight reference expressions.This is intended to provide a more robust identity-fidelity measure at high expression intensity.Source: §3 · p. 4
  • The evaluation reports nearest-gallery similarity in mild (∆Ψ < 4.5), intense (4.5 ≤ ∆Ψ < 8.5), and extreme (∆Ψ ≥ 8.5) expression regimes.The thresholds approximately match the first and third quartiles of intensity among 70 deliberately expressive light-stage photographs, leaving similarly sized mild and extreme tails and most photographs in the intense regime.Source: §3 · p. 4, Figure 4 · p. 5
  • The evaluation uses 48 prompts per subject, combining six basic emotions, four environments, and two triggers to elicit expressions unlike the reference configurations.The distinct prompted expressions are intended to discourage direct copying.Source: §3 · p. 4

4 TRAINING MULTI-REFERENCE SUBJECT-TO-VIDEO MODELS

This section diagnoses identity-similarity failures under expressive motion and describes a multi-reference training pipeline, including how to construct a gallery when only one portrait is available.

Data Curation

  • Training data curation samples 10 frames per video, filters and clusters face crops, then selects up to eight diverse crops from the dominant identity cluster.Videos with more than two clusters or over 40% of crops outside the dominant cluster are discarded; selection starts from the highest-quality crop and greedily adds the farthest remaining crop.Source: §4 · p. 5

Failure Mode of Existing Methods

  • Single-reference models lose identity similarity at high expression intensity, while AnyID (Wang et al., 2026) is more stable but remains low across all regimes.Figure 5 shows representative facial artifacts produced by existing methods under expressive motion. The evaluation compares the single-reference baselines and AnyID using the neutral portrait and expression references, respectively.Source: §4 · p. 5, Figure 5 · p. 5, Figure 6 · p. 6

Architecture and Training

  • The method extends Stand-In (Xue et al., 2026) to multiple references while freezing the Wan2.2-T2V-A14B backbone and adapting self-attention for reference-image tokens.Each reference is independently encoded at 512 × 512 resolution; reference tokens attend only within their own image, while video tokens attend to all tokens.Source: §4 · p. 6
  • Training uses 400k unique clips and shuffles references, randomly varying the number provided to improve robustness to different reference counts.At each step, all available references are used with probability 0.5; otherwise, a uniformly sampled subset size from one to N is selected. Effective batch size is 16.Source: §4 · p. 6

Inference Pipeline with a Single Reference

  • With one available portrait, the inference pipeline uses LivePortrait (Guo et al., 2024) to synthesize seven expressive references and form an eight-image gallery.The synthesized expressions are driven by expressions from a held-out subject.Source: §4 · p. 6

5 RESULTS

Ours (Real) and Ours (Synthetic) outperform all baselines across expression-intensity regimes, with especially pronounced gaps for intense and extreme expressions. The qualitative results also show better identity preservation under strong expressions when conditioning on diverse expressions.

5.1 QUALITATIVE RESULTS

  • Conditioning on diverse expressions helps our model produce frames that better preserve identity under strong expressions.Figure 8 compares generations using real or LivePortrait-generated galleries with Stand-In and AnyID.Source: §5.1 · pp. 6–7

5.2 RESULTS ON OUR EVALUATION DATASET

  • Ours (Real) and Ours (Synthetic) outperform all baselines across mild, intense, and extreme expression regimes, with especially pronounced gaps in the intense and extreme regimes.Table 2 reports identity fidelity on the evaluation dataset; the percentage columns give the fraction of each model’s frames in each expression-intensity regime.Source: §5.2 · p. 7, Table 2 · p. 7
  • Our model achieves higher identity fidelity on matched subject–prompt pairs, including cases where its real-gallery version produces greater peak expression intensity than the competing method.Source: §5.2 · p. 7

6 ABLATION STUDIES

The ablations examine reference count, reference diversity, and temporal embeddings. They find that eight expression-diverse references and shared temporal embeddings improve identity fidelity across expression regimes, while repeated neutral references can suppress expressive output.

6.1 NUMBER OF REFERENCE IMAGES

  • Each added reference improves overall CurricularFace similarity by approximately 1.3 points, with the largest gains for intense and extreme expressions; a ninth adds no overall gain, supporting eight references.Table 4 reports the effect of reference count on identity fidelity; the ninth reference uses a held-out expression outside the standard gallery.Source: §6.1 · p. 7, Table 4 · p. 9

6.2 EXPRESSION DIVERSITY VS. POSE DIVERSITY

  • Expression-diverse references outperform pose-diverse references in all three intensity regimes, while repeated neutral references score highest in mild, similarly in intense, and lower in extreme expressions.Repeated neutral references also produce a much lower share of intense and extreme frames, suggesting their high overall similarity partly reflects suppressed expression intensity.Source: §6.2 · pp. 8–9, Table 5 · p. 9

6.3 TEMPORAL POSITIONAL EMBEDDING

  • The shared-coordinate model achieves higher identity fidelity in all regimes than a variant assigning distinct temporal embeddings to shuffled references.All references receive the same temporal embedding in the shared-coordinate model; the negative-temporal variant assigns distinct embeddings on the video-to-image attention path while retaining shared coordinates for reference-image self-attention.Source: §6.3 · p. 9, Table 6 · p. 9

7 CONCLUSION

The paper concludes that a single neutral portrait is insufficient to capture person-specific facial changes or evaluate identity under strong expressions. Its expression-diverse gallery and multi-reference conditioning improve identity similarity across expression-intensity regimes, especially for extreme expressions.

7 CONCLUSION

  • The expression-diverse gallery and multi-reference conditioning model improve identity similarity across all expression-intensity regimes, especially under extreme expressions.The authors encourage future work to provide larger expression-diverse datasets for benchmarking and training.Source: §7 · p. 9

AI USE STATEMENT

  • Generative AI tools provided feedback on methodology and experiments, helped implement methods, and supported title generation and readability edits.The authors report that they reviewed AI-assisted work, manually audited AI-written code, and did not use AI to propose or refine hypotheses, clean datasets, or interpret results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • All human-face subjects signed consent forms permitting research use and academic publication of their images.The authors state that data handling follows relevant privacy laws and regulations, with sensitive information anonymized when necessary.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors support reproducibility with detailed evaluation, training, and inference descriptions, plus expressions and prompts used in evaluation.They will make code, data, and model weights publicly available upon acceptance, before submitting the camera-ready version.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS · A.1 FACE DETECTION AND EXPRESSION INTENSITY DEFINITION

The implementation aligns detected faces to a standard template for identity scoring and defines expression intensity as the distance of SMIRK-predicted facial parameters from neutral.

A.1 FACE DETECTION AND EXPRESSION INTENSITY DEFINITION

  • SCRFD-10g (Guo et al., 2021) detects faces at 640 × 640 with a 0.5 confidence threshold, and the highest-confidence face is retained when multiple faces appear.Source: §A.1 · p. 13
  • The detector’s five keypoints align each face to the standard ArcFace template, producing a 112 × 112 crop for both recognizers.Source: §A.1 · p. 13
  • ArcFace (Deng et al., 2019) and CurricularFace (Huang et al., 2020) embed the aligned crop; identity similarity is cosine similarity between ℓ2-normalized embeddings, reported ×100 following Wang et al. (2026).Source: §A.1 · p. 13
  • Frames without detectable faces are omitted from frame-level statistics, and videos with no detected face in sampled frames are excluded from paired comparisons.Detected-face coverage exceeds 98.8% of sampled frames for every evaluated method and 99.7% for ours.Source: §A.1 · p. 13
  • SMIRK encodes each crop into a 55-dimensional vector combining 50 FLAME expression coefficients, two eyelid parameters, and three jaw-rotation parameters.Source: §A.1 · p. 13
  • Expression intensity is defined as ∆Ψ = ∥Ψ − Ψneutral∥2, the Euclidean distance between the predicted facial-parameter vector and its neutral counterpart.Source: §A.1 · p. 13

A.2 MEAD DATASET PREPROCESSING · B SUPPLEMENTAL EXPERIMENTS

MEAD preprocessing uses sampled frontal-view videos to analyze expression variation across subjects, emotions, and intensity levels.

A.2 MEAD DATASET PREPROCESSING

  • MEAD preprocessing samples 24,192 frames from frontal videos of 48 subjects across seven non-neutral emotions, three intensity levels, and three utterances per condition.Of these, 24,190 frames have valid expression estimates. Neutral expression is the coordinate-wise median of sampled neutral frames, and neutral identity is the normalized mean of their embeddings.Source: §A.2 · p. 13

B.1 EVALUATION WITH HELD-OUT EXPRESSIONS

Nearest-gallery matching generalizes to expressions excluded from the reference gallery, and model ordering remains consistent when identity is evaluated on expressions not used for conditioning.

B.1 EVALUATION WITH HELD-OUT EXPRESSIONS

  • Nearest-gallery matching reduces expression-level mean ranges from 3.81 to 0.70 for ArcFace and from 3.53 to 1.75 for CurricularFace.This benefit generalizes to four expressions excluded from the eight-image gallery.Source: Table 7 · p. 13, §B.1 · p. 13
  • On four held-out expressions not used for conditioning, model ordering remains consistent with the original evaluation.The evaluation covers frames in the Mild, Intense, and Extreme regimes.Source: §B.1 · p. 13, Table 8 · p. 14

B.2 RESULTS ON OPENS2V-EVAL · C VERBAL INSTRUCTIONS FOR GALLERY EXPRESSIONS · D SYNTHETIC GALLERY EXAMPLE

On OpenS2V-Eval’s 60-video Human-Domain protocol, the model achieves a much higher Face Similarity score than the baselines. The appendices describe the benchmark’s limits, verbal instructions for the gallery expressions, and an example synthetic gallery.

B.2 RESULTS ON OPENS2V-EVAL

  • OpenS2V-Eval is not designed to test identity preservation under controlled, high-intensity expression changes with multiple references; its prompts do not systematically elicit such expressions and it supplies one reference per subject.Source: §B.2 · p. 14
  • Despite that limitation, evaluation uses the full Human-Domain protocol of 60 videos: 30 single-face and 30 single-human entries, each with one reference image and one text prompt.Source: §B.2 · p. 14
  • Because each OpenS2V-Eval entry provides only one reference, the evaluation constructs an eight-image gallery with the LivePortrait-based synthetic-reference pipeline.Source: §B.2 · p. 14
  • Our model obtains a much higher Face Similarity score than the baselines on OpenS2V-Eval using the official evaluator and aggregation.The comparison is reported for the 60-video Human-Domain protocol; baseline scores are published leaderboard values, and our references are generated by LivePortrait.Source: §B.2 · p. 14, Table 9 · p. 14
  • The official evaluator samples 32 frames, averages non-negative CurricularFace cosine similarity over detected-face frames per video, assigns zero when no face is detected, then averages across 60 videos and multiplies by 100.Source: §B.2 · p. 14

C VERBAL INSTRUCTIONS FOR GALLERY EXPRESSIONS

D SYNTHETIC GALLERY EXAMPLE

  • Figure 9 shows a synthetic expression gallery generated from one neutral portrait; it is less natural than real captures but adds expression information with the reenactment model’s image-domain prior.Source: Figure 9 · p. 15

E PROMPT SPECIFICATION

The appendix specifies a controlled set of fixed-camera prompts for evaluating identity preservation under strong expressions, with 48 prompts per subject spanning four environments, two triggers, and six basic emotions.

E.1 DETAILS OF PROMPT GENERATION PROCESS

  • The prompt design combines four environments, two triggers, and Ekman’s six basic emotions to produce 48 prompts per subject.The controlled fixed-camera setting is designed to test identity preservation under strong expressions without adding camera movement or occlusion.Source: §E.1 · p. 15
  • Each prompt starts from a neutral, relaxed face and specifies a medium close-up, a reaction trigger, emotion cues, a fast build to full intensity, and a stationary camera.The subject looks just past the camera.Source: §E.1 · p. 15

E.2 FULL LIST OF PROMPTS

  • The full list applies the shared prompt structure in settings including a kitchen, home office, living room, and cafe.Examples place the subject at a counter, desk, sofa, or cafe table by a window.Source: §E.2 · p. 16
  • Prompts specify visible cues for laughter, astonishment, and fear, including lifted cheeks, widened eyes, and tense brows and lips.The examples pair laughter with a joke or funny message, astonishment with unexpected news, and fear with a startling crash or terrifying message.Source: §E.2 · pp. 16–17
  • Across the prompt list, reactions are triggered either by someone or something just out of frame or by content on the subject’s phone.Examples include a joke or crash heard off camera and unexpected, terrifying, or upsetting messages viewed on a phone.Source: §E.2 · pp. 16–17
  • The remaining prompts cover anger, disgust, and grief, using cues such as clenched teeth, a wrinkled nose, and trembling chin with tear-filled eyes.These expressions are elicited by insults, revolting smells or images, and devastating news, respectively.Source: §E.1 · p. 15, §E.2 · pp. 18–20
Loading 2610.11023v1…