PaperlayerView exact revision on arXiv

Source-linked AI summary

FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation

Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Kai Yu, Sen Liang, Wenyue Li, Tianxiang Zheng, Qinglin Lu, Zhen Cui

arXiv:2605.04702v1cs.CVcs.AI

TL;DR

Existing IPT2V methods can distort facial identity during large pose changes and occlusions. FaithfulFaces aligns facial poses with explicit pose representations and a pose-diverse dataset, achieving superior identity consistency and structural clarity in such scenarios.

  • Problem

    Existing IPT2V methods struggle to preserve facial identity and structure under drastic facial pose changes, emotions, and occlusions because inputs represent only a single pose view.

  • Method

    FaithfulFaces uses a pose-shared identity aligner with dictionary learning, Euler-angle embeddings, and a pose-diverse video dataset to represent global facial pose.

  • Results

    Extensive quantitative and qualitative experiments across diverse identity and dynamic scenarios surpass existing open-source and commercial methods.

  • Takeaways & Limitations

    FaithfulFaces maintains facial identity consistency and structural clarity as facial pose changes and occlusions occur in generated videos.

Abstract

from arXiv · show

Identity-preserving text-to-video generation (IPT2V) empowers users to produce diverse and imaginative videos with consistent human facial identity. Despite recent progress, existing methods often suffer from significant identity distortion under large facial pose variations or facial occlusions. In this paper, we propose \textit{FaithfulFaces}, a pose-faithful facial identity preservation learning framework to improve IPT2V in complex dynamic scenes. The key of FaithfulFaces is a pose-shared identity aligner that refines and aligns facial poses across distinct views via a pose-shared dictionary and a pose variation-identity invariance constraint. By mapping single-view inputs into a global facial pose representation with explicit Euler angle embeddings, FaithfulFaces provides a pose-faithful facial prior that guides generative foundations toward robust identity-preserving generation. In particular, we develop a specialized pipeline to curate a high-quality video dataset featuring substantial facial pose diversity. Extensive experiments demonstrate that FaithfulFaces achieves state-of-the-art performance, maintaining superior identity consistency and structural clarity even as pose changes and occlusions occur.

1 Introduction

IPT2V generates videos from reference images and text while preserving facial identity, but existing methods struggle with pose changes and occlusions. FaithfulFaces addresses this with pose-faithful alignment, pose-diverse data, and demonstrated quantitative and qualitative improvements.

  • Motivation: IPT2V generates diverse videos from reference images and text prompts while maintaining consistent human facial identity across consecutive frames.The task supports applications including film and television, personalized avatars, advertising, and social multimedia content.
  • Motivation: Existing IPT2V methods exhibit facial identity and structural distortion in complex dynamic scenes involving pose changes and occlusions.Figure 1 reports severe facial-structure distortion for ConsisID and significant identity-detail distortion for VACE and Kling.
  • Method: FaithfulFaces introduces a pose-faithful facial identity preservation framework centered on a pose-shared identity aligner.The aligner encodes global facial pose representations from single-view reference images to address identity preservation under complex facial dynamics.
  • Method: The aligner uses a pose-shared dictionary, pose variation–identity invariance constraint, and Euler angle embedding learning to align diverse facial poses.Same-identity faces with different poses form positive pairs, while other identities serve as negative samples.
  • Data: A task-specific dataset pipeline constructs a high-quality video dataset with substantial facial pose diversity for robust FaithfulFaces training.The learned framework extracts global facial pose representations as holistic priors for foundational generative models.
  • Experiments: Extensive experiments across diverse identities and dynamic scenarios show FaithfulFaces surpasses existing open-source and commercial methods quantitatively and qualitatively.The reported results demonstrate improved identity consistency and structural clarity as facial poses change and occlusions occur.

2 Related Work

Related work has progressed from lightweight face adapters built on early text-to-video models to IPT2V methods based on DiT backbones. However, existing open-source and commercial systems still struggle with complex facial dynamics and identity distortion, motivating a new learning framework.

  • Text-to-Video and IPT2V: Continuous-time generative models enabled large-scale text-to-video generation, facilitating the development of identity-preserving text-to-video generation.Early IPT2V work introduced ID-Animator, which combines AnimateDiff with a lightweight face adapter.
  • Text-to-Video and IPT2V: ID-Animator uses the Unet-based AnimateDiff model and builds a face adapter for identity-preserving text-to-video generation.
  • DiT-Based IPT2V: DiT has become a mainstream video-generation backbone, supporting models such as HunyuanVideo, CogVideoX, and Wan, while many recent IPT2V methods extend DiT-based models.ConsisID is cited as an example that uses CogVideoX as its basic generative model and designs global and local facial extractors.
  • Limitations: Despite open-source methods and commercial tools such as Vidu, Pika, and Kling, complex facial dynamics can still distort identity information in generated videos.This limitation motivates the proposed learning framework.

3 Method

FaithfulFaces decomposes identity encoding into basic facial identity and global facial pose representations, using a pose-shared identity aligner to align varied poses and guide identity-preserving text-to-video generation. Its training combines pose-alignment contrastive learning with generative-model optimization, supported by a video-data pipeline containing substantial facial-pose variation.

  • Problem formulation: IPT2V generates a video conditioned on a reference face image and text prompt, requiring textual alignment and facial identity consistency with the reference.The foundational generative model determines semantic alignment, while the identity encoder supplies reference identity information.
  • Training and inference: During training, paired faces from each video form pose-variant identity pairs for contrastive alignment, while the refined representations are injected into noisy videos for joint optimization.The alignment loss constrains different poses, whereas flow matching constrains LoRA parameters to adapt to the global facial pose representation.
  • Main idea: FaithfulFaces partitions identity encoding into a basic facial identity encoder for structure and texture and a global facial pose encoder for pose representation.The global facial pose encoder is designed to represent faithful pose from a single-view reference image.
  • Pose-shared identity aligner: The pose-shared identity aligner tokenizes face images, injects explicit pitch, yaw, and roll embeddings, and uses a shared dictionary to refine and align representations across poses.The dictionary produces global facial pose representations from embeddings that otherwise contain only implicit pixel-level pose information.
  • Training and inference: During inference, a single face image is pose-estimated and passed through the trained aligner, whose global pose representation is combined with the noisy video, text prompt, and face image.The dataset pipeline filters unsuitable videos, selects substantial facial-pose variation, generates prompts with an MLLM, and integrates the resulting data.
  • Information-theoretic justification: Because the pose-alignment loss is equivalent to InfoNCE, minimizing it lower-bounds shared identity information across poses and discourages representation collapse.The paper further reports that visualization confirms the learned facial identity and that similar poses activate particular dictionary elements.

4 Experiments

FaithfulFaces is evaluated on challenging identity-preserving text-to-video scenarios involving facial pose variation, expression changes, and occlusion. It achieves the best performance across four metrics, with qualitative and component analyses supporting improved identity consistency and facial clarity.

  • Evaluation Setup: Evaluation uses 30 identities from ConsisID and 20 challenging prompts featuring facial pose variation, expression changes, and facial occlusions.One image is randomly sampled for each identity, and four standard metrics from prior work measure generated-video quality and identity preservation.
  • Baselines: FaithfulFaces is compared against two commercial products and eight open-source identity-preserving video-generation models.The compared methods include Vidu, Kling, ConsisID, VACE, HunyuanCustom, Phantom, Concat-ID-Wan, SkyReels-A2, Stand-In, and MAGREF.
  • Quantitative Results: FaithfulFaces achieves the best IPT2V performance under all four evaluation metrics, with considerable gains in FaceSim-Cur and FaceSim-Arc.These two metrics assess identity preservation, and the improvement is attributed to a robust global facial pose prior for foundation models.
  • Qualitative Results: Visual comparisons show that FaithfulFaces preserves clear facial structures and consistent identity under diverse pose changes and facial occlusions.Other open-source and commercial methods exhibit varying degrees of identity distortion and facial collapse in the boxing scenario.
  • Ablation Study: Ablations show that the pose-shared identity aligner substantially improves identity consistency, while Euler angle embeddings contribute to the framework’s performance.The aligner represents global facial pose information from a single-view reference image.
  • Identity Visualization: t-SNE visualization across seven unseen identities and eight facial poses per video shows promising identity separability for FaithfulFaces.The visualization indicates that its encoded identity representation is faithful and high-fidelity.

5 Conclusion

FaithfulFaces is proposed as a pose-faithful facial identity preservation framework for identity-preserving text-to-video generation. It addresses existing methods’ limited awareness of global facial pose in intricate facial dynamic scenarios through a pose-shared identity aligner.

  • FaithfulFaces is a pose-faithful facial identity preservation learning framework for IPT2V.
  • Existing methods struggle with intricate facial dynamic scenarios because of insufficient awareness of global facial pose.
  • The pose-shared identity aligner encodes global facial pose from a single-view face image by refining and aligning distinct facial poses.

A Appendix · A.1 Observations in Learned Dictionary

The learned pose-shared dictionary exhibits pose-specific activation patterns across five representative facial poses. Similar poses consistently activate particular dictionary elements, indicating meaningful pose-related facial representations.

  • A.1 Observations in Learned Dictionary: Activations of the pose-shared dictionary were visualized for five representative facial poses in Fig. 7.Face images were screened by Euler angles and passed through the pose-shared identity aligner.
  • A.1 Observations in Learned Dictionary: For each face image, the top-10 dictionary-element indices were recorded from its dictionary weight vector.These indices represent the most prominently activated dictionary elements for each image.
  • A.1 Observations in Learned Dictionary: Similar facial poses consistently activated particular dictionary elements across the visualized examples.The results show pose-dependent recurrence in dictionary activation patterns.
  • A.1 Observations in Learned Dictionary: Frontal faces tended to activate dictionary elements with indices 3, 562, and 2806.This example illustrates a recurring activation pattern associated with the frontal pose.
  • A.1 Observations in Learned Dictionary: Upward-looking faces frequently activated dictionary elements with indices 2, 704, and 1856.The upward-looking pose produced a distinct recurring index pattern from the frontal pose example.
  • A.1 Observations in Learned Dictionary: The observed activation regularities demonstrate that the learned dictionary captures meaningful patterns across facial poses.These patterns potentially enable robust representations of faces exhibiting varied poses.

A.2 Exploring the Effects of Different Dictionary Elements

An ablation study evaluates how the number of dictionary elements in D affects FaithfulFaces under two FaceSim metrics, finding the best performance at 4096 elements.

  • Dictionary-element ablation: The ablation varies the dictionary size across 1024, 2048, 4096, 8192, 16384, and 32768 elements.Six experimental settings are compared.
  • Dictionary-element ablation: FaithfulFaces reaches its best performance with 4096 dictionary elements under both FaceSim metrics.The comparison is illustrated in Fig. 8 and quantified in Table 3.

A.3 Qualitative Analysis of Ablation Study for Key Components

The qualitative ablation visualizations show that global facial pose awareness is crucial for preserving facial structure and details, while the pose-shared identity aligner partially mitigates distortions without Euler Embedding.

  • Qualitative Ablation Results: Removing both the Identity Aligner and Euler Embedding causes obvious distortion of facial structures and facial details.The degradation is attributed to lacking global facial pose awareness.
  • Qualitative Ablation Results: Removing only Euler Embedding mitigates facial distortions through the global facial pose representation supplied by the pose-shared identity aligner.This comparison isolates the aligner’s contribution to facial-pose-aware generation.
  • Qualitative Ablation Results: Figure 9 provides visual comparisons for the ablation study of FaithfulFaces’ key components.The visual analysis complements the quantitative ablation results reported in Table 2.

A.4 Ablation Study of Pooling Operation Type

The ablation study finds that max pooling performs best in the pose-shared identity aligner, likely because it suppresses redundant information shared across poses of the same identity.

  • A.4 Ablation Study of Pooling Operation Type: Max pooling delivers the optimal performance among the evaluated pooling operation types in the pose-shared identity aligner.The results are reported in Table 4.
  • A.4 Ablation Study of Pooling Operation Type: Face images of the same identity across different poses contain substantial similar or redundant information, which max pooling can alleviate while extracting highly informative features.This is the proposed explanation for max pooling’s superior performance.

A.5 Ablation Study of Different Identity Features … A.13 More Visualization Results

The supplementary studies validate FaithfulFaces across identity-feature ablations, optimization stability, non-frontal inputs, Euler-angle perturbations, challenging prompts, ethics, reproducibility, and additional visualizations. Results highlight the pose-shared aligner’s importance for preserving identity under difficult pose conditions while documenting implementation resources and potential misuse risks.

  • A.5 Ablation Study of Different Identity Features: Replacing the pose-shared aligner features with ArcFace or CLIP significantly deteriorates performance, because neither adequately represents the intended global facial identity-pose information.ArcFace lacks global facial pose representation, while CLIP primarily encodes text-image alignment rather than facial identity.
  • A.6 Stability of Contrastive Loss in Optimization Procedure: The aligner’s contrastive loss gradually decreases during training and converges to approximately 0.2, indicating stable optimization.Figure 10 tracks the aligner loss throughout training.
  • A.7 Discussion on Non-frontal View Robustness: With non-frontal reference images, both Ours w/o Aligner and Phantom suffer performance decreases exceeding 50%, whereas FaithfulFaces maintains identity consistency visually.The study uses 10 identities with paired frontal and non-frontal images, with Phantom representing the strongest baseline.
  • A.8 Discussion on the Robustness of Identity Aligner for Euler Angles: Random Euler-angle perturbations produce only minor performance variations within −15°∼+15°, with significant degradation occurring only beyond that range.The experiments evaluate perturbation ranges of −5°∼+5°, −10°∼+10°, −15°∼+15°, and −20°∼+20°.
  • A.9 Prompt Construction: The prompt-construction procedure targets videos with substantial facial pose variations, expression changes, occlusions, and diverse movements such as boxing, dancing, ballet, speech, and turning.GPT-4.13 is used to generate information-rich prompts from representative movement scenes.
  • A.10 Ethics Statement and Broader Impact: FaithfulFaces can generate high-quality, realistic human videos, but its identity-preserving capability raises misinformation and fraudulent-content risks.The ethics statement emphasizes potential misuse for deceptive video generation.
  • A.11 Reproducibility Statement: The paper documents implementation, training, inference, and dataset-construction details, and states that its code and dataset pipeline will be open-source online.These materials are identified as reproducibility resources.
Loading 2605.04702v1…