PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer

Xu Guo, Fulong Ye, Xinghui Li, Pengqi Tu, Pengze Zhang, Qichao Sun, Songtao Zhao, Xiangwang Hou, Qian He

arXiv:2601.01425v1cs.CV

TL;DR

Video face swapping struggles to preserve identity, attributes, and temporal consistency simultaneously. DreamID-V bridges image and video face swapping through supervised data construction, a DiT framework, and targeted training strategies, achieving superior benchmark performance and broad task versatility.

  • Problem

    Video face swapping must preserve identity similarity, target attributes, and temporal dynamics, while existing methods lag behind image face swapping in identity and attribute preservation.

  • Method

    DreamID-V combines SyncID-Pipe’s bidirectional ID quadruplets with a DiT framework using modality-aware conditioning, synthetic-to-real curriculum learning, and identity-coherence reinforcement learning.

  • Results

    DreamID-V achieves superior performance on IDBench-V and demonstrates clear advantages over state-of-the-art methods in quantitative and qualitative evaluations.

  • Takeaways & Limitations

    The framework provides a systematic, high-fidelity solution for video face swapping and can adapt to various swap-related tasks.

Abstract

from arXiv · show

Video Face Swapping (VFS) requires seamlessly injecting a source identity into a target video while meticulously preserving the original pose, expression, lighting, background, and dynamic information. Existing methods struggle to maintain identity similarity and attribute preservation while preserving temporal consistency. To address the challenge, we propose a comprehensive framework to seamlessly transfer the superiority of Image Face Swapping (IFS) to the video domain. We first introduce a novel data pipeline SyncID-Pipe that pre-trains an Identity-Anchored Video Synthesizer and combines it with IFS models to construct bidirectional ID quadruplets for explicit supervision. Building upon paired data, we propose the first Diffusion Transformer-based framework DreamID-V, employing a core Modality-Aware Conditioning module to discriminatively inject multi-model conditions. Meanwhile, we propose a Synthetic-to-Real Curriculum mechanism and an Identity-Coherence Reinforcement Learning strategy to enhance visual realism and identity consistency under challenging scenarios. To address the issue of limited benchmarks, we introduce IDBench-V, a comprehensive benchmark encompassing diverse scenes. Extensive experiments demonstrate DreamID-V outperforms state-of-the-art methods and further exhibits exceptional versatility, which can be seamlessly adapted to various swap-related tasks.

1 Introduction

Video face swapping must preserve identity, attributes, and temporal dynamics, yet existing approaches struggle to match image-swapping fidelity. DreamID-V addresses this gap with SyncID-Pipe, a DiT framework, and a comprehensive benchmark, achieving strong performance and versatility.

  • Video face swapping adds temporal identity continuity, pose consistency, and environment preservation constraints beyond image face swapping.
  • The framework robustly handles complex expressions, animation, large angles, occlusions, and small faces, while adapting to various swap-related tasks.
  • Its Modality-Aware Conditioning, Synthetic-to-Real Curriculum, and Identity-Coherence Reinforcement Learning target condition fusion, realism, and identity consistency under difficult motion.
  • DreamID-V achieves clear quantitative and qualitative advantages over state-of-the-art methods on IDBench-V.
  • DreamID-V combines SyncID-Pipe, the first DiT-based video face-swapping framework, and IDBench-V to transfer image-swapping strengths into video.SyncID-Pipe constructs supervised paired data, while IDBench-V evaluates diverse video conditions.

2 Related Work

Video face swapping has progressed from GAN-based image methods to diffusion-based approaches and Diffusion Transformer video models, improving quality and temporal consistency. However, identity similarity and attribute preservation remain unsatisfactory, motivating explicit supervision from image face swapping and the SyncID-Pipe data pipeline.

  • Diffusion Transformer-based video-generation methods have shown superior quality and consistency compared with earlier video diffusion architectures.
  • Image face swapping has evolved from GAN-based models to diffusion-based systems, with recent methods improving generated quality and temporal consistency.
  • Identity similarity and attribute preservation remain largely unsatisfactory because existing video face-swapping methods lack explicit supervision.
  • SyncID-Pipe addresses this supervision gap by combining a pre-trained Identity-Anchored Video Synthesizer with an Image Face Swapping model to construct bidirectional quadruplet-pair data.

3 Methodology

DreamID-V bridges image and video face swapping through explicit paired supervision, modality-aware conditioning, and training strategies targeting realism and identity coherence. Its framework injects distinct contextual, structural, and identity signals while extending to diverse swapping tasks.

  • 3.1 SyncID-Pipe: DreamID-V uses SyncID-Pipe to bridge IFS and VFS by constructing bidirectional ID quadruplets for explicit supervision.The pipeline uses an Identity-Anchored Video Synthesizer and IFS models to create paired training data.
  • 3.3 DreamID-V Training Pipeline: Synthetic-to-Real Curriculum learning strengthens visual realism while maintaining identity similarity, and Identity-Coherence Reinforcement Learning focuses optimization on difficult frames.IRL dynamically reweights the loss toward low-identity-fidelity segments, reducing temporal flickering.
  • 3.1 SyncID-Pipe: The Identity-Anchored Video Synthesizer reconstructs portrait videos from keyframes and pose sequences, providing identity-preserving, dynamically controllable training data.It conditions a First-Last-Frame video foundation model on initial and final frames plus extracted pose sequences.
  • 3.1 SyncID-Pipe: Enhanced background recomposition augments training data for background preservation while retaining the real video as supervision.The strategy targets inconsistencies caused by static keyframe-driven synthesis, especially under substantial background motion.
  • 3.2 DreamID-V Framework: DreamID-V is a DiT-based framework whose Modality-Aware Conditioning mechanism separates and injects spatio-temporal context, structural guidance, and identity information.Reference video and dilated face masks provide contextual information that must align with latent noise across spatial and temporal dimensions.
  • 3.4 DreamID-V Versatility: DreamID-V demonstrates advantages over state-of-the-art methods across identity similarity, expression preservation, background preservation, and occlusion, while adapting to outfit, accessory, and hairstyle swapping.The framework is designed for diverse and challenging scenarios beyond face swapping.

4 Experiments

DreamID-V is evaluated on the IDBench-V benchmark through quantitative, user, qualitative, ablation, and versatility studies. Across these evaluations, it achieves strong identity similarity, attribute preservation, video quality, and adaptation to related swapping tasks.

  • 4.1 Setup: IDBench-V contains 200 real-world source video-target image pairs covering small faces, extreme poses, occlusions, dynamic expressions, and multi-person scenes.The benchmark is designed for rigorous evaluation in real-world usage scenarios.
  • 4.2 Quantitative Comparisons: DreamID-V comprehensively outperforms state-of-the-art models on identity similarity metrics and is optimal on nearly all attribute-preservation metrics, while also improving video quality over image-based methods.It is slightly inferior to CanonSwap on pose preservation, but CanonSwap has much lower identity similarity.
  • 4.2 Quantitative Comparisons: User evaluation by 19 volunteers ranks DreamID-V best across identity similarity, attribute preservation, and video quality.Each dimension was scored from 1 to 5 on IDBench.
  • 4.3 Qualitative Analysis: The qualitative comparisons show stronger identity, expression, background, and occlusion handling than competing methods, with dynamic expression information improving expression performance over DreamID.The method also handles complex expressions and occlusions robustly.
  • 4.4 Ablation Studies: Ablations show that SyncID-Pipe improves identity similarity, while synthetic-to-real training balances identity similarity and realism and IRL improves profile-view consistency.Without IRL, frontal views remain stronger than profile views; IRL reduces inter-frame similarity variance under complex motion.
  • 4.5 Versatility: Expanding the training data extends DreamID-V to accessory, outfit, headphone, and hairstyle swapping.This demonstrates adaptation beyond video face swapping.

5 Conclusion

The paper presents SyncID-Pipe and DreamID-V as a comprehensive solution for high-fidelity video face swapping. On IDBench-V, the DiT-based model achieves superior performance and supports diverse swap-related tasks.

  • 5 Conclusion: SyncID-Pipe transfers image face-swapping advantages to video, enabling DreamID-V to achieve superior performance on the comprehensive IDBench-V benchmark.DreamID-V is presented as the first DiT-based model for video face swapping.

A Appendix

The appendix supplies supplementary methodological, implementation, benchmark, data-distribution, comparison, and ethical-consideration materials. These sections expand the paper’s reproducibility and evaluation details.

  • A Appendix: Ethical considerations are provided separately in Section A.6.

A.1 Preliminary

DiT uses a transformer denoising network to refine diffusion latents, while flow matching linearly interpolates noise and clean video before velocity regression.

  • DiT uses a transformer as the denoising network for refining diffusion latents.
  • Flow matching defines z_t by linearly interpolating between clean video z_0 and Gaussian noise ε.
  • The model is trained to directly regress the target velocity under condition y.

A.2.1 Inference Details

Inference uses identity-conditioned classifier-free guidance, then purifies its direction to preserve identity while reducing artifacts and maintaining realism.

  • The guidance vector d subtracts unconditional velocity from identity-conditioned velocity predictions.
  • Conventional CFG increases identity similarity but can cause oversaturation and unrealistic artifacts.
  • IDGP decomposes d into components parallel and orthogonal to the normalized conditional prediction.
  • IDGP reweights these components with α > 1 to amplify identity-preserving signal and suppress artifact-inducing signal.
  • The purified guidance enables strong identity preservation without sacrificing realism.

A.2.2 Detailed Parameters

Training proceeds through synthetic, hybrid real-and-synthetic, and identity-focused refinement stages with specified data scales and iteration schedules.

  • The IVS module is trained on 1000 hours of video data.
  • Synthetic training uses 100 hours of IVS-generated video, followed by real augmentation with 150 hours of real and synthetic data.
  • The final IRL stage uses 10 hours of data selected for high identity variance.
  • DreamID-V trains for 50k synthetic-data iterations, then 80k hybrid-data iterations before IRL refinement.

A.2.3 Evaluation Metrics

Evaluation combines automated identity, attribute, and video-quality measures with user studies and diverse qualitative benchmarks, while IDBench-V covers challenging video conditions.

  • A.2.3 Evaluation Metrics: Automated evaluation measures identity consistency, attribute preservation, and video quality, including temporal stability from frame-wise identity similarity variance.
  • A.2.3 Evaluation Metrics: A user study had 19 evaluators rate six anonymized model outputs on identity similarity, attribute preservation, and video quality using 1-to-5 scales.
  • A.3 IDBench-V Details: IDBench-V contains 200 videos paired with identity images spanning small faces, extreme poses, occlusions, expressions, and multi-person scenes.
  • A.2.3 Evaluation Metrics: t-SNE visualizes latent representations from 300 real videos, 300 synthetic videos, and 300 base-DiT outputs.
  • A.2.3 Evaluation Metrics: Qualitative figures present DreamID-V inference results and comparisons with baseline methods, including website-sourced cases for unavailable implementations.
  • A.2.3 Evaluation Metrics: DreamID-V maintains high photorealism and identity similarity across diverse and challenging scenarios.
  • A.6 Ethical Considerations: The released model could be misused for non-consensual deepfakes or disinformation, motivating consent and licensing restrictions.
Loading 2601.01425v1…