PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffFace: Diffusion-based Face Swapping with Facial Guidance

Kihong Kim, Yunho Kim, Seokju Cho, Junyoung Seo, Jisu Nam, Kychul Lee, Seungryong Kim, KwangHee Lee

arXiv:2212.13344v1cs.CV

TL;DR

GAN-based face swapping involves unstable training and extensive hyperparameter tuning. DiffFace uses an identity-conditional DDPM with facial guidance and target-preserving blending during sampling. It reports surpassing state-of-the-art methods on standard face-swapping benchmarks, with guidance and identity–attribute control available without retraining.

  • Problem

    GAN training is inherently unstable and requires extensive hyperparameter tuning, which becomes more challenging when external models are combined.

    Source: §1 · p. 2

  • Method

    DiffFace trains an identity-conditional DDPM and uses pretrained facial guidance and target-preserving blending during sampling.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    DiffFace surpasses other state-of-the-art methods on standard face-swapping benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework can apply varied facial guidance and adjust the identity–attribute trade-off without retraining.

    Source: §1 · p. 2

    The paper reports limited expression, pose, and shape scores, though sampling guidance can be adjusted to compensate.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.

1. Introduction

DiffFace combines an identity-conditioned diffusion model, facial guidance, and target-preserving blending for face swapping. The paper reports stronger benchmark performance than prior state-of-the-art methods and control of the identity–attribute trade-off without retraining.

1. Introduction

  • GAN-based face swapping requires unstable min–max training and repeated hyperparameter tuning, while balancing source identity against target attributes remains challenging.Tuning becomes more difficult when external models are combined.Source: §1 · p. 2
  • Diffusion models offer more stable training and desirable diversity and fidelity, while guidance can be applied through external modules at test time.The paper presents these properties as motivation for using diffusion in face swapping.Source: §1 · p. 2
  • DiffFace trains an identity-conditional DDPM, applies pretrained facial guidance during sampling, and blends to preserve target attributes while transferring source identity.The identity condition and identity-similarity loss make the diffusion model aware of facial identity.Source: §1 · p. 2
  • DiffFace surpasses other state-of-the-art methods on standard face-swapping benchmarks.The paper also describes gains over GAN-based methods in training stability, fidelity, and controllability.Source: §1 · p. 2
  • The target-preserving strategy gradually strengthens the facial mask to retain target-face attributes during denoising, and guidance can be adjusted without retraining.This provides a way to adapt the identity–attribute trade-off to the desired result.Source: §1 · p. 2

2. Related Work

Prior work applies diffusion models to conditional generation and image editing, but had not directly addressed face swapping. Existing face-swap approaches use structural priors, reconstruction, or StyleGAN, while the cited discussion notes fidelity challenges in several approaches.

2.1. Diffusion Model

  • Diffusion methods address conditional generation, local editing, and image translation, but the paper identifies face swapping as an unaddressed task for diffusion models.The authors present their framework as the first high-fidelity, controllable face-swapping approach based on conditional diffusion.Source: §2.1 · p. 2

2.2. Face Swap Model

  • Structural-prior methods use 3D shape information or facial keypoints and segmentation, but inaccurate structural or keypoint information can hinder high-fidelity results.HifiFace combines 3D Morphable Model information with GANs; FSGAN reenacts the source to match the target.Source: §2.2 · p. 2
  • The ID Conditional DDPM training procedure conditions each denoising step on the source identity vector and computes cosine similarity loss between source and predicted-image identity vectors.The figure caption identifies the predicted denoised image at timestep t as bx0.Source: Figure 2 · p. 3
  • Reconstruction-based methods include identity-specific DeepFakes and subject-agnostic Faceshifter and SimSwap, which inject source identity into target features but are described as insufficiently high-fidelity.DeepFakes is limited to swapping between paired identities.Source: §2.2 · pp. 2–3
  • StyleGAN-based methods manipulate latent or multiresolution features for face swapping, but the cited approaches typically fail to synthesize fine details.The passage describes MegaFS latent-space fusion and work focused on separating identity and pose.Source: §2.2 · p. 3

3. Preliminaries: Denoising Diffusion Probabilistic Models

The preliminaries describe diffusion as reversing a gradual noising process, with a neural network estimating noise during denoising. They also introduce direct denoised-image prediction and classifier guidance as background for the paper’s identity-conditioned approach.

3. Preliminaries: Denoising Diffusion Probabilistic Models

  • A diffusion model generates an image by reversing a recurrent noising process that takes data toward a Gaussian distribution.The forward transition is described using a predefined variance schedule.Source: §3 · p. 3
  • The reverse process can use a neural network to parameterize its mean and variance, with a noise-approximation model used in practice.The passage says noise approximation worked best, rather than specifying the completion of the comparison.Source: §3 · p. 3
  • The training algorithm samples a timestep and Gaussian noise, forms a noisy image, predicts a denoised image, and updates the model using identity features.The algorithm passage lists the sampling steps and predicted identity feature.Source: §3 · p. 3
  • The model can derive a fully denoised prediction from a noisy input, which the paper uses as input to facial expert modules.This prediction is denoted x̂0 in the prose.Source: §3 · p. 3
  • Classifier guidance uses a classifier gradient during reverse diffusion, whereas this work uses an identity-conditional DDPM.The passage distinguishes the unconditional diffusion model used by ADM from the paper’s model.Source: §3 · p. 3

4. Methodology

DiffFace is organized around identity-conditioned DDPM training, facial guidance during sampling, and target-preserving blending. Together, these stages transfer source identity while guiding facial attributes and retaining target structure.

4.1. ID Conditional DDPM

  • The ID Conditional DDPM learns to predict noise while preserving source identity, using identity constraints on estimated denoised images at each timestep.The method estimates a fully denoised image at each transition because expert models are trained on clean images.Source: §4.1 · p. 4

4.2. Facial Guidance

  • During sampling, pretrained identity, face-parsing, and gaze experts guide identity transfer, target facial structure, and gaze preservation without retraining.Identity guidance constrains source and generated identity embeddings; semantic guidance matches target facial features, while gaze guidance penalizes differences from the target gaze.Source: §4.2 · p. 5

4.3. Target-Preserving Blending

  • A hard mask can preserve background but noise can erase target structure; gradually strengthening a soft mask blends noise-matched target images with reverse-step predictions to tune identity–structure balance.The mask intensity increases over the diffusion process, with its saturation point controlling how much target structure is maintained.Source: §4.3 · p. 6

5. Experiment

The experiments evaluate DiffFace on FaceForensics++ against GAN-based face-swapping methods and examine implementation, evaluation measures, baseline comparisons, and component ablations. Results report strong identity transfer alongside trade-offs in preserving target attributes.

5.1. Implementation Details

  • Experiments use FFHQ training details and sample uniformly selected frames from 1,000 original FaceForensics++ videos for evaluation.Training uses 70k aligned 256 × 256 faces, 700k steps, batch size 48, and eight NVIDIA A100 GPUs; sampling uses T = 75 and guidance weights λid = 2000, λsem = 150, and λgaze = 200.Source: §5.2 · p. 6, §5.1 · p. 6

5.2. Evaluation Protocol

  • Evaluation compares SimSwap, HifiFace, InfoSwap, MegaFS, FaceShifter, and Deepfakes using identity similarity, expression, pose, and shape measures.Identity is measured with ArcFace and CosFace embeddings and a relative-distance metric; expression, pose, and shape use distances between 3D face-model coefficients.Source: §5.2 · p. 6

5.3. Comparison with Baselines

  • In qualitative comparisons, DiffFace more effectively transfers identity-related attributes, including vivid lips and eyes, than competing models.Other results tend to retain eye and lip colors from target images.Source: §5.3 · pp. 6–7, Figure 4 · p. 7
  • DiffFace achieves the highest identity score and better removes target identity than other models in the FaceForensics++ comparison.Table 1 is the quantitative comparison; the authors report that expression, pose, and shape performance is limited.Source: §5.3 · p. 7, Table 1 · p. 7

5.4. Ablation Study and Analysis

  • With ID Conditional DDPM and identity guidance present, the model attains its highest identity similarity; removing either yields visible artifacts or weaker identity transfer.The ablation reports artifacts without ID Conditional DDPM and poor identity transfer without identity guidance.Source: §5.4 · pp. 7–8
  • Increasing ˆT raises identity similarity but worsens expression, pose, and shape; decreasing it shifts results toward target attributes, including skin tone.The authors describe this as a way to balance target-attribute preservation and source-identity injection without additional training.Source: §5.4 · p. 8
  • Gaze guidance makes the synthesized image follow the target gaze, which identity or semantic guidance alone cannot control.Source: §5.4 · p. 8
  • Increasing ˆT and decreasing λsem prioritizes shape change, while decreasing ˆT and increasing λsem prioritizes target structure without retraining.The illustrated chin shape becomes progressively rounder across the results.Source: §5.4 · p. 8

6. Conclusion

DiffFace combines an identity-conditional diffusion model, pretrained facial guidance, and target-preserving blending for subject-agnostic face swapping. The conclusion reports that it preserves target attributes, supports flexible guidance and identity–attribute control without additional training, and significantly outperforms previous face-swapping methods.

6. Conclusion

  • DiffFace significantly outperforms previous face-swapping methods in extensive experiments.Source: §6 · p. 9
  • Pretrained facial expert models guide sampling to transfer source identity while preserving target attributes.Source: §6 · p. 9
  • Target-preserving blending helps retain target-face attributes from noise while transferring source identity.Source: §6 · p. 9
  • The framework flexibly applies different facial guidance and controls the identity–attribute trade-off without additional training.Source: §6 · p. 9
  • DiffFace targets controllable, high-fidelity, subject-agnostic face swapping.Source: §6 · p. 9

A. Architecture Details

The architecture extracts a source identity vector, injects it into a diffusion U-Net, and uses parsing and gaze models to provide facial guidance during sampling.

A.1. Identity Embedding Model

  • ArcFace and CosFace extract the source-face identity vector, which UnitNorm makes unit length.The embedding models are based on ResNet-101.Source: §A.1 · p. 12

A.2. ID Conditional DDPM

  • The identity vector conditions the diffusion model by being injected into U-Net residual blocks alongside the diffusion timestep.The figure describes feature dimensions in channels × height × width order.Source: §A.2 · p. 12, Figure 7 · p. 12

B.1. Face Parser

  • The face parser outputs 19 classes; the facial mask uses 11 face-swap-related classes and excludes non-facial components.Examples of included classes are skin, nose, and eyes; excluded examples include hair, hats, and clothing.Source: §B.1 · p. 12

B.2. Gaze Estimator

  • Gaze guidance uses detected eye landmarks to crop eye images, resizes them to 96x160, and feeds them to a pretrained gaze estimator.Source: §B.2 · p. 12

C. Additional Results from DiffFace

Target-preserving blending adjusts facial-mask intensity during diffusion to retain target-image details.

C.1. Target-Preserving Blending

  • Facial-mask intensity increases linearly until the masking threshold.Source: §C.1 · p. 12
  • Changing mask intensity determines how strongly the target image is reflected in the blend.Source: §C.1 · p. 12
  • The blending strategy can preserve fine facial details, including expression, pose, and skin tone.Source: §C.1 · p. 12

C.2. Trade-off between ID and Shape

DiffFace can vary facial shape and skin tone during sampling by adjusting facial guidance and the masking threshold.

C.2. Trade-off between ID and Shape

  • Facial guidance and the masking threshold control face shape, allowing emphasis on identity or shape.Source: §C.2 · p. 12
  • The examples include both round-chin and sharp-chin outputs.Source: §C.2 · p. 12
  • Skin tone varies with the time threshold.Source: §C.2 · p. 12

C.3. Comparison and Out-Of-Domain Results

The section presents additional face-swapping comparisons on FaceForensics++ and tests DiffFace on unseen oil-painting and cartoon-face datasets.

C.3. Comparison and Out-Of-Domain Results

  • Additional results compare various face-swapping models on FaceForensics++.The passage identifies the dataset but does not report a winning model or comparison outcome.Source: §C.3 · p. 12
  • On unseen MetFaces and Disney Face images, DiffFace results reflect each domain’s characteristics with shape changing.The model was not trained on these images.Source: §C.3 · p. 13

D. Limitations

The limitation described is that artifacts arising during sequential stochastic denoising can persist into DiffFace outputs.

D. Limitations

  • Artifacts such as wrinkles, hair segments, and glasses can persist once they occur during sequential stochastic transitions.The passage says the model sometimes produces unintended artifacts and identifies correcting artifact-bearing transitions as a research need.Source: D. Limitations · p. 13
  • DiffFace sometimes produces unintended artifacts in its outputs.The passage attributes this concern to artifacts being easy to maintain during denoising.Source: D. Limitations · p. 13
  • Correcting transitions that contain artifacts is identified as a direction for further research.Source: D. Limitations · p. 13
Loading 2212.13344v1…