PaperlayerView exact revision on arXiv

Source-linked AI summary

DCFace: Synthetic Face Generation with Dual Condition Diffusion Model

Minchul Kim, Feng Liu, Anil Jain, Xiaoming Liu

arXiv:2304.07060v1cs.CV

TL;DR

Synthetic face-recognition datasets must generate many subjects, realistic style variation, and consistent identities rather than merely high-fidelity images. DCFace combines identity and style conditions in a two-stage diffusion generator, achieving higher verification accuracy than prior methods while retaining a documented gap from real-data performance and limited 3D pose consistency.

  • Problem

    Synthetic face-dataset generation must jointly provide novel subjects, real-image style distributions, and consistent images of the same subjects.

  • Method

    DCFace uses a two-stage diffusion generator that samples an identity image and a style image, then mixes them with patch-wise style extraction and time-dependent ID loss.

  • Results

    DCFace surpasses previous synthetic-data methods in face-recognition performance, with a 6.11% average improvement over DigiFace on 4 of 5 datasets in the 0.5M-image regime.

  • Takeaways & Limitations

    The results support controlling subject uniqueness, style diversity, and label consistency as important dimensions of synthetic face-recognition datasets.

  • Takeaways & Limitations

    The generator lacks 3D consistency across pose, and synthetic-data performance still trails real-data performance.

Abstract

from arXiv · show

Generating synthetic datasets for training face recognition models is challenging because dataset generation entails more than creating high fidelity images. It involves generating multiple images of same subjects under different factors (\textit{e.g.}, variations in pose, illumination, expression, aging and occlusion) which follows the real image conditional distribution. Previous works have studied the generation of synthetic datasets using GAN or 3D models. In this work, we approach the problem from the aspect of combining subject appearance (ID) and external factor (style) conditions. These two conditions provide a direct way to control the inter-class and intra-class variations. To this end, we propose a Dual Condition Face Generator (DCFace) based on a diffusion model. Our novel Patch-wise style extractor and Time-step dependent ID loss enables DCFace to consistently produce face images of the same subject under different styles with precise control. Face recognition models trained on synthetic images from the proposed DCFace provide higher verification accuracies compared to previous works by $6.11\%$ on average in $4$ out of $5$ test datasets, LFW, CFP-FP, CPLFW, AgeDB and CALFW. Code is available at https://github.com/mk-minchul/dcface

1. Introduction

A useful synthetic face-recognition dataset must combine many unique subjects, substantial within-subject variation, and reliable labels. The paper frames dataset generation as jointly maximizing subject novelty, real-image distribution matching, and consistency across images of each subject.

  • An ideal face-recognition dataset has many unique subjects, large intra-subject variation, and reliable subject labels.
  • Synthetic data can reduce privacy concerns and address informed-consent, long-tail, and demographic-bias issues in real face datasets.
  • The paper asks how to synthesize novel subjects, mimic real-image distributions, and consistently generate multiple images of each subject.
  • Earlier synthetic face-dataset methods addressed individual aspects but did not consider subject uniqueness, distribution matching, and consistency together.

1. Sampling Stage 2. Mixing Stage

DCFace uses a two-stage process to combine independently sampled identity and style conditions, targeting subject uniqueness, style diversity, and label consistency. Its diffusion-based dual-condition generator uses a patch-wise style extractor and time-dependent ID loss to balance diversity with identity preservation.

  • Mixing Stage: The proposed generator targets uniqueness, real-dataset style matching, and label consistency together.
  • Mixing Stage: The dual-condition formulation treats Xid as a person’s appearance and Xsty as identity-independent factors such as pose, expression, and image quality.
  • Mixing Stage: The mixing stage combines identity from Xid with style from Xsty to produce images whose subject label remains consistent across varied styles.
  • Mixing Stage: The method modifies diffusion models with a style-condition extractor and auxiliary time-dependent ID loss to control diversity and label consistency.
  • Mixing Stage: DCFace surpasses previous methods in face-recognition performance and examines how subject uniqueness, diversity, and consistency relate to recognition.

2. Related Works

Prior work spans GAN-based synthesis, 3D face rendering, and diffusion models. These approaches offer different trade-offs among fidelity, controllability, 3D consistency, style variation, ethics, and recognition performance.

  • Face-recognition models are commonly trained on large-scale web-crawled datasets and evaluated with verification benchmarks covering pose and age changes.
  • GANs have been widely used to manipulate, animate, or enhance faces through disentangled latent representations controlling face properties.
  • 3D-prior methods provide fine-grained control and 3D consistency but lack style or domain variation.
  • Diffusion and score-based models provide high-quality image generation with stable MSE-based objectives and support conditional tasks such as text generation and inpainting.
  • Synthetic training datasets can address ethical and class-imbalance concerns, but low recognition performance has limited their use as the sole training data.
  • SynFace explored DiscoFaceGAN, while DigiFace-1M used 3D rendering and augmentation; DCFace aims to combine many subjects with diverse styles close to real data.

3. Proposed Approach

DCFace uses a two-stage diffusion pipeline that samples a novel-subject identity image, selects a style image, and mixes them with dual conditioning. Its patch-wise style extractor and time-step-dependent ID loss separate style from identity while gradually shifting generated images toward the target style.

  • Condition Sampling and Mixing: DCFace first generates a high-quality identity image of a novel subject and samples an arbitrary style image from a real-data style bank.Stage 1 uses an identity generator; Stage 2 mixes the two conditions with a dual-condition generator.
  • Diffusion Backbone: DDPM is selected as the identity-generator backbone because its generated subject count exceeds DiscoFaceGAN’s under the described uniqueness measurement.The comparison uses models trained on FFHQ, with 10,000 generated samples each.
  • Dual Condition Generator Gmix: The dual-condition generator injects identity and style conditions into a conditional DDPM denoiser to predict the target noise.The conditions are provided through feature extractors and cross-attention-based conditioning mechanisms.
  • Training Gmix: The training formulation uses two same-subject images because the desired combination of one subject’s identity with another subject’s style is unavailable in labeled data.Naive training could ignore the identity condition, making it unused during testing.
  • Patch-wise Style Extractor: The patch-wise style extractor computes local style vectors from feature-map patches while constraining them to omit full identity information.It uses patch statistics and learned positional embeddings; a global feature is included as the k=1 case.
  • Time-step-dependent ID Loss: The time-step-dependent ID loss linearly interpolates recognition-space similarity to the identity and style images across diffusion time-steps.The model relies more on identity for an ID-rich image early in denoising and shifts toward the target style later; the total loss is LMSE + λLID.

4. Dataset Evaluation

The evaluation introduces class-dependent metrics for uniqueness, consistency, and diversity, alongside recognition performance, to characterize synthetic face datasets. Results indicate that face-recognition performance is best when consistency and diversity are balanced.

  • Evaluation metrics: Synthetic datasets are evaluated using FID, recognition performance, and three class-dependent metrics for dataset properties.The class-dependent metrics use an evaluation recognition model, Feval, to assess identity and diversity-related properties.
  • Uniqueness: Uniqueness counts non-overlapping subjects in Feval space using a cosine-distance threshold r.For labeled datasets, uniqueness is computed from distances between feature centers for different labels.
  • Consistency: Intra-class consistency measures whether generated samples remain close to their class center under a distance threshold.Higher Cintra values indicate that samples are more likely to represent the same subject under the same label.
  • Diversity: Intra-class diversity measures how much of the real style manifold is covered by generated images under the same label.The metric extends Improved Recall from unconditional to label-conditional distributions using nearest-neighbor feature distances.
  • Results: FR performance is best when consistency and diversity are balanced, while SynFace and DigiFace show high consistency but low diversity relative to DCFace.Figure 6 plots recognition accuracy against consistency and diversity, with color intensity and circle size encoding accuracy.

5. Experiments

Experiments evaluate DCFace through model and sampling ablations, comparisons with prior synthetic datasets, and verification performance relative to real data. The results identify trade-offs between consistency and diversity and show improved performance over prior methods.

  • 5. Experiments: The ablations generate 10K subjects with 50 images per subject and evaluate verification accuracy alongside uniqueness, consistency, and diversity metrics.The ablation setup uses held-out CASIA-WebFace validation data for the latter metrics.
  • 5.1. Model Ablation: Increasing style-extractor grid size increases diversity but can reduce consistency; 5×5 provides the best face-recognition performance as a compromise.Larger grids encode finer-grained style information that may include identity-related content.
  • 5.1. Model Ablation: Time-dependent ID loss outperforms naive alternatives because Lnaive1 limits diversity while Lnaive2 weakens consistency and can produce random labels.An FR performance of 0.5 indicates divergence and random predictions.
  • 5.1. Model Ablation: A larger pretrained ID-loss backbone improves synthetic-dataset quality, although the authors retain IR50 for fair comparison with CASIA-WebFace.The larger backbone is pretrained on WebFace4M without directly showing those images to the generator.
  • 5.2. Sampling Ablation: Balancing or matching gender and ethnicity distributions yields no significant gain, whereas oversampling Xid produces 89.56% average verification accuracy, a 0.52% increase.Oversampling repeats the same Xid five additional times for each label.
  • 5.3. Comparison with Previous Methods: In the 0.5M-image regime, DCFace surpasses DigiFace on 4 of 5 datasets by 6.11% on average, while DigiFace performs better on CFP-FP.The comparison covers verification accuracies on five validation sets and uses the reported SoTA synthetic datasets.
  • 5.3. Comparison with Previous Methods: DCFace reduces the synthetic-to-real performance gap by 57% over the SoTA in the 0.5M setting and reaches a 3.74% gap with 1.2M images.The 1.2M result represents a 60.9% improvement over the previous method’s 9.55% gap.

6. Conclusion

The conclusion presents DCFace as a synthetic face-dataset method targeting subject uniqueness, style diversity, and label consistency. It reports gains over prior methods while acknowledging limitations in pose consistency and the remaining real-versus-synthetic performance gap.

  • 6. Conclusion: DCFace generates synthetic face-recognition datasets by jointly targeting many unique subjects, large style diversity, and label consistency.The authors frame these properties as the central goals of dataset generation.
  • 6. Conclusion: The authors report large face-recognition performance gains over previous synthetic-dataset methods.The conclusion positions the approach as progress toward matching real training datasets with synthetic ones.
  • 6. Conclusion: The generator lacks 3D consistency across pose, an advantage associated with generative models using 3D priors.The authors also report that synthetic training remains separated from real-dataset performance.

A.1. Architecture Detals

The architecture combines identity and style features as dual conditions for a diffusion U-Net. Patch-wise style features, CNN-derived identity features, cross-attention, identity-conditioned residual modulation, and an auxiliary ID loss control the generated images.

  • A.1. Architecture Detals: The dual-condition generator modifies DDPM denoising by conditioning the U-Net on identity Xid and style Xsty features.The two conditioning images are encoded separately by Esty and Eid.
  • A.1. Architecture Detals: The patch-wise style encoder concatenates style vectors from k×k spatial patches with an additional global style vector.This produces R(k^2+1)×C style features.
  • A.1. Architecture Detals: The identity encoder concatenates intermediate CNN features and a final feature vector, then adds learnable positional embeddings.For the described ResNet50 configuration, the resulting identity representation has 50×C features.
  • A.1. Architecture Detals: Cross-attention injects concatenated identity and style conditions into intermediate U-Net features in the final two downsampling residual blocks.The U-Net intermediate representation supplies queries, while encoded conditions supply keys and values.
  • A.1. Architecture Detals: Adding the identity feature vector to the timestep embedding lets residual blocks modulate intermediate features through additional identity-dependent paths.The combined vector is used as the residual-block scaling signal.
  • A.1. Architecture Detals: Training uses an end-to-end LMSE + λLID objective with λ = 0.05.The inputs are 112×112 images.

B.2. Combining Multiple Synthetic Datasets

Combining DCFace and DigiFace synthetic datasets produces higher average verification accuracy than either dataset alone. This suggests that datasets generated by different methods can provide complementary training signals.

  • B.2. Combining Multiple Synthetic Datasets: 93.06 average verification accuracy is achieved when DCFace and DigiFace are combined without real images.The authors interpret this as evidence that differently generated synthetic datasets can be complementary.

C. Analysis

DCFace analysis examines uniqueness, feature distribution, style interpolation, identity control, dataset metrics, and societal concerns. The results support realistic variation and identity consistency while acknowledging trade-offs and misuse risks.

  • Unique Subject Counts: 60,000 unique subjects are generated from 200,000 DDPM samples, with extrapolation estimating up to 80,000 from more samples.Filtering removes sunglasses and subjects similar to CASIA-WebFace; uniqueness eventually reaches a limit.
  • Feature Plot: DCFace feature distributions are closest to CASIA-WebFace for both 50-subject and one-subject, 50-image t-SNE comparisons.Features come from an IR101 face recognition model, and the one-subject setting separates images by identity-related representation.
  • Identity Control: Time-dependent ID loss outperforms naive losses and classifier-free guidance by balancing identity consistency with facial diversity.Naive losses respectively reduce diversity or consistency, while strong guidance can also decrease diversity.
  • FID Scores: DCFace approximates the CASIA-WebFace distribution with FID=13.67, prioritizing realistic inter- and intra-subject variation over high-resolution image quality.The authors note that FID alone cannot capture label consistency or directly relate to face recognition performance.
  • Identity and Gender: DCFace allows factors weakly related to face-recognition identity, such as age and hairstyle, to vary even when perceived gender changes.The method adheres to subject identity as defined by a pretrained face recognition model.
  • Interpolation: Style interpolation preserves identity while smoothly producing intermediate poses and expressions that were not previously present.The interpolation combines two style embeddings while keeping the same identity condition.

G. Implementation Details and Code

The implementation uses existing repositories and preprocessing tools for training-data preparation, model definitions, and evaluation across five face-recognition benchmarks.

  • Implementation Details and Code: CASIA-WebFace preprocessing uses AdaFace references and MTCNN, while TFace defines the backbone and AdaFace evaluates LFW, CFP-FP, CPLFW, AgeDB, and CALFW.The code is released at the project GitHub repository.
Loading 2304.07060v1…