IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models

1. Introduction

• Authentic face databases collected without proper consent have faced legal and ethical criticism, and many widely used training datasets were retracted by their creators.The paper notes that GDPR's right to be forgotten and requirements for face-data collection and use make maintaining such datasets difficult.Source: §1 · p. 1
• Recent synthetic training datasets can lack either identity discrimination or intra-class variation, while effective recognition requires a trade-off between both.The paper identifies that trade-off as necessary for high verification accuracy.Source: §1 · p. 2
• IDiff-Face uses identity-conditioned diffusion to generate synthetic identities with identity-separable images and relatively large intra-class diversity.It trains in a pretrained autoencoder's latent space and uses contextual partial dropout to reduce overfitting to identity context and control the trade-off.Source: §1 · p. 2
• 98.00% accuracy on LFW exceeded the reported human-level result of 97.5% and approached authentic-trained models at 99.52% and 99.82%.The reported average accuracy across benchmarks was 88.20%, versus 83.45% for the best prior synthetic-based approach.Source: §1 · p. 2

2. Related Work

• Many synthetic-face methods rely on attribute annotations, carefully constructed batches, or 3D morphable models, and predefined attributes can limit natural variation and diversity.The related-work discussion also notes that DigiFace-1M uses a rendering pipeline with considerable computational cost.Source: §2 · pp. 2–3
• Existing synthetic face-recognition approaches trade off identity discrimination and intra-class variation, and their average accuracy remains below authentic-trained approaches.The cited comparison reports 83.45% average accuracy for synthetic-based recognition and 94.82% for authentic-based recognition across five benchmarks.Source: §2 · p. 3
• IDiff-Face is presented as the first diffusion-based identity-conditioned approach for synthetic identity-specific images with more realistic intra-class diversity for recognition training.The authors report that it outperforms recent state-of-the-art synthetic-based face-recognition approaches.Source: §2 · p. 3

3.1. Identity-Conditioned Latent Diffusion

• IDiff-Face trains a conditional diffusion model in a pretrained autoencoder's latent space, using face-recognition features injected through cross-attention as identity context.The denoiser learns to predict added noise at each diffusion step, and the iterative sampling process incorporates the identity context.Source: §3.1 · pp. 3–4

3.2. Synthetic Image Generation

• Synthetic identities are generated either from normalized Gaussian-sampled directions on a hypersphere or from embeddings extracted from images generated by an unconditional diffusion model.Keeping a context fixed while varying sampling noise yields multiple images for that identity; the latter pipeline is called Two-Stage IDiff-Face.Source: §3.2 · p. 4

3.3. Enhancing Intra-Class Variation via CPD

• Figure 2 presents samples from prior synthetic datasets, IDiff-Face variants with different CPD probabilities and synthetic embeddings, and variations generated for existing LFW identities.The figure shows four consecutive images for each of two identities per method.Source: Figure 2 · p. 4
• Contextual Partial Dropout randomly drops context-embedding components during training to reduce overfitting to identity context and increase within-identity variation.The paper describes this as partial dropout of the context, with components dropped at a chosen probability.Source: §3.3 · p. 4

4.1. IDiff-Face Training Dataset

• The diffusion model was trained on 70,000 FFHQ face images at 128×128 resolution, spanning attributes including age, lighting, and facial expressions.Training was conducted on eight NVIDIA A100-SXM4-40GB GPUs.Source: §4.1 · p. 4, §4.2 · p. 4

4.2. IDiff-Face Training Setups

• Three IDiff-Face models were trained with CPD probabilities of 0%, 25%, and 50%, using 150,000 global steps, batch size 512, and 1,000 diffusion steps.The setup uses Adam with an initial learning rate of 1e−4, cosine learning-rate annealing with warm restarts, and horizontal flipping at 50% probability.Source: §4.2 · p. 5
• The setup uses a pretrained VQGAN autoencoder for latent-space training and ElasticFace features as 512-dimensional identity contexts.ElasticFace is a ResNet-100 trained on MS1MV2; the U-Net uses four resolution levels and attention blocks in all but the first level.Source: §4.2 · p. 5

Assessment of Intra-Class Diversity and Identity Separability

• DiscoFaceGAN-based data had low intra-class diversity under predefined attribute control, whereas SFace had greater diversity but lower identity separability.DigiFace-1M also maintained some identity discrimination but relied on a computationally expensive rendering process.Source: §5 · pp. 5–6
• Higher CPD increased intra-class variation while slightly reducing identity discrimination, making CPD probability a trade-off between these properties.The paper links this balance to generating useful FR training data and achieving high verification accuracy.Source: §5 · p. 6

Synthetic-based FR

• 79.73% average accuracy was the best result for 80K-sample training with two-stage sampling and CPD50, close to CPD25 at 79.72% and uniform CPD25 at 79.54%.The evaluation trained ResNet-18 models on five benchmarks.Source: §5 · pp. 6–7, Table 2 · p. 7

Comparison to the SOTA Synthetic-based FR

• 83.87% average accuracy with 10K identities ×16 images exceeded 82.86% with 5K identities ×32 images at the same 160K-sample size.The paper reports this width-over-depth pattern across its experiments.Source: §5 · p. 7
• RandAugment improved verification accuracies when added to training of the best model in each setting.The experiments used RandAugment settings from, and prior synthetic FR models also used extensive augmentation.Source: §5 · p. 7
• 83.87% average accuracy on 160K samples exceeded SynFace (74.75%), SFace (77.71%), USynthFace (79.30%), and DigiFace-1M (83.45%).The compared methods used 400K–634K training samples.Source: §5 · p. 7
• 88.20% average accuracy was achieved by the best IDiff-Face model, versus 83.45% for the prior synthetic-based FR best, DigiFace-1M.The comparison reports verification accuracies across five benchmarks.Source: §5 · p. 7, Table 3 · p. 8
• 98.00% accuracy on LFW was below the authentic-trained MS1MV2 result (99.82%) but above the reported human-level performance (97.5%).The paper also reports 99.52% for CASIA-WebFace-trained models.Source: §5 · p. 8

• An ideal face-recognition training dataset combines strong identity discrimination with realistic, large intra-class variation, but legal and ethical concerns may make future use of authentic datasets infeasible.Source: §6 · p. 8
• Synthetic IDiff-Face data achieved new state-of-the-art verification accuracies on five mainstream face-recognition benchmarks, outperforming recent synthetic-based approaches.Source: §6 · p. 8
• IDiff-Face generates identity-specific synthetic face images, and CPD controls the trade-off between identity separability and intra-class variation while helping prevent overfitting to identity context.Source: §6 · p. 8