Source-linked AI summary
SynFace: Face Recognition with Synthetic Data
Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, Dacheng Tao
TL;DR
Face recognition needs alternatives to noisy, privacy-sensitive real-world data and lacks systematic evidence about how facial attributes affect recognition. SynFace evaluates synthetic training, identifies intra-class variation and domain mismatch as key gaps, and addresses them with identity and domain mixup. The study reports that these strategies narrow the real–synthetic gap while enabling analysis of dataset scale and facial attributes.
Problem
Real-world face data are difficult to collect because of label noise and privacy issues, while web datasets provide limited attribute annotations for studying pose, expression, and related factors.
Method
SynFace combines controllable synthetic face generation with identity mixup to enlarge intra-class variation and domain mixup using a few labeled real images to reduce domain mismatch.
Results
Identity mixup significantly improves synthetic-trained recognition, and SynFace IM reaches 97.30% accuracy versus 87.83% for RealFace at R = 0.6 on Syn-LFW-R.
Takeaways & Limitations
Synthetic data can support face recognition while allowing systematic control and analysis of identities, samples per identity, pose, expression, and illumination.
Abstract
from arXiv · showhide
With the recent success of deep neural networks, remarkable progress has been achieved on face recognition. However, collecting large-scale real-world training data for face recognition has turned out to be challenging, especially due to the label noise and privacy issues. Meanwhile, existing face recognition datasets are usually collected from web images, lacking detailed annotations on attributes (e.g., pose and expression), so the influences of different attributes on face recognition have been poorly investigated. In this paper, we address the above-mentioned issues in face recognition using synthetic face images, i.e., SynFace. Specifically, we first explore the performance gap between recent state-of-the-art face recognition models trained with synthetic and real face images. We then analyze the underlying causes behind the performance gap, e.g., the poor intra-class variations and the domain gap between synthetic and real face images. Inspired by this, we devise the SynFace with identity mixup (IM) and domain mixup (DM) to mitigate the above performance gap, demonstrating the great potentials of synthetic data for face recognition. Furthermore, with the controllable face synthesis model, we can easily manage different factors of synthetic face generation, including pose, expression, illumination, the number of identities, and samples per identity. Therefore, we also perform a systematically empirical analysis on synthetic face images to provide some insights on how to effectively utilize synthetic data for face recognition.
1. Introduction
Face recognition has advanced substantially, but further progress is constrained by noisy, privacy-sensitive real-world data and limited attribute annotations. SynFace explores synthetic faces to narrow the real–synthetic performance gap and systematically study dataset and facial-attribute factors.
- Motivation: Large-scale web-collected face data remain difficult to expand because of severe label noise, privacy issues, and long-tailed distributions.Some datasets face consent problems or restricted access, while existing collections often lack detailed annotations for attributes such as pose and expression.
- Motivation: Synthetic face recognition has been insufficiently investigated despite prior work on photo-realistic augmentation and pose-varying synthetic images.The paper positions SynFace as a broader study of synthetic data for recognition.
- SynFace approach: Synthetic-trained models exhibit a performance gap caused mainly by poor intra-class variation and the domain gap between synthetic and real faces.The authors identify these causes through empirical analysis.
- SynFace approach: Identity mixup enlarges intra-class variation by combining identity coefficients, while domain mixup uses a few real images to reduce the synthetic–real domain gap.These two mechanisms are presented as complementary ways to narrow the gap.
- Empirical analysis: The study examines how synthetic-dataset width, depth, pose, expression, and illumination influence face-recognition performance.The contributions emphasize dataset properties and controllable facial attributes as systematic analysis axes.
2. Related Work
Related work spans synthetic data for computer vision, controllable identity-preserving face synthesis, deep face-recognition objectives, and mixup. SynFace extends these directions by analyzing more dataset and facial factors while introducing identity and domain mixup.
- Synthetic Data: Synthetic data address expensive annotation, augment real-data training, and enable systematic analysis of dataset attributes.Prior work covers tasks including crowd counting, vehicle re-identification, segmentation, 3D face reconstruction, and face recognition.
- Synthetic Data: Earlier synthetic-face studies mainly considered augmentation or head pose, whereas SynFace also examines pose, expression, illumination, width, and depth.The paper describes its scope as broader than work focused only on head-pose effects.
- Face Synthesis: GAN- and 3DMM-based methods generate identity-preserving faces by disentangling or controlling identity and facial attributes.The cited methods target properties such as identity, pose, expression, and illumination.
- Deep Face Recognition: Deep face-recognition methods commonly maximize inter-class variation and minimize intra-class variation using objectives such as contrastive, triplet, and center losses.These objectives improve feature embeddings through margins or identity-specific centers.
- Mixup: Vanilla mixup combines data samples, whereas SynFace applies convex combinations to identity coefficients and uses real images as supervision for domain adaptation.The paper motivates identity mixup because pixel-level face mixing may not produce proper faces.
3. Method
SynFace uses controllable synthetic face generation to study the synthetic–real performance gap and addresses it with identity mixup and domain mixup. Identity mixup expands intra-class variation, while domain mixup combines synthetic data with limited labeled real data to reduce the domain gap.
- SynFace pipeline: SynFace explores the performance gap between models trained on synthetic and real face datasets, then introduces identity mixup and domain mixup to address it.The pipeline evaluates SynFace and RealFace before applying the two mixup strategies.
- Face generator: DiscoFaceGAN generates faces by controlling identity, expression, illumination, pose, and other properties through disentangled latent representations.The generator samples latent variables, maps them through trained VAEs, and synthesizes the corresponding face image.
- Identity mixup: Identity mixup linearly interpolates two identity coefficients while changing the corresponding label, producing intermediate identities and enlarging intra-class variation.The interpolation ratio ϕ is sampled from 0.0 to 1.0 in steps of 0.05.
- Identity mixup: 88.75% → 89.47% → 90.95% accuracy accompanies increasing intra-class variation across Syn1, Syn2, and Syn3.Identity mixup also raises accuracy from 88.98% to 91.97% in the reported comparison.
- Identity mixup evaluation: 97.30% accuracy versus 87.83% for RealFace at R=0.6 shows SynFace IM remains robust when synthetic test identities are mixed.SynFace IM achieves nearly perfect accuracy when R is larger than 0.6, whereas RealFace degrades as the primary identity weight decreases.
- Domain mixup: Domain mixup mixes mini-batches of synthetic and labeled real images, improving Mix 2K 10 to 95.78% and Mix 2K 20 to 97.65% on LFW.The method uses large-scale synthetic data with a small set of real-world images and changes labels according to the mixup ratio.
4. Experiments
The experiments evaluate how synthetic dataset construction, identity mixup, imbalance, dataset width and depth, and facial-attribute variation affect face verification on LFW.
- Datasets: Synthetic datasets are generated with controllable identity, expression, pose, and illumination coefficients, while Syn-LFW matches LFW statistics without overlapping training identities.The pipeline reconstructs attributes, generates images with random identities, and samples latent variables for synthetic training data.
- Long-tailed Face Recognition: The long-tailed study keeps 2K identities and 100K images fixed while varying imbalance across synthetic datasets.The balanced dataset uses 50 samples per identity; the two unbalanced datasets have standard deviations of 76.35 and 43.52.
- Effectiveness of Depth and Width: Accuracy rises from 83.85% to 88.75% as dataset width increases, while depth gains saturate beyond 20 samples per identity.With equal total image counts, wider datasets outperform deeper ones by 4.37%; identity mixup consistently improves performance, reaching 91.97%.
- Impacts of Different Facial Attributes: 74.55% and 73.72% are obtained by “Non” and “Expression,” whereas pose and illumination variations significantly improve performance.Expression variation remains limited because generated images mainly depict smiling, while pose and illumination distributions are more diverse and resemble LFW.
5. Conclusion
SynFace’s empirical analysis identifies dataset and attribute factors that affect synthetic face-recognition performance and proposes mixup-based remedies. The conclusion emphasizes intra-class variation, dataset width, facial attributes, and domain mixup as key considerations.
- Conclusion: Identity mixup consistently improves synthetic face-recognition performance by enlarging intra-class variation.The paper also reports domain mixup as a way to use a small subset of real-world face images.
- Conclusion: Increasing dataset width is more important than increasing depth, with saturation appearing first along the depth dimension.Both dimensions significantly influence performance.
- Conclusion: Pose and illumination variation improve performance, while generated expression images suffer from poor diversity.The paper reports differing effects across these three facial attributes.
- Conclusion: Domain mixup enables a small subset of real-world face images to greatly boost SynFace performance.
A. Intra-class Variations
Synthetic face images exhibit substantially simpler within-identity variation than real images, while the synthetic-to-real domain gap further limits generalization to unconstrained face recognition.
- 88.98% vs. 99.18% on LFW shows a clear performance gap between SynFace and RealFace.
- The domain gap between clean synthetic faces and complex in-the-wild faces leaves synthetic-trained models with weaker generalization.
- Real face images in the wild contain larger variations than synthetic images, including motion blur.
B. Expression Diversity
Synthetic expression data has limited within-class diversity: samples mainly differ through small mouth variations and are predominantly smiling.
- Expression samples within the same class differ only marginally, mainly through mouth variations.
- Generated expressions are mainly smiling despite slight differences, revealing limited expression diversity.