Source-linked AI summary
Adaptive Adversarial Augmentation for Controllable Face Synthesis
Saransh Suri, Shivang Agarwal, Mayank Vatsa, Richa Singh
TL;DR
Synthetic-only face-recognition training is limited by insufficient intra-class variation and synthetic–real domain mismatch. EFCS uses ensemble feedback and structured perturbations to produce controllable samples; evaluations report challenging yet authentic samples that surpass traditional methods.
Problem
Synthetic-only face-recognition models underperform real-data models amid insufficient intra-class variation and a synthetic–real domain gap.
Source: §1 · pp. 1–2Method
Results
EFCS surpasses traditional methods, and SSIM, LPIPS, and DISTS evaluations confirm its challenging yet authentic samples.
Source: §6 · p. 11Takeaways & Limitations
The formulation offers insights into balancing synthetic-data complexity for optimal face-recognition training.
Source: §1 · p. 3The robust-optimization interpretation assumes a continuous loss, a smooth data manifold, and bounded perturbations.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Synthetic data provides a scalable alternative to real-world datasets for training face recognition models, particularly under challenging conditions such as low resolution, occlusion, and masks. Yet, most approaches lack diversity and fail to generalize effectively. We propose Ensemble Feedback Controllable Synthesis (EFCS), a guided framework that generates diverse and challenging samples while preserving visual realism. EFCS expands distributional variability, often reflected in higher FID and KID scores compared to single-feedback and random synthesis, while maintaining high precision. Recognition models trained on EFCS data consistently outperform baselines across multiple benchmarks, showing improved generalization to real-world scenarios. Furthermore, we introduce an analytically motivated formulation linking perturbation-induced difficulty, sample utility, and performance degradation, offering principled insights into balancing synthetic data complexity for optimal training. Together, these contributions establish EFCS as an effective and analytically grounded approach for bridging the gap between synthetic and real datasets.
1 Introduction
Synthetic face datasets address privacy concerns, but face-recognition models trained exclusively on them underperform real-data models amid limited variation and synthetic–real mismatch. EFCS uses ensemble feedback to generate challenging samples and offers a formulation for balancing synthetic-data complexity.
1 Introduction
- Synthetic-only FR models underperform amid insufficient intra-class variation and synthetic–real domain mismatch, especially in low-resolution, occluded, or cross-domain settings.Source: §1 · pp. 1–2
- EFCS uses ensemble FR-model feedback and controlled perturbations to generate diverse, appropriately challenging samples, as illustrated by its framework and qualitative comparison.The framework overview depicts synthesis, perturbation, and ensemble feedback.Source: §1 · p. 2, Figure 2 · p. 2, Figure 3 · p. 3
- FR models trained solely on synthetic data without guided feedback consistently fall short of real-data model performance.Source: §1 · p. 2
- The paper introduces a formulation linking perturbation-induced difficulty, sample utility, and performance degradation to inform complexity balancing for FR training.Source: §1 · p. 3
2 Literature Review
Prior controllable synthesis offers attribute control, but GAN- and diffusion-based datasets still face diversity, domain-gap, and training-challenge limitations. These shortcomings are especially relevant to complex face-recognition tasks.
2 Literature Review
- Controllable face synthesis offers fine-grained facial-attribute manipulation, motivated by privacy concerns and noisy labels or demographic imbalance in real datasets.Source: §2 · p. 3
- SFace evaluations report gaps often exceeding 25% in verification accuracy between synthetic-trained and real-trained models; SynFace adds latent-state mixing but remains predominantly frontal-view and uses real-image mixing to address the domain gap.Source: §2 · p. 3
- Diffusion models support precise attribute generation from prompts, yet struggle to provide samples sufficiently diverse and challenging for robust FR training.Source: §2 · p. 3
- Existing synthesis methods remain limited on low-resolution, pose-varying, occluded, and injured-face recognition because diversity, domain mismatch, and challenging samples remain unresolved.Source: §2 · p. 3
3 Ensemble Feedback Controllable Synthesis
EFCS uses ensemble recognition feedback to synthesize identity-preserving, controlled perturbations that increase recognition difficulty, rather than attacking models at inference time. Its pipeline combines a controllable generator, feedback-driven augmentation, and a utility formulation for balancing sample difficulty.
3 Ensemble Feedback Controllable Synthesis
- EFCS optimizes structured latent perturbations against ensemble recognition loss, targeting a controlled difficulty increase so samples are neither trivial nor excessively difficult.Source: §3 · p. 4
3.1 Proposed EFCS Architecture
- EFCS adopts CFSM’s encoder–decoder generator but adds feedback-driven generator retraining, ensemble rather than single-model feedback, and a utility objective controlling sample difficulty.Source: §3.1 · pp. 4–5
3.1.1 Stage 1: Controllable Synthesis Model
- Stage 1 encodes identity-preserving content and uses a style-modulated decoder and orthogonal latent dimensions for coherent variations such as pose, illumination, expression, and occlusion.Source: §3.1.1 · p. 5
3.1.2 Stage 2: Adversarial Augmentation with Ensemble Feedback
- Stage 2 uses per-model PGD feedback to refine identity-preserving perturbations, aggregates model-specific outputs, and selects difficult synthetic samples for balanced real–synthetic batches.Source: §3.1.2 · pp. 5–6
3.2 Analytical Perspective and Intuition
- EFCS constrains perturbations to remain semantically meaningful, while ensemble gradients reduce model-specific bias and promote perturbations that generalize across architectures and loss formulations.The robust-optimization interpretation assumes a continuous loss, a smooth data manifold, and bounded perturbations.Source: §3.2 · p. 6
4 Experimental Details
Experiments assess synthesis quality, realism, and utility with complementary metrics, using controlled dataset construction and specified synthesis, recognition-model, and training settings.
4.1 Evaluation Metrics
- Quality, realism, and utility are evaluated with FID, KID, precision, recall, SSIM, LPIPS, and DISTS.These metrics are selected to align with the EFCS objectives.Source: §4.1 · p. 6
4.2 Implementation Details
- CASIA-WebFace and WiderFace-12K images are aligned using five landmarks and resized to 112 × 112; balanced CASIA subsets use 20 real and 20 EFCS images per identity.The resulting real-only and EFCS datasets each contain 211,500 images, and the mixed dataset contains 423,000.Source: §4.2 · p. 7
- EFCS uses a synthesis network adapted from [21] to generate identity-preserving adversarial variations conditioned on a latent perturbation vector and perturbation depth.Robustness is evaluated with ArcFace, AdaFace [22], and ElasticFace [32], which use distinct loss formulations.Source: §4.2 · p. 7
- Adversarial samples use single-step PGD with ϵ = 0.314 and α = 0.314; ArcFace-ResNet50 training runs for 20 epochs on one NVIDIA A40 GPU.Training uses SGD with learning rate 0.1, momentum 0.9, weight decay 5 × 10−4, polynomial decay, and a 2-epoch warmup.Source: §4.2 · p. 7
5 Results and Analysis
EFCS produces challenging samples while retaining visual consistency, and mixed real–synthetic training performs strongly across custom and public evaluations. Analyses show that intermediate perturbation and difficulty are beneficial, while excessive distortion degrades utility and recognition.
5 Results and Analysis
- 0.7277 SER-FIQ for EFCS is close to single-feedback synthesis at 0.7264, while original images score 0.7776.SER-FIQ reflects image quality through embedding robustness and indicates visual consistency.Source: §5 · p. 7
- EFCS has lower SSIM and higher LPIPS (0.5922) and DISTS (0.3653), consistent with perceptually meaningful variations in harder samples.The table caption frames these metric shifts as a consequence of producing harder samples.Source: §5 · p. 7, Table 3 · p. 8
- 73.947 FID and 0.0152 KID for EFCS exceed single-feedback and random synthesis, while precision remains 0.7566; these shifts indicate intentional hard-region expansion, not reduced quality.Recall is 0.3152, reflecting targeted rather than uniform coverage of the real distribution.Source: §5 · pp. 7–8, Table 4 · p. 8
- 99.4% R1 on Mix and 99.2% on Real for Mixed-50%, while real-only training reaches 73.1% R1 on EFCS; mixed training is most robust overall.EFCS-only training achieves 89.6% R1 on EFCS and transfers to real data at 84.3% R1.Source: §5 · p. 8, Table 5 · p. 9
5.1 Graphical Analysis and Detailed Observations
- Utility peaks at moderate perturbation and target difficulty; excessive distortion reduces utility and recognition performance after a threshold.The degradation curve is stable under mild perturbations, then drops sharply beyond that threshold.Source: §5.1 · p. 8, Figure 4 · p. 9
5.2 Human Evaluation Study
- Among 63 participants, about 65% selected intermediate-noise samples as the hardest recognizable cases; EFCS was perceived as more challenging yet identifiable.The study compared random, single-model, and ensemble feedback conditions using similarity ratings and identity judgments.Source: §5.2 · pp. 8–9
5.3 Model Overview
- Mixed-eval models M2–M4 vary the synthetic share at 50%, 33%, and 25%; M1 is synthetic-only, M5 real-only, and M6–M8 are fine-tuned variants.The model set is designed to examine how synthetic-data quantity relates to accuracy and robustness.Source: §5.3 · p. 10
5.4 Analysis of Public Benchmark Performance
- Across six public benchmarks, mixed models are only marginally below real-only M5; on LFW, M2 scores 98.68% accuracy and M4 scores 98.83%, versus 98.88% for M5.The slight accuracy difference supports synthetic data as a supplement or alternative within the reported evaluation.Source: §5.4 · p. 10, Table 7 · p. 10
6 Conclusion
EFCS balances synthetic-sample complexity with recognizability for face recognition. The conclusion presents it as a way to generate challenging, authentic samples and support robust, privacy-conscious training.
6 Conclusion
- EFCS creates synthetic samples designed to enhance face-recognition robustness and generalization, surpassing traditional methods.The conclusion describes this as balancing sample complexity with recognizability.Source: §6 · p. 11
- SSIM, LPIPS, and DISTS evaluations confirm EFCS’s capability to generate challenging yet authentic samples.Source: §6 · p. 11
- EFCS addresses limitations in existing synthetic datasets and provides analytical motivation for optimized noise injection.Source: §6 · p. 11
- By producing controlled data that closely emulates real-world conditions, EFCS offers a robust, privacy-conscious alternative for training face-recognition models.Source: §6 · p. 11