PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Universal Representation Learning for Deep Face Recognition

Yichun Shi, Xiang Yu, Kihyuk Sohn, Manmohan Chandraker, Anil K. Jain

arXiv:2002.11841v1cs.CV

TL;DR

Face recognition degrades under variations that are underrepresented in training data, while existing solutions may require target-domain information or added model complexity. The paper trains a universal representation with synthetic variations, confidence-aware sub-embeddings, and decorrelation regularization, achieving strong general-benchmark performance and better results on challenging datasets.

  • Problem

    Face-recognition models face large performance drops on challenging variations, while existing approaches may require target-domain distributions or model ensembles.

  • Method

    The framework synthetically augments training variations, uses sample- and sub-embedding-specific confidence, decorrelates sub-embeddings, and aggregates them with uncertainty modeling.

  • Results

    The method maintains high accuracy on general benchmarks and achieves state-of-the-art or top performance across challenging benchmarks including IJB-C, IJB-S, TinyFace, and IJB-A.

  • Takeaways & Limitations

    A single representation can generalize across diverse face-recognition conditions without using target-domain data information.

Abstract

from arXiv · show

Recognizing wild faces is extremely hard as they appear with all kinds of variations. Traditional methods either train with specifically annotated variation data from target domains, or by introducing unlabeled target variation data to adapt from the training data. Instead, we propose a universal representation learning framework that can deal with larger variation unseen in the given training data without leveraging target domain knowledge. We firstly synthesize training data alongside some semantically meaningful variations, such as low resolution, occlusion and head pose. However, directly feeding the augmented data for training will not converge well as the newly introduced samples are mostly hard examples. We propose to split the feature embedding into multiple sub-embeddings, and associate different confidence values for each sub-embedding to smooth the training procedure. The sub-embeddings are further decorrelated by regularizing variation classification loss and variation adversarial loss on different partitions of them. Experiments show that our method achieves top performance on general face recognition datasets such as LFW and MegaFace, while significantly better on extreme benchmarks such as TinyFace and IJB-S.

1. Introduction

The paper targets face-recognition performance drops caused by variation gaps, proposing a single universal representation trained without target-domain data. It combines synthetic variation augmentation, confidence-aware sub-embeddings, decorrelation regularization, and uncertainty-aware aggregation.

  • Motivation: Prior face-recognition systems suffer substantial drops on challenging datasets because large public training sets still contain biases and limited variation.Prior state-of-the-art accuracy on IJB-S or TinyFace is reported as about 30% lower than on IJB-A or LFW.
  • Motivation: Existing approaches handle specific variations, require test-domain distributions, or add runtime complexity through model ensembles.The proposed framework instead seeks one representation that works across diverse situations without test-distribution access.
  • Method: The method augments training with non-frontal pose, low resolution, and heavy occlusion, then uses confidence-aware identification loss to learn from hard examples.The confidence is made sample- and sub-embedding-specific to address the harder optimization introduced by augmentation.
  • Method: Sub-embeddings are decorrelated using variation classification and variation adversarial losses applied to different partitions.The framework also mines additional training variations and aggregates sub-embeddings probabilistically according to their uncertainties.
  • Results: The method maintains high accuracy on general benchmarks and achieves state-of-the-art performance on challenging datasets including IJB-C and IJB-S.The paper reports extensive public-dataset evaluation and ablations of the proposed contributions.

2. Related Work

Related work spans deep face-recognition loss and dataset design, universal representations, and methods for handling domain variation. This paper positions its approach as a model-agnostic similarity-metric method that applies without retraining on unseen domains.

  • Deep Face Recognition: Deep face-recognition research has improved feature discrimination through neural networks, loss functions, and datasets containing within-class variation.The related work discusses methods such as center loss and metric learning.
  • Universal Representation: Universal representation learning seeks a single model that maintains performance across varied visual domains instead of using domain-specific models.The paper distinguishes this setting from domain generalization, which learns from multiple seen domains to adapt to unseen ones.
  • Universal Representation: The proposed work differs from prior universal-representation methods by addressing similarity metric learning rather than detection or classification and by being model-agnostic.Its features can be applied directly to different domains without retraining on unseen domains.

3. Proposed Approach

The method augments training faces with blur, occlusion, and pose, then learns confidence-aware sub-embeddings to handle hard samples and decorrelates them by variation-specific regularization.

  • Training images are augmented with blur, occlusion, and head-pose variations before representation learning.
  • Confidence-aware identification loss: Each sample is modeled with a probabilistic embedding whose confidence controls how identification training uses its feature.The confidence-aware formulation uses sample-specific confidence and constrains confidence growth with an additional regularization term.
  • Confidence-aware sub-embeddings: The feature vector is split into equal-length sub-embeddings, with corresponding partitioned prototypes and confidence values.Each sub-embedding is separately normalized before the final identification loss is applied.
  • Sub-embeddings decorrelation: Variation decorrelation assigns different masked sub-embedding subsets to variations, using classification to promote variation specificity and adversarial loss to promote invariance to others.For each known variation, a fixed mask selects half of the sub-embeddings; distinct masks are used for different variations.
  • Mining more variations: Additional semantic variation labels are mined from CelebA attributes and merged with the original augmentation labels for decorrelation learning.The attribute classifier is trained with an identity-adversarial loss, then applied to the recognition training set to produce soft labels such as smiling or young.

4. Implementation Details

The implementation uses a cleaned MS-Celeb-1M training set, a modified 100-layer ResNet, and 16 feature groups for multi-embedding models. Experiments also synthesize blur, occlusion, and pose variations for testing.

  • Training Details: Training uses 4.8M images from 76.5K classes after cleaning overlapping subjects from MS-Celeb-1M.Images are aligned, cropped to 100 × 100, and trained with a modified 100-layer ResNet.
  • Model Configuration: All models use 512-dimensional embeddings, while multi-embedding models split features into 16 groups.The baseline uses CosFace loss, and domain-augmented models train for 27 epochs to ensure convergence.
  • Dataset Examples: Figure 7 samples LFW, IJB-A, and IJB-S as datasets representing three levels of face-image variation.These examples correspond to general, mixed-quality, and low-quality testing scenarios.
  • Synthetic Variation Testing: Figure 8 evaluates synthetic IJB-A data with blur, occlusion, and pose variations across training augmentation strategies.The proposed method improves monotonically as more augmentations are added.

5. Experiments

Experiments span eight benchmarks grouped by variation level, then assess augmentation, confidence-aware learning, decorrelation, aggregation, and performance on challenging datasets. The proposed framework preserves general-benchmark accuracy while improving results under mixed and low-quality conditions.

  • 5.1. Datasets: The evaluation covers eight benchmarks divided into limited-variation, mixed-quality, and low-quality dataset types.LFW, CFP, YTF, and MegaFace form Type I; IJB-A and IJB-C form Type II; TinyFace and IJB-S form Type III.
  • 5.2.1 Effect of Confidence-aware Learning: The proposed model improves as additional blur, occlusion, and pose augmentations are introduced, unlike the baseline’s declining performance.The controlled experiment applies corresponding synthetic variations to IJB-A testing data.
  • 5.2.1 Effect of Confidence-aware Learning: t-SNE features from the proposed model keep augmented samples clustered with their original identity samples, whereas baseline features become mixed.This indicates that the proposed representation preserves identity under augmentation.
  • 5.2.2 Ablation on All Modules: Confidence-aware learning reduces the negative impact of hard augmented examples, while decorrelation and probabilistic aggregation combine complementary sub-embedding features.Multiple embeddings alone provide little improvement because their confidence values remain strongly correlated.
  • 5.3. Evaluation on General Datasets: The method remains near the top on limited-variation datasets, despite having limited advantage when testing images are mostly high quality.It is better than most compared methods but slightly worse than ArcFace.
  • 5.4. Evaluation on Mixed/Low Quality Datasets: Uncertainty visualizations show distinct patterns across variations, with brighter regions representing higher uncertainty among the 16 sub-embeddings.The values are arranged in 4 × 4 grids without spatial meaning.
  • 5.4. Evaluation on Mixed/Low Quality Datasets: On IJB-A, IJB-C, and IJB-S, the full method consistently outperforms state-of-the-art methods, while variation augmentation alone can worsen IJB-A and IJB-C.Adding probabilistic aggregation further improves performance across challenging datasets and protocols.
  • 5.4. Evaluation on Mixed/Low Quality Datasets: 63.89% rank-1 and 68.67% rank-5 accuracy are achieved on TinyFace, exceeding the reported 44.80% and 60.40% and ArcFace’s 47.39% and 52.28%.The comparison uses rank-1 and rank-5 identification accuracy.

6. Conclusion

The paper proposes a universal face representation framework that combines augmentation, confidence-aware sub-embeddings, decorrelation, and uncertainty-based aggregation. It reports top performance on general benchmarks and significantly better accuracy on challenging benchmarks.

  • Conclusion: The framework augments training with blur, occlusion, and head-pose variations to recognize faces across diverse conditions.It targets recognition without relying on target-domain variation data.
  • Conclusion: Confidence-aware learning partitions embeddings into sub-embeddings with sample- and sub-embedding-specific confidence values.Variation classification and adversarial losses decorrelate the sub-embeddings before uncertainty-based aggregation.
  • Conclusion: The method achieves top performance on general benchmarks and significantly better accuracy on IJB-A, IJB-C, and IJB-S.These results support its performance under challenging variations.

A. Proofs

The proofs derive posterior identity probabilities for latent and sub-embedding representations, then show that confidence values gate gradients during training. This makes confident samples exert greater influence on prototypes and corresponding sub-embeddings.

  • A.1. Proofs: The latent embedding z represents facial appearance, while each identity j is represented by a prototype vector w_j.The class-conditional distribution is modeled as a Dirac delta centered at the identity prototype.
  • A.1. Proofs: The posterior identity probability is derived by integrating the likelihood p(x_i|z) against the class-specific latent distribution.The derivation is identified as Equation (4) in the main paper.
  • A.1. Proofs: For a sub-embedding network, the same derivation produces a sub-embedding likelihood and posterior classification probability.Incorporating the margin m and averaging scores yields the corresponding main-paper formulation.
  • A.2. Gradient of the sub-embeddings: The gradient of prototypes and sub-embeddings depends on both confidence values and posterior classification probabilities.The posterior probability is p(y=j|x_i), with δ_yi,j indicating whether the sample belongs to class j.
  • A.2. Gradient of the sub-embeddings: Confidence acts as a back-propagation gate, giving confident samples larger influence on prototypes and confident sub-embeddings greater impact on prototypes.This mechanism explains how confidence weighting changes the training signal.

B. Additional Implementation Details

The embedding network uses two branches: one produces a 512-D representation split into 16 sub-embeddings, and the other produces their confidence values. A separate four-layer CNN mines additional variations.

  • B. Additional Implementation Details: A modified 100-layer ResNet splits after its final convolution layer into two fully connected branches.One branch produces representations, while the other produces confidence values.
  • B. Additional Implementation Details: The representation branch outputs a 512-D vector divided into 16 sub-embeddings.The separate branch outputs a 16-D vector containing one confidence value for each sub-embedding.
  • B. Additional Implementation Details: The confidence branch uses the exp function to ensure that all sub-embedding confidence values are positive.These confidence values parameterize the probabilistic representation.
  • B. Additional Implementation Details: The variation-mining model is a four-layer CNN with 64, 128, 256, and 512 kernels across successive layers.All convolutional kernels are 3 × 3.

C. Ablation Study on Variation Decorrelation Loss

The ablation varies how many augmentable and additional variations are included in the variation decorrelation loss. Adding more variations makes decorrelation more effective and improves performance.

  • C. Ablation Study on Variation Decorrelation Loss: The ablation compares models trained with different numbers of variations in the variation decorrelation loss.All models retain the paper’s other proposed modules.
  • C. Ablation Study on Variation Decorrelation Loss: The tested augmentable variations are blur, occlusion, and pose, while additional variations include gender, age, and smiling.The comparisons progressively add these variation types.
  • C. Ablation Study on Variation Decorrelation Loss: More variations make the decorrelation more effective and lead to better performance.This conclusion is drawn from the Table 4 comparisons.

D. Additional Results on IJB-S

Additional IJB-S analyses compare protocols and models, examine confidence distributions and uncertainty, and visualize embeddings. The results connect confidence specialization with complementary sub-embeddings and robustness to facial variation.

  • D. Additional Results on IJB-S: IJB-S evaluates surveillance-to-single and surveillance-to-booking protocols with different gallery-template constructions.The former uses one gallery image, while the latter uses multiple images with different poses.
  • D. Additional Results on IJB-S: The proposed method significantly outperforms its baseline on all IJB-S metrics and exceeds state-of-the-art methods on most metrics.These comparisons support the efficacy of the proposed method on IJB-S.
  • D. Additional Results on IJB-S: Different sub-embeddings learn distinct confidence distributions and assign high or low confidence to different image types.Because confidence guides the corresponding feature’s training signal, the sub-embeddings learn complementary features.
  • D. Additional Results on IJB-S: Images with larger facial variations show low uncertainty on some sub-embeddings, whereas extremely varied images show high uncertainty across all sub-embeddings.Distinguishable faces generally have low uncertainty on most sub-embeddings.
  • D. Additional Results on IJB-S: The t-SNE visualization compares baseline and proposed embeddings for original and augmented samples.The baseline assumes equal confidence and fails to converge to a good local minimum when hard augmented samples cluster near noise.

H. Image Examples From the Testing Datasets

The figures illustrate sub-embedding confidence and uncertainty, feature distributions, and testing-dataset examples. The accompanying discussion contrasts relatively uniform LFW images with more varied and challenging IJB-A images, while Table 5 specifies IJB-S evaluation protocols.

  • LFW images are mostly high-quality faces with limited variation, so the evaluated models achieve similar performance on this dataset.
  • IJB-A images contain greater variation, including extremely challenging cases requiring cross-domain matching between high- and low-quality images.
  • Table 5 compares IJB-S rank retrieval and TPIR@FPIR, with 1% and 10% denoting FPIR and “+” indicating results from released author models.
  • Figures 12–14 visualize sub-embedding confidence on training samples and uncertainty on testing images.
  • Figure 15 uses t-SNE to show features in two dimensions, with colors denoting identities and circles versus triangles distinguishing original from augmented training samples.
Loading 2002.11841v1…