PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaFace: Quality Adaptive Margin for Face Recognition

Minchul Kim, Anil K. Jain, Xiaoming Liu

arXiv:2204.00964v2cs.CV

TL;DR

Low-quality face recognition is difficult because degradation can obscure facial attributes and identity information, while hard-sample emphasis may overfocus on unidentifiable images. AdaFace adapts the margin function using feature norms as image-quality proxies, emphasizing hard or easy samples according to quality. The method improves recognition over the state of the art on mixed and low-quality datasets, while its loss does not specially handle mislabeled samples.

  • Problem

    Low-quality face images challenge recognition, and naively emphasizing hard samples can overemphasize unidentifiable images lacking identity clues.

  • Method

    AdaFace uses feature norms as image-quality proxies and adaptively changes the margin function to control the importance of samples with different difficulties.

  • Results

    AdaFace achieves state-of-the-art performance on mixed and low-quality face datasets and improves low-quality recognition while maintaining high-quality performance.

  • Takeaways & Limitations

    Quality-aware adaptive margins let the method avoid emphasizing unidentifiable images while focusing on hard yet recognizable samples.

  • Takeaways & Limitations

    The loss does not specially treat mislabeled samples, so difficult high-quality mislabeled images can be wrongly emphasized.

Abstract

from arXiv · show

Recognition in low quality face datasets is challenging because facial attributes are obscured and degraded. Advances in margin-based loss functions have resulted in enhanced discriminability of faces in the embedding space. Further, previous studies have studied the effect of adaptive losses to assign more importance to misclassified (hard) examples. In this work, we introduce another aspect of adaptiveness in the loss function, namely the image quality. We argue that the strategy to emphasize misclassified samples should be adjusted according to their image quality. Specifically, the relative importance of easy or hard samples should be based on the sample's image quality. We propose a new loss function that emphasizes samples of different difficulties based on their image quality. Our method achieves this in the form of an adaptive margin function by approximating the image quality with feature norms. Extensive experiments show that our method, AdaFace, improves the face recognition performance over the state-of-the-art (SoTA) on four datasets (IJB-B, IJB-C, IJB-S and TinyFace). Code and models are released in https://github.com/mk-minchul/AdaFace.

1. Introduction

Low-quality face images can lose identity information and undermine recognition training, motivating AdaFace’s quality-aware sample weighting. The method uses feature norms to adapt margins, emphasizing hard recognizable samples while avoiding unidentifiable ones.

  • Motivation: Low-quality images are increasingly important in surveillance and drone datasets, while many training datasets remain high quality.This motivates evaluation on challenging datasets such as IJB-B, IJB-C and IJB-S.
  • Motivation: Low-quality face images can become unrecognizable when degradation removes identity information.Such images may cause models to exploit cues like clothing color or resolution instead of identity.
  • Contribution: AdaFace adjusts sample importance by image quality, emphasizing hard high-quality samples and easy low-quality samples.The goal is to avoid emphasizing unidentifiable images while focusing on hard yet recognizable examples.
  • Contribution: AdaFace combines feature norms as image-quality proxies with adaptive margin functions in a unified loss.This avoids requiring a separate image-quality estimation module.
  • Evaluation: The method is evaluated across nine datasets of varying quality, with improved low-quality recognition while maintaining high-quality performance.The listed datasets include LFW, CFP-FP, CPLFW, AgeDB, CALFW, IJB-B, IJB-C, IJB-S and TinyFace.

2. Related Work

Prior face-recognition losses use fixed or training-progress-based margins, whereas AdaFace motivates adapting margins to image quality because hard low-quality samples may lack identity clues.

  • Margin-Based Losses: Margin-based softmax losses add margins to improve feature discriminability, with SphereFace, CosFace and ArcFace using different margin forms.The general formulation applies a margin function to the ground-truth class while scaling logits.
  • Margin-Based Losses: The margin formulations differ across SphereFace, CosFace and ArcFace through multiplicative, additive, and angular adjustments.The supplied equations instantiate these distinct forms for the ground-truth class.
  • Margin-Based Losses: Angular margins scale the learning signal according to sample difficulty, which relates to proximity to the ground-truth classifier weight.The derivative is monotonic with respect to cos θ_yi when the margin is positive.
  • Adaptive Losses: Adaptive losses have addressed hard-sample mining, curriculum scheduling, and hyperparameter selection, while CurricularFace adapts margins with training progression.Its parameter t increases as training progresses.
  • Quality-Aware Adaptation: AdaFace argues that hard samples should receive different emphasis by quality: hard high-quality images are useful, but hard low-quality images may lack identity clues.This contrasts with always emphasizing hard samples.
  • Low-Quality Face Recognition: Related low-quality face-recognition methods use probabilistic uncertainty, rich subspaces, auxiliary context, or synthetic augmentation, whereas AdaFace modifies conventional softmax loss.The paper presents this modification as easier to use during training.

3. Proposed Approach

AdaFace combines gradient-based sample weighting with feature-norm-based image-quality estimation to adapt margin functions during face-recognition training. The resulting loss emphasizes difficult, recognizable samples while reducing emphasis on very hard, low-quality samples.

  • 3.1. Margin Form and the Gradient: Margin functions affect sample importance by scaling the gradient during backpropagation, not only by shifting decision boundaries.The gradient scaling term depends on the margin function and sample difficulty.
  • 3.1. Margin Form and the Gradient: Positive angular margins reduce emphasis on very hard samples, whereas negative angular margins induce the opposite behavior.ArcFace and MagFace use positive angular-margin behavior; AdaFace combines positive, negative, and additive margin functions when needed.
  • 3.2. Norm and Image quality: Feature norm is used as a proxy for image quality, avoiding an additional image-quality estimation module during training.The feature norm correlates with the image-quality score and has higher correlation than the ground-truth probability output, including from an early training stage.
  • 3.3. AdaFace: Adaptive Margin based on Norm: AdaFace adaptively changes the margin function according to normalized feature norm to assign different emphasis across sample difficulties.High-norm features receive greater gradient scale away from the decision boundary, while low-norm features receive greater scale near it.
  • 3.3. AdaFace: Adaptive Margin based on Norm: For low-norm features, AdaFace de-emphasizes harder samples away from the decision boundary, while high-norm features receive stronger emphasis there.The adaptive function becomes ArcFace at d∥z_i∥ = -1, CosFace at d∥z_i∥ = 0, and a shifted negative angular margin at d∥z_i∥ = 1.

4. Experiments

Experiments evaluate AdaFace across datasets spanning high, mixed, and low visual quality, using ablations, training analyses, and comparisons with state-of-the-art methods. The results support feature norms as an image-quality proxy and show stronger low-quality recognition while preserving high-quality performance.

  • Datasets and implementation: AdaFace is evaluated on 9 datasets grouped into high-, mixed-, and low-quality settings, using ResNet-based backbones and specified training protocols.Training uses MS1MV2, MS1MV3, or WebFace4M; evaluation includes LFW, CFP-FP, CPLFW, AgeDB, CALFW, IJB-B, IJB-C, IJB-S, and TinyFace.
  • Augmentation: On-the-fly cropping, rescaling, and photometric jittering introduce unidentifiable images but are designed to test whether AdaFace can handle them.The augmentations are applied randomly and create a trade-off between additional data and degraded samples.
  • Ablation: When h = 0.33, AdaFace performs best, while h = 0.22 and h = 0.66 still outperform CurricularFace.The concentration parameter is not very sensitive when the normalized feature norm retains variation.
  • Ablation: AdaFace uses m = 0.4 in subsequent experiments because it performs well on low-quality datasets without sacrificing high-quality performance.The best high-quality result occurs at m = 0.4, while the best low-quality result occurs at m = 0.75.
  • Augmentation: AdaFace augmentation keeps high-quality performance unchanged while significantly improving low-quality performance, whereas augmentation hurts CurricularFace.The authors attribute this difference to AdaFace avoiding overfitting on unidentifiable images.
  • Analysis: Low-norm samples often remain low-probability through training, and 62.0% of samples below the transition point include augmentation versus 38.5% above it.These trajectories support treating low-norm, hard samples differently from hard but recognizable samples.
  • Comparison with SoTA: AdaFace performs on par with competitive methods on high-quality datasets and achieves SoTA performance across varied image qualities and training sets.The method does not require additional learnable layers or two-stage training.
  • Comparison with SoTA: 11% and 9% relative error reductions on IJB-B and IJB-C, respectively, demonstrate gains on mixed-quality datasets.For low-quality datasets, AdaFace’s averaged gain over the second-best method is 3.5% on four Rank-1 metrics and 2.4% on three TPIR@FPIR=1% metrics.

5. Conclusion

AdaFace addresses unidentifiable training images by using feature norms as image-quality proxies and adaptively changing the margin function to control gradient emphasis. It achieves state-of-the-art performance on mixed- and low-quality face datasets, but does not specially handle mislabeled samples.

  • 5. Conclusion: AdaFace uses feature norms as image-quality proxies and changes the margin function adaptively to control gradient scale across image qualities.The method targets unidentifiable images introduced by data collection or augmentation.
  • 5. Conclusion: AdaFace achieves state-of-the-art results on mixed- and low-quality face datasets.
  • 5. Conclusion: The loss does not specially treat mislabeled samples, so high-quality mislabeled images can receive large importance as difficult samples.The authors identify joint handling of unidentifiability and label noise as future work.

Supplementary Material

The supplementary material defines the gradient scaling term for margin-based softmax losses and contrasts AdaFace's treatment of feature norms with MagFace's optimization of them.

  • Supplementary Material: The gradient scaling term g determines the gradient magnitude during backpropagation and depends on the selected margin function.For the ground-truth index, the supplementary table summarizes each margin function and its corresponding gradient scale term.
  • Supplementary Material: AdaFace treats the feature norm ∥z_i∥ as constant when analyzing the margin effect, unlike MagFace's optimization treatment.The table notes that MagFace's exact gradient scale term therefore differs from the constant-norm analysis.
  • Supplementary Material: The ground-truth probability P_yi also changes with the margin function, causing g to depend on the margin parameter through f(cos θ_yi).For Angular Margin, the margin appears directly in the gradient-scale expression.

A.1. Derivation of Angular Margin

The angular-margin derivation rewrites the ground-truth logit using trigonometric identities and explains how its derivative contributes to difficulty-dependent gradient scaling.

  • A.1. Derivation of Angular Margin: For Angular Margin, the ground-truth function expands as s(cos θ_yi cos m − sin θ_yi sin m).The expression is further rewritten using sin θ_yi = sqrt(1 − cos^2 θ_yi).
  • A.1. Derivation of Angular Margin: The derivative of the Angular Margin function with respect to cos θ_yi is s(cos(m) + cos θ_yi sin(m) / sqrt(1 − cos^2 θ_yi)).This derivative is reported as Equation 4.
  • A.1. Derivation of Angular Margin: With the commonly used s = 64, Softmax's gradient scaling becomes nearly equal for samples far from the decision boundary.The corresponding curve is described as flat except near the decision boundary.
  • A.1. Derivation of Angular Margin: For m > 0, the Angular Margin derivative increases with cos θ_yi, which is related to sample difficulty during training.The derivative can therefore be viewed as scaling sample importance according to difficulty.

B.1. Correlation between Norm and BRISQUE during Training

The supplementary analysis examines the relationship between feature norm and image quality for ArcFace and AdaFace, while distinguishing AdaFace's quality proxy from MagFace's recognizability-oriented norm.

  • B.1. Correlation between Norm and BRISQUE during Training: ArcFace and AdaFace both show a feature-norm trend correlated with image quality during training.The comparison uses 1,534 randomly sampled MS1MV2 training images.
  • B.1. Correlation between Norm and BRISQUE during Training: AdaFace uses feature norm to adjust emphasis by sample difficulty, whereas MagFace aligns feature norm with recognizability through gradient flow.AdaFace treats the norm as constant, while MagFace optimizes it as part of the learned representation.
  • B.1. Correlation between Norm and BRISQUE during Training: AdaFace substantially outperforms MagFace on the reported comparisons, including reduced MagFace errors on IJB-B and IJB-C.

B.2. Training Sample Visualization

Training samples are visualized by partitioning feature angle and feature norm into six zones, with representative images drawn from each zone. A pretrained AdaFace model supplies the features.

  • B.2. Training Sample Visualization: A pretrained AdaFace model is used as the feature extractor for the six-zone examples.
  • B.2. Training Sample Visualization: 1,534 MS1MV2 training images are plotted across six zones using cos θyi on the x-axis and feature norm ∥zi∥ on the y-axis.A few images are sampled from each zone for visualization.
  • B.2. Training Sample Visualization: The gray top-right and bottom-left zones contain relatively few samples.The visualization highlights sparsity in these combinations of angle and feature norm.

B.3. Training Samples’ Gradient Scaling Term for AdaFace

The visualizations examine AdaFace’s gradient scaling term across training samples and compare it with ArcFace. Test examples illustrate cases where AdaFace succeeds on difficult IJB-C matches that ArcFace misses.

  • B.3. Training Samples’ Gradient Scaling Term for AdaFace: The GST visualization colors 1,534 samples by GST magnitude and represents each sample in angular space using cos θyi and ∥zi∥.Angular position uses the angle from Wyi, while radial distance uses the feature norm; both are normalized for visualization.
  • B.3. Training Samples’ Gradient Scaling Term for AdaFace: Figure 7 compares GST magnitudes for unrecognizable images and hard yet recognizable images under ArcFace and AdaFace.Each bar represents one training sample, with height indicating GST magnitude.
  • B.3. Training Samples’ Gradient Scaling Term for AdaFace: AdaFace successfully finds correct IJB-C gallery matches in examples where ArcFace fails, with many such successes involving low-quality samples.The figure pairs probe images with gallery images and reports rank and similarity scores for both methods.

D. Comparison with General Image-Quality Aware Learning Method

AdaFace is compared with a general image-quality-aware method and evaluated for sensitivity to batch size. The comparison emphasizes low-quality-image handling and reports minimal performance variation across tested batch sizes.

  • D. Comparison with General Image-Quality Aware Learning Method: AdaFace outperforms QualNet on the TinyFace test set.QualNet aligns low-quality features to high-quality features with a fixed pretrained decoder, whereas AdaFace de-emphasizes heavily degraded low-quality images.
  • D. Comparison with General Image-Quality Aware Learning Method: AdaFace’s image-quality proxy based on ∥zi∥ does not depend on batch size because it uses an exponential moving average.
  • D. Comparison with General Image-Quality Aware Learning Method: The performance difference across batch sizes 128, 256, and 512 is minimal on IJB-B TAR@FAR=0.01%.The experiment trains R50 models on MS1MV2 and evaluates them on IJB-B.

F. Implementation Details and Code

The implementation releases code and documents the external components used for training, backbone definition, evaluation, and dataset preprocessing.

  • F. Implementation Details and Code: AdaFace code is released at https://github.com/mk-minchul/AdaFace.
  • F. Implementation Details and Code: InsightFace and related repositories are used for MS1MV2 preprocessing, backbone definition, and evaluation on several face-recognition benchmarks.
  • F. Implementation Details and Code: MTCNN is used to align faces when preprocessing IJB-S and TinyFace.
Loading 2204.00964v2…