PaperlayerView exact revision on arXiv

Source-linked AI summary

SphereFace: Deep Hypersphere Embedding for Face Recognition

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song

arXiv:1704.08063v4cs.CV

TL;DR

Open-set face recognition requires discriminative large-margin features, but existing approaches do not effectively achieve the desired separation. SphereFace introduces A-Softmax to learn angularly discriminative features on a hypersphere with adjustable margin, achieving competitive results on LFW, YTF, and MegaFace Challenge 1.

  • Problem

    Open-set face recognition needs discriminative large-margin feature spaces, while few existing algorithms effectively achieve the desired intra-class and inter-class separation.

  • Method

    A-Softmax trains CNNs with an adjustable angular margin that constrains features on a hypersphere manifold.

  • Results

    SphereFace achieves competitive results on LFW, YTF, and MegaFace Challenge 1.

  • Takeaways & Limitations

    Angular margin provides an effective hypersphere-based approach for learning discriminative face features in face recognition.

Abstract

from arXiv · show

This paper addresses deep face recognition (FR) problem under open-set protocol, where ideal face features are expected to have smaller maximal intra-class distance than minimal inter-class distance under a suitably chosen metric space. However, few existing algorithms can effectively achieve this criterion. To this end, we propose the angular softmax (A-Softmax) loss that enables convolutional neural networks (CNNs) to learn angularly discriminative features. Geometrically, A-Softmax loss can be viewed as imposing discriminative constraints on a hypersphere manifold, which intrinsically matches the prior that faces also lie on a manifold. Moreover, the size of angular margin can be quantitatively adjusted by a parameter $m$. We further derive specific $m$ to approximate the ideal feature criterion. Extensive analysis and experiments on Labeled Face in the Wild (LFW), Youtube Faces (YTF) and MegaFace Challenge show the superiority of A-Softmax loss in FR tasks. The code has also been made publicly available.

1. Introduction

The paper targets open-set face recognition by learning angularly discriminative features whose maximal intra-class distance is smaller than minimal inter-class distance. It proposes A-Softmax loss, which imposes adjustable angular margins on a hypersphere manifold and achieves competitive benchmark results.

  • Open-set face recognition: Open-set face recognition requires mapping identities unseen during training into a discriminative feature space rather than classifying them to known identities.Face identification becomes verification between a probe face and every gallery identity.
  • Open-set face recognition: The desired open-set criterion is that maximal intra-class distance be smaller than minimal inter-class distance, but large intra-class variation and high inter-class similarity make it difficult.This criterion is necessary for perfect nearest-neighbor accuracy.
  • Motivation: Euclidean-margin losses are poorly motivated with softmax because softmax-learned features have intrinsic angular distributions.The paper argues that combining these losses can be incompatible with softmax.
  • A-Softmax loss: A-Softmax loss introduces an integer m (m≥1) that quantitatively controls angular margins and separates decision regions for multiple classes.Its binary-class boundaries use cos(mθ1) or cos(mθ2), depending on the class.
  • Geometric interpretation: A-Softmax constrains features to be discriminative on a hypersphere manifold, with angular distance equivalent to geodesic distance and a geometry matching the prior that faces lie on a manifold.The method also derives lower bounds for m to approximate the desired open-set distance criterion.
  • Experimental results: Trained on the publicly available CASIA dataset, SphereFace achieves competitive results on LFW, YTF, and MegaFace Challenge 1.The paper presents this as the first demonstration of angular-margin effectiveness in face recognition.

2. Related Work

Prior work learns distances or discriminative face embeddings using metric-learning objectives and CNN-based supervision. A-Softmax differs by explicitly imposing an adjustable angular margin on a hypersphere to target open-set face recognition.

  • Metric learning: Traditional metric learning learns a matrix A that defines a distance over given features.Its metric is expressed as ∥x1−x2∥A = (x1−x2)^T A(x1−x2).
  • Metric learning: Deep metric learning uses neural networks to learn discriminative features followed by a simple Euclidean distance.Contrastive and triplet losses impose Euclidean margins on the learned features.
  • Deep face recognition: Deep face recognition has treated open-set recognition as multi-class classification or learned face embeddings with contrastive, softmax, or triplet supervision.These approaches include jointly combining contrastive and softmax losses and training a unified face embedding with triplet loss.
  • Deep face recognition: L-Softmax implicitly involves angles and improves closed-set classification, whereas A-Softmax learns discriminative face embeddings for open-set recognition.A-Softmax explicitly connects the learned features to a hypersphere manifold and imposes an angular margin that can be quantitatively controlled.

3. Deep Hypersphere Embedding

This section reformulates softmax classification in angular terms and introduces A-Softmax loss to impose an adjustable angular margin on a hypersphere manifold. Larger m produces stricter, more discriminative constraints and can enforce smaller maximal intra-class than minimal inter-class angular distance.

  • Angular reformulation: After weight normalization and bias removal, softmax classification depends only on the angles between features and class weights.The decision boundary becomes cos(θ1)−cos(θ2)=0.
  • A-Softmax loss: A-Softmax replaces cos(θ1)>cos(θ2) with the stricter requirement cos(mθ1)>cos(θ2), where m≥2 controls angular discrimination.The resulting class-1 boundary is cos(mθ1)=cos(θ2).
  • A-Softmax loss: m≥1 is an integer controlling angular-margin size; when m=1, A-Softmax reduces to modified softmax loss.The angular margin increases with larger m and is zero when m=1.
  • Hypersphere interpretation: A-Softmax learns features discriminatively on a hypersphere manifold, with larger m yielding smaller hypercircle-like correctly classified regions.The decision boundary is mω1=ω2, and class-1 classification requires mω1<ω2.
  • Properties: With larger m, the angular margin grows, the constrained manifold region shrinks, and the learning task becomes more difficult.A-Softmax therefore defines a large angular-margin learning task with adjustable difficulty.
  • Properties: There exists a minimal m such that maximal intra-class angular distance is constrained below minimal inter-class angular distance; for the multi-class case, mmin≥3.This lower bound is obtained by solving the corresponding multi-class inequality.

4. Experiments (more in Appendix)

Experiments evaluate SphereFace with controlled CNN architectures and training protocols, showing that larger angular margins and A-Softmax consistently improve face-recognition performance. SphereFace achieves strong results on LFW, YTF, and MegaFace despite using a relatively small public training dataset.

  • Effect of m: Larger m produces more discriminative hyperspherical feature distributions and improves A-Softmax accuracy on LFW and YTF.The comparison uses a 64-layer CNN for all losses.
  • Effect of CNN architectures: 1.54%∼1.91%: A-Softmax consistently outperforms standard softmax across CNN architectures with different depths.The experiments train A-Softmax with m=4 and compare it against original softmax loss.
  • Effect of CNN architectures: 98.20% to 99.42% on LFW and 93.4% to 95.0% on YTF: A-Softmax benefits substantially from deeper architectures.Standard CNN improvements are smaller, reaching 97.75% on LFW and 93.1% on YTF.
  • LFW and YTF: 99.42% and 95.0%: SphereFace achieves these accuracies on LFW and YTF using a single model trained on publicly available CASIA-WebFace.CASIA-WebFace contains 494,414 images from 10,575 individuals and is described as relatively small with noisy labels.
  • LFW and YTF: SphereFace consistently outperforms softmax, contrastive, center, triplet, and L-Softmax features when all use the same 64-layer CNN architecture.The comparison is conducted for face-recognition tasks on LFW and YTF.
  • MegaFace: 4.8% and 6.3%: SphereFace’s 3-patch ensemble exceeds the second-best MegaFace results in rank-1 identification and verification, respectively.Under the small training dataset protocol, it also exceeds models trained on large datasets by 0.64% in identification rate and 1.4% in verification rate.

5. Concluding Remarks · Appendix

The paper presents SphereFace, a deep hypersphere embedding approach for face recognition based on angular softmax loss. A-Softmax learns angularly discriminative face features by constraining them on a hypersphere manifold that matches the prior that faces lie on a non-linear manifold.

  • 5. Concluding Remarks: SphereFace introduces a deep hypersphere embedding approach for face recognition.The approach is presented as novel.
  • 5. Concluding Remarks: The method proposes angular softmax loss for CNNs to learn discriminative face features with an angular margin.Angular margin is the defining constraint of the proposed loss.
  • 5. Concluding Remarks: A-Softmax provides a geometric interpretation by constraining learned features to be discriminative on a hypersphere manifold.The geometric view connects the loss to the structure of the learned feature space.
  • 5. Concluding Remarks: The hypersphere constraint intrinsically matches the prior that faces lie on a non-linear manifold.This correspondence links the optimization geometry to the assumed structure of face data.

A. The intuition of removing the last ReLU

The section motivates removing the last ReLU because standard CNN features are restricted to the non-negative range, limiting feasible angular learning space. SphereFace therefore removes this nonlinearity, and MNIST visualization shows more reasonable feature distributions without it.

  • The intuition of removing the last ReLU: Standard CNN features after the last ReLU distribute only in the non-negative range [0, +∞), limiting the CNNs’ feasible learning space.The limitation is described specifically in terms of the available angle for feature learning.
  • The intuition of removing the last ReLU: SphereFace and [16] remove the ReLU connected to the bottom of FC1 to address this restriction and provide larger feasible learning space.The passage presents removing the last ReLU as a feature-learning intervention.
  • The intuition of removing the last ReLU: With ReLU, MNIST 2-D features distribute only in the first quadrant, whereas removing the last ReLU produces a much more reasonable feature distribution.This comparison comes from the 2-D visualization of MNIST features with and without the last ReLU.

B. Normalizing the weights could reduce the prior caused by the training data imbalance

Normalizing classifier weights is proposed as a way to reduce the class prior induced by training-data imbalance, such as long-tail distributions. An empirical study finds that classes with more samples tend to have larger associated weight norms.

  • B. Normalizing the weights could reduce the prior caused by the training data imbalance: Weight normalization can partially address training-data imbalance by implicitly reducing its induced prior.The motivation includes long-tail training-data distributions and complements the geometric interpretation of normalized weights.
  • B. Normalizing the weights could reduce the prior caused by the training data imbalance: Classes with more training samples tend to have larger 2-norms for their corresponding classifier weights.The i-th column of W is associated with class i, and the relation was examined on MNIST and CASIA-WebFace.
  • B. Normalizing the weights could reduce the prior caused by the training data imbalance: The study relates each class’s sample count to the norm of its corresponding weight vector.Norms of W_i and sample numbers were computed class by class, as shown in Figure 9.

C. Empirical experiment of zeroing out the biases

The analysis zeroes out the FC2 bias because it obscures A-Softmax’s angular geometry. MNIST visualization indicates that removing the bias does not directly affect feature distribution or use of the learning space.

  • Method: The FC2 bias is set to zero to make A-Softmax’s angular geometry and angular-margin interpretation clearer.Standard CNNs usually retain fully connected-layer biases, but the FC2 bias makes the angle-based analysis more difficult.
  • MNIST visualization: Figure 11 compares 2-D MNIST feature distributions with and without the last fully connected-layer bias.The comparison supports the observation that both settings make full use of the learning space.
  • MNIST visualization: Zeroing out the bias has no direct influence on the 2-D MNIST feature distribution.Features learned with and without bias can both fully use the learning space.

D. 2D visualization of A-Softmax loss on MNIST

The MNIST 2-D visualization shows that increasing m makes A-Softmax features more discriminative through a larger inter-class angular margin, while the learned features generalize well to testing data.

  • D. 2D visualization of A-Softmax loss on MNIST: Increasing m produces more discriminative MNIST features by enlarging the inter-class angular margin.The visualization compares A-Softmax with m = 2, 3, and 4.
  • D. 2D visualization of A-Softmax loss on MNIST: The learned discriminative features generalize well to the MNIST testing set.
  • D. 2D visualization of A-Softmax loss on MNIST: The 2-D MNIST visualization compares features learned with softmax and A-Softmax at m = 2, 3, and 4.

E. Angular Fisher score for evaluating the feature discriminativeness and ablation study on our proposed modifications

This section introduces the angular Fisher score (AFS) to evaluate feature discriminativeness in angular-margin learning and studies the effects of the proposed architectural and loss modifications. The ablation experiments use a 4-layer CNN trained on CASIA and tested on LFW, with all modifications reported to improve performance.

  • Angular Fisher score: The angular Fisher score evaluates feature discriminativeness in angular margin feature learning.It is based on within-class and between-class angular scatter.
  • Angular Fisher score: Within-class scatter uses 1 − cos⟨x_j, m_i⟩, comparing each class sample with its class mean.X_i denotes the samples in class i, and m_i is their feature mean.
  • Angular Fisher score: Between-class scatter uses 1 − cos⟨m_i, m⟩, weighting each class by its sample count n_i.m is the mean vector of the whole dataset.
  • Angular Fisher score: Lower Fisher values indicate more discriminative features.This interpretation applies to the angular Fisher score used for evaluating the learned representations.
  • Ablation study: The ablation study removes the last ReLU and biases, normalizes weights, and applies A-Softmax loss using a 4-layer CNN trained on CASIA and tested on LFW.The experimental setting matches the LFW experiment in the main paper, and Table 6 reports verification accuracy (%).
  • Ablation study: All proposed modifications are reported to improve performance in the ablation experiments.The reported comparison is presented in Table 6 for LFW verification accuracy.

F. Experiments on MegaFace with different convolutional layers · G. The annealing optimization strategy for A-Softmax loss · H. Details of the 3-patch ensemble strategy in MegaFace challenge

On MegaFace, A-Softmax benefits substantially from increased convolutional capacity and can outperform much deeper softmax networks. Training uses annealing from an easy large-λ objective, while the challenge system uses a three-patch concatenation ensemble to balance efficiency and accuracy.

  • F. Experiments on MegaFace with different convolutional layers: SphereFace with four convolutional layers outperforms softmax with 64 convolutional layers on MegaFace, demonstrating A-Softmax’s effective use of network capacity.With more convolutional layers, SphereFace performance improves further.
  • F. Experiments on MegaFace with different convolutional layers: Table 7 reports MegaFace performance in percentage using TAR and FAR, with m = 4 for all SphereFace models.TAR denotes True Accept Rate and FAR denotes False Accept Rate.
  • F. Experiments on MegaFace with different convolutional layers: The authors note that larger m and suitable network optimization could further increase SphereFace performance.This statement accompanies the Table 7 experimental setup.
  • G. The annealing optimization strategy for A-Softmax loss: A-Softmax is trained with annealing optimization, supervising the network from an easy task with large λ toward a difficult task with small λ.The strategy is described as similar to optimization for L-Softmax.
  • G. The annealing optimization strategy for A-Softmax loss: Training starts with a very large λ, equivalent to original softmax optimization, and gradually reduces λ during stochastic gradient descent.Although λ could ideally reach zero, a small value usually suffices in practice.
  • G. The annealing optimization strategy for A-Softmax loss: 5 is the typical final λ value in the face experiments, producing impressive results, while smaller λ may improve performance but makes optimization harder.The passage states that decaying λ to 5 has already led to impressive results.
  • H. Details of the 3-patch ensemble strategy in MegaFace challenge: The MegaFace system uses a three-patch simple concatenation ensemble without PCA to trade off efficiency and accuracy.Using more patches could continue increasing performance, but the authors select three patches for this tradeoff.
  • H. Details of the 3-patch ensemble strategy in MegaFace challenge: The three patches can be selected by cross-validation, and the paper uses the same three patches shown in Figure 13.Figure 13 depicts the three-patch ensemble used for the MegaFace challenge.
Loading 1704.08063v4…