PaperlayerView exact revision on arXiv

Source-linked AI summary

NormFace: L2 Hypersphere Embedding for Face Verification

Feng Wang, Xiang Xiang, Jian Cheng, Alan L. Yuille

arXiv:1704.06369v4cs.CV

TL;DR

Face-verification systems normalize features for comparison, but incorporating normalization into training is non-trivial and can prevent convergence. NormFace analyzes this behavior and proposes scaled cosine-softmax and agent-based metric-learning objectives, improving two models by 0.2%–0.4% on LFW.

  • Problem

    Feature normalization improves testing similarity, but its role in CNN training and the failure of direct cosine optimization were not clearly understood.

  • Method

    The paper analyzes normalized-feature optimization and proposes scaled cosine-softmax plus agent-based reformulations of metric-learning losses for classification.

  • Results

    0.2%–0.4% accuracy improvement on LFW was achieved across two publicly available models using training-time normalization.

  • Takeaways & Limitations

    Training with normalized features can improve face-verification models while avoiding the hard sample mining required by conventional metric learning.

  • Takeaways & Limitations

    The agent strategy can leave marginal features unoptimized unless larger margins are used.

Abstract

from arXiv · show

Thanks to the recent developments of Convolutional Neural Networks, the performance of face verification methods has increased rapidly. In a typical face verification method, feature normalization is a critical step for boosting performance. This motivates us to introduce and study the effect of normalization during training. But we find this is non-trivial, despite normalization being differentiable. We identify and study four issues related to normalization through mathematical analysis, which yields understanding and helps with parameter settings. Based on this analysis we propose two strategies for training using normalized features. The first is a modification of softmax loss, which optimizes cosine similarity instead of inner-product. The second is a reformulation of metric learning by introducing an agent vector for each class. We show that both strategies, and small variants, consistently improve performance by between 0.2% to 0.4% on the LFW dataset based on two models. This is significant because the performance of the two models on LFW dataset is close to saturation at over 98%. Codes and models are released on https://github.com/happynear/NormFace

1 INTRODUCTION

Face verification commonly compares normalized features at test time, while training uses unnormalized inner products. The paper analyzes why normalization helps and proposes trainable normalized-feature objectives that improve strong baselines.

  • Motivation: Feature comparison commonly uses cosine similarity or L2-normalized Euclidean distance, whereas CNN classification training usually uses unnormalized inner products.This creates a mismatch between the similarity metric used during training and testing.
  • Motivation: 0.6%–0.7% improvement followed feature normalization in experiments exceeding 98% accuracy.The result motivates treating normalization as more than a testing-time trick.
  • Problem: Directly normalizing features and weights in a cosine layer failed to converge after sufficient training iterations.The failed experiment prompted mathematical analysis of normalized-feature optimization.
  • Contributions: The paper analyzes why normalization is effective, why direct cosine optimization can fail, and how to optimize normalized features with suitable losses.It also asks whether metric-learning losses can be adapted for normalized features.
  • Methods: The proposed softmax strategy inserts a scale layer after cosine scoring, while the agent strategy reformulates metric learning for classification without pair or triplet sampling.The agent strategy avoids hard-mining algorithms used by conventional metric learning.
  • Results: 0.2%–0.4% improvement on LFW and about 0.6% on YTF resulted from training with normalization across two publicly available models.Additional methods merge original and mirrored image features by summation and use similarity histograms for video classification.

2 RELATED WORKS

The related work covers normalization methods, face-verification objectives, and hyperspherical embeddings. Existing approaches differ in whether they normalize features, weights, or both.

  • Normalization in Neural Network: Neural-network normalization methods include local response, local contrast, batch, weight, and layer normalization.These methods address feature normalization, convergence, weight parameterization, or batch-size dependence.
  • Face Verification: Face verification determines whether two face images represent the same person and supports access-control or re-identification applications.Deep-learning face verification methods primarily use metric-learning or classification-based losses.
  • Softmax Geometry: A softmax toy experiment on MNIST showed that Euclidean distance can place same-class features inconsistently, while feature-vector angles appear more suitable for comparison.The figure visualizes both the two-dimensional feature distribution and softmax probability for class 0.
  • Hypersphere Embeddings: Cosine Loss, vMFMM, and the proposed losses normalize both features and weights, whereas L2-softmax normalizes features only and SphereFace normalizes weights only.This comparison distinguishes hyperspherical embedding designs by which quantities receive L2 normalization.

3 L2 NORMALIZATION LAYER

The section explains why feature normalization is useful yet difficult to train with softmax loss, then introduces a normalized feature layer and scaled cosine softmax to address convergence.

  • 3.1 Necessity of Normalization: Softmax loss encourages well-separated features to have larger magnitudes, producing a radial feature distribution that normalization can remove.This motivates using cosine similarity, which is independent of feature magnitude.
  • 3.1 Necessity of Normalization: Biases can place class clusters near zero, so normalizing those features may spread them across the unit circle and cause misclassification.The issue arises because bias terms can support classification even when class weight vectors coincide.
  • 3.2 Layer Definition: L2 normalization maps features or weights to a unit hypersphere, while its backward gradient lies in the tangent space and can increase the input norm.Weight decay is therefore necessary to prevent the norm from growing without bound.
  • 3.3 Reformulating Softmax Loss: Directly applying softmax to normalized features fails to converge because cosine similarity is restricted to [−1, 1], limiting the attainable class probability.Even well-separated samples can retain large gradients when the number of classes is large.
  • 3.3 Reformulating Softmax Loss: A lower bound explains why unit-normalized softmax loss remains high; on CASIA-Webface, loss decreased from about 9.27 to 8.50 against a bound of 8.27.The close agreement suggests that the bound is tight.
  • 3.3 Reformulating Softmax Loss: Scaled cosine softmax normalizes features and weights to a larger norm through a scale layer, with s = ℓ2 learned by back-propagation or fixed near 20 or 30.The formulation directly optimizes cosine similarity while restoring a sufficient score range for softmax.

4 REFORMULATING METRIC LEARNING

The paper reformulates metric-learning losses for classification with normalized features by assigning each class a learnable agent vector. This preserves normalized-distance objectives while avoiding exhaustive pair or triplet sampling, but agent approximation can omit marginal features.

  • Motivation: Metric learning requires O(N^2) pair or O(N^3) triplet combinations, making sampling and hard mining necessary and difficult.The reformulation targets classification-style O(N) training while retaining compatibility with normalized features.
  • Geometric interpretation: With margin m = 1, agents shift away from class boundaries under other-class marginal features, while features follow agents through the intra-class term.For m = 0, agents are influenced only by features from their own classes and converge toward corresponding class centers.
  • Agent reformulation: Each class receives a learnable agent vector W_i that replaces one feature in the normalized-distance formulation.The agent is a column of W and is trained through back-propagation like an inner-product layer.
  • Agent reformulation: C-contrastive and C-triplet losses convert contrastive and triplet objectives into classification losses using class agents.The classification variants are named C-contrastive loss and C-triplet loss.
  • Limitations and parameter setting: The agent strategy can omit marginal features under the original margin, so larger margins are needed to optimize more features.The distortion bound provides theoretical guidance; recommended margins are 1 for modified contrastive loss and 0.8 for modified triplet loss.
  • Limitations and parameter setting: Normalization fixes feature magnitude, allowing the modified losses to use a fixed margin instead of searching for a new margin during training.The paper does not attempt to train C-contrastive or C-triplet loss without normalization because this is difficult.

5 EXPERIMENT

Experiments evaluate normalized-feature training on LFW and YTF using established CNN models and several loss configurations. Normalization consistently improves verification performance, while histogram-based video scoring provides an additional gain on YTF.

  • Experiment settings: Experiments evaluate two publicly available CNN baselines on LFW and YTF under established verification protocols.The baselines are Wu’s 10-layer Maxout CNN and Wen’s 28-layer ResNet with softmax and center loss.
  • LFW experiments: Normalization is the key factor improving LFW accuracy, while the choice among tested loss functions has a minor influence.The highest reported LFW accuracy is 99.2167% with softmax + 0.01 * C-contrastive; pure normalized softmax also performs reasonably well.
  • LFW experiments: 98.45% accuracy results from normalizing features only, outperforming conventional softmax but remaining below state-of-the-art loss functions.Normalizing weights only causes network collapse, whereas feature normalization requires a scale parameter.
  • LFW experiments: C-contrastive loss is more robust to loss-weight changes than center loss.C-contrastive can train a model alone, while center loss optimizes only intra-class variance and requires other supervised losses.
  • LFW experiments: Normalization boosts BLUFR performance particularly when the false alarm rate is low.BLUFR evaluates all 13,233 LFW images rather than only the 6,000 pairs used by the standard protocol.
  • YTF experiments: On YTF, models performing better on LFW also perform better, and HIK-SVM score histograms further improve accuracy by a significant gap.The histogram encodes pairwise cosine-similarity scores across video frames instead of using only their mean or maximum.

6 CONCLUSION AND FUTURE WORK

The paper proposes normalization-based training methods for classification models and reports superior performance over models without normalization. It also identifies broader applicability while noting that current normalization techniques require fine-tuning from other models.

  • The proposed approach applies L2 normalization to features and last-layer weights during classification training.
  • Two loss strategies train normalized features: scaled cosine-based softmax and metric-learning reformulation with class agents.
  • Three theoretical propositions guide settings for the bias, scale parameter, and margin.
  • Experiments on two models show superior performance over models without normalization.
  • The current normalization techniques can only fine-tune networks based on other models.
  • The methods and analysis may extend to person re-identification and image retrieval, but the paper leaves those applications for future work.

8 APPENDIX

The appendix develops propositions about normalized softmax and agent-based metric learning. Its arguments characterize probability scaling, softmax-loss bounds, geometric equality conditions, and agent approximation distortion.

  • 8.1 Proof of Proposition 1: For no-bias softmax, scaling a feature by s > 1 increases the probability assigned to its highest-scoring class.
  • 8.1 Proof of Proposition 1: The probability inequality becomes equality only when the winning score is zero or all class weights are identical, conditions described as practically unlikely.
  • 8.2 Proof of Proposition 2: Proposition 2 assumes equal class sample counts, well-separated samples, and equal norms for normalized features and class-weight columns.
  • 8.2 Proof of Proposition 2: The softmax-loss equality condition requires equal pairwise inner products, meaning equal distances between features from different classes.
  • 8.2 Proof of Proposition 2: In d-dimensional space, at most d + 1 equidistant vertices form a regular d-simplex, so the equality condition generally cannot hold when class count exceeds feature dimension.
  • 8.2 Proof of Proposition 2: The appendix also uses convexity of the softplus function and characterizes equality through equal sums of distances to other class weights.
  • 8.3 Agent Approximation: Proposition 3 bounds the distortion introduced when one agent represents each class instead of a specific sample.
Loading 1704.06369v4…