PaperlayerView exact revision on arXiv

Source-linked AI summary

L2-constrained Softmax Loss for Discriminative Face Verification

Rajeev Ranjan, Carlos D. Castillo, Rama Chellappa

arXiv:1703.09507v3cs.CV

TL;DR

Softmax-based face verification does not directly enforce higher similarity for positive pairs and lower similarity for negative pairs, especially under unconstrained conditions. The paper adds an L2 constraint that places feature descriptors on a fixed-radius hypersphere and reports consistent gains across three benchmarks, including state-of-the-art IJB-A performance.

  • Problem

    Softmax training does not ensure positive face pairs are close and negative pairs are separated in normalized or angular space, while difficult faces are underrepresented and harder to model.

  • Method

    L2-softmax adds a constraint that fixes each penultimate-layer feature descriptor's L2-norm at α, placing descriptors on a fixed-radius hypersphere.

  • Results

    The method yields a consistent and significant boost on LFW, YouTube Face, and IJB-A, achieving state-of-the-art performance on IJB-A.

  • Takeaways & Limitations

    L2-softmax is presented as a valuable replacement for regular softmax for face verification while retaining a one-network, one-loss design.

  • Takeaways & Limitations

    Very low α values can prevent training convergence; α = 1 achieves 86.37% accuracy on LFW.

Abstract

from arXiv · show

In recent years, the performance of face verification systems has significantly improved using deep convolutional neural networks (DCNNs). A typical pipeline for face verification includes training a deep network for subject classification with softmax loss, using the penultimate layer output as the feature descriptor, and generating a cosine similarity score given a pair of face images. The softmax loss function does not optimize the features to have higher similarity score for positive pairs and lower similarity score for negative pairs, which leads to a performance gap. In this paper, we add an L2-constraint to the feature descriptors which restricts them to lie on a hypersphere of a fixed radius. This module can be easily implemented using existing deep learning frameworks. We show that integrating this simple step in the training pipeline significantly boosts the performance of face verification. Specifically, we achieve state-of-the-art results on the challenging IJB-A dataset, achieving True Accept Rate of 0.909 at False Accept Rate 0.0001 on the face verification protocol. Additionally, we achieve state-of-the-art performance on LFW dataset with an accuracy of 99.78%, and competing performance on YTF dataset with accuracy of 96.08%.

1. Introduction

The paper addresses softmax-based face verification's difficulty with unconstrained, low-quality faces by constraining feature descriptors to a fixed-radius hypersphere. This L2-softmax approach improves verification performance while retaining a simple training pipeline.

  • Motivation: Softmax training can overfit abundant high-quality faces and neglect rare difficult faces affected by pose, resolution, occlusion, and image quality.Feature L2-norms are informative of face quality: high-quality frontal faces have higher norms, while blurry or extreme-pose faces have lower norms.
  • Proposed Method: Equal feature norms give good and bad quality faces similar attention and strengthen separation between same-subject and different-subject features in normalized space.The method maximizes the margin for normalized L2 distance or cosine similarity between positive and negative pairs.
  • Practical Properties: The method retains softmax's one-network, one-loss design, easy framework implementation, fast convergence, and lack of required joint supervision.It adds only a single scaling parameter to the network.
  • Proposed Method: The L2-softmax loss constrains every feature descriptor to have constant L2-norm α, placing descriptors on a fixed-radius hypersphere.The constraint is implemented as an addition to regular softmax loss and introduces a single scaling parameter.
  • Results: The L2-softmax loss gives a consistent and significant boost across LFW, YouTube Face, and IJB-A verification datasets.The paper reports state-of-the-art results on IJB-A and competing results on LFW and YouTube Face.
  • Results: 0.909 TAR at 0.0001 FAR is achieved on IJB-A when L2-softmax is combined with TPE.The gains from L2-softmax are reported as complementary to metric learning and auxiliary loss functions.

2. Related Work

Related work follows two main routes: pair-based metric learning and classification-based learning of discriminative face features. The paper distinguishes L2-softmax from center loss and other feature-normalization methods by its single-loss, single-parameter formulation.

  • Pair-based Metric Learning: Pair-based methods learn embeddings by bringing positive face pairs closer and pushing negative pairs farther apart.Examples include siamese networks with contrastive loss, discriminative deep metric learning, and FaceNet's triplet loss.
  • Classification-based Learning: Classification-based methods train DCNNs with subject labels and softmax loss, then use the learned features for similarity scoring or metric embedding.Some methods jointly train identification and verification tasks.
  • Comparison with Center Loss: Center loss learns more discriminative face features jointly with softmax loss and introduces C × D additional training parameters.L2-softmax instead uses one loss function and a single parameter defining the fixed feature L2-norm.
  • Feature Normalization: SphereFace uses angular softmax and DeepVisage normalizes descriptors before softmax, whereas the proposed method applies an L2-constraint to feature descriptors.The cited passage identifies these approaches as related feature-normalization strategies.

3. Motivation

The DCNN pipeline trains face identity classification with softmax, then compares extracted face descriptors for verification. Its training objective is not directly aligned with pairwise verification and can underrepresent difficult faces, motivating a fixed L2-norm constraint.

  • Pipeline: The pipeline trains a DCNN for identity classification with softmax loss, then extracts penultimate-layer descriptors and computes pairwise similarity at test time.The descriptors are normalized before similarity scoring, and a threshold determines whether a pair represents the same person.
  • Limitations: Softmax training does not necessarily bring positive pairs closer and negative pairs farther apart in normalized or angular feature space.This decouples the classification training objective from the verification decision criterion.
  • Limitations: Softmax can fit high-quality faces while ignoring rare difficult faces because it is biased toward the sample distribution in each mini-batch.The feature L2-norm becomes informative of face quality: good frontal faces tend to have high norms, whereas blurry or extreme-pose faces tend to have low norms.
  • Proposed solution: The proposed L2 constraint fixes every feature descriptor to a hypersphere of constant radius during training.This gives good- and bad-quality faces the same feature norm and is intended to strengthen the verification signal.
  • Proposed solution: On the fixed-radius hypersphere, minimizing softmax loss is equivalent to maximizing cosine similarity for positive pairs and minimizing it for negative pairs.The same constraint also allows softmax to model difficult and extreme faces with equal feature norms.

4. Proposed Method

The proposed L2-softmax loss constrains penultimate-layer feature descriptors to a fixed-radius hypersphere during training. The method is implemented with normalization and scaling layers, while experiments examine feature geometry and how the radius α affects classification probability and convergence.

  • L2-softmax formulation: L2-softmax loss adds an L2-constraint requiring each feature descriptor f(x_i) to have constant norm α.The constrained features lie on a hypersphere of fixed radius.
  • MNIST example: The MNIST setup compares regular softmax with L2-softmax using two-dimensional features for visualizing class structure.The network is trained end-to-end with regular softmax in one setup and with normalization and scaling layers in the other.
  • MNIST example: L2-softmax produces lower intra-class angular variability than regular softmax, yielding thinner class lobes in the two-dimensional visualization.Regular softmax features have larger intra-class angular variance.
  • Implementation details: An L2-normalize layer followed by a scale layer enforces the constraint after the penultimate DCNN layer.The normalization produces a unit vector, and the scale layer sets its radius using α.
  • Implementation details: The module is differentiable and adds one trainable scalar parameter α, while remaining redundant at test time because cosine similarity normalizes features.Gradients are backpropagated through both added layers and α during training.
  • Bounds on parameter α: Higher α is required to reach a fixed classification probability as the number of classes C increases, while very low α can prevent convergence.The paper derives a lower bound on α under assumptions about class centers and feature dimension.

5. Results

Experiments add L2-normalization and scaling after the feature descriptor, then evaluate L2-softmax against regular softmax across training sizes, architectures, and face-verification benchmarks. L2-softmax consistently improves performance, with strongest reported results on IJB-A, LFW, and YTF.

  • Experimental setup: The Face-Resnet experiments add L2-normalize and scale layers after the fully connected feature descriptor before softmax loss.The network contains 27 convolutional layers, two fully connected layers, and a 512-dimensional descriptor.
  • Small training set: 99.28% LFW accuracy with MS-small and L2-softmax exceeds regular softmax’s 98.1% and reduces error by more than 62%.Performance is better for α >12 and remains stable above the threshold.
  • Small training set: 19% higher TAR@FAR=0.0001 on IJB-A is achieved over baseline softmax, with consistent performance for α between 16 and 32.Learning α instead of fixing it slightly decreases performance in this experiment.
  • Large training set: MS-large training achieves 99.6% LFW accuracy, reduces error by 60%, and improves IJB-A TAR@FAR=0.0001 by more than 10%.Performance is consistent for α values from 40 onward, while self-trained α performs equally well to fixed α values of 40 and 50.
  • Architecture consistency: L2-softmax improves All-In-One Face performance on LFW, reducing error by 40% relative to base softmax.The added L2-normalize and scale layers follow the 512-dimensional feature descriptor.
  • Comparison with recent methods: RX101 with L2-softmax reaches 99.78% on LFW and 96.08% on YTF using a single-loss training paradigm.On IJB-A, combining L2-softmax with TPE achieves TAR 0.909 @ FAR = 0.0001 and state-of-the-art verification and identification results.

6. Conclusions

The paper concludes that a fixed-radius L2 constraint is a simple, effective replacement for regular softmax in face verification. Across LFW, YTF, and IJB-A, it reports consistent gains and state-of-the-art IJB-A performance.

  • Conclusion: The L2 constraint forces feature descriptors onto a hypersphere of fixed radius controlled by α.The paper also provides bounds on α for consistent performance.
  • Conclusion: L2-softmax provides a significant and consistent boost over regular softmax across LFW, YTF, and IJB-A.The paper identifies it as a valuable replacement for existing softmax loss in face verification.
  • Future work: The paper proposes exploring manifold-based metric learning to exploit the geometric structure of feature encodings.This is presented as future work beyond the evaluated L2 constraint.
Loading 1703.09507v3…