Source-linked AI summary
AdaCos: Adaptively Scaling Cosine Logits for Effectively Learning Deep Face Representations
Xiao Zhang, Rui Zhao, Yu Qiao, Xiaogang Wang, Hongsheng Li
TL;DR
Cosine-based softmax losses improve face recognition but are highly sensitive to scale and angular-margin hyperparameters that are difficult to tune. AdaCos analyzes how these parameters shape predicted probabilities and adaptively controls the scale during training. It achieves stable, high-performing results and outperforms state-of-the-art softmax losses across the evaluated benchmarks.
Problem
Existing cosine-based softmax losses are sensitive to scale and margin hyperparameters, whose settings affect supervision, convergence, and final recognition performance.
Method
AdaCos analyzes hyperparameter effects through predicted classification probabilities and automatically adjusts an adaptive scale to strengthen training supervision.
Results
AdaCos outperforms state-of-the-art softmax losses on LFW, MegaFace, and IJB-C benchmarks, with dynamic scaling achieving the best performance on MegaFace and IJB-C.
Takeaways & Limitations
Adaptive scaling provides a hyperparameter-free cosine-based loss with faster, more stable convergence and effective supervision during training.
Takeaways & Limitations
Oversized margins can make probabilities unreliable and training difficult to converge, while the adaptive-scale derivation assumes non-target angles distribute around π/2 during training.
Abstract
from arXiv · showhide
The cosine-based softmax losses and their variants achieve great success in deep learning based face recognition. However, hyperparameter settings in these losses have significant influences on the optimization path as well as the final recognition performance. Manually tuning those hyperparameters heavily relies on user experience and requires many training tricks. In this paper, we investigate in depth the effects of two important hyperparameters of cosine-based softmax losses, the scale parameter and angular margin parameter, by analyzing how they modulate the predicted classification probability. Based on these analysis, we propose a novel cosine-based softmax loss, AdaCos, which is hyperparameter-free and leverages an adaptive scale parameter to automatically strengthen the training supervisions during the training process. We apply the proposed AdaCos loss to large-scale face verification and identification datasets, including LFW, MegaFace, and IJB-C 1:1 Verification. Our results show that training deep neural networks with the AdaCos loss is stable and able to achieve high face recognition accuracy. Our method outperforms state-of-the-art softmax losses on all the three datasets.
1. Introduction
Deep face recognition combines large-scale data, deep architectures, and effective losses, but cosine-based softmax methods remain sensitive to difficult hyperparameter choices. AdaCos addresses this by adaptively scaling cosine similarities, enabling stable training and strong benchmark performance.
- 1. Introduction: Deep face recognition advances rely on large training datasets, deep architectures, and effective loss functions.These factors support improvements in face verification and identification.
- 1. Introduction: Cosine-based softmax losses reduce the mismatch between cosine-similarity inference and conventional softmax training while angular margins improve class separation.However, their training processes are often tricky and unstable.
- 1. Introduction: Their convergence and final performance depend substantially on hyperparameters that are difficult to tune and may provide insufficient supervision when poorly set.Finding effective settings can require repeated training with different configurations.
- 1. Introduction: AdaCos automatically tunes an adaptive scale for cosine similarities, generating more effective supervision without additional computational overhead.The method is implemented with built-in deep-learning library functions and is reported to converge faster and more stably.
- 1. Introduction: AdaCos outperforms state-of-the-art cosine-based losses across LFW, MegaFace, and IJB-C face-recognition benchmarks.The evaluation covers both verification and identification settings.
2. Related Works
Prior face-recognition work includes normalized feature losses, metric-targeted objectives, angular-margin softmax losses, and automatic hyperparameter-tuning strategies. These approaches respectively address representation geometry, class variation, angular separation, or parameter selection.
- Cosine similarities for inference: Coco loss, NormFace, and related methods normalize face features to improve recognition accuracy or constrain representations to a hypersphere.Ring loss instead introduces a soft feature-normalization approach.
- Margin-based softmax loss: Triplet and contrastive losses use Euclidean distances, while center loss and range loss reduce within-class variation.These methods form part of the metric-targeted-loss progression in face recognition.
- Margin-based softmax loss: L-softmax and A-softmax integrate angular constraints, while CosFace, AM-softmax, and ArcFace directly maximize angular margins with simpler loss functions.The latter methods use more direct angular-margin formulations.
- Automatic hyperparameter tuning: Grid search, random search, Bayesian optimization, and tree-structured Parzen estimators tune hyperparameters by evaluating multiple settings or trials.These strategies remain trial-based even when the selection process is automated.
3. Investigation of hyperparameters in cosine-based softmax losses
Cosine-based softmax losses use scale and margin parameters that reshape predicted class probabilities, so improper settings can weaken supervision or make training difficult. The analysis motivates choosing parameters by examining probability behavior as angles change.
- Scale s and margin m substantially affect the predicted probability Pi,yi in cosine-based softmax losses.The paper studies their effects through the softmax prediction probability rather than only through the loss form.
- Cosine-based logits use fi,j = s·cos θi,j, while angular-margin variants reduce the correct-class logit using m.ArcFace adds m inside the angle, and CosFace subtracts m from the cosine term; both reduce fi,yi relative to non-margin losses.
- Effects of the scale parameter s: During training, non-corresponding-class angles stay near π/2, making the competing-logit sum Bi almost unchanged.This behavior is illustrated by the red curve in Figure 1 and supports treating Bi as approximately stable in the probability analysis.
- Effects of the scale parameter s: Too-small s prevents Pi,yi from reaching 1 even at θi,yi = 0, so confident correct classifications remain penalized.The paper gives s = 10 as an example for C = 2,000 and C = 20,000.
- Effects of the scale parameter s: Too-large s can produce very high Pi,yi near θi,yi = π/2, making the loss insensitive to misclassified samples and weakening corrective updates.The paper gives s = 64 as an example and states that scaling affects both the probability range and its curve.
- Effects of the margin parameter m: Increasing m shifts the Pi,yi curve left and strengthens supervision, but oversized m can make probabilities near 0 even for very small angles, hindering convergence.The paper gives m = 1.0 as an example of an oversized margin and describes margin selection as ad hoc in previous methods.
4. The cosine-based softmax loss with adaptive scaling
AdaCos replaces manually tuned angular-margin settings with an adaptive scale that maps feature–class angles to effective probabilities and strengthens supervision as training progresses.
- 4. The cosine-based softmax loss with adaptive scaling: AdaCos eliminates manual margin tuning by automatically adapting the scale parameter while removing the angular margin parameter.The design is motivated by the different effects of scale and margin on predicted-probability curves.
- 4. The cosine-based softmax loss with adaptive scaling: The ground-truth probability P_i,y_i is the main supervision signal, so AdaCos focuses its adaptive scaling on controlling this probability.The scale is chosen to make predicted probability change significantly with the ground-truth angle.
- 4. The cosine-based softmax loss with adaptive scaling: A fixed adaptive scale uses the number of training classes C and approximates non-corresponding-class contributions by C−1.This fixed scale serves as a baseline for the dynamically tuned scale.
- 4.2. Dynamically adaptive scale parameter: During training, ground-truth angles decrease while non-corresponding angles stabilize near π/2, weakening supervision unless scaling becomes stricter.AdaCos therefore adjusts the central angle and scale over iterations.
- 4.2. Dynamically adaptive scale parameter: The dynamic scale uses the mini-batch median ground-truth angle to regulate supervision: larger angles receive less strict supervision, while smaller angles receive stricter supervision.The median angle represents the network’s current optimization state on the mini-batch.
- 4.2. Dynamically adaptive scale parameter: The dynamically adaptive scale changes classification probabilities and gradients according to current training convergence, while changing little across successive iterations.The previous iteration’s scale is reused in practice to estimate the current dynamic scale.
5. Experiments
Experiments evaluate AdaCos against cosine-based softmax losses on LFW, MegaFace, and IJB-C, alongside analyses of scale adaptation, feature angles, and convergence. AdaCos achieves stronger benchmark performance and faster convergence under the reported configurations.
- Comparison on LFW: AdaCos fixed and dynamic variants surpass compared cosine-based softmax losses on LFW under the same training configuration.Models use ResNet-50 trained on cleaned WebFace; dynamic AdaCos averages 0.26% higher accuracy than ArcFace and 1.52% higher than l2-softmax.
- Comparison on LFW: 0.26% higher average accuracy than ArcFace and 1.52% higher than l2-softmax are reported for dynamic AdaCos on LFW.All losses were trained and tested three times, with the reported values comparing average recognition accuracy.
- Exploratory Experiments: Dynamic AdaCos adaptively decreases its scale parameter during training, providing stricter supervisions as iterations increase.The reported analysis links the decreasing scale to progressively stronger supervision and reducing ground-truth feature angles.
- Exploratory Experiments: Dynamic AdaCos reduces ground-truth feature-to-category angles while non-ground-truth angles remain nearly π/2, outperforming l2-softmax on the reported angle behavior.The average and median ground-truth angles gradually decrease, whereas the average non-ground-truth angle remains nearly π/2.
- Convergence rates: AdaCos losses show much higher convergence rates than the compared softmax losses, with dynamic AdaCos achieving the best accuracy at equal iterations.The convergence-rate comparison uses the same training configurations as the LFW experiments.
- Results on MegaFace: Dynamic AdaCos outperforms all compared losses on MegaFace and achieves the best performance on IJB-C 1:1 verification.MegaFace models use cleaned WebFace and MS1M training data; IJB-C comparisons use the same training data and Inception-ResNet architecture.
6. Conclusions
The paper attributes limitations of existing cosine-based softmax losses to a mismatch between cosine distance and classification probability. It proposes AdaCos to adaptively reformulate this mapping and reports effective, efficient performance across public benchmarks.
- 6. Conclusions: Existing cosine-based softmax losses may be limited by a mismatch between cosine distance and classification probability.The paper analyzes this mismatch from a probability perspective before proposing its adaptive loss.
- 6. Conclusions: AdaCos automatically adjusts an adaptive scale parameter to reformulate the mapping between cosine distance and classification probability.The proposed loss is described as simple and effective, with the adaptive parameter adjusted during training.
- 6. Conclusions: AdaCos demonstrates effectiveness and efficiency through exploratory experiments and state-of-the-art performance on several public benchmarks.The conclusion summarizes the method’s reported empirical evaluation without specifying individual benchmark values.