Source-linked AI summary
Large-Margin Softmax Loss for Convolutional Neural Networks
Weiyang Liu, Yandong Wen, Zhiding Yu, Meng Yang
TL;DR
CNN softmax supervision does not explicitly enforce discriminative feature geometry, motivating a loss that controls intra-class compactness and inter-class separation. The paper proposes L-Softmax, an adjustable angular-margin generalization of softmax, and reports improved performance across classification and verification tasks. Its learned features become more discriminative while the objective can partially avoid overfitting.
Problem
Standard softmax loss is widely used in CNNs but does not explicitly encourage intra-class compactness and inter-class separability.
Method
L-Softmax generalizes softmax by imposing an adjustable inter-class angular margin controlled by m.
Results
L-Softmax effectively boosts performance in both classification and verification tasks and produces more discriminative, compact, and well-separated features.
Takeaways & Limitations
The loss provides a flexible learning objective whose increasing margin creates a harder target and can partially avoid overfitting.
Abstract
from arXiv · showhide
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a generalized large-margin softmax (L-Softmax) loss which explicitly encourages intra-class compactness and inter-class separability between learned features. Moreover, L-Softmax not only can adjust the desired margin but also can avoid overfitting. We also show that the L-Softmax loss can be optimized by typical stochastic gradient descent. Extensive experiments on four benchmark datasets demonstrate that the deeply-learned features with L-softmax loss become more discriminative, hence significantly boosting the performance on a variety of visual classification and verification tasks.
1. Introduction
CNNs provide strong visual representations, but standard softmax loss does not explicitly promote intra-class compactness and inter-class separability. L-Softmax addresses this gap with an adjustable angular margin and improves discriminativeness across classification and verification tasks.
- Motivation: CNNs combine convolutional feature learning with softmax-based supervision in many visual recognition systems.The paper frames standard CNNs as convolutional feature-learning machines supervised by softmax loss.
- Motivation: Standard softmax loss does not explicitly encourage intra-class compactness or inter-class separability.This limitation motivates adding discriminative structure to CNN feature learning.
- Contribution: L-Softmax imposes an adjustable inter-class angular margin, with larger margins creating progressively harder learning objectives.The method uses the margin parameter m to control the strength of the angular constraint.
- Contribution: L-Softmax is presented as partially avoiding overfitting by defining a more difficult learning target.This reframes overfitting control as a consequence of increasing learning-task difficulty.
- Results: L-Softmax learns more compact and well-separated features, supporting improved classification and verification performance.Experiments and feature visualizations are reported as evidence across both task types.
2. Related Work and Preliminaries
The paper situates L-Softmax among CNN loss functions designed to improve feature discrimination and reformulates softmax through feature–classifier angular geometry. This formulation provides the basis for its large-margin generalization.
- Related Work: Contrastive and triplet losses encourage intra-class similarity and inter-class separation but may require selecting among up to O(N^2) training pairs or triplets.The pair and triplet counts can become costly for large-scale CNN training sets.
- Softmax Formulation: The paper generalizes original softmax loss by defining each input feature x_i together with its class label y_i.This establishes the notation used for the subsequent reformulation.
- Softmax Formulation: Class scores are represented by fully connected-layer activations, with f_j denoting the score for class j and N the number of training samples.The formulation uses K classes and omits the bias term for simplified analysis.
- Angular Reformulation: The fully connected score f_j is expressed as ||W_j||||x_i||cos(theta_j), separating classifier and feature magnitudes from their angular relationship.This angular factorization motivates imposing margins through angles rather than only score amplitudes.
3. Large-Margin Softmax Loss
L-Softmax modifies softmax by multiplying the ground-truth angle by an integer margin parameter, producing stricter decision criteria and an adjustable angular margin between classes. Its geometric interpretation narrows feasible angles, while larger margins increase learning difficulty and may reduce overfitting.
- 3.1. Intuition: L-Softmax replaces the ground-truth term with a function of mθ, where m controls the angular margin and m = 1 recovers standard softmax.The objective becomes harder as m increases.
- 3.1. Intuition: The modified criterion requires a stricter inequality than softmax, yielding a more rigorous decision boundary for the ground-truth class.For binary classification, the ground-truth angle is multiplied by m before comparison with the competing class.
- 3.3. Geometric Interpretation: With equal classifier norms, softmax requires θ1 < θ2, whereas L-Softmax requires mθ1 < θ2, creating an angular margin between classes.The analysis assumes both losses reach the same value and training features are perfectly classified.
- 3.3. Geometric Interpretation: L-Softmax narrows each class’s feasible angle and brings learned features closer to their corresponding classifier vectors.The same conclusion extends to unequal classifier norms, where feasible angles differ across classes.
- 3.4. Discussion: The margin parameter defines an adjustable learning task: larger m produces a larger ideal margin and greater difficulty, while the loss can serve as a drop-in replacement for standard loss.It can also be combined with activation learning, data augmentation, pooling, or modified architectures.
4. Optimization
L-Softmax changes only the ground-truth logit computation relative to standard softmax, so its forward and backward propagation can be handled with typical stochastic gradient descent. The required terms are computed efficiently, including through a lookup table for the integer k.
- 4. Optimization: For each sample, L-Softmax differs from standard softmax only in fyi, while non-ground-truth logits remain unchanged.This limits the additional computation to the ground-truth class term during forward and backward propagation.
- 4. Optimization: The integer k can be computed efficiently using a lookup table, supporting practical forward and backward propagation.The same equations remain applicable when m ≥ 3; m = 2 is given as an explicit example.
5. Experiments and Results
Experiments on MNIST, CIFAR10, CIFAR100, and LFW evaluate L-Softmax against softmax and other CNN baselines across classification and face verification. Results report stronger accuracy, feature discriminativeness, adjustable angular margins, and reduced overfitting.
- Experimental Settings: Experiments cover visual classification on MNIST, CIFAR10, and CIFAR100 and face verification on LFW using single-model CNN comparisons.Both Softmax and L-Softmax use the same CNN architecture in the experiments.
- Visual Classification: L-Softmax outperforms the original softmax with the same network and achieves state-of-the-art performance on MNIST.The comparison is reported in recognition error rates on the MNIST dataset.
- Visual Classification: 1%-2% improvement in CIFAR10 accuracy over the original softmax loss and other state-of-the-art CNNs is achieved by L-Softmax.The reported comparison includes both non-augmented and data-augmented CIFAR10 protocols.
- Visual Classification: More than 2.5% accuracy over the CNN and more than 1% over the current state-of-the-art CNN are achieved on CIFAR100 with L-Softmax.The method also outperforms the CNN with softmax loss and other competitive methods.
- Feature Visualization: L-Softmax enhances intra-class compactness and inter-class separability in confusion-matrix comparisons using normalized features and cosine distance.The comparison uses the softmax baseline and L-Softmax with m=4 on CIFAR10, CIFAR10+, and CIFAR100.
- Optimization and Overfitting: L-Softmax greatly avoids the severe overfitting observed for original softmax, and larger-filter CNNs perform better on both training and testing sets.The error-versus-iteration analyses vary m and, separately, the number of filters.
- Face Verification: The generalized softmax loss achieves the current best LFW results using CASIA-WebFace outside data and is comparable to state-of-the-art CNNs trained with private outside data.The evaluation uses a single feature-extraction network and tests on 6,000 LFW face pairs.
6. Concluding Remarks
The paper proposes Large-Margin Softmax loss with an adjustable margin controlled by m and reports advantages over state-of-the-art CNNs and compared baselines. Its geometric interpretation and benchmark results support the proposed approach.
- Concluding Remarks: Large-Margin Softmax defines a flexible learning task whose adjustable parameter m controls the inter-class decision margin.Larger m produces a larger decision margin.
- Concluding Remarks: Extensive benchmark experiments show clear advantages over current state-of-the-art CNNs and all compared baselines.The conclusion summarizes results across several benchmark datasets.