Source-linked AI summary
Deep Learning using Linear Support Vector Machines
Yichuan Tang
TL;DR
Deep-learning classifiers commonly use softmax, motivating the question of whether an alternative top layer can improve performance. The paper replaces softmax with a jointly trained linear L2-SVM layer and reports superior results across multiple datasets and a facial-expression challenge. The approach is simple to apply, while the experiments suggest the gain is largely associated with the SVM loss's regularization.
Problem
Most deep-learning classification models use softmax objectives, leaving the benefit of replacing that top layer with a jointly trained SVM formulation to be examined.
Method
The paper replaces softmax with a linear SVM top layer, backpropagating through lower-level features while optimizing the differentiable squared-hinge L2-SVM loss.
Results
Superior performance is reported on MNIST, CIFAR-10, and facial-expression recognition compared with networks using a softmax top layer.
Takeaways & Limitations
Switching from softmax to SVMs is simple and appears useful for classification tasks, with gains believed to arise largely from the SVM loss's regularization.
Takeaways & Limitations
The study uses only linear SVMs because kernel SVM optimization has scalability problems.
Abstract
from arXiv · showhide
Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.
1. Introduction
The paper examines replacing the standard softmax classification layer in deep networks with a linear SVM trained jointly through backpropagation. Using the differentiable L2-SVM loss, it reports superior performance across MNIST, CIFAR-10, and facial-expression recognition, with gains attributed largely to regularization.
- Most deep-learning classifiers use softmax, while SVMs provide an alternative classification layer.
- Prior approaches often fed learned hidden representations into separate linear or kernel SVMs, leaving lower-level features unfine-tuned for the SVM objective.
- The paper jointly backpropagates through a linear SVM top layer and lower-level features, using differentiable L2-SVM loss instead of standard hinge loss.
- Compared with softmax networks, the proposed models achieve superior performance on MNIST, CIFAR-10, and a facial-expression recognition competition.
- The authors believe the performance gain primarily reflects stronger regularization from the SVM loss rather than better parameter optimization.
2. The model
The model replaces softmax’s cross-entropy objective with a linear SVM objective while retaining the same prediction rule and enabling backpropagation into lower layers. It uses L2-SVM because its squared hinge loss is differentiable and more strongly penalizes margin violations.
- 2. The model: L2-SVM minimizes squared hinge loss, providing a differentiable alternative to the nondifferentiable standard hinge loss.The squared loss imposes a quadratic rather than linear penalty on margin-violating points.
- 2. The model: Linear SVMs are used because kernel-SVM optimization must occur in the dual and kernel methods present scalability problems.The paper therefore restricts its deep-learning models to linear SVMs.
- 2.3. Multiclass SVMs: For K classes, the model trains K independent linear SVMs using a one-vs-rest construction.Data from the other classes provide the negative cases for each SVM.
- 2. The model: Softmax and multiclass SVMs make identical predictions; they differ in their training objectives.Softmax minimizes cross-entropy, whereas SVMs maximize the margin between classes.
- 2. The model: The paper trains lower-layer neural-network weights by backpropagating gradients from a top-layer linear SVM.The backpropagation procedure after differentiating the SVM objective is otherwise the same as in softmax-based networks.
3. Experiments
The experiments compare softmax and DLSVM classifiers across face-expression recognition, MNIST, and CIFAR-10, with additional analysis of their objectives. DLSVM generally performs better, while the authors attribute the gain mainly to regularization.
- 3.1. Facial Expression Recognition: 71.2% private test accuracy was achieved in the face-expression challenge, compared with 69.4% on the public validation leaderboard.The private score was almost 2% higher than the second-place team’s score.
- 3.1.1. Softmax vs. DLSVM: DLSVM maintains a small but clear performance gain over softmax during eight-fold cross-validation as training proceeds.The comparison uses otherwise similar convolutional models with separately selected hyperparameters.
- 3.2. MNIST: 0.87% test error was obtained by DLSVM on MNIST, versus 0.99% for softmax.The models differed only in their final layer for this experiment.
- 3.3. CIFAR-10: The CIFAR-10 experiment used a convolutional network with two filtering and pooling stages, a 3072-unit penultimate layer, and separately validated hyperparameters.The supplied passages describe the model and comparison but do not provide the corresponding test-error values.
- 3.4. Regularization or Optimization: The authors report limited evidence that DLSVM’s gain is mainly due to its objective function and superior regularization rather than better parameter optimization.Lower cross-entropy did not necessarily correspond to lower error, and continued softmax training from DLSVM weights increased error toward 14%.
4. Conclusions
The paper concludes that DLSVM outperforms softmax across two standard datasets and one recent dataset, while encouraging further study of multiclass formulations and performance gains.
- DLSVM works better than softmax on two standard datasets and a recent dataset.
- Switching from softmax to SVMs is described as simple and useful for classification tasks.
- Further research should explore other multiclass SVM formulations and clarify where and how much the gain is obtained.