Source-linked AI summary
An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification
Abien Fred Agarap
TL;DR
The paper examines whether replacing the conventional Softmax classifier with a linear SVM in a CNN architecture is effective for image classification. It emulates a CNN–SVM design with a simple base CNN and compares it with CNN–Softmax, finding that Softmax achieved better MNIST classification accuracy while matching prior Fashion-MNIST findings. The authors conclude that improved preprocessing and a more sophisticated base CNN could further validate or reproduce prior CNN–SVM results.
Problem
Prior studies challenged the conventional use of Softmax by investigating SVM as an alternative classifier in neural network architectures.
Method
The paper emulates an architecture combining a convolutional neural network with a linear SVM and compares it with CNN–Softmax using a simple CNN model.
Results
CNN–Softmax achieved better MNIST test accuracy than CNN–SVM, while its Fashion-MNIST test accuracy matched prior findings obtained without data preprocessing.
Takeaways & Limitations
Further validation of the CNN–SVM approach should use data preprocessing and a relatively more sophisticated base CNN model.
Takeaways & Limitations
The study's simple procedure and lack of data preprocessing limit direct reproduction of results from a more sophisticated prior methodology.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) are similar to "ordinary" neural networks in the sense that they are made up of hidden layers consisting of neurons with "learnable" parameters. These neurons receive inputs, performs a dot product, and then follows it with a non-linearity. The whole network expresses the mapping between raw image pixels and their class scores. Conventionally, the Softmax function is the classifier used at the last layer of this network. However, there have been studies (Alalshekmubarak and Smith, 2013; Agarap, 2017; Tang, 2013) conducted to challenge this norm. The cited studies introduce the usage of linear support vector machine (SVM) in an artificial neural network architecture. This project is yet another take on the subject, and is inspired by (Tang, 2013). Empirical data has shown that the CNN-SVM model was able to achieve a test accuracy of ~99.04% using the MNIST dataset (LeCun, Cortes, and Burges, 2010). On the other hand, the CNN-Softmax was able to achieve a test accuracy of ~99.23% using the same dataset. Both models were also tested on the recently-published Fashion-MNIST dataset (Xiao, Rasul, and Vollgraf, 2017), which is suppose to be a more difficult image classification dataset than MNIST (Zalandoresearch, 2017). This proved to be the case as CNN-SVM reached a test accuracy of ~90.72%, while the CNN-Softmax reached a test accuracy of ~91.86%. The said results may be improved if data preprocessing techniques were employed on the datasets, and if the base CNN model was a relatively more sophisticated than the one used in this study.
1 INTRODUCTION
Deep-learning image classifiers commonly use Softmax, but prior studies explored SVM as an alternative and reported relatively better results. This paper emulates a CNN–linear SVM architecture while using a simpler CNN and methodology than prior work.
- Deep-learning models commonly employ Softmax at the classification layer across image classification and other tasks.
- Prior studies investigated SVM as an alternative to Softmax in artificial neural network architectures and reported relatively better results.
- SVM classification is restricted to binary problems, while multinomial use becomes a one-versus-all formulation.
- The paper emulates a CNN–linear SVM architecture for image classification using a simple two-convolutional-layer CNN with max pooling.
2 METHODOLOGY
The study combines CNN components with SVM-based classification and evaluates CNN-SVM alongside CNN-Softmax on MNIST and Fashion-MNIST without preprocessing.
- Google TensorFlow was used to implement the deep learning algorithms.
- 2.2 The Dataset: MNIST is a 10-class grayscale dataset with 60,000 training examples and 10,000 test cases, while Fashion-MNIST shares its class count and color profile.
- 2.2 The Dataset: Both datasets were used without preprocessing such as normalization or dimensionality reduction.
- 2.3 Support Vector Machine (SVM): The SVM separates classes using f(w, x) = w · x + b, with parameters learned through an optimization problem and L1- or L2-based formulations.L2-SVM uses the Euclidean norm and squared hinge loss, while the cited text describes it as providing more stable results.
- 2.4 Convolutional Neural Network (CNN): The CNN uses convolutional layers, pooling, and nonlinear activations; convolution produces activation maps, while pooling downsamples inputs and reduces parameters.
- 2.4 Convolutional Neural Network (CNN): The base model uses a simple architecture, and its 10th layer replaces softmax with L2-SVM loss while Adam learns the weights.
- Experiments: The experiments train and test CNN-SVM and CNN-Softmax on both datasets, considering training accuracy, training loss, and test accuracy.Testing followed 10,000 training steps.
3 EXPERIMENTS
The experiments compare CNN-Softmax and CNN-SVM on MNIST and Fashion-MNIST using training accuracy, training loss, and test accuracy. CNN-Softmax achieved higher reported training accuracy and lower training loss in both datasets, while its test accuracy also exceeded CNN-SVM on MNIST.
- MNIST: 98.4765625% average training accuracy and 0.136794931 average training loss were reported for CNN-Softmax on MNIST, versus 97.671875% and 0.268976859 for CNN-SVM.Both models completed 10,000 training steps in 4 minutes and 16 seconds.
- Test results: Table 3 reports test accuracies for CNN-Softmax and CNN-SVM on both MNIST and Fashion-MNIST.Each dataset had 10,000 test cases.
- Test results: CNN-Softmax had better classification accuracy than CNN-SVM on the MNIST test set.The paper states that this result did not corroborate the findings in [11].
4 CONCLUSION AND RECOMMENDATION
The paper recommends improving the methodology to further validate the CNN-SVM approach. It specifically identifies data preprocessing and a more sophisticated base CNN model as possible improvements for reproducing the related study’s results.
- Recommendation: The study recommends methodological improvements to further validate the proposed CNN-SVM approach.This recommendation follows the study’s contradiction of the findings in [11].
- Recommendation: Data preprocessing and a more sophisticated base CNN model are hypothesized to help reproduce the results in [11].The paper describes the implemented base CNN as relatively simple and the procedure as lacking data preprocessing.