Source-linked AI summary
A Neural Network Architecture Combining Gated Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection in Network Traffic Data
Abien Fred Agarap
TL;DR
The paper addresses binary intrusion detection on sequential network traffic, where manual activity analysis is laborious. It replaces GRU’s Softmax output and cross-entropy loss with SVM-based classification and a margin-based objective. GRU-SVM outperformed GRU-Softmax in reported accuracy and training and testing time on the Kyoto University honeypot data.
Problem
Manual analysis of massive user-activity data for intrusion detection is laborious, motivating automated machine-learning approaches for sequential network traffic.
Method
The paper replaces GRU’s Softmax output with linear SVM and replaces cross-entropy with an SVM margin-based loss for binary classification.
Results
GRU-SVM achieved approximately 81.54% training accuracy and 84.15% testing accuracy, versus GRU-Softmax’s approximately 63.07% and 70.75%.
Takeaways & Limitations
Empirical evidence supports GRU-SVM over GRU-Softmax for predictive accuracy and training and testing time in this binary intrusion-detection experiment.
Takeaways & Limitations
Further work is needed to validate GRU-SVM on other binary classification tasks and to study faster multinomial classification.
Abstract
from arXiv · showhide
Gated Recurrent Unit (GRU) is a recently-developed variation of the long short-term memory (LSTM) unit, both of which are types of recurrent neural network (RNN). Through empirical evidence, both models have been proven to be effective in a wide variety of machine learning tasks such as natural language processing (Wen et al., 2015), speech recognition (Chorowski et al., 2015), and text classification (Yang et al., 2016). Conventionally, like most neural networks, both of the aforementioned RNN variants employ the Softmax function as its final output layer for its prediction, and the cross-entropy function for computing its loss. In this paper, we present an amendment to this norm by introducing linear support vector machine (SVM) as the replacement for Softmax in the final output layer of a GRU model. Furthermore, the cross-entropy function shall be replaced with a margin-based function. While there have been similar studies (Alalshekmubarak & Smith, 2013; Tang, 2013), this proposal is primarily intended for binary classification on intrusion detection using the 2013 network traffic data from the honeypot systems of Kyoto University. Results show that the GRU-SVM model performs relatively higher than the conventional GRU-Softmax model. The proposed model reached a training accuracy of ~81.54% and a testing accuracy of ~84.15%, while the latter was able to reach a training accuracy of ~63.07% and a testing accuracy of ~70.75%. In addition, the juxtaposition of these two final output layers indicate that the SVM would outperform Softmax in prediction time - a theoretical implication which was supported by the actual training and testing time in the study.
1 INTRODUCTION
The paper motivates automated intrusion detection because manual analysis of massive user-activity data is laborious, and proposes adapting recurrent models with SVM for sequential network traffic.
- Cybercrime was projected to cost the global economy $2 trillion by 2019.
- Intrusion detection systems identify illegal or unauthorized network or system use by attackers.
- Manual analysis of massive user-activity data is laborious, motivating automation through machine learning.
- SVM separates two classes using a hyperplane, while ANN models information transfer between neurons.
- The proposed approach replaces ESN with GRU RNNs for intrusion detection because network traffic data is sequential.
2 METHODOLOGY
The methodology preprocesses Kyoto University honeypot traffic data and trains a GRU-SVM architecture that replaces Softmax and cross-entropy with SVM-based prediction and loss optimization.
- Data preprocessing: The study uses 2013 Kyoto University honeypot network traffic data with 24 statistical features, although only 22 dataset features were used.
- Data preprocessing: Only 25% of the 16.2 GB dataset, approximately 4.1 GB, was used after normalization, indexing, and discretization.
- Data preprocessing: Continuous features were decile-binned and one-hot encoded, while categorical values were indexed using Scikit-learn preprocessing functions.
- GRU-SVM architecture: The proposed architecture combines GRU recurrent units with SVM as the classifier for binary classification.
- GRU-SVM architecture: GRU parameters are learned through gating, while the final SVM layer is trained by optimizing an SVM objective instead of cross-entropy loss.
- GRU-SVM architecture: The architecture uses differentiable L2-SVM because it is more stable than the nondifferentiable L1-SVM.
- GRU-SVM architecture: SVM scores are converted into predicted class labels by selecting the highest-scoring class with argmax.
- Experimental design: The experiment trains and tests GRU-SVM and GRU-Softmax on the same dataset using 80% for training and 20% for testing after duplicate removal.
3 RESULTS
The experiment compares GRU-SVM and conventional GRU-Softmax on the same intrusion-detection data, measuring performance and runtime during training and testing.
- Experimental setup: Both models were trained for 5 epochs on 1,898,240 network-traffic lines and tested on 420,608 lines.These values were selected because they are divisible by the batch size of 256.
- Reported results: Training and testing results were summarized for both GRU-SVM and GRU-Softmax, while losses were excluded from further comparison because they used different scales.Tables 4 and 5 report binary-classification statistical measures for training and testing.
- Runtime comparison: 16 minutes 43 seconds was the GRU-SVM training time, compared with 17 minutes 11 seconds for GRU-Softmax.The training comparison used 5 epochs over the 1,898,240-line training dataset.
- Runtime comparison: 1 minute 22 seconds was the GRU-SVM testing time, compared with 1 minute 40 seconds for GRU-Softmax.The testing comparison used 5 epochs over 420,608 network-traffic lines, totaling 2,103,040 predictions.
4 DISCUSSION
The study reports that GRU-SVM outperformed GRU-Softmax in prediction accuracy and training and testing time. It attributes the accuracy difference to Softmax’s behavior in binary classification and the computational advantage of SVM’s predictor.
- SVM outperformed Softmax in prediction accuracy when used as a neural-network output layer.
- ≈81.54% training accuracy and ≈84.15% testing accuracy were reported for GRU-SVM, versus ≈63.07% and ≈70.75% for GRU-Softmax.
- Softmax’s probability-distribution objective is described as excessive for binary classification, whereas SVM requires class margins to be satisfied.
- The sigmoid graph is used to infer that small gradients can contribute to vanishing gradients, a problem addressed by LSTM and GRU variants.
- SVM’s predictor has O(1) complexity compared with O(n) for Softmax, and GRU-SVM also outperformed GRU-Softmax in training and testing time.
5 CONCLUSION AND RECOMMENDATION
The paper proposes replacing GRU’s Softmax output layer with SVM for binary classification and evaluates the amendment against GRU-Softmax. The reported experiment found higher effectiveness for GRU-SVM in predictive accuracy and training and testing time, while further validation remains necessary.
- The proposed amendment replaces the GRU RNN’s final output layer with SVM for binary, non-probabilistic classification.
- The experiment compared the proposed GRU-SVM model with the established GRU-Softmax model.
- GRU-SVM was more effective than GRU-Softmax in predictive accuracy, training time, and testing time.
- Further work is required to validate GRU-SVM’s effectiveness in other binary classification tasks.