Source-linked AI summary
IGRF-RFE: A Hybrid Feature Selection Method for MLP-based Network Intrusion Detection on UNSW-NB15 Dataset
Yuhua Yin, Julian Jang-Jaccard, Wen Xu, Amardeep Singh, Jinting Zhu, Fariza Sabrina, Jin Kwak
TL;DR
High-dimensional, redundant, and irrelevant features can impair intrusion-detection models. The paper proposes IGRF-RFE, combining Information Gain, Random Forest importance, and recursive feature elimination with an MLP; on UNSW-NB15, it reduces features from 42 to 23 and raises accuracy from 82.25% to 84.24%.
Problem
High-dimensional, redundant, and irrelevant features can cause overfitting and high false-positive rates in intrusion-detection models.
Method
IGRF-RFE first combines Information Gain and Random Forest importance to reduce the feature search space, then applies wrapper-based recursive feature elimination with an MLP.
Results
84.24% MLP multi-classification accuracy was achieved after reducing the feature dimension from 42 to 23, compared with 82.25% before feature selection.
Takeaways & Limitations
IGRF-RFE effectively selects important features and improves intrusion-detection performance on the UNSW-NB15 dataset.
Takeaways & Limitations
Existing filter methods may rank correlated features similarly and overlook interactions, leaving redundant variables that wrapper selection is intended to address.
Abstract
from arXiv · showhide
The effectiveness of machine learning models is significantly affected by the size of the dataset and the quality of features as redundant and irrelevant features can radically degrade the performance. This paper proposes IGRF-RFE: a hybrid feature selection method tasked for multi-class network anomalies using a Multilayer perceptron (MLP) network. IGRF-RFE can be considered as a feature reduction technique based on both the filter feature selection method and the wrapper feature selection method. In our proposed method, we use the filter feature selection method, which is the combination of Information Gain and Random Forest Importance, to reduce the feature subset search space. Then, we apply recursive feature elimination(RFE) as a wrapper feature selection method to further eliminate redundant features recursively on the reduced feature subsets. Our experimental results obtained based on the UNSW-NB15 dataset confirm that our proposed method can improve the accuracy of anomaly detection while reducing the feature dimension. The results show that the feature dimension is reduced from 42 to 23 while the multi-classification accuracy of MLP is improved from 82.25% to 84.24%.
Introduction
Network attacks and increasingly complex traffic motivate intrusion detection, while high-dimensional redundant features can impair machine-learning performance. The paper proposes IGRF-RFE, combining Information Gain, Random Forest Importance, and RFE for MLP-based multi-class intrusion detection on UNSW-NB15.
- Motivation: Network attacks have become a major challenge as network connections and services increase, motivating intrusion detection systems.The passage distinguishes active attacks, which affect system usability, from passive attacks, which capture computer-system information.
- Motivation: Signature-based IDS can miss unknown attacks because of zero-day attacks and outdated signature databases.Modern IDS are categorized as signature-based, anomaly-based, or hybrid systems.
- Motivation: High-dimensional, redundant, and irrelevant features may cause overfitting and high false-positive rates in real network environments.Feature selection is therefore widely studied to improve IDS performance and reduce false positives.
- Proposed approach: IGRF-RFE combines filter and wrapper selection: Information Gain and Random Forest Importance reduce the search space, then RFE removes redundant features.The method is designed for multi-classification performance on UNSW-NB15 using an MLP-based intrusion detection system.
Related works
Feature-selection research includes filter, embedded, wrapper, and hybrid approaches, but existing methods may mishandle correlated or interacting features. The paper positions wrapper selection as a complement to univariate filtering so feature relevance can be assessed within subsets.
- Feature-selection methods: Filter methods rank features using statistical measures, information distance, or correlations, whereas wrapper methods evaluate subsets through actual model performance.Embedded methods derive feature importance from tree-based algorithms.
- Feature-selection methods: Wrapper methods are model-dependent and train the model for each subset evaluation, making search-time reduction useful.Random search and other methods are typically combined with wrappers to reduce selection time.
- Prior studies: Other related studies used random forests with DNNs, optimizer-based selection, or combined Information Gain and ReliefF methods across intrusion-detection datasets.One cited DNN study did not consider multi-classification performance under its proposed model.
- Prior studies: Prior UNSW-NB15 studies reported varied feature-selection strategies, including Information Gain with SMOTE, XGboost ranking, and correlation-based selection.Reported examples include 85.78% accuracy for IG-TS and selection of 19 features using XGboost.
- Limitations of prior work: Existing tree-based methods may assign equal importance to correlated features, affecting interpretability.The related-work discussion identifies combining filter methods as one way to address limitations of a single method.
- Limitations of prior work: Univariate filters rank features independently and may ignore feature interactions, so redundant variables can remain.Wrapper methods can account for relative relevance across features within the same subset space.
Proposed method
The proposed method preprocesses UNSW-NB15 data, narrows numerical features with Information Gain and Random Forest importance, then applies RFE with an MLP to optimize the remaining subset. The MLP uses two 128-neuron ReLU hidden layers, batch normalization, and Softmax output for multi-class classification.
- UNSW-NB15 data are cleaned, encoded, normalized, and divided into training, validation, and test sets before MLP processing.
- Ensemble Feature Selection: Information Gain and Random Forest importance rank the 39 numerical features, while the 3 categorical features are preserved in the first selection stage.Importance scores are normalized, thresholded, and combined through the union of retained feature subsets.
- Recursive Feature Elimination: Recursive Feature Elimination iteratively removes features from the reduced subset according to machine-learning performance until the best performance or a specified feature count is reached.Its evaluation score averages accuracy across 10 experiments with different random seeds, and a patience parameter can stop the process.
- Model Training and Validation: The MLP is a feed-forward network whose input and output layer sizes correspond to feature and class counts, respectively.The network computes layer outputs using weights and biases, then applies activation functions.
- Model Training and Validation: The specified MLP has two hidden layers with 128 ReLU neurons each, batch normalization after each hidden layer, and Softmax class probabilities at the output.Selected and preprocessed features enter the network, which is trained through forward and backward propagation.
- O(n^2) is the worst-case computational complexity of IGRF-RFE, matching the complexity of RFE after the first-stage IG and RF operations.The two first-stage feature-selection algorithms each have worst-case complexity O(n).
Experiments and results
The experiments use the 10% UNSW-NB15 dataset and a documented software and hardware environment. The dataset includes modern attack traffic and 42 meaningful features after removing label columns and other irrelevant data.
- Experimental environment: The experiments ran on Ubuntu 20.04.4 with 16GB RAM, an AMD Ryzen 7 2700 processor, and an AMD RX580 graphics card.The environment used Python 3.8, TensorFlow 2.4.1, Scikit-Learn, Numpy, pandas, and matplotlib.
- Dataset: UNSW-NB15 was created from 100 GB of normal and modern attack traffic and contains one normal class and nine attack classes.The dataset was developed to address limitations of older intrusion-detection datasets.
- Dataset: The 10% UNSW-NB15 partition contains 175,341 training records and 82,332 test records.The training and test distributions were reported as highly correlated.
- Feature data: The 10% dataset contains 42 features, including 38 numerical features, after removing a few meaningless features.The source describes the 42-feature subset as meaningful features used for modeling.
Data pre‑processing
Pre-processing removes unusable or out-of-scope data, addresses class imbalance, and uses oversampling to balance normal and abnormal samples before modeling.
- Cleaning and labels: The binary label was removed because the MLP model performs multi-class intrusion classification, and 44 rows with null values were cleaned.The multi-class label ‘attack_cat’ was retained for the task.
- Class handling: Four minority classes—Analysis, Backdoor, Shellcode, and Worms—were removed because they represented very small portions of the training set.Their reported proportions were 1.141%, 0.996%, 0.646%, and 0.074%, respectively.
- Class handling: The normal class represented 32.9% of the training set but 45.9% of the test set, creating a distribution imbalance.The authors associate this mismatch with possible overfitting toward abnormal classes.
- Class handling: The normal class was double-sampled to improve balance between normal and abnormal classes.The stated purpose was to address the training–test imbalance in normal-class proportions.
One‑hot encoding
Categorical features are converted into binary indicators through one-hot encoding, and feature values are normalized to a common 0–1 range for MLP training.
- Normalization: MinMax normalization converts feature values to the range 0 to 1 to reduce scale-related bias during MLP training.The transformation uses each feature vector’s minimum and maximum values.
Training, validation, and test set preparation
The preparation pipeline separates data for fitting, validation, and testing while using PCA to visualize class distributions. The PCA visualization shows substantial overlap among attack and normal classes in three dimensions.
- PCA visualization: PCA reduced the original training and test sets to three dimensions to visualize their class distributions.The visualization is presented as an interpretability aid rather than a representation of all data dimensions.
- PCA visualization: Different attack classes and the normal class substantially overlap in the three-dimensional PCA latent space.The passage explicitly cautions that PCA visualization cannot represent all dimensions of the data.
- Dataset splitting: Training data fits the model, validation data estimates training loss, and test data evaluates model performance.The three sets should contain separate samples to avoid performance bias from data leakage.
- Dataset splitting: The paper includes a table recording the training, validation, and test sets used in the model.The supplied table title identifies the three dataset partitions.
Evaluation metrics
The paper evaluates multi-classification using accuracy, recall, precision, false positive rate, F1 score, and AUC-ROC. These metrics quantify classification correctness, class-specific performance, error rates, balance between recall and precision, and threshold-based prediction quality.
- Accuracy measures the ratio of correctly classified samples to all samples.
- Recall measures correctly classified positive samples relative to all samples that should be positive.
- Precision measures correctly classified positive samples relative to all samples predicted as positive.
- False positive rate measures incorrectly classified positive samples relative to samples that should be negative.
- F1 score is the harmonic mean of recall and precision and addresses their limitations with multi-class imbalanced data.
- The ROC curve displays FPR and TPR across thresholds, while AUC summarizes the area under that curve for judging model performance.
Results
The evaluation applies information gain and random-forest ranking before RFE, producing a 23-feature subset for MLP-based multi-class intrusion detection. The resulting model achieves 84.24% accuracy, with performance varying substantially across attack classes and error types.
- Filter feature selection: 22 features were retained by IG and 19 by RF, with 17 shared features yielding a 24-feature numeric union.
- Wrapper feature selection: RFE applied to 24 numeric and 3 categorical features selected 23 important features, comprising 20 numeric and 3 categorical features.The process averaged scores over 10 random seeds from 2022 to 2031 and stopped after five non-improving iterations.
- Multi-classification performance: 84.24% accuracy was achieved by the MLP model using IGRF-RFE on UNSW-NB15.Generic and normal classes had the highest F1 scores, at 98.20% and 93.11%, respectively.
- Multi-classification performance: 11.09% and 42.26% were the F1 scores for DoS and Fuzzer attacks, respectively, which the paper relates to their limited training-set representation.DoS comprised 5.4% of the training set and Fuzzer 8.0%.
- Error rates: 0.0403 was the model’s weighted FPR, while Exploits had the highest class-specific FPR at 0.993.Generic had the lowest FPR at 0.0005, and DoS had an FPR of 0.0062 despite its lowest F1 score.
- ROC analysis: AUC was 1 for Generic and 0.99 for normal, compared with 0.95 for DoS and 0.89 for Fuzzer.The ROC curves were generated using one-versus-all classification for each class.
Comparison
IGRF-RFE outperforms the alternative feature subsets and similar MLP-based methods evaluated on UNSW-NB15. The comparison reports higher accuracy and weighted F1 than using all features, standalone selectors, and their combinations.
- Feature-subset comparison: Nearly 2% higher accuracy and 2.6% higher weighted F1 were obtained than when using all features.
- Feature-subset comparison: IG, RF, and their intersection improved performance over some alternatives but remained below IGRF-RFE on every evaluation metric.
- Comparison with prior work: Among similar MLP-based studies, IGRF-RFE achieved better accuracy and F1 score while selecting 23 important features.
- Comparison with prior work: The reported results remain competitive despite using UNSW-NB15’s 10% pre-partitioned dataset, whereas other studies used different data amounts.
Conclusion and future work
The paper concludes that IGRF-RFE reduces features for MLP-based intrusion detection while achieving reported accuracy and weighted F1 improvements on UNSW-NB15. Future work will test other datasets, resampling techniques, and machine-learning models.
- Conclusion: 84.24% accuracy and 82.85% weighted F1 were achieved with IGRF-RFE on UNSW-NB15.The method used 20 numerical and 3 categorical features after two feature-reduction stages.
- Conclusion: IGRF-RFE has worst-case computational complexity O(n^2), equivalent to normal RFE, and its patient parameter can stop the algorithm earlier.
- Future work: Future work will apply IGRF-RFE to different intrusion-detection datasets with advanced resampling techniques and other machine-learning models.