Source-linked AI summary
Feature Extraction for Machine Learning-based Intrusion Detection in IoT Networks
Mohanad Sarhan, Siamak Layeghy, Nour Moustafa, Marcus Gallagher, Marius Portmann
TL;DR
IoT NIDSs remain unreliable, and prior ML/FE studies provide limited evidence that their gains generalise across differing datasets. This paper evaluates six ML models and three FE algorithms across three benchmark datasets, identifying dataset-dependent optimal dimensions and combinations rather than one universally best method. The findings support greater emphasis on generalisability and a universal, compact feature set for future NIDS datasets.
Problem
Current IoT NIDSs are unreliable, while prior studies often optimise results on individual datasets whose differing feature sets, attack types, and network designs limit evidence of generalisability.
Method
The paper evaluates six ML models with PCA, AE, and LDA across UNSW-NB15, ToN-IoT, and CSE-CIC-IDS2018, analysing extracted dimensions and variance.
Results
No single FE–ML combination performs best across all three datasets; optimal extracted dimensions are dataset-dependent, and LDA degrades ML performance on two datasets.
Takeaways & Limitations
The choice of dataset significantly alters technique performance, motivating a compact universal feature set generated from live traffic headers without dependence on limited protocols or attack types.
Takeaways & Limitations
Finding an FE–ML combination that performs well across diverse datasets and practical application scenarios remains far from trivial and requires further investigation.
Abstract
from arXiv · showhide
A large number of network security breaches in IoT networks have demonstrated the unreliability of current Network Intrusion Detection Systems (NIDSs). Consequently, network interruptions and loss of sensitive data have occurred, which led to an active research area for improving NIDS technologies. In an analysis of related works, it was observed that most researchers aim to obtain better classification results by using a set of untried combinations of Feature Reduction (FR) and Machine Learning (ML) techniques on NIDS datasets. However, these datasets are different in feature sets, attack types, and network design. Therefore, this paper aims to discover whether these techniques can be generalised across various datasets. Six ML models are utilised: a Deep Feed Forward (DFF), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Decision Tree (DT), Logistic Regression (LR), and Naive Bayes (NB). The accuracy of three Feature Extraction (FE) algorithms; Principal Component Analysis (PCA), Auto-encoder (AE), and Linear Discriminant Analysis (LDA), are evaluated using three benchmark datasets: UNSW-NB15, ToN-IoT and CSE-CIC-IDS2018. Although PCA and AE algorithms have been widely used, the determination of their optimal number of extracted dimensions has been overlooked. The results indicate that no clear FE method or ML model can achieve the best scores for all datasets. The optimal number of extracted dimensions has been identified for each dataset, and LDA degrades the performance of the ML models on two datasets. The variance is used to analyse the extracted dimensions of LDA and PCA. Finally, this paper concludes that the choice of datasets significantly alters the performance of the applied techniques. We believe that a universal (benchmark) feature set is needed to facilitate further advancement and progress of research in this field.
1 Introduction
IoT growth and increasingly sophisticated attacks expose weaknesses in conventional NIDSs, while ML-based approaches remain difficult to generalise across datasets. This paper evaluates 18 FE–ML combinations across three benchmark datasets and examines feature dimensions and variance.
- Motivation: IoT expansion has increased cyber-security risks, while current security measures have proven unreliable against attacks on networked devices.NIDSs analyse traffic flows to detect threats and protect confidentiality, integrity, and availability, but existing protections remain insufficient.
- Motivation: Signature-based NIDSs are unreliable against zero-day attacks because their databases lack previously known attack signatures.ML can learn harmful patterns from network traffic, and DL models can extract complex patterns from features such as packet counts, protocols, services, and flags.
- Research gap: Existing ML and feature-reduction studies often pursue better results on specific datasets rather than reliable production IoT deployment or cross-dataset insight.The paper identifies generalisability as a central concern because datasets differ in features, attack types, and network design.
- Approach: The paper evaluates 18 combinations of six ML models and three FE algorithms across UNSW-NB15, ToN-IoT, and CSE-CIC-IDS2018.The study also compares results without feature extraction, investigates extracted-dimension counts, and analyses feature variance and its relationship with detection accuracy.
2 Related works
Prior work reports promising dataset-specific results using diverse ML and FE methods, but limited cross-dataset evaluation weakens claims of generalisability. The reviewed literature also includes unresolved preprocessing, dimension-selection, and realistic-deployment concerns.
- Dataset selection: Recent datasets are preferred because evolving networks produce new attack scenarios and older traffic patterns may have limited significance.The review focuses on UNSW-NB15, ToN-IoT, and CSE-CIC-IDS2018, including ToN-IoT because few studies had examined it at the time.
- Prior ML studies: A CNN study on UNSW-NB15 achieved 85.86% accuracy on balanced data and 91.2% on unbalanced data after minority-class oversampling.The study also compared hidden-layer configurations, LSTM inclusion, and activation functions.
- Feature extraction: On UNSW-NB15, AE outperformed PCA for KNN and DFF, while DT results were similar; 20 dimensions were optimal for UNSW-NB15 but not NSL-KDD.This illustrates that feature-extraction effects and optimal dimensionality can vary by classifier and dataset.
- Methodological concerns: Prior studies varied in preprocessing and model design, with one CSE-CIC-IDS2018 study omitting preprocessing and reporting longer training times as hidden nodes and learning rates increased.Another study removed fuzzers and exploits from some results after poor detection, limiting realism of those evaluations.
- Research gap: Most prior studies used a single dataset, leaving generalisation claims uncertain because each dataset contains its own feature set and information variation.The literature also tends to optimise small numerical gains through new FR methods, ML models, or dataset-specific hyper-parameters.
3 Methodology
The methodology preprocesses three NIDS datasets, extracts reduced feature representations, and compares six ML models across PCA, LDA, and AE configurations. It also varies extracted dimensions for PCA and AE and addresses class imbalance during model training.
- 3.2 Feature extraction: PCA and AE are evaluated with 1, 2, 3, 4, 5, 10, 20, and 30 extracted dimensions to examine their effect on classification performance.The selected dimensions are intended to identify an optimal number of extracted features.
- 3.1 Data processing: Three publicly released NIDS datasets are processed by removing duplicates and identifiers, encoding categorical values, replacing invalid values, and scaling features to 0–1.The preprocessing aims to reduce redundancy and prediction bias while making features numerically suitable and comparably weighted for ML models.
- 3.2 Feature extraction: Feature extraction reduces dataset dimensionality by projecting raw features into fewer features while minimising informational loss.The study evaluates PCA, LDA, and AE as its three feature-extraction algorithms.
- 3.3 Machine learning: Six classifiers are compared: DFF, CNN, RNN, DT, LR, and NB, spanning deep-learning and shallow-learning models.The supplied methodology describes architectures or implementation choices for DFF, CNN, RNN, LR, and DT; NB is included in the comparison set.
- 3.3 Machine learning: All three datasets exhibit class imbalance, which can bias learning toward the majority class and leave minority attack classes poorly fitted.The methodology identifies minority attack classes in two datasets as a critical imbalance case.
4 Results and discussion
Across UNSW-NB15, ToN-IoT, and CSE-CIC-IDS2018, feature-extraction effects depended strongly on the dataset and classifier. The experiments identified dataset-specific dimensional choices and found no universally best FE–ML combination.
- Experimental setup: 18 FE–ML combinations were evaluated across three NIDS datasets using five-fold stratified testing and mean results.The comparison used AUC initially and varied extracted dimensions for each model and dataset.
- ToN-IoT: On ToN-IoT, DT performed best on the full dataset or AE-extracted dimensions, whereas LDA and PCA often produced defective DT results.Using AE with 10 dimensions, DT reached a detection rate of 98.28% and false alarm rate of 3.21%; LDA improved RNN and NB but reduced DFF, CNN, and DT.
- CSE-CIC-IDS2018: On CSE-CIC-IDS2018, DFF and CNN performed best on the full dataset, FE improved RNN, and LDA generally underperformed AE and PCA except for NB.The reported optimal dimensions were 20 for PCA and 10 for AE; LR and NB were ineffective for attack detection.
- Dimension analysis: The first 10 PCA dimensions accounted for most variance, explaining why additional dimensions produced little further classification improvement.The study also associated UNSW-NB15’s higher LDA variance with classification accuracy close to the full feature set.
- UNSW-NB15: On UNSW-NB15, CNN achieved the strongest result with AE, while LDA at one dimension performed close to the full dataset for most classifiers.LDA performed better with NB but worse with DT; AE and PCA generally stabilised after a small number of dimensions for several models.
- Cross-dataset comparison: The best model differed by dataset: CNN with AE on UNSW-NB15, and DT with AE on ToN-IoT and CSE-CIC-IDS2018.The authors attribute the lack of a universal best method to substantial differences in dataset feature sets and argue for a compact, broadly applicable benchmark feature set.
5 Conclusions
The study evaluates feature-extraction and machine-learning combinations across three benchmark NIDS datasets, identifying dataset-specific optimal dimensions. No single combination performs best across all datasets, underscoring the challenge of generalisation.
- PCA, autoencoder, and LDA were evaluated with DFF, CNN, RNN, LR, DT, and NB models across three benchmark NIDS datasets.
- Variance analysis identified an optimal number of extracted dimensions for each dataset and assessed its relationship with classification performance.
- No single FE–ML combination performed best across all three NIDS datasets, making broadly generalisable practical deployments difficult.
- The authors argue that generic, efficient feature sets are needed to enable broader comparisons across datasets and practical network settings.