PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning for Encrypted Malicious Traffic Detection: Approaches, Datasets and Comparative Study

Zihao Wang, Kar-Wai Fok, Vrizlynn L. L. Thing

arXiv:2203.09332v1cs.CRcs.AIcs.LG

TL;DR

Encrypted traffic lets malicious activity evade detection, while the lack of well-recognized datasets and feature sets makes reliable comparison difficult. The paper reviews machine-learning detection methods, combines public datasets, and compares algorithms and features; its experiments identify strong algorithm-feature combinations, including FOS with RF or XGBoost.

  • Problem

    Encrypted traffic can shield malicious activity from detection, and the lack of established datasets and feature sets makes reliable performance comparisons difficult.

    Source: §I · p. 1

  • Method

    The paper proposes a detection framework and systematic review, combines five public datasets, and compares ten algorithms and five feature sets on the same dataset.

    Source: §I · pp. 1–2, §VI · p. 18

  • Results

    FOS with either RF or XGBoost achieves TPR higher than 92% and FPR lower than 6%.

    Source: C. Experiment Setup · p. 16

  • Takeaways & Limitations

    The combined dataset and comparative experiments are intended to facilitate future research and technology evaluation in encrypted malicious traffic detection.

    Source: §VI · p. 18

    Protocol-specific features apply only to their related protocols, while extracting protocol-agnostic numerical features is complex, time-consuming, and requires prior knowledge.

    Source: C. Traffic and Feature Extraction · p. 8

Abstract

from arXiv · show

As people's demand for personal privacy and data security becomes a priority, encrypted traffic has become mainstream in the cyber world. However, traffic encryption is also shielding malicious and illegal traffic introduced by adversaries, from being detected. This is especially so in the post-COVID-19 environment where malicious traffic encryption is growing rapidly. Common security solutions that rely on plain payload content analysis such as deep packet inspection are rendered useless. Thus, machine learning based approaches have become an important direction for encrypted malicious traffic detection. In this paper, we formulate a universal framework of machine learning based encrypted malicious traffic detection techniques and provided a systematic review. Furthermore, current research adopts different datasets to train their models due to the lack of well-recognized datasets and feature sets. As a result, their model performance cannot be compared and analyzed reliably. Therefore, in this paper, we analyse, process and combine datasets from 5 different sources to generate a comprehensive and fair dataset to aid future research in this field. On this basis, we also implement and compare 10 encrypted malicious traffic detection algorithms. We then discuss challenges and propose future directions of research.

I. Introduction

Encryption protects traffic but also lets adversaries evade detection, while conventional inspection cannot decipher encrypted contents. The paper reviews detection methods and builds a public-data training set for more consistent comparisons.

I. Introduction

  • Encrypted traffic enables adversaries to evade detection, while conventional security systems cannot decipher its contents.The paper cites reports of increasing encrypted threats and calls for further study of detection methods.Source: §I · p. 1
  • The paper reviews machine-learning detection methods, proposes a framework for comparing them, and analyzes their strengths, limitations, and applicable scenarios.It also categorizes features and discusses possible optimization directions.Source: §I · pp. 1–2
  • Combining public traffic data and comparing algorithms and feature sets on the same training dataset is intended to make experimental results more reliable and consistent.The authors describe their dataset as composed entirely of open public data sources.Source: §I · pp. 1–2

II. RELATED SURVEYS

  • Earlier surveys covered encrypted-traffic classification or broader network analysis, but often focused on mobile platforms or gave limited attention to malicious traffic detection.This review also covers non-mobile platforms and traditional machine-learning approaches alongside deep learning.Source: §II · p. 2

III. PRELIMINARIES

Encryption limits traditional traffic identification, motivating machine-learning approaches that use traffic characteristics rather than readable payloads. Prior work spans flow-based, traditional, and deep-learning methods.

III. PRELIMINARIES

  • Deep packet inspection cannot inspect encrypted contents, while port-based identification can fail when applications use dynamic ports or nonstandard protocols.The paper notes that DPI also raises user-data privacy concerns.Source: §III · p. 2
  • Different encryption mechanisms operate at different layers, and traffic distributions vary even under the same mechanism, making classification challenging.The paper says most research therefore focuses on identifying malicious traffic among legitimate traffic.Source: §III · p. 3
  • Flow-based machine-learning methods are common, but training-data collection and feature selection remain active research areas.The paper cites works [31-33][75-77] as examples of flow-based methods.Source: §III · p. 3
  • Prior work includes supervised and unsupervised traditional learning and deep-learning models such as CNNs, RNNs, and autoencoders.Examples include encrypted-traffic classifiers using RF, SVM, XGBoost, clustering, and neural models [13][15][81].Source: §III · p. 3
  • The paper adapts an existing three-stage traffic-classification framework into a more detailed analysis of machine-learning-based encrypted traffic classification.Khalife et al. provide the three-stage framework, but the paper says it lacks in-depth technical analysis of these techniques.Source: §III · p. 3

IV. METHODS REVIEW BASED ON OUR FRAMEWORK

The review organizes detection research around a general workflow, from defining objectives and collecting traffic to selecting features and algorithms and evaluating performance. It distinguishes detection objectives, settings, and constraints across prior work.

IV. METHODS REVIEW BASED ON OUR FRAMEWORK

  • The proposed framework covers research objectives, dataset collection, feature selection and extraction, algorithm selection, and experimental evaluation.Manual feature selection precedes extraction, whereas automated selection follows extraction of possible features.Source: §IV · p. 3

A. Research Objective

  • Encrypted-traffic research includes identifying encrypted traffic within mixed traffic, a preliminary step toward detecting malicious encrypted traffic.Examples include an attention-based LSTM model and a heuristic method combining statistics and machine learning.Source: A. Research Objective · p. 3
  • Existing approaches address encrypted malicious traffic across protocols or focus on particular protocols, especially HTTPS and TLS/SSL features.Examples include TLS/SSL-feature models [13][25][28][29][33][92] and methods combining TLS/SSL with traffic statistics.Source: A. Research Objective · p. 4
  • Detection objectives include binary malicious-versus-legitimate classification and multiclass identification of malware families, with multiclass performance generally observed to be worse.The paper notes that no existing work covers all traffic types and malware families.Source: A. Research Objective · pp. 3–5
  • Encrypted traffic analysis also includes IoT settings, where resource constraints and device heterogeneity complicate conventional endpoint and network security.One cited approach uses aggregated traffic statistics at a home gateway to reduce its processing load.Source: A. Research Objective · p. 5
  • Detection choices also depend on deployment conditions: online and offline models face different efficiency, robustness, and accuracy requirements that may require compromise.Source: A. Research Objective · p. 5

B. Traffic Dataset Collection

Training datasets can come from real traffic collection or simulated traffic generation, each with different strengths and limitations. The paper analyzes public datasets because the field lacks a well-established corpus for fair comparisons.

B. Traffic Dataset Collection

  • Real traffic collection preserves authenticity, but rare attacks can leave datasets imbalanced and redundant, requiring substantial screening and labeling.Examples include an IoT honeypot dataset and campus-network traffic collected over three working days.Source: B. Traffic Dataset Collection · p. 5
  • Simulated traffic can contain more attack types and be more balanced, but may not represent real network conditions.The paper describes simulated networks and script-based traffic generation as collection methods.Source: B. Traffic Dataset Collection · p. 5
  • Useful encrypted-malicious-traffic datasets need sufficient volume and attack variety, class balance, confirmed ground truth, and no redundant data.The paper says available public datasets rarely contain encrypted traffic, and those containing encrypted malicious traffic are rarer still.Source: B. Traffic Dataset Collection · p. 5
  • Because no well-established dataset exists, studies often use private or few public datasets, making fair performance comparisons difficult.The paper compiles public datasets and analyzes their characteristics, limitations, and applicability.Source: B. Traffic Dataset Collection · p. 5

C. Traffic and Feature Extraction

Traffic data preparation filters and transforms raw or preprocessed datasets before feature extraction. The paper groups extracted features into protocol-agnostic numerical and protocol-specific types, each with different scope and extraction trade-offs.

Traffic data pre-processing

  • Raw PCAP/PCAPNG data is cleaned by removing irrelevant, duplicate, damaged, or incomplete traffic, then filtered to retain encrypted traffic and encoded as needed.Categorical values such as protocol and certificate information are converted to numeric data; some datasets instead arrive as preprocessed CSV features.Source: C. Traffic and Feature Extraction · pp. 5, 7
  • Dataset imbalance can be addressed by combining datasets or using augmentation, over-sampling, or under-sampling, while normalization scales traffic or feature values.The paper describes normalization ranges of zero to one or negative one to positive one.Source: C. Traffic and Feature Extraction · p. 7

Feature extraction

  • Feature extraction is categorized into protocol-agnostic numerical features and protocol-specific features because no recognized feature classification or naming convention exists.The numerical category includes packet- and session-based features, while the protocol-specific category includes TLS/SSL features.Source: C. Traffic and Feature Extraction · p. 7
  • Packet- and session-based numerical features capture traffic timing, sizes, counts, flow duration, and byte totals, with statistical calculations such as mean, variance, and standard deviation.Examples include packet-size changes and client/server packet or byte counts.Source: C. Traffic and Feature Extraction · p. 7
  • Protocol-agnostic numerical features work across encryption mechanisms, while protocol-specific features apply only to their related protocols and can require complex, time-consuming extraction.Protocol-agnostic extraction also requires prior knowledge; protocol-specific TLS/SSL features are drawn from interconnected conn.log, SSL.log, and X509.log files.Source: C. Traffic and Feature Extraction · pp. 7–8

D. Feature Set Selection

Feature selection may rely on domain expertise or machine-learning-based selection and extraction. The approaches trade human judgment against automation, with limitations in manual reliability and automated interpretability.

Feature selection

  • The paper notes that using too many features increases memory and computation needs, lengthens training, and may affect model accuracy.Thus, feature-set choice involves more than maximizing the number of inputs.Source: D. Feature Set Selection · p. 8

Domain expert based selection

  • Domain experts manually choose features before extraction, but human error and missed non-intuitive features can affect detection performance.Privacy concerns also prevented one cited work from disclosing its dataset and expert-selected features.Source: D. Feature Set Selection · p. 8

Machine learning self-selection

  • Machine-learning selection ranks extracted features or learns them from raw inputs, helping discover non-intuitive features while reducing human error and bias.Examples include feature ranking, sequential forward selection, recursive feature elimination, and deep models that learn from raw data.Source: D. Feature Set Selection · p. 8
  • Automated feature selection, especially with deep learning, can be difficult to interpret or explain, and the authors identify further research as necessary.Sequential forward selection can also be time-consuming and may reach a local optimum.Source: D. Feature Set Selection · pp. 8–9

E. Algorithms Selection

The surveyed approaches span traditional machine learning and deep learning, including unsupervised, supervised, and combined methods. Their reported benefits and limitations vary by algorithm and evaluation setting.

Traditional Machine Learning Approach

  • Unsupervised DPC-GS-MND sampling is reported to outperform several clustering and sampling alternatives, but it cannot handle class imbalance and requires a manually chosen K value [13].The approach is also described as reducing computational cost while improving detection accuracy and efficiency.Source: E. Algorithms Selection · p. 9
  • Supervised models include random forests using cross-layer TLS/IPSec features and feature-selected classifiers; reported comparisons favor the proposed models in their tested settings [52][54].IoT Keeper uses unsupervised fuzzy C-means, but it was not tested on encrypted malicious traffic.Source: E. Algorithms Selection · p. 9
  • The surveyed traditional methods also include heuristic statistical testing, where handshake skipping improved accuracy and the approach was reported as more robust than tested alternatives.Its evaluation used proprietary and private protocols, and the dataset was not publicly released.Source: E. Algorithms Selection · p. 9
  • A hybrid framework first uses GMM and OPTICS to identify new malware classes, then trains 24 XGBoost models for malware detection, outperforming two named alternatives.Source: E. Algorithms Selection · p. 9

Deep Learning Approach

  • Deep-learning examples include CNN, LSTM, autoencoder, and hybrid CNN-RNN approaches, with individual studies reporting favorable comparisons or detection results [42][43][44][45][46].Reported examples include CNN image encoding with Perlin-noise augmentation, LSTM detection of new malware families, and CNN-LSTM autoencoder detection.Source: E. Algorithms Selection · pp. 10–11
  • Deep learning can learn traffic features automatically, but research applying it to encrypted malicious traffic detection and classification remains limited.The paper also describes multimodal and multitask learning frameworks for encrypted and mobile traffic classification.Source: E. Algorithms Selection · pp. 9, 11

V. COMPARATIVE EXPERIMENT

The comparative experiments use a curated public-data dataset to compare algorithms and feature sets, and separately examine protocol-agnostic versus protocol-specific features. The passage states the experimental aims but does not report outcomes.

Experiment 1

  • Experiment 1 compares algorithms and feature sets on a curated dataset assembled from public datasets applicable to encrypted malicious traffic detection.The authors describe the composed dataset as more comprehensive and objective than datasets used in existing works.Source: §V · p. 11

Experiment 2

  • A cross-dataset validation in Experiment 2 is intended to assess the composed dataset's comprehensiveness and objectivity.Source: §V · p. 11
  • Experiment 2 also examines whether protocol-agnostic numerical or protocol-specific features perform better.Source: §V · p. 11

A. Data Collection

The authors combine five public traffic datasets to create a dataset for encrypted malicious traffic detection and classification. Selection and sampling aim to balance traffic classes and source contributions while including conventional and IoT devices.

A. Data Collection

  • Combining public datasets from varied sources is intended to reduce bias and support more objective comparisons across research.The authors note that using different datasets can impede objective comparison of proposed work.Source: A. Data Collection · p. 11
  • The selection criteria include encrypted malicious and legitimate traffic from conventional and IoT devices, while balancing traffic classes and dataset contributions.The authors extract approximate numbers of each class from each dataset and avoid allowing one source to dominate by traffic size.Source: A. Data Collection · p. 11
  • Five public datasets contribute approximately 20% each, except CICDS-2012; malicious and legitimate traffic sizes are nearly equal.These balances minimize bias toward any one source; the dataset was released in Mendeley Data.Source: A. Data Collection · p. 11

B. Feature extraction and selection

The authors extract protocol-agnostic numerical and TLS/SSL features, then assemble feature sets based on features used in prior research. These sets support comparisons between numerical and protocol-specific features.

B. Feature extraction and selection

  • The experiments extract more than 113 unique protocol-agnostic numerical features alongside TLS/SSL features.The numerical features are drawn from those applicable features mentioned in prior encrypted-traffic detection research.Source: B. Feature extraction and selection · p. 11
  • The 14-feature FOS set comprises frequently used numerical features, and the five experimental sets also include [12]’s top 10 ranked features.FOS features were selected through statistical analysis of features appearing frequently in prior work.Source: B. Feature extraction and selection · p. 11
  • Other numerical sets use [14]’s five packet features capped at 15 packets per session, ’s tamper-resistant features, and ’s time-based features.The 15-packet choice follows analysis in reporting better performance metrics when retaining 5 to 15 packets per session.Source: B. Feature extraction and selection · p. 12
  • The 22-feature FOTS set collects frequently used TLS/SSL features from Zeek logs, but its protocol-specific features apply only to related encrypted protocols.FOTS is compared with protocol-agnostic numerical features to assess their relative performance.Source: B. Feature extraction and selection · p. 12

C. Experiment Setup

The experiments compare ten algorithms across five feature sets, finding that RF with FOS performs best overall while cross-dataset validation exposes weak transfer between datasets. Results also favor combining session- and packet-level features over packet-only features.

C. Experiment Setup

  • RF, KNN, CART, C4.5, MLP, Naïve Bayes, XGBoost, AdaBoost, Linear R, and Logistic R were evaluated using default hyperparameters, five-fold stratified cross-validation, and the specified RF and KNN settings.RF used 200 estimators and KNN used 6 neighbors.Source: C. Experiment Setup · p. 12
  • 94.71% accuracy, 99.09% ROC-AUC, and 4.18% FPR for RF with FOS accompany its 93.54% TPR; RF with Top 10 ranked features reaches 93.69% TPR.The authors judge FOS with RF to provide the better TPR–FPR trade-off.Source: Table VI · p. 14, C. Experiment Setup · pp. 16–17
  • More than six algorithms exceeded 80% accuracy with accuracy standard deviation below 0.5% across feature sets, while RF and XGBoost achieved ROC-AUC above 90%.For RF and XGBoost, ROC-AUC standard deviation was below 0.13%.Source: C. Experiment Setup · p. 15, Figure 4 · p. 15, Figure 5 · p. 15
  • FOS produced the highest accuracy and ROC-AUC for RF, XGBoost, C4.5, AdaBoost, and CART; other feature sets led for the remaining five algorithms, but at lower performance.The comparisons report each algorithm’s best result among the five feature sets.Source: Figure 8 · p. 15, C. Experiment Setup · p. 16, Figure 9 · p. 16
  • FOS with RF achieved 93.54% TPR and 4.18% FPR, versus 93.69% TPR and 5.71% FPR with Top 10 ranked features; the authors favor FOS’s trade-off.FOS’s TPR was 0.15% lower and its FPR 1.53% lower.Source: C. Experiment Setup · pp. 16–17, Figure 10 · p. 16, Figure 11 · p. 16
  • Combining time-based and traffic-numerical features performed better than either alone, while packet-only side-channel features had the worst performance among the five feature sets.The authors suggest session-based features, alone or combined with packet-based features, are likely to outperform packet-only features.Source: C. Experiment Setup · p. 17
  • Cross-dataset validation produced poor accuracy, ROC-AUC, FPR, and TPR; conventional-device training yielded 0 FPR and 0 TPR on IoT data, underscoring dataset-distribution differences.The authors conclude that a single public dataset may lack traffic variety and that using separate datasets can introduce unpredictable bias, including possible overfitting.Source: C. Experiment Setup · p. 17

E. Experiment 2: Compare the performance of Protocol-agnostic numerical and TLS/SSL feature sets

The comparison tests protocol-agnostic numerical features against TLS/SSL features on TLS/SSL-only traffic. FOS matched or exceeded TLS/SSL performance for nine algorithms on accuracy and ROC-AUC, and the authors conclude protocol-agnostic features are feasible to consider instead.

E. Experiment 2: Compare the performance of Protocol-agnostic numerical and TLS/SSL feature sets

  • The experiment used TLS/SSL-only traffic and compared FOTS and FOS feature sets with TLS/SSL features across ten algorithms.The figures report accuracy and ROC-AUC comparisons.Source: E. Experiment 2: Compare the… · p. 17
  • For nine of ten algorithms, FOS achieved higher or similar accuracy and ROC-AUC to TLS/SSL features; among top performers RF and XGBoost, FOS performed better on both measures.Naïve Bayes was the exception: its FOS model had higher ROC-AUC but lower accuracy than its TLS/SSL-feature model, and its performance was lower than most other algorithms.Source: E. Experiment 2: Compare the… · pp. 17–18

VI. CONCLUSION

The paper proposes a framework for reviewing encrypted-traffic detection research and combines public datasets to support comparative evaluation. It also identifies limits in protocol coverage and binary classification as directions for future work.

VI. CONCLUSION

  • The proposed framework organizes the review around research objectives, dataset collection and preprocessing, feature extraction and selection, algorithm choice, and performance evaluation.Source: §VI · p. 18
  • Five publicly available datasets were analysed, processed, and combined into a large, comprehensive dataset to support future research and technology evaluation.The authors identify the lack of a comprehensive, class-balanced, realistic public dataset as a key challenge.Source: §VI · p. 18
  • Comparative experiments used one dataset to assess 10 machine-learning algorithms and five feature sets, including protocol-agnostic numerical features as possible replacements for TLS/SSL features.The experiments compare feature sets and algorithms using the same dataset.Source: §VI · p. 18
  • The authors advise against studying only one or two kinds of encrypted traffic and note that multiclass classification could provide finer-grained detection results.They point to the growth of encryption and proprietary protocol types and note that current approaches are mostly binary classification.Source: §VI · p. 18

Related papers

Loading 2203.09332v1…