PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Unknown Encrypted Malicious Traffic in Real Time via Flow Interaction Graph Analysis

Chuanpu Fu, Qi Li, Ke Xu

arXiv:2301.13686v1cs.CR

TL;DR

Encrypted malicious traffic with unknown patterns is difficult to detect because individual flows can resemble benign traffic and existing methods rely on known attack signatures or labeled data. HyperVision uses a compact in-memory flow-interaction graph with unsupervised graph learning and an information-theoretic recording analysis. Across real-world datasets, it achieves strong detection accuracy, near-optimal retained information, and high throughput with low latency.

  • Problem

    Existing encrypted-traffic detectors rely on single-flow features and labeled examples of known attacks, limiting detection of unknown malicious patterns.

  • Method

    HyperVision builds a compact in-memory graph of flow interactions and applies unsupervised learning to connectivity, sparsity, and statistical graph features.

  • Results

    HyperVision achieves at least 0.92 AUC and 0.86 F1 across 92 datasets, while reaching at least 80.6 Gb/s detection throughput and 0.83s average latency.

  • Takeaways & Limitations

    Flow-interaction graphs preserve near-optimal traffic information and support real-time unsupervised detection without requiring labeled datasets of known attacks.

Abstract

from arXiv · show

In this paper, we propose HyperVision, a realtime unsupervised machine learning (ML) based malicious traffic detection system. Particularly, HyperVision is able to detect unknown patterns of encrypted malicious traffic by utilizing a compact inmemory graph built upon the traffic patterns. The graph captures flow interaction patterns represented by the graph structural features, instead of the features of specific known attacks. We develop an unsupervised graph learning method to detect abnormal interaction patterns by analyzing the connectivity, sparsity, and statistical features of the graph, which allows HyperVision to detect various encrypted attack traffic without requiring any labeled datasets of known attacks. Moreover, we establish an information theory model to demonstrate that the information preserved by the graph approaches the ideal theoretical bound. We show the performance of HyperVision by real-world experiments with 92 datasets including 48 attacks with encrypted malicious traffic. The experimental results illustrate that HyperVision achieves at least 0.92 AUC and 0.86 F1, which significantly outperform the state-of-the-art methods. In particular, more than 50% attacks in our experiments can evade all these methods. Moreover, HyperVision achieves at least 80.6 Gb/s detection throughput with the average detection latency of 0.83s.

I. INTRODUCTION

Encrypted malicious traffic increasingly evades payload- and single-flow-based detectors because its individual flows resemble benign traffic, while multi-flow interactions retain distinguishing attack patterns. HyperVision addresses this gap with a compact in-memory flow-interaction graph, unsupervised graph learning, and information-theoretic analysis.

  • Motivation: Over 70% of malicious Internet traffic is encrypted, but encrypted flows can resemble benign traffic and evade existing detectors.DPI signatures fail on encrypted payloads, while supervised ML methods depend on labeled examples of known attack patterns.
  • Motivation: Distinct interactions among attackers and victims provide detectable structure even when individual encrypted malicious flows resemble benign flows.Multi-step attacks generate flow relationships that differ from legitimate interactions, such as spam-bot communication with SMTP servers.
  • HyperVision: HyperVision detects unknown encrypted malicious traffic by learning connectivity, sparsity, and statistical features of a compact flow-interaction graph without labeled training data.The graph avoids packet-payload access and supports unsupervised detection based on abnormal interaction patterns.
  • HyperVision: Short-flow aggregation and separate handling of long flows reduce graph density and help preserve interaction patterns for real-time detection.The design responds to dependence explosion in a naive graph built from IP vertices and four-tuple flow edges.
  • Analysis and evaluation: HyperVision’s graph approaches the ideal information bound and provides higher information density than sampling- and event-based traffic records.The information-theoretic framework argues that existing sources cannot retain high-fidelity flow interactions, whereas the graph preserves near-optimal information.
  • Analysis and evaluation: The prototype is evaluated on 92 attack datasets, including 48 encrypted malicious-traffic attacks, using extensive real-world experiments.The supplied introduction identifies broad evaluation coverage and a DPDK-based prototype, while detailed performance results are reported elsewhere.

II. THREAT MODEL AND DESIGN GOALS

HyperVision targets real-time detection of encrypted malicious traffic by representing flow interactions in a compact graph and learning abnormal graph patterns without prior attack knowledge.

  • II. THREAT MODEL AND DESIGN GOALS: The threat model focuses on active attacks that generate traffic, excluding passive attacks such as eavesdropping and passive traffic analysis.Detection uses traffic replicated through router port mirroring without interfering with forwarding.
  • III. OVERVIEW OF HYPERVISION: HyperVision detects encrypted malicious traffic in real time by analyzing abnormal interaction patterns among flows rather than relying on single-flow signatures.The system constructs a compact graph whose vertices and edges represent interactions among addresses and flows.
  • III. OVERVIEW OF HYPERVISION: HyperVision classifies flows as short or long, aggregates similar short flows, and fits distributions for long flows to reduce graph density and storage overhead.These strategies address dependency explosion while preserving interaction information for graph-based detection.
  • III. OVERVIEW OF HYPERVISION: Graph preprocessing extracts connected components, filters components classified as benign, and pre-clusters edges before graph learning.Critical vertices are then selected through a vertex-cover formulation so connected edges can be clustered with reduced overhead.
  • B. Short Flow Aggregation: Short-flow aggregation reduces the graph by 93.94% of vertices and 94.04% of edges, making in-memory maintenance feasible for real-time detection.The comparison uses traditional flow edges versus the aggregated graph on real-world backbone traffic.

C. Feature Distribution Fitting for Long Flows

For long flows, HyperVision replaces lengthy per-packet sequences with histograms of packet-feature distributions, while graph preprocessing isolates and prioritizes abnormal components for efficient detection.

  • C. Feature Distribution Fitting for Long Flows: HyperVision represents long flows with histograms of per-packet feature distributions instead of storing their full feature sequences.It uses bucket widths of 10 bytes for packet length and 1 ms for arrival interval.
  • C. Feature Distribution Fitting for Long Flows: The long-flow fitting is effective because packet lengths and arrival intervals are centrally distributed, allowing compact histogram representations.Packet-length distributions use 11 buckets on average, while most buckets collect more than 200 packets.
  • A. Connectivity Analysis: Connected-component analysis identifies abnormal components by clustering five high-level statistical features and treating large clustering-loss outliers as abnormal.The method uses depth-first search to obtain connected components before clustering their statistics.
  • A. Connectivity Analysis: Edges from normal components are labeled benign, while edges from abnormal components proceed to later processing stages.Most components contain few edges with similar interaction patterns, whereas a high ratio of huge components is classified as abnormal.
  • A. Connectivity Analysis: Pre-clustering exploits sparse edge distributions to reduce the scale of abnormal components before graph learning.HyperVision extracts separate structural feature sets for short- and long-flow edges and normalizes them before DBSCAN clustering.

B. Edge Feature Clustering for Detection

HyperVision clusters edges around critical vertices using flow and structural features, while its information-theoretic analysis compares graph recording with idealized, sampling, and event-based alternatives.

  • B. Edge Feature Clustering for Detection: Edges connected to each critical vertex are clustered using flow and structural features, and abnormal interaction patterns are identified from clustering loss.Short- and long-flow edges are clustered separately with lightweight K-Means.
  • B. Edge Feature Clustering for Detection: The clustering loss combines distance from cluster centers, the time range of pre-clusters, and the number of represented flows.These terms respectively capture feature difference, persistence of interactions, and bursts of massive flows.
  • A. Information Entropy Based Analysis: The information-theoretic framework evaluates recording modes by information amount, data scale, and information density under a DTMC model of packet-feature sequences.The model assumes a stationary distribution and geometrically distributed flow lengths for its analysis.
  • A. Information Entropy Based Analysis: The idealized recording mode reaches the optimal information value under the data processing inequality, providing the theoretical reference for HyperVision’s graph recording mode.The idealized mode records every random variable without processing and assumes infinite storage.
  • A. Information Entropy Based Analysis: HyperVision’s graph recording mode records all variables for short flows but fits histograms for long flows to derive a lower bound on retained information entropy.The analysis also derives expected data scale and information density for this mode.

B. Analysis Results

HyperVision’s graph recording preserves substantially more information and higher information density than existing modes while approaching the theoretical optimum. These properties provide high-fidelity, low-redundancy flow interactions for realtime unsupervised detection of unknown encrypted attacks.

  • Information retention: 2.37 and 1.34 times more information entropy than traditional flow sampling and event-based recording, respectively.HyperVision uses 69.69% of the sampling-based data scale while retaining more information.
  • Information retention: 4.6 × 10^-9 to 2.6 nat separates HyperVision’s retained information from the theoretical optimum across the feasible parameter region.The graph and optimal modes have reported gaps of 0.056 and 0.021 in the comparison summarized by the paper.
  • Information density: 1.46, 1.54, and 2.39 times higher information density than the existing methods, respectively.The idealized system’s density is only 78.55% of HyperVision’s in the worst case.
  • Information density: 35.51% to 47.27% density increases are achieved across all per-packet feature types through distinct short- and long-flow recording strategies.Short and long flows are handled differently to improve graph information density.
  • Implication: The resulting flow interaction graph provides high-fidelity, low-redundancy information that supports realtime unsupervised detection of unknown encrypted malicious traffic.Its interaction patterns are represented structurally rather than through specific attack signatures.

VIII. EXPERIMENTAL EVALUATION

The evaluation uses real-world background and attack traffic, multiple baselines, standard metrics, and cross-validation on a physical high-speed testbed. It covers traditional and encrypted attacks, long-run datasets, and robustness against evasion.

  • Implementation and testbed: The testbed uses DPDK-based high-speed packet processing on DELL servers with a 10 Gb/s NIC and 24 GB memory.HyperVision’s graph is maintained in memory for realtime detection.
  • Datasets: 80 new datasets combine MAWI backbone background traffic with attacks generated across four groups, including three groups of encrypted malicious traffic.The experiments also include 12 existing real-world datasets to reduce dataset-bias effects.
  • Traffic replay: 0.01 to 8.79 Kpps encrypted attack flows consume only 0.01% to 0.72% of bandwidth during 45-second replays.Each dataset contains 12–15 million packets, with the first 75% of replay time free of malicious traffic for interaction collection and baseline training.
  • Baselines and metrics: Five generic detection methods serve as baselines, while DPI-based and task-specific encrypted-traffic methods are excluded because the paper states they cannot investigate encrypted payloads or achieve acceptable accuracy.The evaluation uses AUC and F1 primarily, plus precision, recall, F2, ACC, FPR, and EER.
  • Evaluation protocol: Four-fold cross-validation averages results across validation and testing splits to reduce overfitting and hyper-parameter bias.The ablation study reports at most a 5.2% accuracy change from different threshold settings.

B. Accuracy Evaluation

Across traditional, low-rate, flooding, web, and malware attacks, HyperVision consistently detects encrypted malicious traffic more accurately than the evaluated baselines. Its graph captures interaction patterns that remain informative when individual encrypted flows resemble benign traffic.

  • Overall results: 0.927 to 0.978 average F1 and 0.974 to 0.993 average AUC across 80 datasets yield 35% and 13% improvements over the best baselines.In 44 datasets, no baseline achieves F1 above 0.80.
  • Traditional brute-force attacks: 0.992–0.999 AUC and 0.929–0.999 F1 are achieved on traditional brute-force attacks, with up to 13.4% F1 and 1.3% AUC improvement.HyperVision also has fewer false positives and higher precision at equal recall in the ROC and PRC comparisons.
  • Low-rate probing: 0.920–0.994 F1 and 0.916–0.999 AUC are achieved for low-rate probing with 6–268 attackers and 17.6–97.9 Kpps total bandwidth.The graph preserves slow attackers’ interactions and identifies vertices with large out-degrees.
  • Encrypted flooding: 0.856–0.981 F1 and 0.917–0.998 AUC are achieved for encrypted flooding, improving over capable baselines by 58.7% and 25.3%.HyperVision detects low-rate Crossfire, encrypted video TCP DoS, and SSH connection-injection attacks through graph-based interaction analysis.
  • Web and malware traffic: 0.985 average AUC and 0.957 average F1 are achieved for encrypted web attacks, while flow-based ML methods fail when attack flows resemble benign web access.Encrypted malware traffic is detected with at least 0.942 F1.

C. Performance Results

HyperVision combines task-agnostic encrypted-traffic detection with high-throughput graph construction and learning. Its compact graph supports sustained throughput, subsecond average detection latency, and reduced storage requirements.

  • Throughput: HyperVision constructs graphs at 28.21 Gb/s average traffic throughput and analyzes flow interactions at 121.14 Gb/s on average.Detection throughput is 4.2 times higher than construction throughput, allowing iterative analysis of recorded traffic.
  • Throughput: 80.6 ∼ 148.9 Gb/s throughput is achieved when HyperVision’s performance is stable.Throughput is lower on April and June 2020 datasets because their original traffic volume is low.
  • Latency: 0.83s average detection latency is measured for the learning module, with a 99th-percentile latency of 4.48s.Pre-clustering contributes 75.8% of latency while reducing subsequent processing overhead.
  • Resource Consumption: 1.78 GB of memory maintains flow interaction patterns extracted from 2.82 TB of ongoing traffic.The compact graph reduces memory consumption through long-flow distribution fitting and short-flow aggregation, and the graph can be exported for forensic analysis.
  • Resource Consumption: 8.99%, 55.7%, and 98.1% storage reductions are achieved over Suricata and Zeek monitors and raw packet headers, respectively.The comparison records the first 45 seconds of MAWI traffic.
  • Graph Based Anomaly Detection: HyperVision provides task-agnostic encrypted-traffic detection without parsing specific application-layer headers, unlike prior graph-based approaches relying on DPI.Related encrypted-traffic methods are described as task-specific or supervised, whereas HyperVision targets unknown patterns.

APPENDIX

The appendix specifies HyperVision’s edge features, datasets, recommended hyperparameters, and short-flow aggregation procedure. The procedure groups flows by protocol and shared sources or destinations before constructing edges.

  • APPENDIX: Table V lists HyperVision’s edge features, Table VI details malicious-traffic datasets, and Table VII gives recommended hyperparameter configurations.The appendix identifies these tables as the sources for edge features, dataset properties, and tuning values.
  • APPENDIX: Short flows are aggregated by protocol and shared source or destination, producing constructed ShortEdge outputs from ShortFlow inputs.The procedure uses source aggregation, destination aggregation, or both, and retains unaggregated flows when needed.
  • APPENDIX: Flows are inserted into protocol-hash groups, then processed for source aggregation among flows with the same protocols.The protocol hash table is initialized before flows are grouped and appended by protocol hash.
  • APPENDIX: For each source group, the algorithm records flow destinations and emits an edge when aggregation conditions are met.A unique destination is saved for source-and-destination aggregation; otherwise, source-only aggregation is used.
  • APPENDIX: Destination groups are processed similarly, and remaining short flows are added to ShortEdge when they cannot be aggregated.The destination table collects flows with the same destinations before residual flows are retained.

B. Details of Experiments

Experiments evaluate HyperVision across newly collected and existing datasets, including long-run operation and obfuscation attacks. Results report improvements over baselines, sustained detection, and limited accuracy degradation under injected benign traffic.

  • B. Details of Experiments: The 80 newly collected datasets are documented by attacker and victim counts, attack-flow packet rates, and encrypted-traffic ratios.They are collected and labeled using the same method as the MAWI and CIC datasets.
  • B. Details of Experiments: 7.8%, 11.0%, and 5.1% F1 improvements over the best baselines are achieved on Kitsune, CIC-IDS2017, and CIC-DDoS2019 datasets, respectively.Twelve existing datasets are used to reduce dataset-bias effects, and Kitsune validates deployed baselines.
  • B. Details of Experiments: Over 0.95 F1 and 0.99 AUC are achieved during 6∼8 hours of long-run detection.The experiments also report simultaneous multi-attack detection and resilience to changed addresses.
  • B. Details of Experiments: 4.3% F1 is the maximum reported accuracy decrease under obfuscation using injected benign TLS, UDP video, or normal ICMP traffic.Average F1 decreases by 1.7%, 0.9%, and 2.4% for the three injected traffic types, respectively.
  • B. Details of Experiments: HyperVision remains able to detect obfuscation attacks because the injected traffic manipulates single-flow patterns while detection learns interactions among flows.This mechanism is presented as the reason for the limited accuracy decrease.

C. Details of Theoretical Analysis

The theoretical analysis models information retained by event-based and sampling-based recording modes. It derives event entropy and then analyzes expected recorded data scale and information density.

  • C. Details of Theoretical Analysis: H_Eve. is defined as the entropy of the event-recording indicator I_Eve.The entropy is expressed from the probabilities that the event-based mode records or does not record a flow.
  • C. Details of Theoretical Analysis: The event-recording probability is derived from q and p_s before entropy is evaluated.The supplied expression gives the probability distribution for I_Eve. under the event-based mode.
  • C. Details of Theoretical Analysis: A second-order Taylor expansion is used to approximate H_Eve. when q > 0.5.The analysis then derives the expected data scale L_Eve. and information density D_Eve.
  • C. Details of Theoretical Analysis: The event-based analysis is completed before the sampling-based mode models recorded flow information as the sum of observed per-packet features.The sampling-mode random variable is denoted X_Samp.

2) Analysis of Sampling based Mode:

The sampling-based mode quantifies recorded information with Shannon entropy, decomposed into conditional entropy and mutual information. Under a near-zero mutual-information assumption, it derives and numerically validates a tight lower bound, then estimates the entropy’s second term with a second-order Taylor series.

  • Shannon entropy measures the information recorded by the sampling-based mode and is decomposed into conditional entropy and mutual information.
  • A near-zero mutual information between sequence length L and accumulative statistic X_Samp. yields an entropy lower bound that numerical analysis verifies as tight.
  • The second-order Taylor series accurately approximates the entropy’s second term.
  • The analysis completes by deriving the expected data scale and information density for the sampling-based mode.

3) Analysis of Graph based Mode in HyperVision:

HyperVision’s graph-based recording mode uses detailed per-packet information for short flows and histogram counters for long flows. Its entropy analysis treats the two flow regimes separately, models long-flow counters probabilistically, and derives expected data scale and information density.

  • Short Flow Information: Short flows retain detailed per-packet feature sequences, whose information growth equals the entropy rate of the DTMC.
  • Long Flow Information: For long flows, HyperVision records s histogram counters over DTMC states for distribution fitting.
  • Long Flow Information: Long-flow histogram counters are modeled as a binomial process and approach Poisson distributions when per-packet sequences are relatively long.
  • Entropy Analysis: Assuming q > 0.5, Taylor expansion approximates the final entropy term, while a quadratic term in s is used in the expansion of P_s.
  • Entropy Analysis: The completed graph-based analysis derives expected data scale and information density, with information density defined as D_H.V. = H_H.V./L_H.V.
Loading 2301.13686v1…