PaperlayerView exact revision on arXiv

Source-linked AI summary

ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification

Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, Jing Yu

arXiv:2202.06335v2cs.CRcs.AIcs.NI

TL;DR

Encrypted traffic classification needs robust representations that generalize beyond content-invisible, imbalanced data and limited labeled samples. ET-BERT pre-trains contextual datagram representations on large-scale unlabeled traffic, then fine-tunes them for downstream tasks. It achieves state-of-the-art performance across five classification tasks, with reported gains of up to 10.0%.

  • Problem

    Encrypted traffic classification requires robust representations for content-invisible and imbalanced data, while existing solutions depend heavily on data size and generalize poorly to unseen data.

  • Method

    ET-BERT pre-trains contextual datagram-level representations from large-scale unlabeled encrypted traffic and fine-tunes all parameters with a small amount of task-specific labeled data.

  • Results

    ET-BERT achieves new state-of-the-art performance across five encrypted traffic classification tasks, with reported improvements of 5.4%, 0.2%, 5.2%, 4.4%, and 10.0% over existing works.

  • Takeaways & Limitations

    The model provides generic datagram representations that transfer across multiple encrypted traffic classification scenarios and supports fine-tuning with limited labeled data.

  • Takeaways & Limitations

    ET-BERT depends on clean pre-training data, and poisoned pre-training data could create a backdoored model that forces target-class predictions on specific tasks.

Abstract

from arXiv · show

Encrypted traffic classification requires discriminative and robust traffic representation captured from content-invisible and imbalanced traffic data for accurate classification, which is challenging but indispensable to achieve network security and network management. The major limitation of existing solutions is that they highly rely on the deep features, which are overly dependent on data size and hard to generalize on unseen data. How to leverage the open-domain unlabeled traffic data to learn representation with strong generalization ability remains a key challenge. In this paper,we propose a new traffic representation model called Encrypted Traffic Bidirectional Encoder Representations from Transformer (ET-BERT), which pre-trains deep contextualized datagram-level representation from large-scale unlabeled data. The pre-trained model can be fine-tuned on a small number of task-specific labeled data and achieves state-of-the-art performance across five encrypted traffic classification tasks, remarkably pushing the F1 of ISCX-Tor to 99.2% (4.4% absolute improvement), ISCX-VPN-Service to 98.9% (5.2% absolute improvement), Cross-Platform (Android) to 92.5% (5.4% absolute improvement), CSTNET-TLS 1.3 to 97.4% (10.0% absolute improvement). Notably, we provide explanation of the empirically powerful pre-training model by analyzing the randomness of ciphers. It gives us insights in understanding the boundary of classification ability over encrypted traffic. The code is available at: https://github.com/linwhitehat/ET-BERT.

1 INTRODUCTION

Encrypted traffic makes payload-based classification ineffective and weakens methods tied to fixed features or known encryption patterns. ET-BERT addresses this with contextual pre-training on unlabeled traffic and reports state-of-the-art results across five tasks.

  • Encrypted traffic prevents deep packet inspection from using payload patterns and enables evasion through privacy-enhanced encryption techniques such as Tor and VPN.
  • Plaintext-based methods fail as emerging techniques make plaintext sparser or more obfuscated, while expert-designed statistical features have limited generalization ability.
  • Pre-training learns transferable representations from large unlabeled data that can be fine-tuned on limited labeled data, but prior encrypted-traffic work lacked task-specific pre-training design.
  • ET-BERT transforms datagrams into language-like tokens, represents flows with BURST structures, and uses MBM and SBP tasks to learn byte- and BURST-level contextual patterns.
  • ET-BERT leverages large-scale unlabeled encrypted traffic to learn generic datagram representations for five encrypted traffic classification tasks.
  • 5.4%, 0.2%, 5.2%, 4.4%, and 10.0% are the reported improvements over existing works across the five encrypted traffic classification tasks.

2 RELATED WORK

Related work progresses from plaintext fingerprints and expert-designed statistics to supervised deep learning and generic pre-training. ET-BERT is positioned as a traffic-specific pre-training approach designed for stronger generalization across encryption settings.

  • Fingerprint Construction: FlowPrint constructs flow fingerprints from device, certificate, size, and temporal features, but such fingerprints are easily tampered with in virtual communication networks.
  • Statistical Methods: Statistical methods use packet-size or temporal features, yet generic statistical features are difficult to design for increasingly complex applications and websites.
  • Deep Learning Models: Supervised deep models automatically extract representations from raw packet-size sequences or payloads using architectures including CNNs and RNNs.
  • Pre-training Models: Transformer pre-training extends across language tasks through designs such as dynamic masking and sentence-order prediction.
  • Encrypted-Traffic Pre-training: Ciphertexts can differ in randomness across applications, indicating that encrypted traffic may retain implicit patterns despite lacking payload semantics.
  • Encrypted-Traffic Pre-training: PERT achieves 93.23% F1 on ISCX-VPN-Service but lacks encrypted-traffic-specific representation and pre-training-task designs for new encryption techniques such as TLS 1.3.

3 ET-BERT

ET-BERT learns generic encrypted-traffic representations from unlabeled flows by converting datagrams into pattern-preserving tokens and pre-training with traffic-specific contextual objectives. The representation is then fine-tuned end-to-end for packet- or flow-level classification.

  • Framework: ET-BERT uses two stages: pre-training on unlabeled traffic representations and end-to-end fine-tuning for downstream classification.Task-specific packet or flow representations are input to the pre-trained model, whose [CLS] representation is classified.
  • Traffic representation: Datagram2Token transforms flows into language-like tokens by extracting directional BURSTs and encoding each BURST with adjacent-byte units.A BURST contains time-adjacent packets from either the request or response direction within one session flow.
  • Traffic representation: Each token representation sums token, relative-position, and segment embeddings before Transformer encoding.The embedding dimension is 768, and segment embeddings distinguish sub-BURST A from sub-BURST B.
  • Pre-training: Masked BURST Model and Same-origin BURST Prediction learn byte-level context and dependencies between packets within BURSTs.SBP predicts whether two sub-BURSTs share an origin, using actual next sub-BURSTs or randomly selected sub-BURSTs.
  • Fine-tuning: Fine-tuning supports packet-level ET-BERT(packet) and flow-level ET-BERT(flow) classification with stitched datagrams from M=5 consecutive packets.The [CLS] output represents the complete input traffic and is fed to a multi-class classifier.

4 EXPERIMENTS

The experiments evaluate ET-BERT across five encrypted traffic classification tasks on six public datasets and one newly proposed dataset, using standard metrics and controlled preprocessing. They also compare against existing methods, analyze key components, and test interpretability and few-shot capability.

  • Evaluation scope: Five encrypted traffic classification tasks span standard encryption, malware, VPN, Tor, and TLS 1.3 scenarios.The evaluation uses six public datasets and one newly proposed dataset.
  • Evaluation scope: The GEAC task covers Cross-Platform iOS and Android traffic, with 196 and 215 applications and long-tail class distributions.The applications were collected from the top 100 apps in the US, China, and India.
  • Evaluation scope: The EMC task uses USTC-TFC, containing 10 benign and 10 malicious traffic categories.
  • Evaluation scope: The ETCV task evaluates ISCX-VPN traffic by service and application under VPN and non-VPN communication.The dataset contains six communication applications and presents protocol-obfuscation and imbalance challenges.
  • Evaluation scope: The EACT and EAC-1.3 tasks use ISCX-Tor with 16 applications and CSTNET-TLS 1.3 with 120 applications, respectively.CSTNET-TLS 1.3 was collected from March to July 2021 and labeled using server name indication.
  • Experimental protocol: Performance is assessed with Accuracy, Precision, Recall, and F1, using macro averages to reduce bias from imbalanced categories.Fine-tuning randomly selects at most 500 flows and 5,000 packets per class.

4.2 Comparison with State-of-the-Art Methods

ET-BERT outperforms the compared state-of-the-art methods across Cross-Platform, VPN, malware, Tor, and TLS 1.3 evaluations. The results are attributed to contextual modeling of ciphertext and transmission structure rather than reliance on plaintext fields.

  • GEAC: ET-BERT improves over existing state-of-the-art results by 1.7% on Cross-Platform (iOS) and 5.4% on Cross-Platform (Android).Both packet- and flow-level variants outperform all compared methods.
  • GEAC: ET-BERT learns contextual relationships on ciphertext without plaintext fields and models traffic transmission structure that PERT does not master.
  • EMC: 99.30% F1 is achieved on USTC-TFC, the best result among the compared methods.Other methods can exploit unencrypted application-layer data in the malicious traffic.
  • ETCV: 5.69% and 1.72% improvements over Deeppacket are obtained on ISCX-VPN-Service and ISCX-VPN-App, respectively.The datasets pose imbalanced-data challenges, and ET-BERT reports average F1 improvements of 25.55% and 42.89% over methods excluding PERT.
  • EACT: 4.41% improvement over TSCRNN is obtained on ISCX-Tor, where ET-BERT exploits intrinsic packet relationships under multilayer encryption and adversarial obfuscation.
  • EAC-1.3: 97.41% F1 is achieved on CSTNET-TLS 1.3, a 10.0% improvement over the existing state-of-the-art result of 87.41%.The result indicates that TLS 1.3 datagrams retain implicit patterns leveraged by ET-BERT.

4.3 Ablation Study

Ablation experiments on ISCX-VPN-App show that both self-supervised tasks, BURST inputs, flow-level fine-tuning, and pre-training contribute to ET-BERT’s performance. Removing pre-training causes the largest reported degradation.

  • Pre-training components: 3.97% and 9.33% F1 decreases show that both self-supervised pre-training tasks provide complementary classification patterns.
  • Pre-training components: 1.37% F1 decreases when packets replace BURST inputs, supporting BURST’s role in learning relationships between packets.
  • Fine-tuning inputs: 4.26% F1 drops when flow inputs switch to concatenated-flow inputs, indicating that interdependent packets benefit flow classification.
  • Pre-training contribution: 37.57% lower F1 results from removing the pre-trained model and training the Transformer directly on labeled data.

4.4 Interpretability

ET-BERT’s performance is linked to cipher randomness: practical ciphers are imperfectly random, and datasets containing weaker-randomness ciphers yield especially strong results. Few-shot experiments further show that pre-training remains effective as labeled data decreases.

  • Randomness Analysis: 15 statistical test sets show that the five evaluated ciphers fail to achieve perfect randomness.The tests use p-value = 1 as the indicator of perfect randomness.
  • Cipher Distribution: Datasets containing weaker-randomness ciphers such as RC4 and 3DES include ISCX-VPN, ISCX-Tor, and USTC-TFC.These datasets contain at least three ciphers, whereas other datasets mainly consist of one cipher.
  • Cipher Distribution: ET-BERT achieves F1 close to 100% on datasets with weaker-randomness ciphers.
  • Few-shot Analysis: ET-BERT(packet) obtains F1 scores of 95.78%, 98.33%, and 91.55% with 40%, 20%, and 10% of the data, respectively.It achieves the best results among the compared methods in these few-shot settings.
  • Few-shot Analysis: Deeppacket’s F1 decreases by 40.22% when the sample size is reduced from the full size to 10%.

5 DISCUSSION

The discussion identifies two boundaries for ET-BERT: changing Internet-service content can challenge fixed patterns, and poisoned pre-training data could create targeted backdoors.

  • Generalizability: Changes in Internet-service content over time can challenge ET-BERT’s ability to generalize from fixed patterns learned from fixed data.
  • Generalizability: TLS 1.3 traffic cannot be labeled through SNI, so the authors use active visiting and unique process identifiers to test generalizability under ECH.
  • Pre-training Security: ET-BERT depends on clean pre-training data because poisoned data may add low-frequency toxic subwords that create a backdoor targeting specific predictions.

6 CONCLUSION

ET-BERT pre-trains contextual datagram representations from large-scale unlabeled traffic and fine-tunes them with limited labeled data, achieving state-of-the-art results across five encrypted-traffic tasks.

  • Conclusion: ET-BERT pre-trains contextual datagram-level representations from large-scale unlabeled data and fine-tunes them for multiple encrypted-traffic scenarios.
  • Conclusion: ET-BERT achieves state-of-the-art performance across five encrypted traffic classification tasks, with reported improvements of 5.4%, 0.2%, 5.2%, 4.4%, and 10.0%.

A.1 Qualitative Analysis

The qualitative analysis compares packet- and flow-level visualizations across five models, showing ET-BERT’s stronger separation of encrypted applications under newer encryption protocols.

  • Qualitative Analysis: Five comparison models are evaluated alongside ET-BERT, including flow- and packet-level Transformers, DF, Deeppacket, and PERT.
  • Qualitative Analysis: t-SNE projects ISCX-VPN-App test samples into two-dimensional visualizations for packet-level and flow-level model results.
  • Qualitative Analysis: ET-BERT shows the best classification performance by capturing distinguishing patterns under more secure newer encryption protocols.
  • Qualitative Analysis: Packet-level baselines confuse chat-related applications such as AIM, ICQ, and Gmail, while flow-level methods confuse streaming services including YouTube, Vimeo, Netflix, and Spotify.PERT performs relatively better at the flow level but remains affected by applications providing the same services.
Loading 2202.06335v2…