PaperlayerView exact revision on arXiv

Source-linked AI summary

PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classification

Wumei Du, Jiarong Wen, Kaiyu Zhang, Zi Yang, Yiqin Lv, Longfei Zhang, Dong Liang, Zheng Xie

arXiv:2608.15504v1cs.LGstat.ML

TL;DR

Encrypted traffic foundation-model post-training must address high-risk tail errors, but identifying such samples with full-model inference is prohibitively expensive. PERO uses a lightweight proxy for selective tail-focused updates, consistently improving tail robustness while reducing computational and memory costs across benchmarks.

  • Problem

    Robust post-training must target high-risk encrypted-traffic errors, yet full-model inference for identifying such samples is prohibitively expensive.

  • Method

    PERO uses a lightweight pre-evaluation proxy to estimate sample-wise risk and selectively update the foundation model on high-risk samples.

  • Results

    PERO consistently improves tail robustness over representative baselines while substantially reducing computational and memory costs across three benchmarks.

  • Takeaways & Limitations

    PERO supports scalable robust post-training for continual-update settings where limited update windows make direct CVaR-style optimization impractical.

  • Takeaways & Limitations

    The convergence analysis assumes a uniformly bounded loss-Hessian spectral radius, whose global validity can be restrictive for large Transformer models.

Abstract

from arXiv · show

Encrypted traffic classification is vital for network security, yet real-world deployments are inherently sensitive to rare but high-loss errors such as misclassification of malicious traffic. The encrypted traffic foundation model, as a promising general-purpose technique, can achieve impressive overall performance. However, employing standard objectives such as empirical risk minimization often overlooks high-risk tail events, and commonly used performance metrics hardly reflect robustness limitations in risk-sensitive scenarios. Directly applying robust optimization objectives, such as conditional value-at-risk, to post-training is computationally prohibitive for large models, as identifying high-loss samples exhausts substantial computation. To this end, we propose Pre-Evaluation Robust Optimization (PERO), an efficient robust post-training framework for encrypted traffic foundation models. PERO employs a lightweight proxy to estimate sample-wise risk and selects a subset of high-risk samples to update the foundation model, decoupling risk estimation from expensive large-model optimization. Extensive experiments on typical encrypted traffic datasets show that PERO achieves competitive or superior robustness and average performance compared to outstanding robust post-training methods, while significantly reducing computational and memory costs.

1. Introduction

Encrypted traffic foundation models improve classification but standard ERM overlooks high-risk tail events, while direct CVaR optimization is too costly at scale. PERO addresses this through lightweight proxy-guided selective optimization, with theoretical analysis and experiments showing competitive or superior robustness at lower computational and memory cost.

  • Motivation: Encrypted traffic classification is increasingly critical for distinguishing normal and abnormal traffic or identifying application types amid expanding Internet usage.Its importance follows from growing demands for network management, quality of service, planning, and security.
  • Existing Challenges: Existing foundation-model post-training predominantly uses ERM, which prioritizes majority patterns, overlooks high-risk tail events, and degrades high-loss-sample performance.Foundation models learn hierarchical packet- and flow-level representations through large-scale self- or weakly supervised pre-training before downstream adaptation.
  • Existing Challenges: CVaR robust optimization focuses updates on highest-loss samples, but repeated evaluations and parameter updates across millions of parameters make it computationally and memory prohibitive.Memory-limited batch processing also restricts full loss-landscape exploration and tail-risk estimation at scale.
  • Developed Method: PERO uses a lightweight pre-evaluation proxy to screen high-risk samples and guide selective optimization, reducing computational and memory overhead without replacing the CVaR framework.The method decouples costly sample screening from large-model optimization under limited computational and memory constraints.
  • Primary Contributions: PERO includes theoretical analysis characterizing post-training convergence and deriving a generalization bound for its pre-evaluation module.This provides analytical insight into proxy-guided robust post-training.
  • Primary Contributions: Extensive benchmark experiments show competitive or superior robustness versus outstanding robust post-training methods while substantially lowering computational and memory costs.The experiments evaluate the proposed framework on widely used benchmarks.

2. Related Work

Encrypted traffic classification has progressed from early Transformer-based pre-training to diverse foundation-model architectures with strong cross-environment performance. However, standard objectives overlook rare high-loss samples, while existing robust post-training approaches incur scalability or risk-modeling limitations.

  • Encrypted Traffic Foundation Models: Foundation models pre-train on massive unlabeled traffic data, then fine-tune for downstream tasks to learn transferable representations across network environments.This paradigm has become increasingly prominent with advances in deep learning and Transformer architectures such as BERT.
  • Encrypted Traffic Foundation Models: PERT established Transformer-based pre-training for encrypted traffic, followed by ET-BERT, generative models, encoder-decoder Lens, Mamba-based NetMamba, and attention-based MIETT.MIETT jointly captures intra-packet and inter-packet dependencies and outperforms prior token-centric foundation models on multiple benchmarks.
  • Motivation for Robust Post-Training: Despite strong average performance, encrypted traffic foundation models typically underemphasize rare high-loss samples through standard empirical objectives.Robust optimization during post-training is prohibitively costly for models at this scale.
  • Robust Optimization: Robust optimization, often formulated through distributionally robust optimization, seeks reliability under worst-case conditions amid uncertainty in data distributions.Its foundations include convex uncertainty and moment-based ambiguity sets.
  • Limitations of Existing Robust Methods: CVaR requires repeated tail-loss estimation and re-optimization, GroupDRO depends on potentially misaligned static groups, and OHTM heuristically selects high-loss samples without explicit tail-risk modeling.Memory-limited batching further degrades CVaR tail-risk estimation, while OHTM can behave unstably under nonstationarity.

3. Preliminary

This section formalizes encrypted traffic classification with cross-entropy loss and reviews ERM, CVaR, and GroupDRO as foundational optimization principles. It highlights that ERM can favor majority patterns when high-risk traffic is underrepresented in downstream data.

  • Problem formulation: Encrypted traffic classification learns a mapping fθ: X → Y from labeled samples to minimize classification errors under the data distribution p(x, y).The model uses parameters θ, dataset D with N training pairs, and cross-entropy loss ℓ for optimization.
  • Empirical Risk Minimization: ERM minimizes expected loss under the empirical distribution induced by training data because the true distribution p(x, y) is unknown.ERM is the standard supervised-learning principle and is typically used for post-training through stochastic gradient descent.
  • Conditional Value-at-Risk: CVaR targets the upper loss tail, minimizing expected loss among the worst-performing (1 − α) fraction of samples.For example, α = 0.9 corresponds to the worst 10% of cases; large-scale optimization commonly uses Monte Carlo stochastic approximation, called MC-CVaR.
  • Group Distributionally Robust Optimization: GroupDRO minimizes the maximum group-wise expected loss, focusing optimization on the worst-performing subpopulation.Groups may represent subpopulations such as application types.
  • Post-training limitation: ERM adapts pre-trained representations when downstream data represent the target distribution, but underrepresented high-risk traffic causes it to favor majority patterns and severe performance degradation.This limitation is especially relevant to encrypted traffic classification, where high-risk examples are significantly underrepresented.

4. Method

PERO enables resource-efficient robust post-training by estimating sample-wise risk with a lightweight proxy and selectively updating the foundation model on high-risk samples. Its top-B selection avoids complex robust optimization, while the analysis provides convergence and generalization guarantees under explicit assumptions.

  • Framework Overview: PERO separates risk estimation from large-model optimization through a pre-evaluation module and subset selection module.The pre-evaluation module estimates candidate losses, while the selection module identifies risk-informative samples for classifier updates.
  • Pre-Evaluation Module: The lightweight pre-evaluation module predicts sample-wise cross-entropy risk under the evolving classifier, highlighting uncertain or tail-risk samples.It is trained online from losses produced by the current classifier and learns expected classification difficulty for future candidates.
  • Subset Selection: PERO expands each batch into a candidate pool of size ˆB and selects B samples with the highest predicted losses for classifier updates.Because preprocessed samples have uniform cost, the method sets the budget to B and uses predicted loss as the acquisition score.
  • Subset Selection: Setting λ = 0 reduces the constrained selection objective to top-B acquisition-score selection, focusing post-training on challenging samples without complex robust optimization.The selected high-risk subset is then used for a standard stochastic gradient update of the classifier.
  • Theoretical Analysis: Under stated assumptions, Theorem 4.1 establishes convergence of the coupled parameter sequence to a stationary point, while Theorem 4.2 bounds proxy generalization error with probability at least 1 −ϵ.The convergence result relies on a uniformly bounded loss-Hessian spectral radius, and the generalization bound assumes i.i.d. data and sufficiently large ˆB.
  • Complexity Analysis: PERO achieves robust screening with computational overhead close to ERM, avoiding MC-CVaR’s costly O(ˆB·Cfwd complexity.Table 1 characterizes ERM as efficient but not robust and MC-CVaR as computationally costly.

5. Experimental Results and Analysis

Experiments across encrypted traffic benchmarks evaluate PERO’s performance, tail robustness, efficiency, risk prediction, and hyperparameter sensitivity. PERO achieves strong overall and tail-conditioned results while reducing computational overhead, with a lightweight proxy that identifies informative high-risk samples.

  • Overall and Tail Performance Comparison: PERO consistently achieves strong overall and tail-conditioned accuracy, with its advantage becoming clearer as the tail-risk level increases.At α = 0.9, PERO slightly outperforms MC-CVaR on USTC-TFC and shows a clearer advantage on the other benchmark.
  • Overall and Tail Performance Comparison: 87.63% F1_0.9 on USTC-TFC exceeds MC-CVaR’s 75.73% and OHTM’s 72.87%, while also surpassing the remaining baselines.On ISCX-VPN-Service, PERO achieves the best F1_0.9, improving over ERM, Random, Focal, OHTM, GroupDRO, TDRO, and MC-CVaR by 18.42%, 24.82%, 21.64%, 13.43%, 25.08%, 24.30%, and 7.29%, respectively.
  • Computational and Memory Efficiency: Approximately half the runtime of MC-CVaR makes PERO among the most efficient robustness-oriented methods, while its memory footprint is within 0.16 GB of the lowest.PERO reduces computation by using a lightweight pre-evaluation module and optimizing only selected high-risk samples.
  • Risk Prediction Fidelity: Pearson and Spearman correlations stabilize around 0.5–0.6, while Precision@k stabilizes around 0.65–0.7, indicating informative relative-risk ranking.Early weak correlations are attributed to rapid classifier updates weakening alignment between exact losses and proxy predictions.

6. Conclusion

PERO is an efficient robust post-training framework for encrypted traffic foundation models that decouples risk estimation from expensive optimization and targets critical tail examples with low computational overhead. Experiments on three main benchmarks show consistent tail-robustness improvements over representative robust post-training baselines.

  • PERO provides an efficient robust post-training framework for encrypted traffic foundation models.The framework is the paper’s central proposal.
  • A lightweight pre-evaluation module decouples sample-wise risk estimation from expensive foundation-model optimization.This design enables risk estimation without directly applying costly optimization to the full model.
  • PERO targets critical tail examples for model updates while maintaining low computational overhead.The targeted-update strategy focuses optimization on high-risk tail samples.
  • Experiments on three main benchmarks show that PERO consistently improves tail robustness over representative robust post-training baselines.The conclusion reports consistent gains across the three benchmarks.

Appendix · A. Limitations and Ethical Considerations

The appendix addresses privacy, fairness, misuse risks, and theoretical limitations of PERO. It emphasizes that the method preserves data privacy but cannot remove dataset bias, requires safeguards against intrusive monitoring, and relies on a strong convergence assumption.

  • A. Limitations and Ethical Considerations: PERO uses publicly released encrypted traffic datasets provided by the original authors for research purposes.The datasets are used for research purposes.
  • A. Limitations and Ethical Considerations: PERO does not inspect communication content, attempt payload decryption, or reconstruct user identities or behaviors.These constraints limit direct access to communication content and personal information.
  • A. Limitations and Ethical Considerations: Although PERO emphasizes high-risk samples to improve tail-risk robustness, it does not remove biases inherent in the training data.The method’s focus on high-risk samples does not eliminate bias originating in the data.
  • A. Limitations and Ethical Considerations: Tail-risk performance may vary across network environments because training-data biases remain.This variation is identified as a consequence of biases inherent in the training data.
  • A. Limitations and Ethical Considerations: Encrypted traffic classification technology may be misused for intrusive monitoring without adequate security measures.The appendix identifies misuse as a potential deployment risk.
  • A. Limitations and Ethical Considerations: Responsible use requires transparency and appropriate access safeguards.These safeguards address the misuse risk associated with deployment.
  • A. Limitations and Ethical Considerations: Theorem 4.1’s convergence result relies on a uniformly bounded loss-Hessian spectral radius throughout optimization.This is described as a standard smoothness condition in convergence analyses of gradient-based methods, but it is strong.
  • A. Limitations and Ethical Considerations: Theorem 4.1 may not strictly hold for large Transformer models and should be interpreted as analytical insight under explicit simplifying conditions.The passage does not present the theorem as a complete characterization of practical optimization.

B. Dataset Details

PERO is evaluated across multiple encrypted traffic classification benchmarks, with ET-BERT tested on three datasets and YaTC validated on four broader settings.

  • Evaluation Benchmarks: ET-BERT experiments use USTC-TFC, ISCX-VPN-Service, and ISCX-VPN-App, while backbone validation applies PERO to YaTC on USTC-TFC, ISCX-VPN, ISCX-Tor, and CICIoT2022.This evaluation spans malware, benign applications, VPN and non-VPN traffic, Tor traffic, and IoT attack behaviors.
  • Dataset Characteristics: 97,115 encrypted traffic samples in USTC-TFC cover 10 malware families and 10 benign applications.USTC-TFC is attributed to Wang et al. (2017).
  • Dataset Characteristics: 60,000 samples and 12 service-level classes define ISCX-VPN-Service, while 77,163 samples and 17 application-level classes define ISCX-VPN-App.Both datasets contain encrypted traffic under VPN and non-VPN scenarios and follow ET-BERT’s settings.
  • Dataset Characteristics: ISCX-Tor contains Tor traffic across eight categories, and CICIoT2022 provides a large-scale benchmark spanning diverse IoT devices and attack behaviors.These datasets are used in YaTC-based experiments.

C. Implementation Details

PERO uses ET-BERT as its primary backbone, with a lightweight pre-evaluation module that regresses sample losses from sequence, label, and probability representations. Experiments use standardized preprocessing, 8:1:1 class-preserving splits, specified optimization protocols, and supplementary validation with YaTC; the paradigm may also extend to RL post-training.

  • Model Architecture: PERO primarily uses pre-trained ET-BERT, while supplementary experiments validate backbone generality with YaTC.ET-BERT is selected for broad adoption, strong cross-benchmark performance, and large parameter scale.
  • Model Architecture: The pre-evaluation module uses pooled sequence embeddings, hard-label lookup embeddings, and linearly projected soft targets to regress loss through a three-layer MLP.Its regression head has dimensions 770, 385, and 385, following a two-layer sequence encoder and global mean pooling.
  • Data Preprocessing and Splits: All datasets are randomly divided into training, validation, and test sets in an 8:1:1 ratio while preserving class distributions.ET-BERT and YaTC experiments follow their respective original preprocessing and tokenization pipelines.
  • Training Protocol: The default batch size is B = 32, selection methods use candidate pools of ˆB = 2B, and Adam or AdamW optimize the modules at learning rates of 5 × 10−6 and 1 × 10−6.The pre-evaluation module learns from historical classifier risks, while the primary classifier follows the standard ET-BERT fine-tuning protocol.
  • Potential Extension to RL Post-training: PERO’s pre-evaluation and selective optimization paradigm may extend to RL post-training by predicting sampled-trajectory utility and focusing expensive updates on informative trajectories.The proposed extension is described as a natural application beyond PERO’s supervised post-training setting.

D. Additional Backbone Evaluation

Using YaTC as an additional backbone, PERO is evaluated across four encrypted-traffic benchmarks to assess scalability and backbone generality. It achieves leading overall and tail-conditioned performance, including a substantial AC0.9 improvement on CICIoT2022.

  • Evaluation Setup: Using YaTC as the backbone, PERO is evaluated on USTC-TFC, ISCX-VPN, ISCX-Tor, and CICIoT2022.The evaluation follows YaTC’s setting, with results reported in Tables 5–6.
  • Overall Performance: PERO obtains the best F1 on ISCX-VPN, USTC-TFC, and CICIoT2022, while remaining within 0.39% of the best F1 on ISCX-Tor.This demonstrates competitive overall performance across all four benchmarks.
  • Tail-Conditioned Performance: PERO achieves the best AC0.9 on three datasets and the best F10.9 on ISCX-VPN and ISCX-Tor.Under tail-conditioned metrics, PERO leads multiple datasets despite MC-CVaR attaining the highest F10.9 overall.
  • Tail-Conditioned Performance: 60.11% AC0.9 on CICIoT2022 improves over the second-best 57.83%.The improvement is from 57.83% to 60.11%.
Loading 2608.15504v1…