PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems

Liam Daly Manocchio, Siamak Layeghy, Wai Weng Lo, Gayan K. Kulatilleke, Mohanad Sarhan, Marius Portmann

arXiv:2304.14746v1cs.CRcs.AIcs.CLcs.NEcs.NI

TL;DR

Existing NIDS research often overlooks sequential flow behavior, motivating a framework for systematically evaluating transformer-based NIDS designs. FlowTransformer enables interchangeable components and evaluates them across three benchmark datasets, finding that classification-head choice is most important and that selected components can reduce model size by over 50% without accuracy loss.

  • Problem

    Existing ML-based NIDS research often classifies individual flows in isolation despite the need to model sequential network behavior and long-range traffic patterns.

  • Method

    FlowTransformer is a modular framework for evaluating interchangeable input encodings, transformer architectures, and classification heads on flow-based NIDS data.

  • Results

    Across three benchmark datasets, classification-head choice most strongly affected performance, while record-level encoding reduced model size by more than half without reducing classification performance.

  • Takeaways & Limitations

    Last Token was the best classification head, and shallow transformer models were sufficient for certain tested NIDS tasks.

Abstract

from arXiv · show

This paper presents the FlowTransformer framework, a novel approach for implementing transformer-based Network Intrusion Detection Systems (NIDSs). FlowTransformer leverages the strengths of transformer models in identifying the long-term behaviour and characteristics of networks, which are often overlooked by most existing NIDSs. By capturing these complex patterns in network traffic, FlowTransformer offers a flexible and efficient tool for researchers and practitioners in the cybersecurity community who are seeking to implement NIDSs using transformer-based models. FlowTransformer allows the direct substitution of various transformer components, including the input encoding, transformer, classification head, and the evaluation of these across any flow-based network dataset. To demonstrate the effectiveness and efficiency of the FlowTransformer framework, we utilise it to provide an extensive evaluation of various common transformer architectures, such as GPT 2.0 and BERT, on three commonly used public NIDS benchmark datasets. We provide results for accuracy, model size and speed. A key finding of our evaluation is that the choice of classification head has the most significant impact on the model performance. Surprisingly, Global Average Pooling, which is commonly used in text classification, performs very poorly in the context of NIDS. In addition, we show that model size can be reduced by over 50\%, and inference and training times improved, with no loss of accuracy, by making specific choices of input encoding and classification head instead of other commonly used alternatives.

I. INTRODUCTION

FlowTransformer addresses the limited use of sequential flow data in NIDS by enabling systematic evaluation of transformer architectures and their components. The paper evaluates trade-offs among accuracy, model size, and speed across flow-based network datasets.

  • Current ML-based NIDS research often classifies individual flow records in isolation despite network communications being sequential.
  • Transformers can capture long-range information and complex traffic patterns distributed across extended periods.
  • FlowTransformer enables systematic interchange and evaluation of input encodings, transformer architectures, and classification heads for flow-based NIDS.
  • The study uses flow data because flow records compactly represent communication, reduce processing volume, and are already collected by many networks.
  • Over 50% model-size reduction and lower inference time are achieved at the same accuracy through selected transformer components.

IV. TRANSFORMERS FOR NIDS

Transformer-based NIDSs process sequences of network flows by combining input encoding, transformer blocks, and a classification head. FlowTransformer supports this pipeline for online, flow-based intrusion detection while exposing model-selection trade-offs.

  • FlowTransformer provides a framework for implementing, testing, and evaluating transformer-based NIDS pipelines on flow-based networking data.
  • A transformer-based NIDS analyzes sequences of flow records and produces classifications for those records.
  • Transformers support parallel execution and can capture complex relationships between network flows for scalable NetFlow processing.
  • Selecting a transformer model requires considering the distinct strengths and weaknesses of available architectures.
  • The architecture comprises an input encoding module, transformer blocks, and an output head that converts sequential outputs into one classification result.

A. Input Encoding

FlowTransformer treats input encoding as a learned transformation from network flows to fixed-length vectors, then applies transformer blocks and a classification head. Its modular design supports rapid testing of these components.

  • A. Input Encoding: Input encoding transforms network flows into fixed-length vectors that a transformer can ingest, analogous to tokenisation in NLP.
  • A. Input Encoding: The encoding is part of the model and is learned during training, unlike preprocessing performed before model execution.
  • B. Transformer Blocks: Encoder blocks produce fixed-length representations for flows while considering each flow’s relation to other inputs in the sequence.
  • B. Transformer Blocks: For NIDS classification, decoder layers can be replaced by a classification head, as in encoder-only BERT, whereas GPT uses decoder blocks exclusively.
  • C. Classification Head: The classification head converts sequence outputs into class predictions while avoiding parameter growth caused by directly flattening arbitrarily long sequences.
  • FlowTransformer offers common implementations of transformer components and supports extending the framework with new components.

A. Input Encoding Options

FlowTransformer supports several input encodings for tabular flow data, accommodating categorical and numerical fields before transformer processing. Record-level encodings can represent an entire flow in one pass and reduce redundant information.

  • A. Input Encoding Options: Flow records contain numerical fields with meaningful distances and categorical fields whose numeric values do not necessarily reflect contextual similarity.
  • Categorical Encoding: Categorical inputs can use lookup-based embeddings, dense embeddings, or linear projection layers.
  • Categorical + Numerical - Record Level:: Record-level encoding embeds an entire flow in one pass instead of transforming each categorical field separately.
  • Categorical + Numerical - Record Level:: Lookup-based embedding cannot encode a complete flow record because it does not operate on continuous numerical fields.
  • Categorical + Numerical - Record Level:: Record-level dense embedding one-hot encodes categorical fields, concatenates them with numerical fields, and maps the result to a continuous vector.
  • Categorical + Numerical - Record Level:: Record-level projection uses the same construction as dense record-level embedding but omits the nonlinear activation and bias.

3) Other Approaches:

The paper evaluates alternative input and classification-head approaches for adapting transformer outputs to flow-based NIDS classification. It emphasizes that sequence handling and the choice of output representation must match the task of classifying flows.

  • Input encoding: No encoding passes pre-processed flow records directly to the transformer, but one-hot categorical fields can significantly increase dimensionality.The approach is potentially feasible for lightweight flow formats with few features.
  • Input encoding: Handling flow records as text is not explored because it can discard structured feature–value relationships and hinder modeling of fixed-format dependencies.The paper states that this may lead to suboptimal performance.
  • Classification head: Flattening sends all sequential transformer outputs to an MLP, but its parameter count rises exponentially with sequence length.The classification head must convert sequence outputs into a single NIDS classification result.
  • Classification head: Global Average Pooling averages each feature across the sequence, which may be unsuitable when only the last flow is classified.Previous flows may not be appropriate to combine uniformly for the target flow.
  • Classification head: Last Token uses the final contextual embedding, while Featurewise Projection applies featurewise dense transformations that can weight flows differently.Last Token is designed for tasks concerned with the class of the last flow.
  • Classification head: A special classification token appends a task-directed position whose transformer output feeds a dense classifier.The paper evaluates this approach for NIDS classification and notes its suitability for binary, multi-class, and other tasks.

VI. IMPLEMENTATION

FlowTransformer implements transformer-based NIDS pipelines as interchangeable processing blocks for dataset ingestion, preprocessing, encoding, modeling, evaluation, and export. This design supports customization and repeated experimentation on flow-based datasets.

  • Pipeline architecture: FlowTransformer forms a processing pipeline from interchangeable blocks for customization, training, and evaluation on flow-based NIDS datasets.The implementation includes source code, documentation, and reference transformer implementations for public use.
  • Data ingestion: Dataset ingestion accepts tabular datasets and automatically handles out-of-range or missing values using a dataset specification.The specification identifies categorical and numerical features and the class column for later stages.
  • Preprocessing: Preprocessing supports configurable splits, categorical one-hot or integer encoding, logarithm-based numerical transformation, and custom fit/transform methods.The paper uses a 90% to 10% training/evaluation split for its experiments.
  • Model construction: After preprocessing determines the final input size, FlowTransformer constructs a TensorFlow Keras model that can be compiled and trained independently.The model is built only after the processed input dimensions are known.
  • Interchangeable components: The model exposes interchangeable input encoder, transformer, and classification-head components.The transformer component can be replaced by any model accepting a 3-dimensional input, including the provided transformer blocks and decoder implementations.
  • Evaluation: Evaluation methods optimize dataset loading between runs and produce experiment outputs containing parameters, size, timing, performance, and training progress.These outputs were used to collect the paper’s experimental results.

VII. EXPERIMENTAL METHODOLOGY

The experiments systematically compare transformer components and hyperparameters across three public flow-based NIDS benchmark datasets. They evaluate performance alongside model and runtime considerations using repeated grid-search experiments.

  • Experimental design: The study tests input encodings, transformer blocks, output encodings, and related components across three benchmark NIDS datasets.The FlowTransformer framework is used to compare applicable transformer configurations.
  • Experimental design: The grid search varies input encoding, transformer block type, depth, feed-forward size, attention heads, classification head, and learning rate.These dimensions cover both architectural choices and training configuration.
  • Experimental design: Each grid-search experiment is repeated at least three times, with the best repeat selected to control for poor model initialization.Training uses early stopping with patience 5 and a maximum of 20 epochs.
  • Experimental setup: Results were collected using TensorFlow with GPU support on an NVIDIA GeForce RTX 2070 Max-Q and a six-core Intel Core-i7-10750H CPU.The software environment used Python 3.9 on Windows.
  • Datasets: The evaluation uses flow-format versions of NSL-KDD, UNSW-NB15, and CSE-CIC-IDS2018.These are widely used NIDS datasets converted to a standardized flow-based format.
  • Metrics: Performance is assessed with F1 score, false alarm rate, and detection rate, with F1 score used as the primary comparison metric.The metrics are computed from true and false positive and negative counts.
  • Hyperparameters: The study also compares transformer configurations and hyperparameters listed in Table I.The table provides the hyperparameter values used in the experiments.

E. Inference & Training Time

The evaluation measures training and inference efficiency alongside predictive performance, showing that input-encoding choices can substantially reduce model size and improve throughput without sacrificing performance.

  • Timing methodology: Training time includes backpropagation and is averaged per batch after clearing the GPU and using a warm TensorFlow state.Inference timing uses repeated measurements across randomly selected batches, with median within-batch timings and a mean across 50 batches.
  • Timing methodology: Throughput is reported as the number of flows ingested by the model per second.
  • Evaluation measures: The study compares input encodings and classification heads using parameter count, F1 score, false alarm rate, true positive rate, and training and inference throughput.The basic transformer results are reported for CSE-CIC-IDS2018, with corresponding NSL-KDD and UNSW-NB15 results described as largely consistent.
  • Input-encoding efficiency: Input encodings produce minimal F1-score differences, but record-level embedding achieves comparable performance with approximately half the parameters of categorical feature embedding.With the Last Token head, categorical feature embedding has approximately 500K parameters versus approximately 200K for record-level embedding.
  • Input-encoding efficiency: Projection and embedding perform equivalently overall, while projection is a simpler and smaller transformation that can reduce model size.
  • Input-encoding efficiency: Record-level approaches can produce smaller models with faster inference and training-related efficiency advantages without sacrificing performance relative to more common approaches.The passage reports that record-level approaches outperformed lookup embedding in flows processed per second during inference, while also noting slower training for the record-level approach.

B. Transformer Size

Across the benchmark datasets, smaller transformer configurations generally match or outperform larger ones, while classification-head selection has the largest and most consistent effect on F1 performance.

  • Transformer depth and width: The shallowest 2-layer transformer outperformed deeper models across all three benchmark datasets.
  • Transformer depth and width: The 2-head transformer outperformed models with more attention heads across the three benchmark datasets.
  • Transformer depth and width: The smallest internal feed-forward size performed best on CSE-CIC-IDS2018 and UNSW-NB15, whereas the largest size performed best on NSL-KDD.
  • Transformer depth and width: Shallow models showed comparable performance to deeper models across the tested datasets, and fewer parameters can reduce training-data requirements and overfitting risk.
  • Classification heads: Classification heads had the largest overall effect on performance, consistently across all datasets.
  • Classification heads: Last Token performed best consistently, followed by Flatten and featurewise embedding, while CLS and Global Average Pooling performed very poorly.Global Average Pooling is especially notable because it is commonly used for text classification, but averaging can include previous-flow information irrelevant to classifying the last flow.

D. Comparative Evaluation

The comparative evaluation finds that shallow and deep transformer models achieve similar F1 scores, but shallow models are far smaller and faster, making large GPT and BERT configurations unattractive for this NIDS setting.

  • Evaluation design: The comparison evaluates Basic Dense Encoder, Basic Dense Decoder, GPT, and BERT models using parameter count, training and inference times, F1 score, detection rate, and false alarm rate.For each model, the study reports the two highest-performing combinations of input encoding and classification head.
  • Efficiency comparison: Shallow and deep models have similar F1 scores, but the deep models have orders-of-magnitude more parameters and much lower throughput.
  • Efficiency comparison: ≈200K parameters and ≈8.5K flows per second were reported for the best shallow CSE model, versus ≈29M parameters and ≈300 flows per second for the deep encoder.
  • Efficiency comparison: The deeper models did not outperform shallow transformer models even when evaluated with longer training and no early stopping.
  • Architecture comparison: Decoder models including GPT performed comparably to encoder models, and autoregressive attention suits classifying the most recent flow in a real-time stream.
  • Selected configuration: The selected high-performing configuration uses record-level projection, a decoder transformer block, an internal size of 1–2x the input size, and the Last Token head.
  • Practical guidance: The recommended starting choices are the input encoding and classification head, while learning rates should be determined through repeated training iterations because training volatility was observed.

IX. CONCLUSION

FlowTransformer provides a modular framework for implementing and systematically evaluating transformer-based NIDSs on flow-based networking data. Across three benchmark datasets, the evaluation identifies classification-head choice and record-level representations as important performance, size, and speed factors.

  • Framework: FlowTransformer enables rapid implementation and evaluation by combining input encodings, classification heads, transformer constructions, and flow-based training and inference pipelines.The framework is designed for NIDS experimentation with interchangeable transformer components.
  • Evaluation: Across 3 widely used NIDS benchmark datasets, the evaluation examined a wide range of transformer model configurations, including GPT 2.0 and GPT 3.0-based technologies.The experiments considered model size and performance trade-offs.
  • Classification heads: The classification head was the most critical factor in model performance, with ‘Last Token’ identified as the best choice.This finding concerns the evaluated transformer configurations for NIDS classification.
  • Efficiency: Record-level embedding or projection reduced model size by more than half without reducing classification performance and provided the fastest inference and training times.The speed advantage applied to both inference and training.
  • Model depth: Shallow transformer models were sufficient for certain NIDS tasks, including those represented in the tested benchmark datasets.The conclusion is limited to the tasks and datasets evaluated in the paper.
  • Architecture trade-offs: Encoders and decoder blocks were both effective, but encoder blocks were favored because of their smaller size and higher applicability; larger GPT models matched shallow-model performance but had lower throughput.The paper therefore identifies larger GPT models as unlikely to be optimal for most NIDS tasks.

APPENDIX

The appendix includes tables for the NSL-KDD and UNSW-NB15 datasets using the format established by Table II.

  • Dataset tables: Table IV presents the NSL-KDD dataset in the same format as Table II.
  • Dataset tables: The appendix dataset tables use a shared format established by Table II.
  • Dataset tables: Table V presents the UNSW-NB15 dataset in the same format as Table II.
Loading 2304.14746v1…