PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Model Scaling: Rethinking Sequence Construction for Transformer-based Network Intrusion Detection

Zeeshan Zulkifl Shah, Muhammad Ikram, Dali Kaafar, Yipeng Zhou

arXiv:2610.17725v1cs.CRcs.NI

TL;DR

Flow-based NIDS need sequences that preserve relationships among communications, yet fixed temporalization can split related activity and combine unrelated flows. The paper proposes a pipeline framework and session-coherent temporalization; across three datasets, it reports more than doubled Macro-F1 over timestamp-based windowing. Its results point to sequence construction, rather than transformer scale, as the more consequential design choice, while cross-dataset transfer remains affected by domain shift.

  • Problem

    Fixed sequence construction can fragment related attack activity and merge unrelated communications, while model-centric evaluations confound sequence construction with model capacity.

    Source: §I · p. 1

  • Method

    The paper decomposes transformer-based NIDS into four design dimensions and constructs session-coherent sequences by organizing flows according to communication sessions.

    Source: §I · pp. 1–2

  • Results

    Session-coherent temporalization more than doubles Macro-F1 over timestamp-based windowing, and compact transformers reach up to 0.95 Macro-F1.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across the evaluated datasets, preserving communication-session semantics contributes more to transformer-based NIDS performance than increasing model capacity.

    Source: §I · p. 2

    Cross-dataset transfer remains affected by domain shift, and robust deployment also requires online adaptation.

    Source: E. RQ5: Cross-Dataset Generalization · p. 9

Abstract

from arXiv · show

Transformer-based network intrusion detection systems (NIDS) have shown strong potential for detecting multi-stage cyberattacks, yet most methods construct input sequences using fixed temporalization schemes rather than treating sequence construction as a systems design decision. In flow-based intrusion detection, timestamp-based windowing often fragments related attack activities while merging unrelated communications, reducing the behavioral context available to transformers. We present a unified pipeline abstraction that decomposes transformer-based NIDS into four design dimensions (traffic representation, temporalization, transformer backbone, and prediction aggregation) and propose session-coherent temporalization, which constructs sequences from communication sessions instead of fixed time windows. Experiments on three complementary intrusion detection datasets show that temporalization is the dominant factor governing transformer performance: session-coherent temporalization more than doubles Macro-F1 over timestamp-based windowing, enabling lightweight transformers to achieve up to 0.95 Macro-F1 at substantially higher inference throughput than pretrained models. We further vary the transformer size from 0.65M to 3.38M parameters and find that larger transformer models provide no measurable accuracy improvement despite substantially higher computational cost, while zero-shot cross-dataset evaluation shows only modest gains under domain shift. These results indicate that preserving communication-session semantics, rather than scaling transformer capacity, is the key to accurate and efficient transformer-based intrusion detection.

I. INTRODUCTION

Flow-based NIDS must preserve relationships among network interactions as encrypted traffic limits payload inspection, but fixed sequence construction can distort that context. The paper treats temporalization as a systems design choice and proposes session-coherent sequences, reporting larger gains from preserving session semantics than from scaling models.

I. INTRODUCTION

  • Because encrypted payload inspection is increasingly impractical, operational NIDS rely on flow telemetry to detect coordinated interactions such as reconnaissance, exploitation, lateral movement, and command-and-control.Effective detection must preserve temporal relationships while meeting high-speed network latency and throughput requirements.Source: §I · p. 1
  • Existing fixed sequence heuristics can split related attack stages and combine unrelated communications, leaving transformers with distorted behavioral context.This motivates treating sequence construction as a systems design decision rather than a fixed implementation detail.Source: §I · p. 1
  • The unified transformer-NIDS framework separates traffic representation, temporalization, transformer backbone, and prediction aggregation to support controlled evaluation of each design dimension.The paper argues that preserving communication-session semantics contributes larger gains than increasing model capacity.Source: §I · p. 2
  • Session-coherent temporalization organizes flows by communication session rather than arbitrary timestamp windows, preserving within-session behavioral continuity and reducing fragmentation and cross-session contamination.It is intended to let transformers learn from communication-consistent contexts.Source: §I · p. 2
  • Session-coherent temporalization more than doubles Macro-F1 over timestamp-based windowing, with compact transformers reaching up to 0.95 Macro-F1 and throughput approaching 106 processed flow records per second.Experiments cover UNSW-NB15, CICIDS2018, and MQCIDS.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Network flows lack an intrinsic sequence, so temporalization determines which interactions become context for sequential learning. Prior work includes flow-based and transformer-based traffic models, but fixed sequence construction leaves its effect confounded with model capacity.

A. Flow-based Intrusion Detection and Temporalization

  • Fixed timestamp windows and batching heuristics can fragment related attack activity and group unrelated communications, limiting behavioral information available for inference.Flow records must be ordered into sequences before sequential models can use them.Source: A. Flow-based Intrusion Detection and… · p. 2

B. Related Work and Research Gap

  • Prior NIDS work spans signature-based and learning-based detection, sequence-oriented traffic models, transformer models, and online adaptation approaches.Examples include FS-Net, ACID, RIDS, CTI-BERT, and AOC-IDS,,,,.Source: B. Related Work and Research Gap · p. 2
  • Existing evaluations often inherit fixed temporalization, confounding sequence construction with model capacity; the paper's framework varies representation, temporalization, backbone, and aggregation independently.The paper reports that preserving behavioral coherence contributes larger performance gains than increasing transformer capacity.Source: B. Related Work and Research Gap · p. 3

III. SYSTEM DESIGN

The paper frames transformer-based intrusion detection as a pipeline of configurable design dimensions rather than a comparison of architectures under fixed preprocessing. Its framework supports controlled evaluation, with temporalization prioritized because it determines behavioral information before learning.

III. SYSTEM DESIGN

  • The proposed system design formulates transformer-based intrusion detection as a pipeline of independently configurable dimensions rather than another transformer architecture.Source: §III · p. 3

A. Pipeline Abstraction

  • The pipeline abstraction independently controls traffic representation, temporalization, transformer backbone, and prediction aggregation, enabling controlled evaluation that isolates temporalization from architecture.The framework is presented as a way to address confounding in prior comparisons under fixed preprocessing.Source: A. Pipeline Abstraction · p. 3

B. Independent Design Dimensions

  • Temporalization receives the paper's focus because it alone determines behavioral information available before learning; other dimensions are fixed or varied through controlled ablations.Source: B. Independent Design Dimensions · p. 3

C. Session-Coherent Temporalization

Session-coherent temporalization constructs sliding windows within individual communication sessions, preserving related-flow context that global timestamp-based windows can fragment or contaminate. Its structural metrics show substantially stronger preservation than timestamp-based construction.

C. Session-Coherent Temporalization

  • Related-flow retention measures the fraction of emittable within-session flow pairs that co-occur in at least one window; purity measures the dominant session’s share of a window.Fragmentation is defined as 1 − Ret(A); higher retention and purity and lower fragmentation indicate stronger preservation of behavioral structure.Source: C. Session-Coherent Temporalization · p. 3
  • Related-flow retention rose from 0.096 to 0.752, purity from 0.190 to 1.000, and fragmentation fell from 0.904 to 0.248 with session-coherent temporalization across three datasets.The comparison uses matched window length and stride (k = 32, s = 16).Source: C. Session-Coherent Temporalization · p. 3, Table I · p. 4
  • Each emitted window contains flows from exactly one communication session, preserving behavioral continuity and preventing cross-session contamination.Flows are grouped by session, chronologically ordered, then windowed independently within each session.Source: C. Session-Coherent Temporalization · p. 4
  • Session-coherent windows omit sessions shorter than k, and under matched overlapping-window settings their count is no greater than global sliding-window construction.For (k, s) = (32, 16), each flow appears in at most two windows.Source: C. Session-Coherent Temporalization · p. 4

D. Transformer Learning

The framework separates behavioral sequence construction from transformer learning by converting each emitted flow window into model-specific tokens, contextualizing them, and aggregating a prediction. Its model-agnostic design supports controlled comparisons across backbone types.

D. Transformer Learning

  • Each emitted flow window is transformed into model-specific tokens, contextualized by a transformer, and passed to an aggregation operator for the final prediction.The formulation is T = (ϕ(x1), …, ϕ(xn)), H = fθ(T), and ŷ = Agg(H).Source: D. Transformer Learning · p. 4
  • The pipeline decouples behavioral sequence construction from transformer learning, with prediction aggregation applied after contextualized embeddings are computed.The aggregation operator produces the final prediction from H.Source: D. Transformer Learning · p. 4
  • Different transformer backbones operate on identical temporalized inputs, enabling controlled comparisons of temporalization and model capacity while other pipeline components are fixed.The evaluated backbones include lightweight transformers trained from scratch and representative pretrained encoder- and decoder-based models.Source: D. Transformer Learning · p. 4

E. Computational Complexity and Deployment Considerations

Session-coherent temporalization adds active-session state to online processing while retaining linear preprocessing complexity. The evaluation varies pipeline dimensions in controlled experiments to examine temporalization, model capacity, deployment trade-offs, and cross-dataset transfer.

E. Computational Complexity and Deployment Considerations

  • Session-coherent temporalization required 2.25× the processing time of timestamp-based global sliding-window construction on average.The main and complete-session robustness benchmark runtime ratios were 2.1669× and 2.3304×, respectively.Source: E. Computational Complexity and… · p. 5
  • The method sustained 1.04 million input flows/s while emitting 54% fewer windows for downstream inference, though this benchmark isolates temporalization rather than end-to-end throughput.Both approaches remain linear in processed flows under bounded active-session state, while the session-coherent approach requires state proportional to concurrently active sessions.Source: E. Computational Complexity and… · pp. 4–5

IV. EVALUATION METHODOLOGY

  • The evaluation varies only the pipeline components needed for each research question, holding the remaining components fixed.Five questions address temporalization, capacity, foundation-model comparison, deployment trade-offs, and cross-dataset generalization.Source: §IV · p. 5, A. Research Questions · p. 5

A. Research Questions

  • Table II identifies which primary pipeline components are varied or fixed for the research questions.The questions include whether session-coherent temporalization improves zero-shot transfer across heterogeneous network environments.Source: Table II · p. 5, A. Research Questions · p. 5

B. Experimental Setup

The experiments use three complementary flow-based intrusion-detection datasets and session-aware splits, alongside from-scratch and pretrained transformer models. Evaluation covers detection, deployment efficiency, and robustness under domain shift.

B. Experimental Setup

  • The study evaluates UNSW-NB15, CICIDS2018, and MQCIDS, spanning benchmark enterprise traffic, contemporary attacks, and deception-based operational environments.Cross-dataset evaluation maps dataset-specific features and labels to a common schema.Source: B. Experimental Setup · p. 5
  • Sessions are reconstructed from five-tuples and inactivity timeouts for UNSW-NB15 and CICIDS2018, while MQCIDS uses Suricata-derived behavioral session identifiers.The default configuration is (k, s) = (32, 16).Source: B. Experimental Setup · p. 5
  • Datasets are split into 70% training, 15% validation, and 15% testing before window construction, with sessions disjoint across partitions.Normalization statistics are measured only on training data to prevent validation or test information from entering preprocessing parameters.Source: B. Experimental Setup · p. 5
  • From-scratch lightweight transformers span hidden dimensions 64, 128, and 256, depths 1, 2, and 4, and 2, 4, or 8 attention heads; pretrained models include ModernBERT and Qwen2.5-0.5B.Additional pretrained comparisons include DeBERTa-v3 and CTI-BERT.Source: B. Experimental Setup · pp. 5–6
  • Detection effectiveness is primarily measured by Macro-F1, with PR-AUC and accuracy as complementary metrics; deployment efficiency includes throughput, latency, memory, utilization, and model size.Throughput counts processed flow positions, including repeats in overlapping windows, divided by measured inference time.Source: B. Experimental Setup · p. 6
  • Cross-dataset robustness is assessed using transfer robustness gap, defined as within-dataset Macro-F1 minus unseen-dataset Macro-F1; lower values indicate stronger transfer.Source: B. Experimental Setup · p. 6

V. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments isolate sequence construction as a key determinant of transformer-based NIDS performance. Session-coherent temporalization preserves behavioral context and outperforms timestamp-based windowing, while representation differences are smaller by comparison.

A. RQ1: Temporalization Dominates Detection Performance

  • Session-coherent temporalization more than doubles Macro-F1 over timestamp-based windowing while maintaining essentially identical processed-flow inference throughput.The comparison holds the shallow-transformer architecture constant, isolating sequence construction as the changed factor.Source: A. RQ1: Temporalization Dominates… · p. 6, Figure 2 · p. 6
  • The fixed-architecture comparison attributes the performance difference to behavioral context: session-coherent sequences preserve communication-session semantics, unlike fragmented or mixed timestamp windows.Independent per-flow classification also ignores relationships across communications.Source: A. RQ1: Temporalization Dominates… · p. 6, Figure 2 · p. 6
  • Per-flow representation reaches Macro-F1 0.761, versus 0.580 for fieldtoken and 0.562 for serialized-text, smaller differences than temporalization’s 2.2× improvement.The reported retention and purity gains help explain the detection improvement through preserved within-session context.Source: A. RQ1: Temporalization Dominates… · pp. 6–7
  • Once sequence construction loses behavioral continuity, changing the input representation alone cannot recover that context.The section therefore frames behavior-preserving temporalization as a systems design decision rather than fixed preprocessing.Source: A. RQ1: Temporalization Dominates… · p. 7

B. RQ2: Behavioral Context over Model Scale

With session-coherent temporalization fixed, increasing transformer capacity yields diminishing returns, and aggregation choices have a smaller effect than temporalization. The optimized shallow pipeline matches or exceeds pretrained models’ detection performance at much higher throughput in the evaluated flow-based NIDS setting.

B. RQ2: Behavioral Context over Model Scale

  • A 0.65M-parameter model reaches Macro-F1 0.947, while scaling to 3.38M parameters lowers Macro-F1 to 0.934 and throughput from 0.40M to 0.13M flows/s.The larger configuration is 5.2× the size and has 3.1× lower processed-flow throughput.Source: Figure 3 · p. 7
  • Under fixed capacity, mean pooling is strongest for the encoder and attention pooling for the decoder; decoder CLS pooling yields Macro-F1 0.197 despite 97% overall accuracy.The contrast underscores the importance of balanced metrics in highly imbalanced intrusion detection.Source: Figure 4 · p. 7
  • Aggregation differences are considerably smaller than temporalization differences, and moderate-capacity transformers provide nearly identical detection quality with higher throughput and lower computational cost.Detection performance saturates quickly after moving from very small to moderate-capacity models.Source: B. RQ2: Behavioral Context over Model… · p. 7

C. RQ3: Pipeline Optimization versus Foundation Models

  • The optimized shallow transformer matches or exceeds substantially larger pretrained models while sustaining one to two orders of magnitude higher inference throughput.ModernBERT and Qwen2.5-0.5B are approximately 229× and 759× larger than the 0.65M-parameter ST-B3, yet neither improves Macro-F1.Source: C. RQ3: Pipeline Optimization versus… · pp. 7–8, Table IV · p. 8
  • The pretrained-model comparison tests scale only after communication-session semantics and the remaining pipeline components are optimized, not general capabilities across unrelated tasks.Under this evaluated flow-based NIDS pipeline, the findings do not imply that foundation models are ineffective.Source: C. RQ3: Pipeline Optimization versus… · p. 8
  • The authors recommend prioritizing behavior-preserving pipeline design before increasing model capacity for transformer-based intrusion detection.With communication-session semantics preserved, lightweight transformers balance detection quality, efficiency, and throughput for high-speed deployment.Source: C. RQ3: Pipeline Optimization versus… · p. 8

D. RQ4: Deployment Trade-offs

The selected shallow-transformer configuration offers a deployment operating point with high detection quality, throughput, and performance across datasets. Longer contexts reduce ModernBERT throughput, while shallow-transformer quality stays nearly constant; session-coherent window construction also sustains high input-flow throughput despite overhead.

D. RQ4: Deployment Trade-offs

  • Longer contexts gradually improve ModernBERT Macro-F1 but sharply reduce throughput, whereas the shallow transformer maintains near-constant detection quality and much higher throughput.The cited passage describes the trend without specifying the context-length values.Source: D. RQ4: Deployment Trade-offs · p. 8
  • The selected shallow-transformer configuration performs strongly across UNSW, CICIDS, and MQCIDS, with Macro-F1 values of 0.710, 0.922, and 0.945, respectively.Accuracy ranges from 0.987 to 0.996; the lower UNSW performance reflects greater class imbalance and attack diversity.Source: Figure 8 · p. 9
  • Session-coherent window construction sustains 1.04 million input flows/s despite measurable overhead.Across the three benchmarks, lightweight transformers perform strongly at substantially lower inference cost than larger models.Source: D. RQ4: Deployment Trade-offs · p. 9
  • The authors recommend prioritizing behavior-preserving sequence construction before increasing model capacity.They frame it as a more effective strategy for improving the accuracy–efficiency trade-off than simply increasing model complexity.Source: D. RQ4: Deployment Trade-offs · p. 9

E. RQ5: Cross-Dataset Generalization

In zero-shot transfer, session-coherent temporalization yields modest average gains, while domain shift remains the dominant challenge. The conclusion frames preserving behavioral context as a key systems design principle and identifies online adaptation as future work.

E. RQ5: Cross-Dataset Generalization

  • Macro-F1 rises from 0.414 to 0.423 and average TRG falls from 0.429 to 0.419 with session-coherent temporalization, with improvement or parity in five of six transfer directions.The positive trends are statistically inconclusive for both Macro-F1 (p = 0.19) and TRG (p = 0.28).Source: Table V · p. 9, E. RQ5: Cross-Dataset Generalization · p. 9
  • Domain shift remains the dominant challenge because zero-shot transfer spans differing traffic distributions, attack compositions, feature characteristics, and measurement environments.The paper says session-coherent temporalization does not eliminate the need for target-domain validation or lightweight adaptation in heterogeneous environments.Source: E. RQ5: Cross-Dataset Generalization · p. 9
  • For robust cross-domain deployment, the authors identify online adaptation, including integration with AOC-IDS, as a promising direction.AOC-IDS combines contrastive representation learning and pseudo-label-driven updates.Source: E. RQ5: Cross-Dataset Generalization · p. 9

VI. CONCLUSION

  • The conclusion identifies sequence construction, rather than model capacity, as the primary bottleneck and says lightweight transformers can match or outperform larger pretrained models with higher deployment efficiency.It presents preserving behavioral context as a systems design principle and proposes online contrastive adaptation over session-coherent flow sequences for future work.Source: §VI · pp. 9–10

Related papers

Loading 2610.17725v1…