PaperlayerView exact revision on arXiv

Source-linked AI summary

A First-Principles Evaluation of Graph-Based Network Intrusion Detection Systems

Rui Zhao, Wajih Ul Hassan

arXiv:2609.12263v1cs.CR

TL;DR

Benchmark detection scores alone do not establish whether graph-based intrusion detectors are deployable. The paper evaluates representative systems under a matched protocol with GIDS-Eval, finding that added complexity does not consistently improve detection quality while increasing operational and attack-surface costs.

  • Problem

    High benchmark metrics for graph-based intrusion detectors provide limited evidence about deployability, including resilience to graph-shaping attackers, streaming operation, and the effects of undocumented pipeline choices.

    Source: §1 · p. 1

  • Method

    The paper builds GIDS-Eval to attribute detector performance to pipeline stages under a matched protocol and introduces GIDS-Lite as an encoder-free control.

    Source: §1 · p. 3, §7 · p. 14

  • Results

    Added graph and temporal complexity does not consistently improve clean detection, while GIDS-Lite ranks first by AP on half the datasets at a fraction of the runtime.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    GIDS complexity should be justified under shared evaluation by clear benefits in detection quality, computational cost, stability, and attack applicability.

    Source: §6 · p. 13

    The evaluated benchmarks have fidelity limits: Campus uses scripted APT campaigns that may not match production attack distributions, and OpTC is simulated.

    Source: §3 · p. 4

Abstract

from arXiv · show

Graph-based network intrusion detection systems (GIDS) report strong benchmark detection metrics, but those metrics establish little about deployability. We approach the problem from first principles: rather than inheriting the preprocessing, windowing, and thresholding conventions of each published system, we ask what a controlled comparison requires and impose it uniformly. The result is GIDS-Eval, an evaluation framework that decomposes a GIDS into six interchangeable stages and turns those conventions into explicit experimental variables, so reported performance can be attributed to individual stages instead of whole pipelines. We survey nine representative GIDS, reimplement five of them within GIDS-Eval, and evaluate them on four datasets under one matched protocol. We identify nine recurring evaluation gaps and quantify the impact of each: two crafted edges achieve full evasion against three of the eight detector-dataset pairs with anything to hide; the snapshot window alone accounts for a mean 38.3% relative swing in average precision (AP); aligning preprocessing across systems moves AP by up to 61.8 percentage points for a single detector; and none of the 18 detector-dataset pairs we replay can alert as events arrive. We introduce GIDS-Lite, an encoder-free control built in the same framework, which ranks first by AP on two of the four datasets at up to 575$\times$ lower runtime. Architectural complexity is therefore not a consistent driver of detection quality under our matched protocol on current benchmarks, but it does enlarge the runtime, calibration, and attack surfaces operators must defend.

1 Introduction

The introduction argues that benchmark detection scores alone do not establish deployability, given attack, streaming, and pipeline-attribution questions. It presents GIDS-Eval as a way to test these questions under a matched protocol and reports that added architectural complexity does not consistently improve detection quality.

1 Introduction

  • GIDS-Eval decomposes detectors into six interchangeable stages and makes pipeline choices explicit, enabling performance differences to be attributed to stages rather than whole systems.The study surveys nine representative GIDS, implements five, and runs 1,370 experimental jobs, including a 240-configuration component grid.Source: §1 · p. 2
  • The snapshot window alone swings AP by a mean 38.3%, preprocessing alignment shifts AP by up to 61.8 percentage points, and none of 18 replayed detector–dataset pairs can alert as events arrive.These are among nine recurring evaluation gaps identified by the study.Source: §1 · p. 2
  • GIDS-Eval extends prior black-box GIDS re-runs with stage-level attribution and a control, while adapting PIDS evaluation concerns to network-flow data and its distinct gaps.The paper identifies nine gaps, including measurements neither prior study provides and a streaming-readiness replay measurement.Source: §1 · pp. 1–2
  • 2 of 4 datasets have GIDS-Lite rank first by AP, at up to ∼575× lower end-to-end runtime than the slowest system.Pikachu and VGRNN rank first on the other two datasets.Source: §1 · p. 2
  • Two inserted edges hide every attack edge caught by state-of-the-art GIDS; six of eight detector–dataset pairs with anything to hide fall within twenty.The study identifies this covering-edge attack as a graph-side attack surface.Source: §1 · p. 3

2 Background & Study Scope

The section surveys GIDS across increasingly coupled graph and temporal designs, then narrows the study to systems that can be faithfully rebuilt as interchangeable modules. GIDS-Eval is introduced in §3.

2.1 Evolving GIDS Complexity

  • Recent GIDS range from graph encoders or reconstruction objectives to temporal graph models and designs coupling graph, temporal, edge-feature, and attack-aware components.The systems’ anomaly signals can depend jointly on graph construction, temporal windowing, sequence modeling, and prediction targets.Source: §2.1 · p. 3

2.2 Representative Systems

  • The survey covers nine representative systems, while five are selected for in-depth evaluation based on network-level intrusion detection, peer review, and sufficient implementation detail for faithful modularization.Systems outside full evaluation lack implementation detail, require supervision outside the protocol, or face scale constraints.Source: §2.2 · p. 3
  • VGRNN is evaluated as a temporal graph encoder backbone adopted by later GIDS, not as a purpose-built intrusion detector.The intrusion-detection instantiation was built on the model by later GIDS work.Source: §2.2 · p. 3

3 The GIDS-Eval Framework and Protocol

GIDS-Eval makes pipeline stages and evaluation choices interchangeable, enabling controlled attribution rather than whole-pipeline comparisons. The framework applies a matched protocol to five integrated detectors across four datasets.

3 The GIDS-Eval Framework and Protocol

  • GIDS-Eval divides each detector into six interchangeable stages, from feature extraction through optimization, so modules within a stage can be varied independently.A configuration selects one module and its hyperparameters for each stage.Source: §3 · p. 3
  • Changing a stage reuses cached outputs for unchanged upstream inputs, making the 240-configuration component-attribution sweep tractable across four datasets.Replacing an encoder reruns encoding, decoding, and optimization; changing the loss reruns only optimization.Source: §3 · p. 4, Figure 1 · p. 4
  • The evaluation uses four datasets spanning different telemetry, scales, environments, and attack-labeling mechanisms, with temporal order preserved in train, validation, and test splits.Campus uses scripted APT injections and OpTC is simulated; neither should be treated as a deployment proxy.Source: §3 · p. 4
  • Metrics cover score ranking, thresholded detection, campaign coverage, and execution cost; AP is the primary threshold-free metric for rare attack events.CDR measures the fraction of test-period attack campaigns with at least one flagged positive event.Source: §3 · p. 4
  • The five integrated systems retain their released architectures, preprocessing, objectives, hyperparameters, and scoring logic; wrappers are validated against original-code runs.For a design-choice experiment, only the corresponding module is changed while the rest of the released pipeline stays fixed.Source: §3 · p. 4

4 Evaluation Gaps

This section examines evaluation gaps spanning experimental control and deployment-relevant measurements, including sensitivity to windows, thresholds, preprocessing, and pipeline interactions. It also considers edge-feature value, scalability, adversarial robustness, and how snapshot buffering prevents current systems from alerting as events arrive.

EG1: Snapshot-Window Sensitivity

  • 11 of 16 detector–dataset pairs show more than 20% relative AP swing across snapshot windows, with a mean swing of 38.3%.The best window differs by detector and dataset: on OpTC it is 6 minutes for Argus and Euler but 150 minutes for VGRNN and Pikachu, so no single window is uniformly safe.Source: EG1: Snapshot-Window Sensitivity · p. 5, Figure 2 · p. 6

EG2: Threshold-Selection Instability

  • Euler on Campus has a 70.4-percentage-point TPR range across threshold-calibration seeds, while Argus on LANL has a 42.1-point CDR range.Random validation samples change the calibrated threshold and resulting alerts across runs; threshold-free metrics hide this operating-point instability.Source: EG2: Threshold-Selection Instability · pp. 5–6, Figure 2 · p. 6

EG3: Transductive Thresholding

  • On LANL, inductive thresholding drops CDR from 59.6% to 0.0% and TPR by 66.4 percentage points; on CI-2017, CDR falls to 0.0%, TPR by 83.6 points, and F1 by 75.3 points.Pikachu chooses its threshold from the labeled evaluation edges, a transductive choice unavailable before a future test period; effects also vary by dataset and metric.Source: EG3: Transductive Thresholding · p. 6, Figure 2 · p. 6

EG4: Untested Edge-Feature Value

  • On OpTC, Argus edge features reduce AP from 0.760 to 0.033 and CDR from 100.0% to 0.0%, showing that features can eliminate campaign coverage.Effects vary by dataset: on LANL AP rises from 0.014 to 0.105 while CDR drops from 71.9% to 64.9%; attack and benign feature means are nearly collinear on every dataset (cosine 0.979 to 1.000).Source: EG3: Transductive Thresholding · p. 7, Figure 4 · p. 7

EG5: Scalability Limitations

  • At full scale, the largest workload has 49.3M events, the longest run takes 17.3 hours, and peak memory reaches 67.9 GB; across measurements, memory peaks at 102.2 GB.The largest runtime growth is 14.9× and memory growth is 4.5×; all five systems lack original-evaluation reporting of resource cost and its growth with input scale.Source: EG3: Transductive Thresholding · pp. 7–8

EG6: Unevaluated Adversarial Robustness

  • 2 covering edges achieve full evasion on three of eight detector–dataset pairs with flagged attack edges; six of eight lose all flagged edges within 20 inserted edges.The attack perturbs only edges the detector initially flags; four of 12 pairs flag no attack edge, so there is nothing to evade. Argus on CI-2017 resists at all tested budgets, while VGRNN on OpTC reaches 93.8% ASR at 50 edges without full evasion.Source: Figure 5 · p. 7, EG3: Transductive Thresholding · pp. 8–9

EG9: Missing Real-Time Detection

  • 0 of 18 detector–dataset pairs were real-time ready; median buffering delay at 1× replay was 40 minutes.Four short-window Campus pairs were near-real-time, while the other 14 were batch only, even though most systems kept up with replay.Source: Table 4 · p. 9, EG3: Transductive Thresholding · p. 10

EG7: Lack of Systematic Component Attribution

  • Removing the graph encoder is best marginally on LANL and within 0.005 AP of the best option on OpTC, while Campus and CI-2017 favor MPNN.Across 240 configurations, reconstruction is best on Campus, CI-2017, and LANL, but link prediction is best on OpTC; no single recipe generalizes across datasets.Source: EG3: Transductive Thresholding · p. 9

EG8: No Common Preprocessing Contract

  • VGRNN on LANL rises from .002 to .620 AP under aligned preprocessing, while Anomal-E on CI-2017 drops from .875 to .605 AP.The comparison fixes a shared event table, graph construction, label semantics, split, and snapshot window, isolating preprocessing as the variable.Source: EG3: Transductive Thresholding · p. 10

5 GIDS-Lite vs. Existing GIDS

Under a shared evaluation protocol, GIDS-Lite is an encoder-free control for testing whether added GIDS complexity provides measurable value. Its detection ranking varies by dataset, and it retains snapshot-window sensitivity and other deployment constraints.

5.1 GIDS-Lite Design

  • GIDS-Lite is a single-configuration encoder-free control with a 12-component PCA reconstruction head and 296-dimensional edge vectors, using endpoint states and pair-history features.The predictor is fitted on early trace snapshots, and its threshold is calibrated on chronological validation data and frozen for testing.Source: §5.1 · p. 11, Figure 7 · p. 11
  • GIDS-Lite is a control, not a universal replacement: it closes six evaluation gaps, while window sensitivity and snapshot buffering remain unresolved.The covering-edge attack cannot be instantiated against a detector without a graph-scoring path; this concerns attack applicability, not general robustness.Source: §5.1 · p. 12, Table 6 · p. 12

5.2 Detection Performance

  • At a 0.1% false-positive-rate budget, no dataset retains its reported recall; at 1%, recall ranges from 94% on CI-2017 to zero on OpTC.The reported operating points use alert budgets the authors characterize as questionable, including tens of percent on OpTC.Source: Figure 8 · p. 12, §5.2 · p. 12

5.3 Runtime and Memory

  • GIDS-Lite takes 107.2 seconds and 11.0 GB, while Pikachu takes 17.1 hours; cost rankings do not match detection rankings.Euler and VGRNN use substantially more memory without a consistent detection advantage.Source: §5.3 · pp. 12–13

5.2 Detection Performance

  • GIDS-Lite reaches 0.980 AP on CI-2017 and 0.993 AP on OpTC, while Pikachu leads Campus at 0.915 AP and VGRNN leads LANL at 0.620 AP.The detection benefit of added complexity is dataset dependent; thresholded operating points and alert burden also affect deployment choices.Source: §5.2 · p. 12, Table 7 · p. 13

5.4 Snapshot-Window Sensitivity

  • GIDS-Lite’s AP swing ranges from 0.8% on CI-2017 to 78.1% on LANL as snapshot-window size changes, so removing encoders does not remove window sensitivity.On LANL, campaign exposure also shifts by 23.5 percentage points of CDR; the authors attribute this sensitivity to turning a continuous stream into graphs.Source: §5.4 · p. 13

5.5 Attack Surface

  • GIDS-Lite avoids the specific covering-edge graph-propagation attack because it has no multi-hop aggregation at inference, but this does not establish general adversarial robustness.A white-box feature-perturbation attack remains applicable; its cost depends on whether endpoint representations can be changed.Source: §5.5 · p. 13

6 Discussion & Future Work

The discussion argues that architectural complexity should be justified by empirical net benefit, while current benchmarks leave important factors entangled. It also identifies evaluation dimensions that remain unmeasured or need shared infrastructure.

Added Complexity Needs Clear Benefits

  • Added graph and temporal complexity does not consistently improve clean detection, while increasing runtime and memory costs and potentially exposing attack surfaces.The discussion frames complexity as a claim to be earned empirically, not as a reason to prefer simple designs in every case.Source: §6 · pp. 13–14

Current Benchmarks Still Mix Key Factors

  • Current datasets bundle network setting, logging source, feature visibility, attack type, and temporal granularity, leaving claims about when graph structure helps unresolved.The authors call for benchmarks that vary these factors deliberately while holding the rest fixed.Source: §6 · p. 14

Interpretability Still Matters in Practice

  • The study does not evaluate explanation quality, although analysts need to trace alerts to concrete evidence.The authors recommend including alert traceability alongside detection metrics in future GIDS evaluations.Source: §6 · p. 14

Robustness Must Be Evaluated Explicitly

  • The discussion cautions that simpler detectors are not universally robust and recommends reporting attack success together with attack applicability.Future work should study detector-agnostic attack models, defenses for graph-based and encoder-light detectors, and threat-model-relevant robustness metrics.Source: §6 · p. 14

Shared Evaluation Reduces Repeated Work

  • Incompatible splits, preprocessing, and reporting rules make rigorous comparisons costly to repeat without producing cumulative evidence.The authors identify shared preprocessing artifacts, canonical splits, common metric suites, and reusable harnesses as part of the research agenda.Source: §6 · p. 14

7 Conclusion

The conclusion argues that GIDS-Eval exposes the costs of graph and temporal complexity under a matched protocol, while GIDS-Lite competes on half the datasets at much lower runtime. The appendices document the artifact, ethical scope, AI assistance, and reporting conventions.

7 Conclusion

  • Under a matched protocol, added graph and temporal complexity does not reliably improve detection quality, while GIDS-Lite competes on half the datasets at a fraction of the runtime.The paper concludes that GIDS complexity should be introduced only when it earns its cost under a shared protocol.Source: §7 · p. 14

A Open Science

  • The artifact includes GIDS-Eval, rebuilt detector implementations, sweep configurations, per-job results, and setup instructions; the four public datasets are fetched from their original hosts.Appendix E reports dataset statistics.Source: A Open Science · p. 16

B Ethical Considerations

  • The study uses public datasets, released implementations, and offline traces, with no human subjects, live network experiments, production systems, real credentials, or third parties involved.Source: B Ethical Considerations · p. 16

C Generative AI Usage

  • Claude and GitHub Copilot assisted with editing and coding, while the authors reviewed and revised the output and conducted the study design, experiments, analysis, and conclusions.Source: C Generative AI Usage · p. 16

D Reporting Notes

  • Appendix F reports wall-clock times in seconds, minutes, or hours by duration, memory as peak resident set size in GB, and snapshot windows in their sweep-grid units.The reported measurements derive from the artifact and use the hardware specified in §3.Source: D Reporting Notes · p. 16

E Dataset Details

The evaluation uses four datasets selected to vary in deployment environment, telemetry, scale, and attack-labeling mechanism. The shared protocol defines malicious events and dataset splits using dataset-specific rules.

E Dataset Details

  • The four datasets vary in deployment environment, telemetry type, scale, and attack-labeling mechanism; Campus includes three controlled APT-style campaigns, and CI-2017 uses corrected labels.The corrected-label dataset is a variant of CIC-IDS-2017.Source: E Dataset Details · p. 16
  • The protocol defines an edge as a dataset-specific source-to-destination event and labels it malicious by red-team ground truth, matching event time and endpoints, corrected flow labels, or campaign membership.Source: E Dataset Details · p. 16
  • Under the GIDS-Lite chronological split, training is capped at 50,000 edges, and the train, validation, and test counts differ across datasets.The stated validation and test counts separate benign and attack edges; the passage gives the full counts for Campus and LANL, with CI-2017 and OpTC counts continued in the adjacent passage.Source: E Dataset Details · p. 16
  • Table 8 counts unique endpoints by each dataset’s native identifier: host names for Campus, LANL, and OpTC, and IP addresses for CI-2017.Source: Table 8 · p. 16

F.1 EG1: Snapshot-Window Sensitivity

The appendix compares snapshot-window choices across detector–dataset pairs and reports a separate sensitivity sweep for GIDS-Lite. The native default window is often not the best tested window.

F.1 EG1: Snapshot-Window Sensitivity

  • The default window is best in only four of sixteen detector–dataset pairs, and Pikachu’s CI-2017 one-hour default falls outside the tested 10–50-minute grid.Its default-window AP is therefore dashed, and the reported swing spans the tested windows.Source: §F.1 · p. 16
  • Table 9 defines window swing as the relative gap between the best and worst tested window for the same detector–dataset pair, computed as (max − min)/max.Source: Table 9 · p. 16
  • GIDS-Lite ranks first by AP on the two datasets where window choice barely matters, while its AP range shifts substantially on Campus and LANL.On LANL the one-hour window is best; the maxima on Campus, CI-2017, and OpTC occur at 450 s, 10 min, and 150 min, respectively.Source: §F.1 · p. 16

F.2 EG2: Threshold-Selection Instability

The appendix examines threshold-seed sensitivity, inductive versus transductive thresholding, and Argus edge features. Its findings emphasize protocol-sensitive operating points and the limited evidence for added edge-feature signal.

F.2 EG2: Threshold-Selection Instability

  • Table 10 measures native-pipeline wall-clock time and peak resident set size at four input scales, whereas Figure 9 prices shared-contract runs.Source: Table 10 · p. 17
  • Table 11 defines GIDS-Lite AP swing as (max AP − min AP)/max AP across the EG1 window grid, and CDR swing in percentage points.Source: Table 11 · p. 17
  • For Argus and Euler, threshold-seed variation can shift event detection, false alerts, and campaign coverage together, so instability is rarely confined to one metric.The other three systems do not draw a random validation sample for threshold calibration.Source: §F.2 · pp. 16–17, Table 12 · p. 17

F.3 EG3: Transductive Thresholding

  • Inductive-minus-transductive deltas are operating-point shifts, not only accuracy losses; unchanged TPR can accompany a large FPR increase.The LANL shift is also shaped by its attack-vector distribution, and an inductive split is fixed without knowing where attacks fall.Source: §F.3 · p. 17, Table 13 · p. 17

F.4 EG4: Edge Features in Argus

  • For Argus, the with/without edge-feature deltas provide the evidence that the feature path adds no reliable signal; cosine overlap alone is weak evidence because non-negative attributes bound it above zero.Table 16 reports absolute AP and CDR values under both edge-feature settings, while Figure 4 shows deltas.Source: §F.4 · p. 17

F.5 EG5: Cost and Scalability

The evaluation sweeps cost, evasion, component choices, and preprocessing to show that these factors can materially change GIDS comparisons. Across component configurations, lighter choices can remain competitive, while preprocessing changes can shift detection and campaign coverage differently.

F.5 EG5: Cost and Scalability

  • Scaling exponents range from 0.1 to 2.0 across detector–dataset pairs, so reduced-scale runs cannot be extrapolated with one global factor.The exponent measures wall-clock growth from 25% to 100% input scale; values above 1.5 indicate super-linear growth.Source: §F.5 · p. 17, AP AUC CDR · p. 18

F.6 EG6: Covering-Edge Evasion

  • When the detector flags attack edges, low-budget success recurs across reproductions and protocol extensions; with no flagged edges, it misses the campaign before an adversary acts.Table 19 reports candidate counts, attack success rates, and the smallest budget at which no flagged edge survives.Source: §F.6 · p. 18

F.7 EG7: Component Sweep

  • Removing graph or temporal encoders remains competitive in several datasets, supporting controlled component sweeps rather than assuming graph neural network blocks improve results.Table 14 lists each option by dataset and design dimension, ranked by mean AP across the EG7 configuration grid.Source: §F.7 · p. 18, Table 14 · p. 18

F.8 EG8: Native Versus Aligned Preprocessing

  • Preprocessing changes can improve AP while campaign coverage stays flat or moves the other way, altering both separability and the operational alert set.Table 15 compares native and aligned preprocessing using AP, AUC, and CDR; its Δ columns are aligned minus native scores in percentage points.Source: §F.8 · p. 18, Table 15 · p. 18

F.9 EG9: Streaming Readiness

Replay measurements show that throughput alone does not establish streaming readiness: detectors may keep up with ingest while their first alert is delayed by buffering or graph rebuilding.

F.9 EG9: Streaming Readiness

  • Many detector–dataset pairs keep up even at 5× native ingest, but snapshot buffering or static-graph rebuilding still delays the first alert.Table 18 records replay performance at 1×, 2×, and 5×, the snapshot window, and the dominant bottleneck.Source: §F.9 · pp. 18–19

G Comparison with Prior Studies

The comparison groups findings by how they relate to prior GIDS and provenance-based IDS studies, while emphasizing that their data models differ. It finds that some prior conclusions carry over to network flows, while others do not.

G Comparison with Prior Studies

  • PIDS graphs derive edges from kernel audit causality and use symbolic features, whereas GIDS builds graphs from continuous flow streams using chosen windows, preprocessing, and statistical edge attributes.The comparison notes that these data-model differences constrain which PIDS findings transfer to GIDS.Source: G Comparison with Prior Studies · p. 19
  • Sweeping snapshot windows uniformly across systems and datasets yields a mean 38.3% relative AP swing; imposing one shared preprocessing contract moves Euler on LANL from .035 to .423 AP.Source: G Comparison with Prior Studies · p. 20
  • On network flows, enabling edge features drops Argus on OpTC from 0.760 to 0.033 AP, while two inserted edges fully evade graph scoring on LANL.Six of eight measurable pairs fall within a budget of twenty inserted edges; the comparison attributes the difference from PIDS to the data model and attacker-observable traffic.Source: G Comparison with Prior Studies · p. 20, Table 19 · p. 20
  • Seed choice moves TPR by up to 70.4 percentage points and test-period calibration by up to 83.6 percentage points; GIDS-Lite ranks first by AP on two of four datasets.These results confirm prior observations about GIDS fragility and simple baselines under matched evaluation.Source: G Comparison with Prior Studies · p. 20
  • None of the 18 detector–dataset pairs is real-time ready, extending prior deployment-focused work to replay measurements for GIDS.The staged decomposition adapts a methodology used in PIDS, redefining it for network flows and extending it to per-component attribution.Source: G Comparison with Prior Studies · pp. 19–20, Table 18 · p. 20
Loading 2609.12263v1…