PaperlayerView exact revision on arXiv

Source-linked AI summary

Jev-IDS: System One Models for Network Intrusion Detection

Paulo Severo, Silvio E. Quincozes, Amanda Dias

arXiv:2610.01079v1cs.CRcs.AI

TL;DR

NIDSs need to detect attacks despite scarce representative labels, while balancing false alarms and inference overhead. JEV-IDS uses JEV's typed attack-probability and category questions for flow detection without task-specific fine-tuning. The reported findings position JEV as an efficiency–performance trade-off against a generative LLM, with novel-attack results bounded to withheld attack types in a benchmark.

  • Problem

    NIDS detection must contend with limited representative labels, while missed attacks, false alarms, and inference overhead affect operational usefulness.

    Source: §1 · p. 2

  • Method

    JEV-IDS serializes flows for JEV, which answers typed attack-probability and traffic-category questions without task-specific fine-tuning.

    Source: §1 · p. 2, §4 · p. 4, §4.3 · p. 5

  • Results

    JEV provides a distinct predictive-performance and inference-efficiency trade-off: Gemini has higher F1 across evaluated example budgets, while JEV has comparable precision with lower latency and estimated cost.

    Source: §7 · p. 10

  • Takeaways & Limitations

    JEV's observed efficiency may suit repeated flow invocations, although hosted model approaches remain considerably slower than local machine-learning inference.

    Source: §7 · p. 10

    Novel-attack recall reflects generalization across withheld attack types in this benchmark, not direct evidence of zero-day detection in operational networks.

    Source: §7 · p. 11

Abstract

from arXiv · show

Machine-learning Network Intrusion Detection Systems (IDS) depend on substantial labeled datasets and task-specific training, whereas Large Language Models (LLMs) detection can analyze flow records directly but incurs higher inference cost and latency, with less constrained outputs. This paper presents JEV-IDS, an open experimental general NIDS based on the Jev System One Model (SOM) to detect zero day intrusions Under label scarcity. JEV-IDS serializes one flow per request and asks JEV two questions: a binary attack probability and a finite-choice traffic category. Our results show that, at k=1, JEV was 4.8 times faster and 3.8 times cheaper than GPT-5.6 Luna, with 1.5 times higher novel-attack recall; it also produced 15 times fewer false alarms than a low-data Random Forest. Across 5,400 decisions on a 300-flow NSL-KDD pilot split, JEV achieved F1-Score 0.859, precision 0.941, recall 0.790, and novel-attack recall 0.838. Increasing k to 2 reduced its F1-Score to 0.839.

1. Introduction

NIDSs must balance missed attacks, false alarms, and inference overhead, while conventional detectors depend on representative labeled data. JEV-IDS tests whether a System One Model can detect flows under label scarcity and for attack categories absent from the example pool.

1. Introduction

  • Conventional supervised detectors depend on labeled data that represent encountered traffic, a particular challenge when attacks are rare or newly observed.Prompted LLMs can classify text-encoded flows without updating their parameters, but may incur high inference overhead and require output control.Source: §1 · p. 2
  • SOM performance for distinguishing benign and malicious flows, benefits from labeled examples, and operational advantages in controlled NIDS comparisons remain unknown.The passage describes JEV as a model for typed probabilistic decisions, with existing performance claims concerning general automation workflows rather than NIDS.Source: §1 · p. 2
  • JEV-IDS is introduced to test whether typed probabilistic and finite-choice questions can support flow-based intrusion detection without task-specific training.The comparison uses the same flow attributes, labeled examples, and binary decision threshold for JEV, a structured-output GPT-5.6 Luna detector, and a low-data Random Forest.Source: §1 · p. 2
  • The evaluation protocol varies labeled-example counts and assesses simulated zero-day scenarios, comparing three detectors on two intrusion datasets with one or two examples per category.It jointly measures detection, labeled-data use, latency, and cost, and uses frozen splits to distinguish known attacks from attacks absent from the pool.Source: §1 · p. 2
  • The paper releases its implementation, prompts, frozen splits, and per-flow pilot predictions to support reconstruction of the reported comparisons.The introduction describes a 5,400-decision pilot and evaluation across three seeds.Source: §1 · p. 3

2. Background

The background frames flow classification under limited supervision, contrasts prompting and autoregressive generation with typed decisions, and notes that returned probabilities still require calibration assessment.

2.1. Flow classification with limited supervision

  • A flow-based detector learns from traffic attributes, while its effectiveness depends on whether training data represent deployment traffic.With in-context inference, labeled examples accompany requests rather than updating model parameters; withholding an attack type measures recognition of an unseen attack.Source: §2.1 · p. 3
  • This work evaluates few-shot classification for represented attack types and novel-attack detection when attack types are absent from the example pool.Source: §2.1 · p. 3

2.2. Language models as decision components

  • Autoregressive LLMs generate tokens sequentially, so structured-output constraints control response format but do not remove token-generation overhead.Sequential decoding adds time and computational demand; per-token API charges also accumulate across processed flows.Source: §2.2 · p. 3

2.3. System One models and typed decisions

  • SOMs return bounded probabilistic judgments through typed questions, allowing applications to combine answers through ordinary program logic.For intrusion detection, the relevant outputs are a binary maliciousness judgment and a traffic-category choice; mapping these to alerts remains an application-level decision.Source: §2.3 · p. 3
  • Typed probabilities are not automatically calibrated, so calibration must be assessed for each model and task.Constrained answer spaces simplify integration, but detection accuracy and probability reliability still require empirical evaluation.Source: §2.3 · pp. 3–4

3. Related work

Related work spans prompted traffic classification, constrained generation, and systems that use LLMs outside per-flow decisions. JEV-IDS positions its contribution as a joint, matched evaluation of a typed decision model, structured-output LLM, and low-data Random Forest.

3.1. Language models as traffic classifiers

  • Prior prompted IDS studies moved task adaptation into instructions, demonstrations, or retrieved context, while verdicts remained part of generated responses.Later approaches added retrieval or grammar-constrained JSON, but a constrained generated score differs from an interface whose admissible answers are specified before inference.Source: §3.1 · p. 4
  • Related studies also examine throughput, resource use, few-shot traffic classification, transfer, and predictive-performance versus computational-cost trade-offs.One reported measurement range is approximately 0.8–4.0 seconds per sample, depending on the LLM; eliminating training does not eliminate per-record inference overhead.Source: §3.1 · p. 4

3.2. LLMs as NIDS support components

  • Hybrid NIDS systems use LLMs for threat interpretation or bounded adaptation decisions rather than classifying every flow.Their operational measurements characterize alert interpretation or occasional adaptation, not the cost of per-flow classification by a general model.Source: §3.2 · p. 4

3.3. Positioning of JEV-IDS

  • The authors state that, to their knowledge, no NIDS study evaluates a SOM as the direct flow classifier.JEV-IDS compares typed decisions, structured-output LLMs, and a low-data Random Forest while matching flow representation and labeled examples within each experimental cell.Source: §3.3 · p. 4

4. Jev-IDS architecture

JEV-IDS processes one flow per request, supplies JEV with an ordered flow representation and optional labeled examples, and derives detection outputs from two typed questions. Its intended placement is an asynchronous side channel alongside existing network monitoring.

4. Jev-IDS architecture

  • The pipeline constructs an ordered flow representation, builds a request with optional examples, evaluates two typed questions, and extracts detection outputs.One target flow is processed per JEV request.Source: §4 · p. 4, Figure 1 · p. 5

4.1. Flow representation and Dataset Card

  • A Dataset Card defines ordered features and categories, and target-flow feature values are serialized in Card order regardless of source CSV column order.The target flow's category and attack status are excluded; concrete attack names are also not part of the supplied representation.Source: §4.1 · pp. 4–5

4.2. Jev request construction

  • The request template supplies task instructions, ordered column names, category descriptions, and two typed questions, with the target flow and optional labeled examples added at request time.Examples expose a reference flow's category, not its concrete attack name.Source: §4.2 · p. 5

4.3. Typed Jev decisions

  • For each target flow, JEV answers a binary intrusion-probability question and a choice question over the dataset's categories.The category answer supplies the predicted category and its associated confidence.Source: §4.3 · p. 5

4.4. Detection output and local verdict

  • JEV-IDS maps typed answers to attack probability, predicted category, and confidence, then computes the binary verdict locally from the probability.The resulting output distinguishes JEV's probability and category decision from the locally derived verdict.Source: §4.4 · p. 5

4.5. Request execution and failure handling

  • Transient network failures, rate limits, and selected server errors trigger exponential backoff for up to five total attempts; exhausted requests return an error.The error behavior separates communication failures from successful JEV responses.Source: §4.5 · p. 6

4.6. Operational placement

  • The intended deployment sends flow copies asynchronously beside existing monitoring, rather than placing JEV-IDS in the packet-processing path.When using the hosted service, selected flow features are transmitted remotely while packet payloads are not included in the request.Source: §4.6 · p. 6

5. Experimental methodology

The methodology controls labeled-example availability and sampling while comparing detectors on the same evaluation flows, with separate analyses for unseen attack types and operational trade-offs.

5.1. Experimental design

  • Across finite-k conditions, three independently sampled example sets are nested across k and shared across detectors, with evaluation flows kept separate from example selection and training.The protocol also includes zero-shot and fully supervised reference conditions where applicable.Source: §5.1 · p. 6, §5.3 · p. 6

5.2. Comparison methods

  • JEV and Gemini receive labeled examples in prompts, while Random Forest is trained on the same examples and Isolation Forest uses only benign training flows.The Random Forest also has a full-training-pool condition; Gemini produces a structured output including attack probability, binary classification, and traffic category.Source: §5.2 · p. 6

5.3. Dataset and evaluation protocol

  • NSL-KDD is used as a controlled historical benchmark, with training examples drawn from KDDTrain+ and evaluation flows from KDDTest+; 300 test attacks are novel by attack type.The paper treats the dataset as a methodological benchmark rather than a proxy for current operational networks.Source: §5.3 · pp. 6–7

5.4. Evaluation metrics

  • Attack-class F1 is the primary detection metric, complemented by precision, recall, AUCs, category-specific recall, and separate known- and novel-attack recall.Finite-k metrics are computed per seed and summarized across three seeds; latency and estimated hosted-model cost are also measured.Source: §5.4 · p. 7

5.5. Statistical analysis

  • Paired detector comparisons use exact two-sided McNemar tests on discordant predictions, while comparisons across supervision budgets are identified separately.This tests whether paired detectors’ error patterns differ on the same observations.Source: §5.5 · p. 7

5.6. Reproducibility

  • Fixed evaluation sets, example selections, configurations, prompts, and code revision are recorded, and prediction-level outputs are retained for reproducibility.The implementation and experimental artifacts are publicly available.Source: §5.6 · p. 7

6. Results

On NSL-KDD, Gemini had higher overall F1 than JEV across evaluated example budgets, while JEV had higher aggregate novel-attack recall with examples and lower hosted latency and estimated cost.

6.1. Overall detection performance

  • At k=1, Random Forest detected all attacks but had 0.598 precision and approximately 764 false alarms; full-pool training yielded 0.263 novel-attack recall.At k=8, its F1 was 0.865; the Isolation Forest reached 0.974 precision and 0.624 recall.Source: §6.1 · pp. 7–8
  • At k=1, Gemini’s F1 was 0.880 versus JEV’s 0.856, and Gemini had higher overall F1 than JEV at every evaluated k.Table 2 reports predictive results on the 2,000-flow evaluation set, with F1 summarized across three example seeds.Source: §6.1 · p. 7, Table 2 · p. 8
  • At k=1, JEV PR-AUC was 0.952 and ROC-AUC 0.950, versus Gemini’s 0.948 and 0.959; probability calibration was not evaluated.The fully supervised Random Forest had PR-AUC 0.963 and ROC-AUC 0.964 despite F1 of 0.765 at the common threshold.Source: §6.1 · p. 8

6.2. Generalization to novel attacks

  • With examples, JEV’s novel-attack recall exceeded Gemini’s at each evaluated k, but the aggregate difference masked differing patterns across attack families.JEV’s advantage was largest for novel R2L attacks, while Gemini remained stronger for several other types, notably probe traffic.Source: §6.2 · p. 8

6.3. Paired statistical comparison

  • Gemini was favored on paired overall comparisons at every k (p<0.001), while JEV was favored on novel-attack discordances at k=1 and k=2, but not significantly at k=4 or k=8.At k=1, JEV was correct on 74 versus 44 discordant novel-attack decisions (p=0.007); at k=2, the counts were 105 versus 57 (p<0.001).Source: §6.3 · pp. 8–9

6.4. Operational efficiency

  • At k=1, JEV latency was 7.7× lower and estimated list-price cost 22× lower than Gemini, alongside a 0.024 F1 reduction.Latency includes network and provider overhead; cost is a list-price estimate under the experimental conditions and pricing at measurement time.Source: §6.4 · p. 9, Table 4 · p. 10

7. Discussion

JEV offers a distinct efficiency–performance trade-off: Gemini has higher overall F1, while JEV has comparable precision and lower latency and estimated cost. Novel-attack results and other findings are limited to the benchmark setting and its definition of novelty.

7. Discussion

  • Gemini achieves higher overall F1 across labeled-example budgets, while JEV maintains comparable precision with substantially lower inference latency and estimated cost.The discussion frames this as a trade-off, not consistent SOM superiority in detection accuracy.Source: §7 · p. 10
  • JEV consistently has higher novel-attack recall than Gemini after labeled examples are introduced, with the strongest statistical evidence at the smallest example budgets.The paper cautions that this does not establish general superiority for detecting unseen attacks.Source: §7 · p. 10
  • JEV’s improvement over zero-shot inference largely occurs with few examples, and the value of additional examples depends more on their information content than their quantity in this setting.The category-level difference is substantially associated with R2L traffic, while Gemini is stronger for other attack families.Source: §7 · p. 10
  • Under very limited supervision, Random Forest reaches high attack recall by classifying much benign traffic as malicious, illustrating why similar F1 scores can carry different operational consequences.The discussion notes that false alarms burden downstream analysis.Source: §7 · p. 10
  • The fully supervised Random Forest has strong PR-AUC and ROC-AUC despite lower F1 at the common threshold, so detector differences partly reflect operating-point choice.The common threshold supports controlled comparison but is not an optimized deployment configuration for every detector.Source: §7 · p. 10
  • Across labeled-example budgets, JEV has lower latency and estimated cost than Gemini, but both hosted approaches are considerably slower than local machine-learning inference.The authors favor roles such as alert enrichment, secondary analysis, or selective escalation rather than claiming line-rate suitability.Source: §7 · p. 10
  • A cascade combining JEV with a more expensive generative model or human analyst is proposed as future work, not an evaluated benefit of JEV-IDS.The proposed design would route straightforward cases to a lower-cost component and escalate uncertain cases.Source: §7 · p. 10
  • The findings concern generalization across withheld attack types in NSL-KDD, not direct evidence of operational zero-day detection; broader datasets and analyses accounting for repeated flows and few-shot variability remain future work.The paper also calls for temporal or organizational separation, calibration, independent threshold selection, and stronger tabular baselines.Source: §7 · p. 11

8. Conclusion

The conclusion presents JEV-IDS as a flow-based NIDS pipeline using JEV without task-specific training. In the NSL-KDD pilot, it reports strong results with few labels, while identifying broader evaluation as future work.

8. Conclusion

  • JEV-IDS combines an ordered flow representation, optional category-labeled examples, typed attack and category questions, and structured outputs without task-specific model training.The conclusion describes the pipeline as providing traceable inference records.Source: §8 · p. 11
  • In the NSL-KDD pilot, JEV achieved strong results with few labeled examples, fewer false alarms than Random Forest, better novel-attack detection than GPT-5.6 Luna, and favorable latency and estimated cost at the smallest example budget.These results apply to the considered settings.Source: §8 · p. 11
  • Future work will evaluate additional k values and datasets and strengthen analysis of seed and repetition dependence, calibration, novel-attack robustness, and realistic network conditions.Source: §8 · p. 11

Data and code availability

The paper makes its experiment materials public while directing readers to obtain NSL-KDD from its original distribution points under their terms.

Data and code availability

Related papers

Loading 2610.01079v1…