PaperlayerView exact revision on arXiv

Source-linked AI summary

HydroJEV: A one-second, training-free screen for cyber-attack and fault attribution in water distribution networks

Tianwei Mu, Shengyan Jiang, Mingzhe Yuan, Qing Luo, Min Xiao, Wenhong Wang, Jun Li, Manhong Huang

arXiv:2610.02048v1cs.AI

TL;DR

SCADA alarms require fast attribution, but labeled incidents are rare and frontier LLMs can be slow. This study evaluates a training-free typed-judgment screen and an agreement-gated reviewer cascade; Jev generalized to unseen subtypes, while gating reduced reviewer workload without macro-F1 loss.

  • Problem

    Operators must distinguish cyberattacks, physical faults, normal transients, and sensor faults, while labeled incidents are rare and frontier LLMs take tens of seconds or longer per decision.

    Source: §1 · p. 2

  • Method

    The study compared Jev with a rule tree, supervised classifier, and seven cloud LLMs on identical evidence in sealed, pre-registered rounds.

    Source: §1 · p. 3

  • Results

    Across four rounds, Jev exceeded supervised learning by 0.36-0.42 macro-F1 on unseen subtypes; agreement gating spared reviewers 35-38% of windows without reducing macro-F1.

    Source: §5 · p. 16

  • Takeaways & Limitations

    A tiered screen can accept benign verdicts only with independent agreement, while routing unsettled windows to rule-based or LLM review and retaining physical interlocks.

    Source: §5 · p. 16

    All windows were simulated, and one hydraulic model generated both the events and the twin, so field data would add model error.

    Source: §4.4 · p. 16

Abstract

from arXiv · show

When a SCADA alarm is raised in a water distribution network, operators must decide quickly whether it reflects a cyberattack, a physical fault, a normal transient or a faulty sensor. Supervised classifiers need labelled incidents that utilities rarely have, and frontier large language models (LLMs) take tens of seconds per decision. We tested whether Jev, a training-free model that returns class probabilities in about one second, can serve as the first tier of this triage. On a four-class cause-attribution benchmark built on the C-Town network in EPANET, Jev was compared with a hand-written rule tree, a supervised classifier and seven cloud LLMs on identical evidence in four sealed, pre-registered rounds. With only a label-free prior correction, Jev matched the rule tree (macro-F1 0.62-0.64 against 0.56-0.61 in distribution) and exceeded the supervised classifier by 0.36-0.42 on event subtypes absent from its labels, in all four rounds, and it outperformed the classifier whenever fewer than about four labelled events per class were available. Jev also decided 20-40 times faster than frontier LLMs. Accepting only benign Jev verdicts confirmed by the rule tree spared an LLM reviewer 35-38% of windows on fresh sealed sets without loss of macro-F1. Transferred unchanged to two further networks, this gated cascade stayed within the non-inferiority margin of its reviewer on all four sets. A fast, training-free screen can therefore take over about a third of the review load in SCADA anomaly triage while preserving the accuracy of deliberate review.

Graphical abstract (TOC graphic)

HydroJEV frames anomaly cause attribution as rapid triage and evaluates a training-free screen with a reviewer cascade. Its study design covers sealed evaluation and transfer to two further networks.

Graphical abstract (TOC graphic)

  • A one-second, training-free screen can spare an LLM reviewer about a third of calls without loss of accuracy, while retaining review for concealed attacks.The graphical abstract presents HydroJEV as a first-tier screen, not a final verdict.Source: Graphical abstract (TOC graphic) · p. 2

1 Introduction

  • HydroJEV tests whether typed judgments can generalize beyond supervised labels and whether agreement-gated review preserves accuracy while reducing workload.The study also asks what frontier LLM accuracy costs in time and characterizes difficult classes and network transfer.Source: §1 · p. 3

2.1 Benchmark

  • The benchmark defines four cause classes and uses three unseen event-subtype families for sealed evaluation, including Family C’s cross-class look-alikes.Events are generated in C-Town simulations, with a digital twin replaying reported actuator states to compare against SCADA records.Source: §2.1 · pp. 3–4

2.2 Evidence state

  • Each window is represented by checks, twin-residual z-scores, a signed summary, and a source-meter balance, with no class or subtype names in the evidence.Checks compare hydraulic measurements and actuator behavior with the twin, controls, and related channels.Source: §2.2 · p. 4

2.3 Jev screen

  • Jev returns probabilities over four causes plus insufficient evidence, then applies a label-free prior correction estimated from unlabelled development windows.No evaluated-window label enters the correction or calibrated decision.Source: §2.3 · p. 6

2.4 Baselines; 2.5 LLM reviewers

  • Baselines include a six-branch rule tree and logistic regression; seven cloud LLMs receive the same evidence and question as Jev.The classifier is evaluated with full labels, left-out subtypes, and varying label budgets.Source: §2.4 · p. 6, §2.5 · p. 6

2.6 Screening cascades

  • The cascades settle only benign Jev verdicts, with the LLM cascade requiring R1 agreement and sending other windows to a reviewer.The reviewer is glm-5.2 primarily and deepseek-v4-pro secondarily; failed Jev calls are always reviewed.Source: §2.6 · pp. 6–7

2.7 Sealed evaluation; 2.9 Timing; 2.10 Transfer networks

  • Round designs were frozen before sealed windows; the round-4 screen transferred unchanged to Net3 and Net1, with reviewer timing measured one request at a time.Transfer recomputed network properties using frozen code while retaining the C-Town question, evidence format, priors, and fitted models.Source: §2.7 · p. 7, §2.9 · p. 8, §2.10 · pp. 8–9

3 Results

Across C-Town sealed sets, Jev matched the rule tree in-distribution and exceeded a classifier retrained without the evaluated subtype. Its agreement-gated cascade was non-inferior to LLM review on fresh sets while settling 34.5–38.0% of windows.

3.1 Accuracy without training

  • Mean macro-F1 was 0.63 for calibrated Jev and 0.59 for rule tree R1 across the four in-distribution sets; their differences were not individually significant.On new-family sets, Jev and R1 were also statistically indistinguishable, at 0.51 and 0.50 mean macro-F1.Source: §3.1 · p. 9

3.2 Unseen event subtypes

  • On genuinely new families, fully supervised R2 remained more accurate than Jev, with mean macro-F1 of 0.69 versus 0.51.R2 nevertheless missed pump wear entirely on one new physical subtype, while Jev recalled 0.65 of those windows.Source: §3.2 · p. 9
  • +0.361 to +0.417 macro-F1: Jev exceeded R2 retrained without the evaluated subtype in all four rounds, with each primary comparison surviving Holm correction.Jev’s class-definition-based judgments retained recall for bursts and demand changes, while subtype-withheld R2 recalled none of those event types.Source: §3.2 · p. 9, Figure 2 · p. 10

3.3 Label requirements

  • R2 overtook Jev at four labelled events per class in-distribution and needed four to eight on new-family sets; the E2 exception required 40.With two labels per class, R2 remained below Jev in 64% of in-distribution draws and 82% of new-family draws.Source: §3.3 · p. 11

3.4 Comparison with LLMs

  • Four of 14 Holm-corrected LLM comparisons significantly favoured an LLM, while Jev decided 20–40 times faster than frontier models.On the two round-2 sets, glm-5.2 scored 0.706 and 0.697 and deepseek-v4-pro scored 0.722 and 0.637, against Jev’s 0.628 and 0.434; their per-decision times were tens of seconds versus about one second for Jev.Source: §3.4 · pp. 11–12, Figure 4 · p. 12

3.5 Screening cascades

  • On fresh round-4 sets, the LLM cascade was non-inferior to its reviewer and the agreement gate settled 34.5% and 38.0% of windows.Across C-Town sets, requiring rule-tree agreement raised accept precision from 0.75 to 0.83 and reduced accepted physical faults from 8.4 to 1.9 per set; on E4c, 21 of 76 accepted windows were attacks.Source: §3.5 · pp. 12–13, Figure 5 · p. 13

3.6 Screen components

  • The label-free prior raised Jev’s mean macro-F1 from 0.41 to 0.57; three physics checks did not improve Jev, but raised R1 to 0.67 and its Jev override to 0.68.On pump wear, the override assigned every window to the cyber class.Source: §3.6 · p. 14
  • Concealed manipulation had mean recall 0.34 for Jev and 0.32 for R1, versus 0.87 for supervised R2; in preliminary BATADAL evaluation, Jev’s veto cut false positives from 25.6% to 7.0%.The BATADAL veto result was at a true-positive rate of 81.2%.Source: §3.6 · p. 14

4 Discussion

Jev provides a fast first tier, with accuracy on unseen event subtypes and a gated cascade that reduces review workload. Transfer results support use on two further networks, while simulation-only evaluation and concealed-attack performance constrain deployment.

4.1 A fast first tier

  • Jev matched the expert-written rule tree, retained accuracy on unseen subtypes with a 0.36-0.42 macro-F1 margin, and outperformed supervised learning below about four labelled events per class.It decided 20-40 times faster than frontier LLMs; agreement with the rule tree on benign verdicts spared the reviewer about a third of windows.Source: §4.1 · p. 15

4.2 Rival explanations

  • R2's loss on unseen subtypes was subtype-specific, while Jev's screen contributed by deciding when not to call the reviewer; fresh-set non-inferiority supports that decision as safe on average.R2 remained strong in distribution (0.887-0.934), and its losses were complete on bursts and demand changes but absent on sensor offsets.Source: §4.2 · p. 15

4.3 Applicability to other networks

  • Applied without refitting to Net3 and Net1, the gated cascade stayed within its reviewer's non-inferiority margin on all four transfer sets.With the evaluated subtype removed from C-Town labels, Jev again exceeded the supervised model on both transfer networks.Source: §4.3 · p. 15

4.4 Limitations

  • Evaluation used simulated windows from one hydraulic model, concealed manipulation remained Jev's hardest class, and field validation is needed before settling windows on a utility's network.The authors recommend checking on simulated events that every class trips a check, and retaining physical interlocks; the benign gate can also be tightened.Source: §4.4 · p. 16

5 Conclusions

The study concludes that Jev is best suited to first-pass triage, especially when supervised labels omit event subtypes. Requiring an independent rule-tree agreement for benign verdicts reduced reviewer workload on fresh sealed sets, supporting a tiered workflow with further validation needed before deployment.

5 Conclusions

  • On unseen C-Town event subtypes, Jev exceeded the supervised classifier by 0.36-0.42 macro-F1 in all four rounds and outperformed it below about four labelled events per class.Its recall of concealed cyberattacks remained low, so the authors recommend passing suspected attacks onward rather than dismissing them.Source: §5 · p. 16
  • Requiring rule-tree agreement for benign Jev verdicts spared an LLM reviewer 35-38% of windows on fresh sealed sets with no loss of macro-F1.Accepting every benign Jev verdict did not improve on its components; agreement with an independent judge was essential.Source: §5 · p. 16
  • The recommended utility workflow accepts benign screen verdicts only with independent agreement and sends unsettled windows to rule-based or LLM review, retaining physical interlocks.Field SCADA validation and stronger treatment of concealed manipulation remain next steps before operational use.Source: §5 · p. 16

CRediT authorship contribution statement

The author-contribution statement assigns research, implementation, analysis, and supervisory roles across the team. The paper also identifies the public network assets and makes its library, scenarios, prompts, analyses, and source data available.

CRediT authorship contribution statement

  • Mu led conceptualization, methodology, software, investigation, analysis, visualization, and original drafting; other authors contributed data curation, validation, software, supervision, resources, and review.The contribution statement also assigns funding acquisition to Mu and project administration to Xiao.Source: CRediT authorship contribution statement · p. 17

Code and Data availability

  • The study uses public C-Town, Net3, and Net1 assets and provides the hydrojev library, scenario generators, prompts, design files, responses, scripts, results, and source data.The availability passage provides the project repository URL.Source: Code and Data availability · p. 17

Supplementary Information

The supplementary notes describe simulated benchmark generation and disjoint splits, the evidence state and fixed decision prompts, and the protocols and outcomes for baselines and LLMs.

Note S1. Benchmark generation and splits

  • C-Town scenarios used hourly EPANET simulations over 48-54 hours, with noisy demand and telemetry, random event onsets, and regenerated windows when an event was not physically expressed.The analyzed window ended 2-6 hours after onset.Source: Note S1. Benchmark generation and splits · p. 20
  • Splits used disjoint seed ladders, but round 1 had four window overlaps that the authors retained, creating a slight optimistic bias; round 3 also lacked one failed Jev call.Later split seed bases were at least 5,050,000, and 199 round-3 E1r3 windows were scored.Source: Note S1. Benchmark generation and splits · p. 20

Note S2. Evidence state, screen question and baselines

  • The evidence state encoded SCADA and control context, consistency checks, leading twin residuals, signed system summaries, and source-meter balance.Format 2 included ten checks spanning pumps, tanks, network quantities, control logic, and source meters, with thresholds based on event-free runs.Source: Note S2. Evidence state, screen… · p. 20
  • The fixed screen question barred class and mechanism labels from the state, treated the twin as a nominal model rather than a measurement, and asked for cause attribution using a specified reasoning order.The question was fixed using round-1 development windows and used unchanged; Jev was instructed to give each cause a non-zero probability.Source: Note S2. Evidence state, screen… · pp. 20–21
  • The four cause definitions distinguish concealed manipulation, a real but unmodeled asset fault, coherent honest operation changes, and a single misreporting instrument.Insufficient evidence was reserved for missing or genuinely contradictory evidence that did not favor another explanation.Source: Note S2. Evidence state, screen… · p. 21
  • R1 was a frozen ordered rule tree, while R2 was a standardized multinomial logistic regression; R2's transfer and target-network variants differed in whether target-network labels were used.R1b added a physics override based on three format-3 checks.Source: Note S2. Evidence state, screen… · p. 21

Note S3. LLM protocol

  • LLMs received the same state and screen question as Jev, used fixed sampling and constrained JSON outputs, and were calibrated using their own mean answers on unlabelled development windows.Timed-out calls were retried only after absence was confirmed; answers were not resampled for content.Source: Note S3. LLM protocol · pp. 21–22
  • On the two round-2 sealed sets, deepseek-v4-pro and glm-5.2 outscored Jev on macro-F1, with four of 14 primary tests significant after Holm correction, all favoring an LLM.The significant comparisons were deepseek-v4-pro on both sets, glm-5.2 on E2r2, and gpt-oss:120b on E2r2.Source: Note S3. LLM protocol · p. 22

Note S4. Prior calibration

These analyses show that a label-free prior correction improves Jev’s macro-F1, while performance varies by event subtype and network. Additional checks show trade-offs for physics overrides and benefits of screening when labels are scarce or alarms are benign.

Note S4. Prior calibration

  • Frozen prior correction raised macro-F1 on every set, from 0.447 to 0.620 on E1 and 0.448 to 0.577 on E3.The transductive prior changed macro-F1 by −0.061 to +0.056 relative to frozen calibration, with no consistent gain.Source: Note S4. Prior calibration · p. 22

Note S5. Recall by subtype

  • For unseen new families, pipe closure was hardest for all methods (Jev 0.24, R1 0.14, R2 full 0.16); pump-wear recall was 0.65 for Jev, 0.83 for R1, and zero for R2 full.On in-distribution sets, Jev recalled 0.74–0.82 of bursts and 0.88–0.95 of demand rises, but only 0.08–0.25 of masked overpump attacks.Source: Note S5. Recall by subtype · p. 22

Note S6. Evidence format 3 and physics override

  • Format 3 lowered Jev’s macro-F1 slightly on both round-3 sets: 0.628 against 0.643 on E1r3 and 0.577 against 0.613 on E3.The physics override raised pump-off-attack recall from 0.34 to 0.97 on E1r3, but pump-wear recall fell from 0.65 to 0.00 on E3.Source: Note S6. Evidence format 3 and physics… · p. 23

Note S7. Label scarcity

  • On seven of eight C-Town sets, R2’s label-scarcity crossover was k*=4; at k=2, its mean macro-F1 remained below Jev’s.On the round-1 new-family set E2, R2 matched Jev’s 0.556 only at k=40; transfer-network crossovers ranged from 1 to 8 labelled windows per class.Source: Note S7. Label scarcity · p. 23

Note S8. Benign-alarm veto on BATADAL

  • On 107 flagged BATADAL windows, Jev reached ROC AUC 0.884 and the Jev-confirmed R1 veto reduced R1’s false-positive rate from 25.6% to 7.0% at 81.2% true-positive rate.Jev was significantly better than six of seven prior detectors, statistically indistinguishable from Mahalanobis, and worse than R2 trained on development data.Source: Note S8. Benign-alarm veto on BATADAL · p. 23

Note S9. Time to decision

Jev’s decisions took about a second, substantially less time than frontier LLM decisions. The tested agreement-gated cascades reduced reviewer time while retaining reviewer-level macro-F1 on several sealed sets, though transfer results varied by network and set.

Note S9. Time to decision

  • Jev’s median client wall time was 1.463 s, compared with 30.6–46.4 s for frontier LLMs sent one request at a time, making them 20–40 times slower.R1 and R2 decide in well under 1 ms on a single CPU core.Source: Note S9. Time to decision · p. 24

Note S10. Cascades and round-4 results

  • On round-4 sets E1c and E4c, the glm-5.2 cascade scored 0.745 and 0.565 against 0.735 and 0.549 for glm-5.2 alone.The agreement gate accepted 69 and 76 windows; with glm-5.2, mean decision time fell from 42.7 to 31.3 s on E1c and 44.2 to 30.0 s on E4c. Neither Jev nor either cascade reached R2 full, which scored 0.887 and 0.726.Source: Note S10. Cascades and round-4 results · pp. 24–25

Note S11. Transfer to EPANET Net3 and Net1 (rounds 5 and 6)

  • The glm-5.2 cascade met the non-inferiority criterion on N3-Ref and both Net1 sets; on N3-C, its interval was inside the margin but the corrected test was not met.On Net3, R2 transfer was more accurate than Jev on both sets; on Net1, Jev was superior to R2 transfer on N1-C but not N1-Ref.Source: Note S11. Transfer to EPANET Net3 and… · pp. 25–26
  • On Net1, event classes were not uniformly recognized: Jev recalled only 0.02 of bursts, and the paper recommends confirming that every event class trips a check before letting the gate settle windows.The rule cascade scored below R1 on both Net1 sets, and R2 refitted on Net1 labels overtook Jev with one or four labels per class, depending on the set.Source: Note S11. Transfer to EPANET Net3 and… · p. 26

Note S12. Secondary comparisons

Secondary comparisons report differences between Jev, cascades, and baselines across later rounds, while distinguishing uncorrected comparisons from pre-registered primary tests. The tables also specify how paired macro-F1 differences and hypothesis tests are reported.

Note S12. Secondary comparisons

  • Across sets where defined, Jev beat R2 LOSO by +0.417 on E1c, +0.446 on N3-Ref, and +0.138 on N1-Ref; against R2 full, Jev was lower by 0.243 and 0.237 on E1c and E4c.The comparisons with R2 own network are references rather than claims because that model needs labelled data from the evaluated network.Source: Note S12. Secondary comparisons · p. 26

Tables

  • Table S3 lists the pre-registered primary tests; ∆ is the paired macro-F1 difference, with a class-stratified bootstrap 95% interval and Holm correction within each round.The secondary, uncorrected round-4–6 comparisons are collected separately in Fig. S13.Source: Note S12. Secondary comparisons · p. 26, Table S3 · p. 28

1 E1-LOSO Jev > R2 LOSO +0.361 [0.189, 0.523] <0.0001 0.0125 yes

  • The E1-LOSO comparison reports Jev superior to R2 LOSO, with ∆ +0.387 and 95% CI [0.318, 0.455].The displayed comparison has p<0.0001 and Holm threshold 0.0167.Source: §2 · p. 28

2 E1r2-LOSO

The supplied comparison rows report Jev’s comparisons with R2 LOSO and two LLMs across E1r2 and E2r2. The available entries show positive separation from R2 LOSO, while the listed comparisons with gemma4 do not meet their reported tests.

2 E1r2-LOSO

  • On E1r2, Jev’s comparison with gemma4 was ∆−0.013 [−0.087, 0.061] (p=0.7170), reported as not met.Source: §2 · p. 28
  • On E2r2, Jev’s comparison with gemma4 was ∆−0.036 [−0.102, 0.031] (p=0.2898), reported as not met.Source: §2 · p. 29
  • On E1r2, Jev exceeded R2 LOSO by +0.390, with 95% CI [0.321, 0.458] and p<0.0001.Source: §3 · p. 29

3 E1r3-LOSO

The E1c result rows report Jev outperforming R2 LOSO, the rule cascade outperforming both R1 and Jev, and the glm-5.2 cascade meeting non-inferiority.

3 E1r3-LOSO

  • The rule cascade outperformed R1 and Jev on E1c.Source: §3 · p. 29
  • The glm-5.2 LLM cascade met non-inferiority on E1c.Source: §3 · p. 29
  • Jev > R2 LOSO by +0.417, with a 95% CI of [0.344, 0.488].The reported p-value is <0.0001, with Holm-adjusted p = 0.0100.Source: §4 · p. 29

4 E1c-LOSO

The supplied rows report rule-cascade comparisons on E4c, Jev comparisons with transferred R2 across Net3 and Net1 sets, and LLM-cascade non-inferiority results.

4 E1c-LOSO

  • On E4c, the rule cascade outperformed R1.Source: §4 · p. 29
  • On E4c, the rule cascade outperformed Jev by +0.016, with a 95% CI of [0.003, 0.033].The reported p-value is <0.0001, with Holm-adjusted p = 0.0071.Source: §4 · p. 29
  • The glm-5.2 LLM cascade met non-inferiority on E4c, N3-Ref, N3-C, N1-Ref and N1-C.Source: §4 · pp. 29–30
  • On both N3-Ref and N3-C, Jev outperformed R2 transfer.Source: §4 · p. 29
  • On N1-Ref, Jev outperformed R2 transfer.Source: §4 · p. 29
  • On N1-C, Jev exceeded R2 transfer by +0.150, with a 95% CI of [0.086, 0.213].The reported p-value is <0.0001, with Holm-adjusted p = 0.0125.Source: §4 · p. 30

Figures

The supplementary figures map the C-Town benchmark, subtype recalls and confusion matrices, then present transfer-network evaluations and secondary comparisons. Captions describe what each visual encodes but do not report comparative outcomes.

C-Town and early rounds

Round 4

Transfer networks

Transfer analyses

Related papers

Loading 2610.02048v1…