Source-linked AI summary
A Resilient Runtime-Verification Fabric for Security Monitoring of Critical Edge-IoT Infrastructure
Nikolaos Kekatos, Marinelio Chintri, Panagiotis Katsaros, Alexios Lekidis, Tom Nianios, Ioannis Seitoglou, Anastasios Temperekidis, Stylianos Basagiannis
TL;DR
Shared-log runtime-verification pipelines can silently lose or misorder evidence, undermining monitoring of critical IoT fleets. RV-Fabric adds broker-based delivery mechanisms and carries evidence completeness into verdict semantics. Against a fault-free oracle using real MonPoly, it preserves all seven injected incidents, compared with one for the shared-log baseline.
Problem
Shared-log runtime-verification pipelines lack resilience to event loss, clock skew, network silence, and slow consumers, leaving monitors to issue verdicts from corrupted evidence.
Source: §1 · p. 2Method
Results
The shared-log baseline preserves one of seven injected incidents, whereas the full fabric preserves all seven against a fault-free oracle through the real MonPoly engine.
Source: §11 · p. 19Takeaways & Limitations
The transport-verdict link makes evidence completeness part of runtime-verification semantics, so an incomplete stream cannot yield an unqualified all-clear.
Source: §11 · pp. 18–19A compromised gateway that emits plausible heartbeats while suppressing evidence is out of scope because its sidecar and outbox are trusted.
Source: §10 · p. 18
Abstract
from arXiv · showhide
Protecting critical infrastructure increasingly depends on continuously verifying large IoT fleets against formal security specifications at runtime. Yet the runtime-verification (RV) pipelines proposed for this task are typically single-host prototypes whose monitors read a shared log file, with no resilience to the failures such deployments incur: a crash or overload silently drops events, clock skew corrupts the ordering metric monitors require, a time-triggered "node has gone silent" property cannot fire when the network itself falls silent, and one slow consumer stalls the pipeline. Each failure is silent: the monitor keeps emitting verdicts over a corrupted view. We present RV-Fabric, a resilient delivery layer that carries the hierarchy over two brokers (MQTT for device ingest, a durable stream broker for backend delivery) and re-establishes five continuity guarantees: durable delivery under crashes, a trusted event order, progress under total silence, consumer isolation and flow control under bounded overload, each an invariant conditioned on broker durability. Above the transport, RV-Fabric makes evidence completeness part of runtime-verification semantics: every verdict carries a status (sound, degraded, incomplete or unavailable) derived from delivery gaps, retention pressure and liveness, so an incomplete stream cannot yield an unqualified all-clear. Under controlled fault injection on a containerised testbed, measured against a fault-free oracle using the real MonPoly engine, the shared-log baseline misses six of seven injected incidents, reporting each as an unqualified all-clear, whereas RV-Fabric preserves all seven; removing a delivery mechanism reintroduces silent loss, removing isolation costs only timeliness. Two published critical-infrastructure datasets, water-SCADA and IoT/IIoT, replay end-to-end.
1 Introduction
RV-Fabric addresses silent corruption in shared-log runtime-verification pipelines by adding resilient delivery and making evidence completeness part of verdicts. In the fault-injection evaluation, the shared-log baseline preserves one of seven incidents, while the full fabric preserves all seven.
1 Introduction
- Shared-log prototypes can silently lose or misorder events, break silence-triggered monitoring, and let a slow consumer stall the pipeline.Monitors may continue issuing verdicts from a corrupted fleet view, so evidence completeness must travel with the verdict.Source: §1 · p. 2
- RV-Fabric carries the existing three-layer monitoring hierarchy across the edge-cloud continuum while restoring delivery, ordering, liveness, isolation, and flow-control guarantees.The paper frames these guarantees as broker-contingent invariants.Source: §1 · p. 2
- Each verdict carries a continuity status so delivery, ordering, retention, or liveness problems cannot produce an unqualified all-clear.The paper reports that removing a delivery mechanism reintroduces a specific silent failure; removing the tick alone yields two false all-clears.Source: §1 · p. 3
- The shared-log baseline preserves one of seven injected incidents, whereas the full fabric preserves all seven against a fault-free oracle through the real MonPoly engine.The baseline misreports the other six as unqualified all-clears.Source: §1 · p. 3
2 Background: The Hierarchy and the Shared-Log Prototype
RV-Fabric carries forward a hierarchical monitoring workload and replaces its single-host shared-log prototype. The background describes the hierarchy’s tiered responsibilities and the prototype’s delivery gaps.
The three-layer hierarchy
- The inherited hierarchy places per-device checks at L1, gateway aggregation at L2, and fleet-wide temporal-logic monitoring at L3.L3 can relate events across gateways on one ordered timeline, including coordinated attacks and silent-node failures.Source: §2 · p. 4
The shared-log prototype and its gaps
- The prototype runs all three layers on one host over a shared log, exposing loss, clock skew, silence-liveness, and lack of flow control.Its reported ∼1% ingest loss was not reproduced in a controlled same-host replay; the cited exposure is loss under crash or overload, broken ordering under skew, and silence-liveness.Source: §2 · p. 4
- Prior work names an event bus as a fix, but the paper argues that the monitor must also know when the bus has failed.Fig. 2 presents the resulting architecture.Source: §2 · p. 4
3 The RV-Fabric Architecture
RV-Fabric turns the hierarchy’s input assumptions into transport requirements and distributes monitoring over two brokers. Its architecture adds independent ordering, durability, liveness, and consumer-progress mechanisms, alongside fabric-enabled fleet properties.
From RV assumptions to transport requirements
- RV-Fabric translates sound monitoring into five transport requirements: complete delivery, ordered input, progress under silence, independent consumer reads, and loss avoidance under load.The paper treats each guarantee as conditional on broker durability.Source: §3 · p. 4
The RV-Fabric architecture
- Devices publish at QoS 1 to gateway-partitioned MQTT ingest; gateways run L1/L2 and republish verdicts to backend NATS JetStream for independent L3 durable cursors.The prototype routes gateway partitions with topic prefixes on one shared broker, so it lacks broker-level fault isolation.Source: Figure 2 · p. 5
- MQTT QoS 1 provides at-least-once device-to-gateway delivery with PUBACKs, while idempotent L1 processing makes duplicates harmless.JetStream durable cursors let L3 monitors read at their own pace; a dedicated 1 Hz tick advances time-triggered monitors, and subjects preserve per-device FIFO order.Source: §3 · p. 5
- A gateway sidecar stamps events with trusted, non-decreasing ingest time and orders by it; device-reported time is retained only for spoof detection.Original device event time is not reconstructed.Source: §3 · p. 5
- A persistent event ID, fsynced write-ahead outbox, broker acknowledgement, and confirmation cursor let a restarted relay replay only its unacknowledged suffix.Backend deduplication gives exactly-once monitoring effect over at-least-once transport; a narrow in-memory window remains before the local WAL fsync.Source: §3 · pp. 5–6
Properties the fabric carries
- The carried catalogue includes cross-device properties and adds P3.6 for overflow across at least two gateways within 30 seconds and P3.7 for gateway silence.A backend-resident tick makes P3.7 independent of any gateway staying up; a per-gateway tick would freeze when that gateway goes silent.Source: §3 · p. 6
4 Threat Model and Trust Assumptions
The threat model protects the monitoring verdict stream against attempts to corrupt, blind, or exhaust the evidence path, while explicitly trusting the backend, brokers, and gateway-side components. It distinguishes detectable link or gateway silence from malicious behavior by a compromised gateway, which is out of scope.
Asset and adversary goal
- The protected asset is the verdict stream: accepted security-relevant events must reach the correlator in trusted gateway-ingest order, and absence of events must be observable.IoT payloads are out of scope; the adversary seeks to hide attacks by dropping, reordering, replaying, silencing, or exhausting the evidence path.Source: §4 · p. 6
Adversary treatment and scope
- The threat-model table maps adversary capabilities to mitigations, continuity mechanisms or monitored properties, and residual risks.Entries marked † lie outside the trust boundary and are assumed rather than enforced.Source: Table 1 · p. 7
Trusted computing base
- The trusted computing base includes the cloud backend and L3 monitors, brokers under configured durability, and each gateway’s timestamping sidecar and durable outbox.Devices and network links are untrusted; TLS and authentication are assumed to prevent forging a broker-authenticated identity.Source: §4 · p. 7
Adversary treatment and scope
- Silencing a link cannot hide the silence from P3.5/P3.7, though detection may be delayed during a prolonged partition.Malicious behavior that conforms to the specification remains a residual risk; an actively compromised gateway is out of scope because its sidecar and outbox are trusted.Source: §4 · pp. 7–8
5 Continuity Mechanisms and Their Invariants
RV-Fabric states five broker-contingent continuity guarantees and couples tracked evidence completeness to verdicts. Its soundness claims depend on observable loss or misordering, durable acceptance, and—in cross-gateway cases—bounded clock skew.
Capacity boundary
- Flow control queues transient overload, but sustained overload beyond retention capacity can exhaust it and prevents G1 from preserving delivery.The resulting exhaustion is observable, but delivery remains conditional on capacity.Source: §5 · p. 8
Guarantees as invariants
- The five guarantees are at-least-once delivery on protected segments, same-device order, a 1 Hz tick, consumer isolation, and flow control, each contingent on broker durability.Drops before durable acceptance lie outside the delivery guarantee; flow control queues transient overload but cannot preserve delivery beyond retention capacity.Source: §5 · p. 8
Evidence-aware verdicts
- Verdicts pair a security result with an evidence status: sound, degraded, incomplete, or unavailable, based on continuity evidence tracked by the fabric.A retention high-water mark yields degraded; a gap, overflow, or consumed-stream monotonicity violation yields incomplete; a live tick without evidence yields unavailable.Source: §5 · pp. 8–9
Verdict algebra
- A negative finding over non-sound evidence becomes unknown, while a positive finding for a monotone witness property remains valid under unrelated missing evidence.For fleet-wide properties, the least-complete gateway status governs the L3 evidence status.Source: §5 · p. 9
Conditional soundness
- For monotone witness properties, a sound no-violation verdict agrees with the fault-free oracle over the accepted stream, assuming G2 and the stated cross-gateway skew bound for P3.6.The sound status presupposes G1, G3, and G5 over the window; P3.6 additionally requires skew ϵ < W, with W = 30 s.Source: §5 · pp. 9–10
Observability boundary
- A missed violation is downgraded only when loss is observable through event-id discontinuity, consumed-stream order checking, or the tick.The tick supports absence-based P3.5/P3.7; without it, genuine silence is unobservable, and the algebra cannot flag that loss class.Source: §5 · p. 10
6 Implementation
The implementation connects MQTT device ingest to NATS JetStream backend delivery while reusing the existing monitoring hierarchy and specifications. The containerised testbed deploys edge, gateway, and cloud components and provides a public artefact.
Brokers and subjects
- Gateways ingest device telemetry over MQTT at QoS 1, then publish per-window verdicts to NATS JetStream subjects, including a fleet-wide 1 Hz tick subject.The testbed uses Mosquitto; device topics are routed by gateway and device identifiers.Source: §6 · p. 10
Monitors
- RV-Fabric reuses the L1/L2/L3 monitors and leaves their specifications unchanged; L3 combines MonPoly for metric first-order correlation with RTLola for time-triggered silent-node monitoring.The silent-node monitor is driven by the fabric’s 1 Hz tick subject.Source: §6 · p. 11
Deployment topology
- L1 runs at the edge, gateway nodes run L1/L2 consumers and sidecars over disjoint fleets, and the cloud backend runs JetStream and L3 monitors.The testbed has two gateway nodes with four emulated devices each and scales by adding nodes.Source: §6 · p. 11
Reproducibility
- The reproducibility setup uses Docker Compose, a single-command launch, and a public artefact containing specifications, compose files, and monitor wiring.The containerised testbed runs on a laptop; the clock-skew experiment uses two separate cloud VMs with NTP disabled on one.Source: §6 · p. 11
7 Evaluation on the Controlled Testbed
The controlled testbed evaluates RV-Fabric’s continuity mechanisms through transport tests and a seven-incident oracle replay. RV-Fabric matches the oracle, while the shared-log and mechanism-removal cases reveal missed incidents, false all-clears, and latency costs.
Where the fabric differs (Q1)
- 76 of 1,200 verdicts were out of order under device-time ordering, while the gateway sidecar had 0; a 1 Hz tick fired P3.5 within one tick of its 5 s deadline.The shared-log design delivered every event in the steady-state ingest comparison; the distinction was ordering and silence-liveness.Source: §7 · p. 12
Per-event latency under a wide-area link (Q2)
- At 20 ms one-way WAN delay, mean latency was 60 ms on MQTT and 23 ms on JetStream, while p95 stayed under 75 ms.The MQTT hop grows at approximately 3δ and JetStream at approximately δ; the reported transport latency remained below the second-scale L3 detection latency.Source: §7 · p. 12
Throughput and fleet scale (Q3)
- ≈14K msg/s was the zero-loss plateau; 250/500/1000-device fleets sustained zero loss, with 1000 devices at ≈1930 msg/s and ∼60 ms p95.These measurements validate message handling, not the resource limits of 1000 physical devices; the bounds are per stream and per gateway, and real cross-region WAN and correlated multi-gateway partitions remain untested.Source: §7 · pp. 12, 15, 16
Detection (Q4)
- All six P3.1–P3.6 properties fired on every five-run test; P3.7 flagged gateway-stream loss within 10 s.On two independent hosts, P3.6 stayed sound while clock offset was below its 30 s window and flipped once the offset exceeded it.Source: §7 · pp. 12–13
Failure mode on mechanism removal (Q5)
- A fast L3 consumer drained 2,000 verdicts in 0.043 ± 0.001 s while its slow peer had processed 3, a ∼940× isolation versus 40 s shared-cursor blocking.The consumers used separate durable cursors.Source: §7 · p. 13
Crash recovery of the durable outbox (Q6)
- S2/S3 crashes lost 0 verdicts, with S3’s duplicate deduplicated; full replay took 8.8/95/474 ms for 100/1,000/5,000 entries.A crash before the local WAL fsync lost the one in-flight verdict; recovery was approximately 0.095 ms per entry.Source: §7 · pp. 13–14
Overload buffer horizon (Q7)
- At capacity 1,000, discard-old silently evicted 1,000 of 2,000 verdicts, whereas discard-new raised 1,000 publish errors.At ≈1930 msg/s, a million-message buffer covers ≈8.6 min; crossing the soft high-water mark marks the stream degraded, and the first rejected or evicted publication marks it incomplete.Source: §7 · p. 14
Verdict preservation against a fault-free oracle (Q8)
- The shared log missed six of seven oracle incidents as false all-clears, while RV-Fabric reproduced the oracle; removing G1, G2, or G5 surfaced misses as incomplete, but removing G3 restored false all-clears.The seeded campaign is a controlled causal experiment across seven property classes, not a statistically representative benchmark.Source: §7 · pp. 14–15, Table 3 · p. 15
8 Real-Data Validation: Water-SCADA and IoT/IIoT Datasets
The real-data replay sends two published flow-log datasets through the real MonPoly engine to test end-to-end transport and fleet-property firing, not detection accuracy.
8 Real-Data Validation: Water-SCADA and IoT/IIoT Datasets
- The replay validates detection through unmodified MonPoly on flow logs, not G1–G5 mechanisms; missing gateway assignments require a synthetic partition independent of labels and attack class.Application-layer attacks requiring content-level specifications are out of scope, and episode counts collapse consecutive satisfaction timepoints.Source: §8 · p. 16
- Three unchanged flow monitors detect port scans, DDoS floods, and C2 beacons, with L3 schemas requiring multi-identity or multi-gateway satisfaction.Their thresholds are 20 failed destinations in 60 seconds, 100 connections to one destination in 10 seconds, and 50 periodic retries with CV<0.5, respectively.Source: §8 · p. 16
- Per-identity precision/recall was 0.83/0.83 on WUSTL and 0.64/0.47 on TON_IoT; dedicated detectors score higher.WUSTL contains approximately 1 million Argus flows with 6 of 14 source identities attacking; TON_IoT has 461,000 records, 19 of 11,536 identities attacking over nine classes.Source: §8 · p. 16
- P3.2 fired 17/11 episodes (WUSTL/TON_IoT); P3.6 was continuously satisfied on WUSTL and fired in 168 TON_IoT episodes, up to four sources across two gateways.P3.6’s TON_IoT count depends on the synthetic gateway partition; P3.2 quantifies over devices.Source: §8 · pp. 16–17
9 Related Work
Related work develops hierarchical RV logic, fault-tolerant monitoring, and cloud-native messaging, while RV-Fabric contributes a delivery layer that carries the existing hierarchy across networks.
9 Related Work
- Two prior hierarchical edge-IoT designs introduced the three-layer hierarchy and formal security properties, but validated single-host shared-log prototypes that left delivery out of scope.RV-Fabric carries that same unmodified hierarchy across the edge-cloud continuum and makes evidence completeness part of each verdict.Source: §9 · p. 17
- The shared-log prototype and MQTT-only monitoring lack five guarantees, while generic stream processors provide durability and isolation but not RV-specific ordering or a critical-infrastructure threat model.The comparison covers durable delivery, flow control, independent consumers, a silence-aware clock, and trusted ordering.Source: §9 · p. 17
- Tsigkanos et al. place MFOTL monitors on edge nodes but treat lost-event handling as integrable rather than provided; RV-Fabric supplies that delivery layer and verdict completeness.STREL and SaSTL formalise hierarchical and spatial monitoring, while RV-Fabric addresses deployment across a real network.Source: §9 · p. 17
- Fault-tolerant RV work addresses monitor logic under faults or reordering while abstracting transport; RV-Fabric makes delivery loss, ordering, and liveness first-class and measurable.The cited line includes fault-tolerant distributed monitoring, crash-resilient decentralised algorithms [9], and MFOTL over out-of-order streams.Source: §9 · p. 17
- Cloud-native transports and security-monitoring research span multiple infrastructure sectors, but RV-Fabric contributes the delivery layer rather than detection logic.The paper also situates its delivery-layer guarantees alongside MQTT, NATS, and Kafka.Source: §9 · p. 18
10 Limitations and Threats to Validity
The evaluation establishes mechanism behavior under specified faults, but its deployment scale, trusted components, and real-data setup constrain what it validates.
10 Limitations and Threats to Validity
- The containerised throughput and crash-recovery tests do not establish absolute throughput or latency, and crash tests cover one gateway relay path rather than correlated failures.The skew-resilience test does span two independent machines; multi-host fleet scale, cross-region WANs, and correlated multi-gateway partitions remain future evaluations.Source: §10 · p. 18
- The backend remains a dependency for silence-liveness, and a compromised gateway that sends plausible heartbeats while suppressing evidence is out of scope.The sidecar and outbox are inside the trusted computing base; hash-linked event-sequence commitments are proposed to make suppression tamper-evident.Source: §10 · p. 18
- The real-data study validates fleet properties on traces, not continuity mechanisms; it uses a synthetic partition with untuned detection, and device-order counts vary with randomized time-spoof workload.Crash counts and zero gateway-order violations reproduce exactly.Source: §10 · p. 18
11 Conclusion
RV-Fabric adds broker-contingent continuity mechanisms and evidence-aware verdicts to hierarchical runtime verification; against a fault-free oracle, it preserves all seven injected incidents.
11 Conclusion
- Five broker-contingent continuity mechanisms add crash- and overload-resilient delivery and address ordering and silence-liveness gaps at overhead small relative to second-scale detection.The guarantees are conditioned on broker durability.Source: §11 · p. 18
- Against a fault-free oracle through real MonPoly, RV-Fabric preserved all seven injected incidents, while the shared log preserved one and misreported six as unqualified all-clears.The comparison links transport continuity to the trustworthiness of security verdicts.Source: §11 · p. 19
- By making evidence completeness part of runtime-verification semantics, the fabric prevents faults and overload from silently converting missing evidence into a false all-clear.Each verdict carries a completeness status.Source: §11 · p. 19