PaperlayerView exact revision on arXiv

Source-linked AI summary

IoMT-SecAlarmBench: A Counterfactual Benchmark for Integrity Attacks in IoMT

Emmanuel C. Ugwuabonyi, Dmitri Perkins

arXiv:2608.26416v1cs.CR

TL;DR

IoMT alarms require distinguishing physiology, faults, and attacks, but existing datasets lack the counterfactual ground truth needed for that attribution. IoMT-SecAlarmBench injects controlled integrity attacks into genuine coupled ECG+PPG recordings with retained clean signals and cause labels, then evaluates detectors under threshold-independent and matched-false-alarm protocols. No method consistently handles the hardest cases, while attack detection remains entangled with fault discrimination.

  • Problem

    Existing IoMT datasets do not provide counterfactual ground truth that distinguishes cyberattacks, sensor faults, and genuine physiological events at per-patient granularity.

  • Method

    IoMT-SecAlarmBench injects controlled integrity attacks into genuine coupled physiological recordings and pairs them with dual-modality network data, retaining clean counterfactual signals and cause labels.

  • Results

    No detector consistently exceeds the benchmark’s performance limit, and replay and low-amplitude transient spikes remain close to chance-level detection across detectors.

  • Takeaways & Limitations

    Cross-channel inconsistency indicates an integrity problem but does not by itself resolve whether the cause is an attack, fault, or physiological deterioration.

  • Takeaways & Limitations

    The benchmark targets coupled ECG+PPG integrity attacks and lacks equivalent signal-level counterfactual information for genuine physiological events.

Abstract

from arXiv · show

The Internet of Medical Things (IoMT) combines clinical physiological data with cyber-system information, creating challenges in determining whether an abnormal reading reflects a genuine physiological event, a device fault, or a cyber-attack within the expected physiological range. Answering this requires counterfactual ground truth, which no existing dataset provides. We present IoMT-SecAlarmBench, a semi-synthetic benchmark that injects controlled integrity attacks into genuine coupled ECG+PPG recordings using a structured experimental design combining four attack morphologies, four severity levels, two physiological plausibility conditions, and replay attacks. Each injected window retains its cause, attack subtype, and the clean signal that would have been observed without the attack. We evaluate six detectors from five method families using threshold-independent measures and a matched false-alarm budget. Results show no method consistently detects the most difficult cases: replay attacks and low-amplitude transient spikes remain close to chance-level performance across detectors. Results also reveal a trade-off between detecting attacks and distinguishing them from sensor faults: the best-performing detector on hard cases flags fault/artifact windows at 5.3 times its false-alarm rate on normal data. Three-way classification performs poorly for genuine physiological events, and a leakage audit of a dual-modality network dataset indicates previously reported IoMT intrusion-detection performance is partly driven by identifying information. Benchmark, generation code, preprocessing, evaluation tools, and datasheet are released.

I. INTRODUCTION

IoMT alarms can arise from physiological deterioration, sensor faults, or measurement-path attacks, yet existing data lack the counterfactual ground truth needed to distinguish them. IoMT-SecAlarmBench addresses this gap by injecting controlled attacks into genuine coupled recordings while retaining cause labels and clean counterfactual signals.

  • ICU false-alarm rates can reach 99% because sensor drift, motion, calibration error, and patient variability make abnormal observations difficult to interpret.
  • An alarm may reflect physiological deterioration, sensor failure, packet loss, or spoofing, and each cause requires a different response.
  • The central question is whether the alarm would have occurred if the network had remained healthy.
  • Existing causal-RCA and industrial sensor-attack methods do not provide per-patient counterfactual attribution for coupled physiological and cyber signals because the required ground truth is structurally absent.
  • IoMT-SecAlarmBench injects controlled integrity attacks into genuine coupled cardiorespiratory recordings and pairs them with dual-modality network data.
  • The benchmark combines four attack morphologies, four severity levels, and two plausibility regimes, while treating replay attacks as a separate family.

A. The attack, fault and physiology ambiguity in IoMT

IoMT attribution requires separating cyberattacks, faults, and genuine physiology across coupled clinical and cyber modalities. Existing datasets cover parts of this space but none supplies the required dual-modality, per-patient counterfactual ground truth.

  • An identical alarm can originate from a cyberattack, device fault, or genuine physiological event across the clinical and cyber layers.
  • The benchmark requirements include dual-modality data, labels distinguishing attacks, faults, and physiology, per-patient structure, and measurement-path integrity attacks.
  • CICIoMT2024 is cyber-only, while WUSTL-EHMS-2020 provides the closest cross-modality bridge but has narrow attacks and summarized rather than high-rate physiological data.
  • PhysioNet supplies genuine physiology without attacks, and CinC 2015 pairs ECG with PPG while labeling true and false life-threatening alarms.
  • No existing dataset satisfies all five requirements or provides counterfactual ground truth for rigorous attribution.
  • Controlled injection into genuine recordings is used because native integrity-attack labels are unavailable and live clinical red-teaming is unacceptable.

D. Benchmarking and documentation standards

The benchmark follows patient-wise evaluation, leakage auditing, event-aware metrics, and explicit stealthiness stratification to support credible IoMT attribution results. Its threat model treats clinical events and attacks as different interventions on latent physiology versus measurement channels.

  • Causal-RCA benchmarks have shown many methods perform close to a random baseline, motivating careful evaluation outside the medical domain.
  • The benchmark uses patient- or session-wise splits, audits leakage, avoids point adjustment, stratifies stealthiness, labels artifacts, and releases reproducible documentation and code.
  • A clinical event changes the latent physiological state and propagates coherently through intact measurement mechanisms, whereas an attack changes measurements while leaving that state untouched.
  • Attacks target the coupled ECG and PPG/PLETH channels, with ABP and respiration providing additional consistency constraints when available.
  • The CinC cohort contains 627 ECG+PPG records, including 231 true and 396 false alarms, with splits stratified by alarm type and verdict.
  • MIMIC-III provides continuous waveform depth from six patients, while WUSTL-EHMS-2020 supplies paired network-flow and biometric data for linkage checks and leakage auditing.

B. Preprocessing

Preprocessing standardizes coupled physiological windows, preserves artifacts as fault evidence, audits network-feature leakage, and defines attacks by how much measurement mechanism and plausibility they preserve. This yields a nested difficulty ordering from replay to spoofing.

  • All physiological channels are resampled to 125 Hz, segmented into 10-second windows with 5-second stride, and assigned validity and cause labels.
  • ECG and pulsatile channels are filtered separately, while flat-line, clipping, and physiologically implausible segments are labeled artifact or fault rather than deleted.
  • The network pipeline removes identifying features such as IP addresses, ports, timestamps, flow IDs, and MACs before session-wise evaluation with macro-F1 and AUPRC.
  • Replay substitutes a channel with a real past output, preserving single-channel statistics while breaking current cross-channel coherence.
  • FDI remains within the plausibility interval but is not a valid output of the true mechanism at the current time, allowing it to evade residual and threshold detectors.
  • Value spoofing leaves the physiological manifold and is therefore easiest to detect, producing the ordering replay, FDI, then spoofing by increasing intervention severity.

2) Factorial injection design:

IoMT-SecAlarmBench uses a factorial injection design that varies perturbation morphology, severity, and physiological plausibility, while keeping replay attacks separate. The design yields controlled combinations for distinguishing attack properties.

  • Factorial structure: Four injection morphologies are crossed with four ordered severity levels and two plausibility regimes, separating shape, magnitude, and physiological plausibility.The regimes distinguish in-bound false-data injection from out-of-bound spoofing.
  • Replay attacks: Replay attacks form a separate, mechanism-preserving family rather than being crossed with the crafted-injection factors.The benchmark includes same-patient and cross-patient replay conditions.
  • Design space: The crafted design produces 32 cells, with additional same-patient and cross-patient replay conditions.The crossing is fully realized for in-bound injections; out-of-bound spoofing has a qualification on morphology.

3) Morphology equations and severity schedule:

The benchmark defines four attack morphologies with morphology-specific equations and severity schedules. Severity is ordinal within each morphology, while plausibility bounds constrain in-bound attacks and replay preserves signal plausibility without severity scaling.

  • Morphologies: Spike applies a brief additive change over 3–5 samples, whereas stuck-at freezes a channel at a selected value for the injection duration.These mechanisms respectively model a transient perturbation and a stalled signal that destroys natural variance.
  • Morphologies: Drift adds a linearly ramping offset to a maximum value, while bias applies a constant level shift that preserves waveform shape and dynamics.The two mechanisms represent gradual sensor drift and an incorrect baseline or calibration.
  • Severity schedule: Severity is scheduled separately for each morphology because a shared magnitude grid is not comparable across perturbation types.Within each morphology, levels L1→L4 are monotone and stealthiness is reported by morphology and severity.
  • Plausibility: In-bound FDI is clipped to channel-specific plausibility intervals, while out-of-bound spoofing places the modified signal beyond those intervals.The bounds are recomputed per window from the pre-onset quarter and constrain amplitude, not sample-to-sample rate of change.
  • Replay: Replay substitutes an earlier contiguous segment from the same or another patient, preserving within-channel shape and marginal plausibility while breaking cross-channel coherence and temporal freshness.Replay is defined by source and staleness rather than severity.
  • Severity schedule: The released severity schedule uses additive offsets in per-patient standard-deviation units, with ordinal levels advancing magnitude or duration alternately.For drift, the scheduled magnitudes are 1/2/4/8 standard-deviation seconds; in-bound clipping can reduce realized excursions.

D. Ground-truth labeling and the counterfactual clean signal

The benchmark assigns deterministic four-way cause labels and retains the exact clean signal for every attacked window. Its evaluation protocol uses patient or session-wise splits, imbalance-safe training procedures, and threshold-aware metrics.

  • Ground-truth records: Every window records its primary cause, attack family, morphology, regime, injected channel, onset, duration, severity, and retained clean pre-injection signal.These fields support attribution and signal-recovery evaluation against known counterfactual outcomes.
  • Cause labeling: The four primary classes are physiology-normal, physiology-event, fault or artifact, and attack, assigned deterministically from native verdicts, onset timing, artifact masks, and injection status.Physiological classes retain CinC’s expert-reviewed labels, while fault combines artifact-mask and monitor false-alarm cases.
  • Cause labeling: The fault class contains 4,368 mask-driven windows with artifact_frac > 0.10 and 982 verdict-driven windows with artifact_frac ≤ 0.10.The artifact-mask test is evaluated first, partitioning the heterogeneous fault class exactly.
  • Counterfactual signal: The benchmark’s counterfactual signal is the exact clean measurement that would have appeared without the attack, enabling signal-level evaluation of counterfactual correctness.This ground truth is retained for every attacked window.
  • Evaluation protocol: Detection uses precision, recall, F1, and AUPRC without point adjustment, while discrimination uses confusion matrices, macro-F1, MCC, and Cohen’s κ.The protocol also reports results across multiple runs with dispersion measures.
  • Operating points: AUPRC is interpreted against the positive-class prevalence near 0.54, and detectors are compared threshold-free or at matched false-alarm budgets.Maximum-F1 operating points can vary by up to ±0.15 across seeds and may reverse method rankings.

VI. EXPERIMENTS

The experiments evaluate six unsupervised detectors spanning five method families under a common training and evaluation procedure. Research questions cover attack and event detection, morphology-specific stealthiness, multi-class discrimination, and efficiency.

  • Methods: Six unsupervised detectors span classical, reconstruction-based, probabilistic, graph-attention, and transformer method families.The set includes Isolation Forest, LSTM-AE, USAD, OmniAnomaly, GDN, and TranAD.
  • Experimental procedure: The five deep-learning models operate on [N, C, T] windows within a common reconstruction framework and are trained on normal data before individual-window evaluation.Training uses 40 epochs, gradient clipping, and five random seeds with deterministic kernels.
  • Research questions: RQ1 tests detection of injected integrity attacks and genuine physiological events.The question evaluates whether existing detectors identify both attack and physiological-event windows.
  • Research questions: RQ2 examines how detection performance degrades as injections become stealthier, stratified by morphology.Stealthiness is evaluated per morphology rather than only as an aggregate condition.
  • Research questions: RQ3 evaluates whether methods distinguish attack, fault, and physiological-event classes.This extends binary attack detection to cause discrimination among clinically different outcomes.
  • Research questions: RQ4 asks whether the evaluated methods can operate within bedside or edge-computing budgets.Per-patient attribution is supported by the benchmark but is not a separate research question in this study.

B. Implementation notes

The benchmark fixes reproducible data, feature, split, and evaluation settings, then shows moderate overall detection with strong variation by attack mechanism and a persistent attack–fault ambiguity.

  • Implementation notes: Whole records are held out using a 436/91/100 train/validation/test split, with a unified record/patient identifier supporting leakage-free splitting.The construction also fixes the alarm at sample 37,500.
  • Implementation notes: 22 ECG–PPG features combine per-channel statistics and spectral descriptors with zero-lag cross-correlation and best-lag position, while deep detectors use raw tensors with 10× temporal pooling.The feature matrix uses only the ECG+PPG core channels and contains no missing values across enrichment tiers.
  • Evaluation setup: 100 held-out records provide 11,825 windows, including 6,392 attack windows, with attack prevalence 0.541 as the no-skill AUPRC baseline.Results use unadjusted, event-aware scoring and mean ± SD over five deterministic-seed runs.
  • Overall detection: AUPRC ranges from 0.82 to 0.90, equivalent to a 1.5–1.7× improvement over the 0.54 no-skill baseline, with no detector dominating.Isolation Forest achieves the highest overall performance among the six detectors, while validation-max-F1 false-alarm rates can be seed-unstable.
  • Attack difficulty: At a matched 10% false-alarm budget, out-of-bound spoofing reaches detection rates of 0.89–0.98, whereas sustained in-bound FDI reaches 0.38–0.68 depending on severity.Threshold-free and matched-budget evaluations show the same mechanism-based difficulty ordering.
  • Hard cases and selectivity: Replay and low-amplitude transient spikes remain near chance across detectors, while Isolation Forest flags fault/artifact windows at 5.3 times its ordinary-normal false-alarm rate.Replay AUPRC is 0.076–0.110 against a 0.065 chance floor; transient spikes affect only 3–5 samples, and cross-channel inconsistency can also reflect sensor faults.

C. RQ3 — Discrimination: Attack vs. Fault vs. Physiology

Three-way discrimination separates attack and fault windows reasonably well but performs poorly on genuine physiological events, with aggregate metrics masking this imbalance.

  • RQ3 — Discrimination: F1 is 0.967 for attack windows, 0.674 for fault windows, and 0.040 for physiology-event windows.Overall metrics are Macro-F1 0.560, MCC 0.623, and Cohen’s κ 0.610.
  • RQ3 — Discrimination: 77.2% of genuine physiological deteriorations are misclassified as attacks, while 20.7% are classified as artifacts.This makes physiology-event discrimination the weakest of the three classes.
  • RQ3 — Discrimination: 40.2% of fault/artifact windows are classified as attacks.The reverse confusion remains substantial and is consistent with ambiguity observed for coupling-based detection.
  • RQ3 — Discrimination: Aggregate MCC and Cohen’s κ appear relatively strong because the large, well-separated attack class drives much of the overall performance.The authors therefore caution against interpreting aggregate metrics as reliable discrimination among all three causes.

D. RQ4 — Efficiency

Efficiency is compatible with bedside inference, but training cost varies substantially and evaluation choices affect reported rankings. The leakage audit further shows that identifier features can inflate apparent intrusion-detection performance, while several benchmark scope limitations remain.

  • RQ4 — Efficiency: Per-window inference latency is 0.0075–0.0396 ms for all six detectors, within a bedside budget.Isolation Forest is fastest and TranAD slowest, so inference efficiency does not distinguish the methods.
  • RQ4 — Efficiency: Fit time ranges from 0.4 s for Isolation Forest to 20–90 s for deep detectors and 187.5 s for GDN.GDN’s training cost is an order of magnitude above the classical baseline without a corresponding detection gain.
  • Leakage audit: With identifiers, every leakage-audit metric is 1.000; without them, attack recall falls to 0.484 and attack F1 to 0.644.The stratified 70/30 random split is vulnerable to identifier leakage because the dataset is single-session.
  • Leakage audit: The audit indicates previously reported WUSTL-EHMS-2020 performance is substantially attributable to identifying information, not absence of attack-related signal.Without identifiers, AUPRC remains 0.787 at attack precision 0.96.
  • Evaluation protocol: Maximum-F1 false-alarm rates vary by as much as ±0.234 across seeds, versus at most ±0.013 for threshold-free AUPRC.The authors recommend threshold-free comparison or matched false-alarm budgets, with deep-detector results averaged over at least three seeds.
  • Attribution scope: Channel-level attribution is degenerate because every released attack targets the pulsatile channel.Always predicting the pulsatile channel achieves AC@1 = 1.000, exceeding PCMCI+ at 0.720 and a marginal-deviation ranker at 0.824.
  • Threats to validity: Physiological results come from the 627-record CinC corpus, leaving persistence of the performance limit on long continuous ICU recordings untested.The single dual-modality corpus also does not support multi-protocol coverage for the leakage analysis.
  • Limitations: The benchmark is semi-synthetic, limited to integrity attacks on coupled ECG+PPG core channels, and lacks equivalent counterfactual information for genuine physiological events.The alarm-level counterfactual oracle is specified but not implemented in the current release.

IX. CONCLUSION AND FUTURE WORK

IoMT-SecAlarmBench provides a controlled benchmark for per-patient attribution of integrity attacks using coupled physiological and cyber data with exact clean-signal ground truth. Its baseline results expose method limits and attack-versus-fault ambiguity, while future work extends scope toward longer recordings, multi-channel attacks, and clinically safe mitigation.

  • IoMT-SecAlarmBench pairs coupled physiological and cyber data with exact per-patient ground truth while retaining corresponding clean signals.The benchmark is designed for controlled study of causal attribution in medical devices.
  • Baseline results identify a performance limit that the evaluated methods do not consistently exceed.
  • Attack-versus-fault ambiguity is tied to ECG–PPG coupling, motivating methods that determine an abnormality’s cause rather than only its presence.
  • Future work targets longer continuous ICU recordings, multi-protocol captures, multi-channel attacks, alarm-level counterfactuals, calibrated foundation models, and clinically safe mitigation.
  • 78,729 labeled 10 s windows form the dataset, including ECG+PPG signals, cause labels, attack metadata, clean signals, and alarm information.The windows span 627 records and include 22 per-window features.
  • The release records single-target injection, a deferred alarm-state oracle, and severity saturation, and cautions against clinical deployment decisions without prospective validation.
Loading 2608.26416v1…