PaperlayerView exact revision on arXiv

Source-linked AI summary

How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection

Syeda Anshrah Gillani, Mirza Samad Ahmed Baig, Shahid Munir Shah, Asher Ali, Hamzah Siddiqui

arXiv:2608.18155v1quant-phcs.AIcs.ETcs.LG

TL;DR

Quantum intrusion-detection studies lack a fair, reproducible way to determine whether reported gains are genuinely quantum. This paper benchmarks quantum and tuned classical models with an attribution audit, finding classical models match or exceed quantum models on aggregate detection while two narrowly defined advantages survive correction.

  • Problem

    NIDS research lacks a fair, reproducible evaluation methodology and principled attribution of measured performance differences to quantum components.

  • Method

    The study applies one leakage-controlled benchmark across four datasets, comparing hybrid VQCs and QSVMs with tuned classical baselines and matched attribution controls.

  • Results

    Across all four datasets, tuned classical models match or exceed quantum models on aggregate detection; two advantages survive correction, including p = 0.005 at NSL-KDD’s 1%-FPR point.

  • Takeaways & Limitations

    Practitioners should choose tuned Random Forest or XGBoost for aggregate detection, while quantum methods warrant targeted follow-up at specific operating points.

  • Takeaways & Limitations

    All quantum results are simulated, and conclusions are conditioned on an information-reduced input and a limited qubit budget.

Abstract

from arXiv · show

Quantum machine learning (QML) for network intrusion detection (NIDS) is routinely reported to reach near-perfect accuracy, yet the most rigorous studies find that well-tuned classical models remain competitive, and that apparent quantum gains may be artefacts of classical dimensionality reduction and implicit regularisation rather than genuine quantum effects. We ask not whether a quantum model can post a high accuracy, but how quantum the advantage really is. We present a unified, reproducible QML-IDS benchmark evaluating hybrid variational quantum circuits and quantum-kernel SVMs against five honestly-tuned classical baselines across four standard NIDS datasets (NSL-KDD, UNSW-NB15, CICIDS2017, NF-ToN-IoT-v2) under one leakage-controlled protocol, with an equal-budget feature view, imbalance- and calibration-aware metrics with significance testing, and a simulated NISQ noise sweep. We introduce a quantum-attribution audit (parameter-matched classical controls, a random-feature kernel, and a regularisation sweep) that quantifies how much of any gain is genuinely attributable to the quantum component. Tuned classical models (Random Forest, XGBoost) match or exceed the quantum models on aggregate detection on every dataset, and the audit attributes this to classical preprocessing and regularisation rather than quantum effects. Two advantages survive false-discovery-rate correction: the quantum-kernel SVM out-ranks its direct classical surrogate (a random-feature kernel) on AUPRC and ROC-AUC, and a small four-qubit hybrid out-detects the best classical baseline at the 1% false-positive operating point on the distribution-shifted NSL-KDD task (p = 0.005, BH q = 0.030). Code, seeds, and splits are released; our contribution stands whether quantum wins, ties, or loses.

1. Introduction

This paper reframes QML-based network intrusion detection around a fair, reproducible, leakage-controlled evaluation rather than another headline accuracy record. It benchmarks hybrid VQCs and QSVMs against tuned classical models across four datasets while auditing attribution, operational performance, noise, calibration, and statistical rigour.

  • Motivation: QML-NIDS headline results are questioned because dimensionality reduction and random cross-validation can overstate real-world detection.Strong quantum results often use PCA-reduced inputs with one feature per qubit and binary tasks under random cross-validation.
  • Motivation: The paper identifies a fair, reproducible evaluation methodology and principled attribution of measured differences as the field’s central deficit.It rejects the goal of producing another accuracy record as scientifically fragile and strategically weak.
  • Benchmark design: The benchmark evaluates hybrid VQCs and QSVMs against five honestly tuned classical baselines across NSL-KDD, UNSW-NB15, CICIDS2017, and NF-ToN-IoT-v2.The protocol uses strict leakage control, binary detection, and official splits where applicable.
  • Attribution audit: Parameter-matched classical controls and a regularisation sweep decompose observed gains into quantum and classical components.This provides a constructive test of the Bellante et al. critique in the practical NISQ regime.
  • Operational evaluation: The evaluation reports detection rate, false-positive rate, low-FPR operating points, and calibration beyond accuracy/F1, while characterising degradation under four noise channels.The channels are depolarising, amplitude-damping, phase-damping, and readout-error.
  • Statistical evaluation: Results are statistically framed as means over three to five seeds with statistical rigour and honest reporting.The supplied passage specifies three to five seeds but does not provide further statistical values here.

2. Related Work

Related work spans foundational QML intrusion-detection studies, evolving quantum methods, and increasingly sceptical evaluations. This paper positions itself through broader datasets, fairer evaluation, and explicit attribution of apparent quantum gains.

  • Foundational and breadth-oriented QML-IDS: Foundational studies report ∼98% accuracy on a custom IoT stream and 97.21% precision for a VQCNN on KDD/NSL-KDD.Kalinin and Krundyshev applied quantum SVM and quantum convolutional networks; Gong et al. reported the prominent variational result.
  • Methods landscape: Parameterized quantum circuits dominate QML-IDS, while quantum-kernel SVMs scale poorly and often rely on subsampled, PCA-reduced data.Recurring concerns include barren plateaus and encoding choice; amplitude embedding is reported as less stable than angle embedding.
  • Rigorous and sceptical evaluations: Rigorous evaluations find classical ensembles can remain superior, with reported advantages explained by regularisation rather than quantum effects.These studies establish a comparison standard focused on fairness and attribution rather than higher accuracy alone.
  • Datasets and evaluation practice: Most studies use one or two datasets and omit imbalance-aware metrics, calibration, significance testing, and honestly tuned baselines.Genuine multi-dataset and cross-dataset evaluation is rare, with Bhatnagar et al. identified as the principal exception.
  • Contribution relative to prior work: This work adds tuned baselines, equal-budget features, operating-point and calibration metrics, significance testing, reproducibility, and causal decomposition.It extends the regularisation critique beyond Bellante et al.’s fault-tolerant, PCA-specific case study toward practical NISQ hybrid-VQC evaluation.

3. Threat Model and Background

The evaluation targets flow- and connection-record intrusion detection under low false-positive costs, distribution shift, unseen attacks, and class imbalance, while excluding adaptive evasion. QML pipelines combine quantum state processing with classical feature compression and read-out, motivating an attribution audit to separate quantum effects from classical processing and regularisation.

  • Threat model: Low-FPR operation, distribution shift, unseen attack signatures, and class imbalance define the evaluation constraints; adaptive adversarial evasion is out of scope.The official NSL-KDD split is retained to represent training–deployment distribution shift.
  • QML background: An n-qubit QML model uses a 2^n-dimensional state vector, while noisy simulation tracks a 2^n × 2^n density matrix, limiting noisy runs to twelve qubits.The benchmark ablates angle, amplitude, IQP/ZZ, and data-re-uploading encodings before variational processing and Pauli-Z measurement.
  • QML background: Every hybrid pipeline compresses high-dimensional records classically to the per-qubit budget and uses classical read-out, so performance reflects quantum processing alongside classical processing and implicit regularisation.The attribution audit is designed to disentangle these contributions.

4. Methodology

The benchmark uses a leakage-controlled, equal-budget protocol with official or seeded splits, training-only preprocessing, and cross-validation-based model selection. It compares hybrid quantum and quantum-kernel models with classical controls while auditing whether apparent advantages arise from quantum circuits, preprocessing, or regularisation.

  • Evaluation protocol: Official or seeded splits, training-only preprocessing, dropped identifiers, and within-training cross-validation prevent test-distribution information from influencing model development.NSL-KDD uses KDDTrain+/KDDTest+; UNSW-NB15 uses official CSVs; CICIDS2017 and NF-ToN-IoT-v2 use seeded stratified 70/30 splits.
  • Evaluation protocol: The study reports full-view performance and a same-budget view in which every model receives identical PCA-reduced input with nfeatures = nqubits = 8.Min-max scaling before PCA retains 0.841 on NSL-KDD, compared with ∼27% variance for standard scaling before reduction.
  • Model families: A single model factory evaluates six hybrid VQCs, two QSVMs, a parameter-matched classical MLP, and five classical baselines across encoding and ansatz configurations.The VQC grid includes angle, IQP/ZZ, amplitude, and reuploading encodings with strongly-entangling and basic-entangler ansätze; QSVMs use IQP fidelity and angle projected kernels.
  • Attribution audit: The attribution audit compares each quantum model with parameter-matched classical controls and varies classical regularisation to test whether observed gaps close without the quantum component.Hybrid VQCs share their front-end with matched MLPs, while QSVMs are compared with RBF-SVM and random-feature kernel controls; regularisation varies ridge, weight decay, RF depth, and SVM bandwidth.
  • Noise robustness: Noise robustness is evaluated with depolarising, amplitude-damping, phase-damping, and bit-flip readout channels at p ∈{0, 0.001, 0.005, 0.01, 0.05}, capped at twelve qubits.The sweep uses PennyLane’s default.mixed density-matrix backend.

5. Experimental Setup

The benchmark uses four standard NIDS datasets under a disclosed binary-task protocol, with explicit dataset scopes and stratified downsampling. It reports threshold, ranking, and calibration metrics across uneven but specified seed budgets, while recording reproducible experiment metadata and quantum-simulation constraints.

  • Datasets: Four standard NIDS datasets are evaluated on the main binary task, with CICIDS2017 using the three-day MachineLearningCVE subset and NF-ToN-IoT-v2 stratified downsampled to 200,000 rows.The NF-ToN-IoT-v2 sample is drawn from 16.9M rows; both dataset scopes are explicitly disclosed.
  • Metrics: The evaluation reports threshold metrics plus ROC-AUC, AUPRC, TPR@0.1%FPR, TPR@1%FPR, Brier, and ECE.Threshold metrics include accuracy, precision, recall, F1, detection rate DR, and false-positive rate FPR.
  • Seed budget: Classical baselines use five seeds {42, 43, 44, 45, 46} on NSL-KDD and three seeds {42, 43, 44} on the other three datasets.The paper states the seed budget exactly because it is uneven across model classes.
  • Reproducibility: Each run writes self-describing JSON records containing configuration, per-seed metrics, tuned hyperparameters, reference-seed predictions, environment snapshots, and source checksums.Code, seeds, and splits are released.
  • Simulation and hardware: Quantum simulation dominates runtime at QML-IDS qubit counts, and the main runs use eight qubits with ceilings of sixteen noiseless and twelve noisy qubits.Experiments run on one laptop with an NVIDIA RTX 3050 Ti, 4 GB, under WSL2; the CPU lightning.qubit simulator is reported as faster than the GPU lightning.gpu simulator.

6. Results

Tuned classical models match or exceed quantum models on aggregate detection across all four datasets, while two quantum advantages survive false-discovery-rate correction. The results also show that a small hybrid retains a strict low-false-positive edge on distribution-shifted NSL-KDD and that mild noise can preserve performance.

  • Aggregate results: Tuned classical models match or exceed quantum models on aggregate detection across every dataset, while a small hybrid retains a significant strict low-FPR edge on shifted NSL-KDD.This is the section’s overall result.
  • Aggregate results: 0.782 vs. 0.730 (NSL-KDD), 0.891 vs. 0.836 (UNSW-NB15), 1.000 vs. 0.968 (CICIDS2017), and 0.991 vs. 0.949 (NF-ToN-IoT-v2) are the same-budget F1 results favoring classical models.The comparison is between the best classical baseline and the leading 8-qubit quantum model.
  • Operating-point results: 0.517 vs. 0.467 at TPR@1%FPR gives the four-qubit hybrid a 0.050 advantage over Random Forest on NSL-KDD, with p = 0.005.At TPR@0.1%FPR, the corresponding values are 0.407 vs. 0.272.
  • Noise robustness: 0.421 to 0.430 low-FPR detection under mild depolarising, amplitude-damping, and phase-damping noise remained essentially unchanged, while ROC-AUC rose to 0.878–0.897.The noiseless reference had ROC-AUC 0.846 and TPR@1%FPR 0.422; mild noise was consistent with regularisation.
  • False-discovery-rate audit: 13 quantum-favouring and 27 classical-favouring differences survive Benjamini-Hochberg correction, leaving corrected evidence that favors classical models on aggregate.The audit covered 108 direction-aware comparisons on NSL-KDD.

7. Discussion

The discussion finds almost no quantum advantage on aggregate detection: tuned classical models win or tie broadly, while a small but significant quantum edge survives at the operational 1%-FPR point on shifted NSL-KDD. The findings attribute aggregate gains to classical preprocessing and regularisation, and redirect attention toward calibration, leakage control, and realistic deployment conditions.

  • Aggregate advantage: Comparable classical front-ends and regularisation eliminate aggregate quantum advantage, with tuned classical baselines significantly outperforming hybrid VQC and QSVM models on F1 and AUPRC.The audit identifies classical preprocessing and regularisation, rather than the quantum component, as the source of apparent aggregate gains.
  • Operational exception: At NSL-KDD’s shifted-test 1%-FPR operating point, the four-qubit hybrid significantly out-detects the best classical baseline (p = 0.005).The exception occurs on the discriminative dataset and survives mild simulated noise.
  • Practical recommendation: For current deployment, tuned Random Forest or XGBoost is the honest recommendation because these models win or tie on every aggregate metric and dataset at lower cost without quantum hardware.The discussion frames this as a corrective to inflated accuracy records and a practical redirection of research effort.
  • Operational exception: Only the four-qubit circuit exceeds Random Forest’s TPR@1%FPR of 0.467, while the advantage vanishes by twelve qubits.Both F1 and low-FPR detection decline with circuit width, consistent with implicit regularisation rather than raw expressivity.
  • Methodological implications: The ≈0.20 F1 CV-to-test gap cautions the QML-IDS literature, while a GPU backend running an order of magnitude slower at these qubit counts makes backend choice consequential.The discussion links inflated records to random cross-validation and presents the backend crossover as a practical time-saving consideration.

8. Limitations and Threats to Validity

The study is limited by simulated quantum results, constrained qubit width, dataset coverage choices, indirect trainability analysis, subsampled QSVM training, and small, uneven seed budgets that weaken statistical power.

  • Simulation and hardware validity: All quantum results are simulated, so the noise study omits correlated and non-Markovian noise, crosstalk, connectivity, and transpilation effects.The authors describe simulated results as an optimistic bound and note that real-hardware confirmation would strengthen the conclusions.
  • Resource constraints: A 4 GB GPU constrains the available qubit width, limiting the scale of the evaluated quantum configurations.The passage states that the GPU caps width at roughly an incomplete value.
  • Dataset coverage: NSL-KDD is dated, the IoT evaluation uses a single NetFlow dataset, and CICIDS2017 and NF-ToN-IoT-v2 use standard subsets rather than necessarily full corpora.The dataset limitations are partly mitigated by including three further datasets, while the latter subsets are configurable to full corpora.
  • Model and computational limitations: Trainability is not characterised directly, and QSVMs use subsampled data because fidelity-kernel computation scales as O(N 2).The passage specifically identifies barren plateaus and gradient variance as unmeasured trainability concerns.
  • Statistical power: The seed budget is small and uneven, so paired tests over seeds have low statistical power.The study uses five classical-baseline seeds on NSL-KDD, three elsewhere and for all quantum runs, and two for the noise sweep.

9. Conclusion

Across four datasets, a leakage-controlled, equal-budget benchmark finds that well-tuned classical models match or exceed quantum models on aggregate detection. The study releases code, configurations, seeds, split scripts, and per-run provenance to support reproducibility.

  • Conclusion: Well-tuned classical models match or exceed quantum models on aggregate detection across four datasets.The comparison uses one leakage-controlled protocol and an equal-budget feature view.
  • Conclusion: The benchmark combines operating-point and calibration metrics, significance testing, a noise sweep, and a quantum-attribution audit.These components replace recurring headline accuracy claims with a fairer, reproducible evaluation yardstick.
  • Conclusion: Code, configurations, fixed seeds, split scripts, and per-run provenance are released at https://github.com/Orqly-AI/quantum-ids-benchmark.Preprocessed datasets will be released on request.

CRediT authorship contribution statement

Syeda Anshrah Gillani and Mirza Samad Ahmed Baig are identified as the core contributors, contributing equally across the work. Gillani is credited across the full CRediT taxonomy listed, while Baig is credited for the corresponding contributions shown.

  • CRediT authorship contribution statement: Syeda Anshrah Gillani and Mirza Samad Ahmed Baig contributed equally to all aspects of the work.
  • CRediT authorship contribution statement: Gillani contributed to conceptualization, methodology, software, validation, formal analysis, investigation, and data curation.
  • CRediT authorship contribution statement: Gillani also contributed to original-draft writing, review and editing, visualization, and supervision.
Loading 2608.18155v1…