Source-linked AI summary
Pareto-optimal quantum kernel selection for unsupervised anomaly detection on real malware beaconing data
Boaz Micah, Nadia Milazzo, Maissa Beji, Borja Aizpurua, Llorenç Espinosa-Portalés, Esteban Payares, Ghada Ben Slama, Luc Andrea, Michel Kurek, Thomas Cope, Olivier Salomon
TL;DR
Assessing quantum kernels requires evaluating both task performance and separation from classical kernels, but prior work had not jointly optimized these under a fully unsupervised protocol. This paper combines NPD and geometric difference in a Pareto search for malware-beaconing detection, where the results show a tension between detection quality and geometric separation.
Problem
Quantum-kernel task performance and geometric separation from classical kernels are often assessed separately, and joint optimization without labeled validation data had not been established.
Source: §I · pp. 1–2Method
A fully unsupervised Pareto search jointly optimizes NPD and geometric difference for fidelity and projected kernels on malware-beaconing data.
Source: §I · p. 2, C. Hardware results · p. 9Results
AUC 0.840 versus 0.765 for the NPD-selected fidelity kernel, but its GD ≃7 is below √N = 14.14; projected kernels reach GD up to 272, while their Pareto knee has AUC 0.782.
Source: §V · p. 10Takeaways & Limitations
NPD-guided selection can find strong detection performance, while joint selection exposes that greater geometric separation does not ensure a corresponding detection gain.
Source: §V · p. 10The conclusions rest on a single dataset with N = 200 training samples.
Source: §V · p. 10
Abstract
from arXiv · showhide
Quantum kernel methods are leading candidates for a practical quantum advantage in machine learning, but assessing that potential requires two quantities usually reported separately: how well a kernel performs on the task, and how far its geometry departs from the classical kernels available for the same problem. We introduce a fully unsupervised, multi-objective protocol that optimises simultaneously the normalised pseudo discrepancy (NPD), a label-free proxy for anomaly detection quality, and the geometric difference (GD) to a tuned classical reference kernel, selecting models from the resulting Pareto front. We apply it to malware beaconing detection in real network traffic, using a one-class support vector machine with fidelity and projected quantum kernels over four data encodings, on simulators and on IQM's 20-qubit Garnet processor. NPD-guided selection alone finds a fidelity kernel that beats the tuned classical baseline, but with a geometric difference too small to certify the gain as quantum. Projected kernels reach far larger geometric differences; the Pareto-selected one only marginally exceeds the baseline (AUC $0.782$ versus $0.765$, $g_{C\to Q}\approx 89>\sqrt{N}$ relative to that reference kernel), still below the NPD-selected fidelity kernel ($0.840$).
I. INTRODUCTION
The paper addresses the challenge of assessing quantum-kernel performance and separation from classical kernels when labels are unavailable. It proposes a joint NPD–GD search and applies it to real malware beaconing, finding a trade-off between detection performance and geometric separation.
I. INTRODUCTION
- Quantum kernels use circuit-estimated, data-dependent similarities in a classical SVM, offering a geometry that may differ from classical kernels.The paper uses a classical one-class SVM rather than a fully quantum SVM algorithm.Source: §I · p. 1
- Real malware beaconing provides a setting where unsupervised detection is required because malware data are scarce and fragmented.Beaconing is periodic communication between an infected host and a command-and-control server.Source: §I · p. 2
- The proposed Pareto framework jointly optimizes label-free NPD and geometric difference to search empirically, but not prove, quantum advantage.The approach is presented as applicable beyond the malware-beaconing use case.Source: §I · p. 2
- Fidelity kernels give the strongest detection, while projected kernels exceed the geometric-difference scale without a corresponding detection gain.On hardware, neither kernel family reaches the classical baseline at current noise levels.Source: §I · p. 2
II. RELATED WORK
Prior work studies quantum advantage, label-free anomaly-model selection, and quantum methods for cybersecurity, but these strands do not combine unsupervised quantum-kernel analysis with real cybersecurity data.
II. RELATED WORK
- Quantum kernels can be provably advantageous on carefully designed problems, but realistic-data comparisons often find classical kernels matching or exceeding them.Geometric separation alone is insufficient; the feature space must also carry an inductive bias aligned with the task.Source: §II · p. 2
- NPD measures separation between anomaly scores on held-out normal and synthetic out-of-distribution samples, but prior use guided classical model selection.Labels are unavailable for the usual validation-based tuning in unsupervised settings.Source: §II · p. 2
- Existing cybersecurity QML work covers supervised or semisupervised tasks, autoencoders, or quantum kernels outside cybersecurity, not all three ingredients together.The paper identifies its intersection as an unsupervised protocol, a geometrically analyzable quantum kernel, and real cybersecurity data.Source: §II · pp. 2–3
A. Kernel methods
Quantum kernels let learning algorithms use similarities induced by data-encoding quantum states without explicitly computing the feature map; this paper compares fidelity and projected kernels in a classical OCSVM pipeline.
A. Kernel methods
- Kernel methods use pairwise similarities in a feature space instead of explicitly computing the nonlinear feature map.The resulting kernel values can be supplied to a learning algorithm through the kernel trick.Source: A. Kernel methods · p. 3
- The fidelity kernel is the inner product of quantum feature states, while the projected quantum kernel uses single-qubit reduced states.The projected-kernel bandwidth γ controls similarity through the Frobenius distance between reduced density matrices.Source: A. Kernel methods · p. 3
- A kernel matrix feeds a classical OCSVM that learns a boundary enclosing normal data; encoding choice is part of the multi-objective search.The study uses four existing data-encoding circuits rather than engineering a new feature map.Source: A. Kernel methods · p. 3
B. Multi-objective quantum kernel selection
The selection method jointly optimizes a label-free detection-quality proxy and kernel separation from a tuned classical reference. It searches the Pareto front and selects a knee-point compromise.
B. Multi-objective quantum kernel selection
- The method’s main contribution is fully unsupervised quantum-kernel selection that jointly optimizes NPD and geometric difference.NPD serves as a label-free proxy for anomaly-detection quality; GD measures separation from a classical reference.Source: B. Multi-objective quantum kernel… · p. 3
- NPD compares anomaly scores for held-out training data and synthetic samples drawn from a Gaussian fitted to training data.The synthetic Gaussian has per-dimension variances matching the training subset; larger score separation empirically correlates with better detection on unseen data.Source: B. Multi-objective quantum kernel… · pp. 3–4
- GD compares the quantum kernel matrix with the tuned classical OCSVM reference; small GD means similar or better classical performance is guaranteed.The matrices are normalized to trace N, and greater deviation between their geometries increases GD.Source: B. Multi-objective quantum kernel… · p. 4
- The search selects non-dominated candidates across NPD and GD, then chooses the knee nearest the utopia point after rescaling both objectives.The pipeline also runs each objective independently and evaluates the selected configurations on the test set.Source: B. Multi-objective quantum kernel… · p. 4, Figure 1 · p. 4
C. Dataset
The dataset combines normal network traffic with real botnet beacon traffic from Stratosphere Research Laboratory sources. Traffic is represented as flows with six retained features, with normal-only training and beacon traffic held for testing.
C. Dataset
- Normal traffic datasets from the Stratosphere Research Laboratory define typical-user traffic, while malware beaconing is periodic communication with an external command-and-control server that can blend into benign traffic.The laboratory captures real malware traffic and provides open-access network traffic datasets.Source: C. Dataset · pp. 4–5
- CTU-13 provides large-scale captures of real botnet activity across thirteen scenarios, with external control IPs used to filter botnet traffic from normal and background traffic.Each scenario corresponds to a different botnet sample.Source: C. Dataset · p. 5
- The test set contains 1,000 flows, including 457 anomalous and 543 normal flows; the training set contains 200 randomly sampled normal flows.Malicious flows are exclusively in the test set.Source: C. Dataset · p. 5
- Network traffic was aggregated into unique source–destination–protocol–port flows, then reduced to a nonredundant set of six statistical features.Feature selection was unsupervised and used no label information.Source: C. Dataset · p. 5, Table II · p. 5
IV. RESULTS
Model performance is evaluated with AUC and F1, with F1 reported for both choices of positive class.
IV. RESULTS
- AUC measures the ability to rank anomalies above normal instances across decision thresholds, while F1 is the harmonic mean of precision and recall at a fixed threshold.These are the two metrics used to evaluate model performance.Source: §IV · p. 5
- F1 is reported twice, treating anomalies and normal instances in turn as the positive class, because the score depends on that designation.The authors describe both values as a balanced and transparent view of performance.Source: §IV · p. 5
- The evaluation uses AUC and F1 to assess model performance, with the latter capturing precision–recall balance at a fixed threshold.AUC instead assesses ranking across thresholds.Source: §IV · p. 5
A. Classical baselines
The classical OCSVM establishes the malware-dataset reference performance, with its RBF configuration selected by maximizing NPD.
A. Classical baselines
- On the malware dataset, tuning the classical OCSVM to maximize NPD selects an RBF kernel with γ = 8.99 and ν = 0.472.Source: A. Classical baselines · p. 5
- AUC 0.765, F1 (A) 0.599, and F1 (N) 0.561 are the classical OCSVM's test-set results on the malware dataset.These results establish the classical reference baseline for comparison with quantum kernels.Source: Table III · p. 5, A. Classical baselines · pp. 5–6
B. Fidelity and projected quantum kernels: NPD, GD, and Pareto optimization
The NPD-selected fidelity kernel achieves the highest detection AUC, but its geometric difference remains too small to rule out a classical match. Projected kernels reach much larger geometric differences without matching that detection performance, exposing a trade-off between detection and separation from the reference kernel.
B. Fidelity and projected quantum kernels: NPD, GD, and Pareto optimization
- Projected-kernel GD can grow as its tunable RBF bandwidth drives the kernel matrix toward identity-like geometry, which a classical RBF kernel can also reproduce.The GD-maximising configuration sits at γ = 9.99; the paper notes this narrowbandwidth regime is associated with deteriorating generalisation.Source: B. Fidelity and projected quantum… · p. 6
- On the statevector simulator, joint NPD–GD maximization yields a fidelity-kernel front with GD ≲7, while every projected-kernel configuration lies above the threshold where the reference classical kernel is not guaranteed to reproduce its geometry.The projected front trades high GD at low NPD for improved NPD and passes through a knee at NPD = 1.421, GD = 88.6.Source: B. Fidelity and projected quantum… · pp. 6–7, Figure 2 · p. 8
- AUC 0.840 at GD 4.297 for the NPD-selected fidelity kernel beats the classical baseline (AUC 0.765), while the GD-maximising projected kernel reaches AUC 0.738 at GD 272.The NPD-selected projected kernel reaches AUC 0.777 at GD 32; no projected model reaches the best fidelity model.Source: B. Fidelity and projected quantum… · pp. 7–8
- Large GD did not reliably improve detection: projected kernels reached gk ∼270 without a systematic AUC trend, while the strongest AUCs came from fidelity kernels below the threshold.Across NPD-selected configurations, the largest-NPD fidelity and projected models reached AUC 0.840 and 0.777, respectively; NPD identifies a good region rather than finely ranking models.Source: B. Fidelity and projected quantum… · p. 7
- The joint Pareto fronts are not converged approximations: after 100 NSGA-II trials, single-objective searches found configurations beyond both families’ explored extremes.The family separation relative to the √N threshold held across the three optimisation runs.Source: B. Fidelity and projected quantum… · p. 7
- Under IQM noisy simulation, the NPD-selected fidelity AUC fell from 0.840 to 0.790, while the Pareto-selected projected AUC fell from 0.782 to 0.647.The authors attribute projected-kernel robustness under NPD and GD selection to shallow circuits, while the five-layer Pareto encoding is more noise-sensitive.Source: B. Fidelity and projected quantum… · pp. 7–8
C. Hardware results
On IQM Garnet, the hardware evaluation used NPD-selected configurations and required practical circuit and measurement choices. The fidelity kernel's detection metrics declined on hardware, while the projected kernel's AUC was lower than in simulation; the projected test-score-based ν choice makes those results optimistic.
C. Hardware results
- The hardware benchmark used NPD-optimised fidelity and projected kernels on IQM Garnet, a 20-qubit superconducting processor, with circuits chosen for practical implementation.The Hamiltonian map used two layers; the projected Z separable map used five layers and classical shadows limited to size 100.Source: C. Hardware results · pp. 8–9
- In statevector runs, fidelity kernels reached the highest AUC (≈0.84) below the GD threshold, whereas projected kernels reached gk ∼270 without a systematic AUC trend.The figures compare test AUC against GD and NPD, respectively.Source: Figure 3 · p. 9, Figure 4 · p. 9
- On IQM Garnet, fidelity-kernel AUC and F1(N) decreased considerably, while projected-kernel AUC was lower than simulation but its F1 scores were higher.The projected-kernel ν was chosen to maximise test scores, so those hardware figures are optimistic.Source: C. Hardware results · p. 9
- Hardware evaluation was limited by finite classical-shadow size and hyperparameters not optimised under more hardware-realistic noise models.The authors suggest larger or multi-shots shadows and further hardware-aware optimisation as possible improvements.Source: C. Hardware results · pp. 9–10
V. DISCUSSION
The discussion finds a tension between detection performance and geometric separation: the strongest fidelity result has low GD, while the projected Pareto selection has higher GD but lower AUC. It also identifies limits in the dataset, reference-kernel family, and hardware evaluation.
V. DISCUSSION
- AUC 0.840 versus 0.765 for the NPD-selected fidelity kernel, but its GD of about 7 stays below the √N threshold, so the gain is not evidence of quantum origin.The discussion gives the threshold as 14.14 and says the reference classical kernel may close the gap with more training data.Source: §V · p. 10
- AUC 0.782 for the Pareto-selected projected knee, below the fidelity knee (0.812) and NPD-selected fidelity result (0.840), while its GD exceeds √N.The projected knee was also the most noise-sensitive projected configuration, with AUC 0.647 on the IQM noisy simulator.Source: §V · p. 10
- Large GD separates the quantum feature space from the chosen reference, but does not establish that its inductive bias suits the task.The largest GD occurs near the projected kernel’s top bandwidth, where its kernel approaches identity-like geometry that a classical bandwidth can reproduce.Source: §V · p. 10
- NPD-guided selection found the best model in two of four settings and came within 0.01 AUC of the best in the other two; NPD and detection quality remain weakly related within a family.Making GD a co-objective reveals the trade-off that either NPD-only or GD-only reporting would miss.Source: §V · p. 10
- The evidence is bounded by one dataset with 200 training flows and six features, a finite search budget, and a classical reference drawn from a bounded kernel family.The N threshold and GD depend on sample size; the feature maps came from the literature rather than being designed for network traffic.Source: §V · pp. 10–11
- On IQM Garnet, projected kernels degrade less under hardware noise than fidelity kernels, consistent with shallower circuits and local observables, but neither reaches the classical baseline (AUC 0.765).The comparison uses a different evaluation set; in simulation, fidelity reaches AUC 0.840 versus 0.777 for projected kernels.Source: §V · p. 11
- The authors identify feature maps tailored to beaconing periodicity and burstiness as a natural continuation of this first application.They frame the work as a blueprint rather than a negative result about quantum kernels on cybersecurity data.Source: §V · p. 11
Appendix A: Data-encoding feature maps
The appendix describes four quantum data encodings and an unsupervised procedure for reducing network-flow features. The encodings range from separable rotations to entangling and Trotterized circuits, while feature selection uses clustering and silhouette scoring.
Appendix A: Data-encoding feature maps
- The Z feature map applies Hadamards followed by parametrized Z rotations on individual qubits, without entangling gates, leaving the resulting state fully separable.Source: Appendix A · p. 12
- The IQP-style encoding uses commuting, data-dependent Pauli-Z terms between Hadamard walls; its data-dependent circuit elements are diagonal and can be applied in any order.Source: Appendix A · p. 12
- The Hamiltonian evolution map uses Trotterized many-body dynamics and requires n + 1 qubits for n input features; evolution time and Trotter-step count are hyperparameters.Source: Appendix A · pp. 12–13
- The hardware-inspired CNOT map combines single-qubit rotations with nearest-neighbour entanglement using controlled-X gates.Source: Appendix A · p. 13
Appendix B: Feature engineering and feature selection
- After aggregating traffic into unique flows and extracting statistics, the authors used unsupervised hierarchical clustering and silhouette scores to select six representative features via cluster medoids.The medoid is the feature with the smallest average correlation distance to other features in its cluster.Source: Appendix B · p. 13