PaperlayerView exact revision on arXiv

Source-linked AI summary

Better than classical? The subtle art of benchmarking quantum machine learning models

Joseph Bowles, Shahnawaz Ahmed, Maria Schuld

arXiv:2403.07059v2quant-phcs.LG

TL;DR

The paper examines how dataset choice, publication incentives, and software costs complicate quantum machine-learning benchmarks. It compares out-of-the-box classical models with quantum-inspired classically simulable models and finds no systematic quantum advantage as task difficulty increases, motivating design-focused benchmarks.

  • Problem

    Benchmark results are sensitive to dataset selection and positive-publication incentives, while benchmarking hybrid quantum-classical pipelines imposes substantial software and resource challenges.

  • Method

    The study compares out-of-the-box classical competitors with quantum-inspired classically simulable models, including separable classifiers that remove entangling operations.

  • Results

    Quantum models show no systematic performance improvement as problem difficulty increases; on the hardest two-curves datasets, classical baselines exceed 90% accuracy while quantum models struggle.

  • Takeaways & Limitations

    Benchmarks should examine which model components are essential, including non-invasive removal of quantumness, and should investigate the distance measures induced by quantum kernels.

  • Takeaways & Limitations

    Results on small datasets cannot be used to reason about larger datasets, and hyperparameter optimisation has immense resource requirements.

Abstract

from arXiv · show

Benchmarking models via classical simulations is one of the main ways to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scales within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we develop an open-source package based on the PennyLane software framework and use it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models outperform the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that "quantumness" may not be the crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.

I. WHAT CAN WE LEARN FROM BENCHMARKS?

Benchmark results are highly dependent on dataset and study-design choices, so model performance is a context-dependent signal rather than an intrinsic property. Selection and reporting practices can also reverse apparent comparisons between quantum and classical models.

  • Dataset dependence: Small dataset rearrangements can switch the same quantum model from nearly unlearnable to perfectly accurate.Figure 2 illustrates this sensitivity using two closely varied binary classification datasets.
  • Study-design dependence: Benchmark suites, score aggregation, labeling corrections, and model-design choices can substantially change comparative rankings.Prior studies report different leaderboards after modifying datasets, scoring procedures, or labels.
  • Positivity bias: Selecting only the best of 20 quantum candidates can reverse the observed average against classical models, despite lower true quantum performance.The thought experiment reports published means of 0.74 for quantum models versus 0.65 for classical models, while the assumed true means are 0.55 and 0.65.
  • Positivity bias: Positivity bias is built into leaderboard-driven methodology rather than necessarily reflecting ethical misconduct.Research incentives favor searching for and publishing models that outperform existing ones.
  • Scientific rigour: Rigorous benchmarks should ask narrowly defined questions about how controlled changes affect specified metrics and tasks.The paper contrasts broad claims about whether quantum models beat classical machine learning with testable experimental questions.
  • MNIST example: MNIST studies report accuracies from 70–99.6%, making their results mixed and difficult to interpret against established classical baselines.Preprocessing can alter the hardness and nature of the task, weakening direct comparison with the original benchmark.

B. Summary of the models

The paper organizes its 12 quantum models into quantum neural networks, quantum kernel methods, and quantum convolutional neural networks. These families encode data into quantum states or circuits and use trainable transformations, measurements, or state similarities for binary classification.

  • Model families: The 12 implemented models belong to three families: quantum neural networks, quantum kernel methods, and quantum convolutional neural networks.The section introduces each family before describing individual models.
  • Quantum neural networks: Quantum neural networks are trainable variational quantum circuits that encode data and classify labels through measured observables.For binary classification, the prediction can be the sign of an observable with eigenvalues ±1.
  • Quantum neural networks: Training minimizes a differentiable cost over labeled examples, using stochastic gradient descent and parameter-shift rules for quantum gradients.The loss aggregates per-example performance across the training inputs and labels.
  • Quantum neural networks: The studied neural-network variants include amplitude-embedding models and trainable angle-embedding models with classical feature scaling.The latter aims to maximize output-qubit fidelities to class states.
  • Other constructions: Additional models use classical preprocessing with quantum circuits, IQP-inspired angle embeddings, Gibbs-state observables, or trainable state embeddings based on distances.These examples span hybrid, circuit-based, and metric-learning constructions.

• TreeTensorClassifier

The model overview covers kernel-based and convolutional quantum approaches alongside their classical analogues, with differing embeddings, feature maps, and training losses. Quantum kernels compare embedded states through similarity functions, while quantum convolutional models adapt convolutional architectures to quantum circuits.

  • Training choices: Model training uses varied losses: two cross-entropy, three square, one linear, and one distance-based loss.The paper notes that square loss is more naturally associated with regression than classification.
  • Quantum kernel methods: Kernel classifiers weight training examples by kernel-mediated similarity and can be viewed as linear classifiers in a feature space.Their parameters are commonly trained through maximum-margin convex optimization.
  • Quantum kernel methods: Quantum kernel methods evaluate a positive-definite similarity function using quantum-computer-assisted state embeddings.A common kernel is the overlap k(x_i, x_j) = tr[ρ(x_i)ρ(x_j)].
  • Quantum kernel methods: The benchmark implements three quantum kernel methods, including IQP-inspired, projected-density-matrix, and quantum-kitchen-sinks constructions.QuantumKitchenSinks uses quantum-generated features with a linear classifier and an SVM implementation.
  • Quantum convolutional models: Quantum convolutional models adapt convolutional-network ideas using random quantum filters or amplitude-embedding layers based on linear combinations of unitaries.One design seeks fewer trainable parameters than a classical convolutional neural network.

4. Classical models

Classical competitors are selected as scalable, out-of-the-box reference models rather than parameter-matched or state-of-the-art systems. The benchmark also includes classically simulable quantum-inspired models, while implementation choices and simulation costs constrain the experiments.

  • Classical competitors: Standard classical models are defined as algorithms classically simulable at scale, including neural, support-vector, and convolutional models.The competitors are intended as practical small-dataset baselines, not state-of-the-art systems.
  • Classical competitors: The study rejects matching parameter counts or layer numbers because those components have different computational roles across architectures.The comparison instead uses natural equivalents for the three quantum model families.
  • Quantum-inspired controls: Separable quantum-inspired classifiers remove entangling gates or use non-entangling embeddings, allowing simulation with single-qubit circuits.These models provide a non-entangling comparison for quantum architectures.
  • Implementation: The software pipeline combines PennyLane, JAX, optax, and scikit-learn for differentiable simulation, compilation, optimization, preprocessing, validation, and metrics.The implementation and datasets are released through the project repository and dataset site.
  • Implementation: Hyperparameter choices were not uniformly optimized for predictive performance: batch size was selected for simulation resources, and convergence criteria were partly arbitrary.Variational models may stop while optimization is flat, and further training can improve or worsen test accuracy.
  • Computational constraints: Quantum simulations become computationally demanding because cross-validation and hyperparameter search require repeated circuit evaluations, with some datasets exceeding a day of computation.For N = 250 datapoints, IQPKernelClassifier requires around 30,000 circuit evaluations for training.

QuantumMetricLearner,

The study builds a diverse benchmark suite and shows that dataset design, computational cost, and hyperparameter choices strongly shape comparisons between quantum and classical models.

  • Computational constraints: Some quantum models require over a day to evaluate one dataset, while reaching roughly 20 qubits becomes infeasible for the study’s scope.Bottlenecks include many circuit evaluations, density-matrix memory costs, and slow or failed convergence for larger circuits.
  • Dataset design: Artificially generated data enables controlled variation but may limit conclusions about real-world data.The study uses artificial data because quantum machine learning remains at an early research stage.
  • Dataset design: 160 individual datasets span 10 benchmarks generated by 6 procedures that vary input dimension or a parameter controlling learning difficulty.The procedures prioritize structural diversity, comparability, controllable variables, and theoretical relevance.
  • Dataset design: The benchmark suite includes linearly separable, bars-and-stripes, downscaled MNIST, hidden-manifold, two-curves, and hyperplane classification procedures.Examples vary dimensions, image resolutions, manifold dimensions, Fourier-series degree, and the number of hyperplanes.
  • Dataset design: A margin subtlety affects how easily linearly separable data generalizes as dimensionality increases.The class margin changes with increasing dimensions and influences generalization from training data.
  • Hyperparameter tuning: Hyperparameter choices can move test accuracy from almost random guessing to near-perfect classification, making tuning crucial for fair comparisons.The best settings vary by dataset: increasing layers helps on MNIST PCA, hurts on linearly separable data, and is nonsignificant for hidden-manifold DIFF.

V. RESULTS

Across the benchmark experiments, out-of-the-box classical models systematically outperform quantum models. Additional analyses find no systematic quantum advantage from increasing difficulty or dimension, while non-entangled models often match or exceed entangled quantum models.

  • Out-of-the-box classical models outperform quantum models: The three prototypical classical models consistently rank above the quantum models across the benchmark experiments.Alternative reasonable ranking and aggregation methods changed the overall picture only slightly and not significantly.
  • Out-of-the-box classical models outperform quantum models: Model rankings are surprisingly consistent across benchmarks, with MLPClassifier often first among QNN comparisons and amplitude-encoding models typically last.In four of seven QNN benchmarks, MLPClassifier ranked first and DressedQuantumCircuitClassifier second; the order reversed in the other three.
  • Out-of-the-box classical models outperform quantum models: SVC performs best on four of seven kernel benchmarks, while QuantumKitchenSinks is consistently worst-performing among the kernel methods.QuantumKitchenSinks uses a random, non-trainable feature map based on computational-basis samples from a quantum circuit.
  • Out-of-the-box classical models outperform quantum models: The vanilla ConvolutionalNeuralNetwork outperforms both quantum convolutional neural networks, while WeiNet fails to learn the bars & stripes task.The bars & stripes task was considered easy for a model of WeiNet’s kind.
  • No systematic overfitting or scaling effects: Quantum models show no systematic improvement relative to classical baselines as problem difficulty increases, and generally do not scale differently with dimension except on the linearly separable benchmark.For the hardest two-curves datasets, classical baselines exceed 90% test accuracy while all quantum models appear to struggle.
  • Testing quantumness: Fully separable, classically simulable models perform similarly to or better than most quantum models, indicating that entanglement may not be essential on these tasks.The SeparableKernelClassifier ranks second after SVC among kernel methods, while the SeparableVariationalClassifier is generally surpassed only by MLPClassifier among QNNs.

VI. QUESTIONS RAISED ABOUT QUANTUM MODEL DESIGN

The benchmarks move beyond leaderboard comparisons by probing how quantum components transform data, which design features matter, and what distance measures quantum kernels induce. These analyses often link quantum models to classical counterparts while exposing unresolved questions about their distinctiveness and behavior in higher dimensions.

  • A. Do quantum components in hybrid models help?: The DressedQuantumCircuitClassifier’s quantum and classical replacement layers similarly reshape simple data into a one-dimensional manifold for linear classification.Its performance followed the classical neural network closely in most experiments.
  • A. Do quantum components in hybrid models help?: The QuanvolutionalNeuralNetwork’s initial quantum feature map often creates noisy versions of images and performs worse than the classical convolutional baseline.The observed map can make subsequent classical learning more difficult on the small datasets tested.
  • B. What makes data reuploading work?: Trainable input rescaling and trainable embedding are crucial to DataReuploadingClassifier performance, whereas its special cost function is not.Their interplay may matter, but follow-up work often considers only trainable embedding.
  • C. What distance measures do quantum kernels define?: Quantum kernel methods generally perform similarly to a Gaussian-kernel support vector machine, and their two-dimensional kernel shapes usually resemble the Gaussian kernel.Exceptions include cases involving very narrow bandwidths and periodic regions.
  • C. What distance measures do quantum kernels define?: In higher dimensions, only ProjectedQuantumKernel resembles the SVC model, while the other three quantum kernels produce similar Gram matrices.This raises whether the projected circuit or the subsequent Gaussian kernel is chiefly responsible for learning.
  • C. What distance measures do quantum kernels define?: Understanding quantum-kernel distance measures in high dimensions, rather than focusing only on classical intractability, remains an important open challenge.Benchmarking can support this investigation beyond two-dimensional visualisations.

D. Why are polynomial features not working?

The poor performance of amplitude-encoding models may reflect limited expressivity, unsuitable feature scaling, or inductive biases mismatched to benchmark structure. These results motivate testing alternative data structures, hyperparameters, and model designs rather than treating polynomial features as universally ineffective.

  • Amplitude encoding: Amplitude-encoding QNNs perform poorly; TreeTensorClassifier has limited expressivity because its parameter count grows only arithmetically with qubit number.For 16 features, it has only 7 variational parameters, severely limiting expressivity.
  • Amplitude encoding: CircuitCentricClassifier can construct up to 3rd-order polynomial features, but its optimal number of input copies is often 1 or 2 rather than 3.This weak correlation between input copies and test accuracy raises questions about whether higher-order features help these datasets.
  • Hyperparameters: Smaller input-scaling values can improve average accuracy on selected six-dimensional datasets, indicating that default scaling may disadvantage amplitude-encoding models.The figure varies feature scaling while selecting hyperparameters found under the default scaling.
  • Inductive bias: On the linearly separable benchmark, nearly all quantum models worsen with dimensionality while classical models achieve near-perfect classification.Selected angle-embedding models form periodic decision regions, which may be unsuitable for higher-dimensional linear boundaries.
  • Inductive bias: The hyperplanes diff benchmark shows quantum models performing almost like classical baselines, suggesting that model biases may align differently across data structures.This task tests detection of a global labeling rule determined by several hyperplanes in a low-dimensional manifold.
  • Benchmarking implications: Meaningful benchmarking requires rigorous methodological design and extensive reporting because model rankings depend on difficult experimental choices.Future studies should also examine how results scale beyond small datasets and test which design components are replaceable.

Appendix A: CO2 Emission Table

The supplied passages combine an emissions table with glossary and appendix material describing model definitions and training procedures. The emissions calculation reports approximately 2,600 kWh of simulation energy and 340 kg of CO2 emissions.

  • CO2 Emission Table: ≈2600 kWh of simulation energy corresponds to ≈340 kg of numerical-simulation CO2 emissions.The calculation uses ≈27 core years, ≈11 W per core, and Canada’s ≈0.13 kg/kWh average emissions factor.
  • CO2 Emission Table: Transport contributes 0 kg of CO2 emissions in the reported accounting.
  • Glossary and model setup: Amplitude embedding maps an input vector into a normalized quantum state, while angle embedding encodes features through parameterized rotations.Repeated angle embeddings are called data reuploading; amplitude embedding can behave differently when feature dimension is a power of 2.
  • Glossary and model setup: Variational models are trained with gradient descent using Adam, while the appendix specifies learning-rate and architecture hyperparameter grids.The listed grids vary learning rate, layer count, input copies, and observable type across models.
  • Glossary and model setup: The DataReuploadingClassifier splits features into three-dimensional groups, embeds them on separate qubits, and repeats the process across trainable layers.Training uses output-qubit fidelities to class states, with the number of evaluated qubits controlled by nmax.

3. DressedQuantumCircuitClassifier [51]

The DressedQuantumCircuitClassifier combines classical neural-network layers with a parameterized quantum circuit. Other supplied passages describe related Gibbs-state classifiers, including assumptions and simulation constraints affecting their evaluation.

  • DressedQuantumCircuitClassifier: The DressedQuantumCircuitClassifier maps inputs through a classical network, an angle-encoded quantum circuit, and a final classical layer producing two outputs.The input and output classical layers use fully connected networks; the quantum circuit applies Y rotations and ring-pattern CNOTs.
  • DressedQuantumCircuitClassifier: The output is converted to class probabilities with a softmax layer, and cross entropy jointly trains the classical and quantum parameters.
  • Related classifiers: The IQPVariationalClassifier uses IQP-inspired angle embedding followed by a trainable parameterized circuit and predicts from a Z1Z2 measurement.Its loss is the linear loss, with hyperparameters covering learning rate, variational layers, and repeated embedding layers.
  • Related classifiers: The QuantumBoltzmannMachine encodes data into a Gibbs state and predicts from a diagonal ±1 observable measured on selected visible qubits.The implementation assumes access to a perfect Gibbs state because exact Gibbs-state preparation is computationally difficult.
  • Related classifiers: Mixed-state simulation increases memory costs, preventing the QuantumBoltzmannMachine from being tested at as many qubits as other models.
  • Related classifiers: The separable QuantumBoltzmannMachine uses a product Hamiltonian, yielding a product mixed Gibbs state while otherwise matching the QuantumBoltzmannMachine setup.

7. QuantumMetricLearner [56]

The QuantumMetricLearner trains a quantum embedding by contrasting fidelities within and across classes, then classifies new points by average fidelity to embedded training examples. The supplied appendix also describes several kernel and tree-tensor alternatives.

  • QuantumMetricLearner: The QuantumMetricLearner uses layer-wise QAOA-inspired embeddings with an additional latent-feature qubit.Each layer applies feature-dependent X rotations followed by parameterized ZZ and Y rotations.
  • QuantumMetricLearner: Training minimizes fidelity for same-class pairs and maximizes fidelity for different-class pairs.The implementation samples random training pairs rather than comparing every pair within a batch.
  • QuantumMetricLearner: Prediction assigns a new input to the class whose random batch of embedded training points has the highest average fidelity.This corresponds to the paper’s fidelity classifier.
  • QuantumMetricLearner: The number of circuits per training step and prediction scales linearly with the sampled pair-batch size, fixed at 32 here.Larger batches provide more reliable cost and label estimates.
  • Other appendix models: TreeTensorClassifier uses amplitude encoding and a tree-structured variational circuit with only 2n −1 parameters for n qubits.The implementation assumes direct access to the exact amplitude-encoded state and measures Z on the first qubit.
  • Other appendix models: IQPKernelClassifier feeds an IQP-inspired kernel matrix to a support vector machine, while ProjectedQuantumKernel applies an RBF kernel to measured feature vectors.

11. QuantumKitchenSinks [55]

QuantumKitchenSinks constructs randomized quantum feature vectors from linearly transformed inputs and classifies them with logistic regression. The benchmark also describes several related classical and quantum convolutional and neural-network models.

  • QuantumKitchenSinks: QuantumKitchenSinks linearly transforms each input into kmax vectors of dimension n before circuit processing.The transformed vectors use randomly sampled Wk and bk, with k ranging from 1 to kmax.
  • QuantumKitchenSinks: Each transformed vector is angle encoded and processed by X rotations plus two CNOT layers coupling qubits at distances 1 and 2.The circuit returns a bitstring sample for each transformed vector.
  • QuantumKitchenSinks: The concatenated bitstrings form a feature vector of size n · kmax, which is passed to a linear classifier.The implementation uses scikit-learn’s LogisticRegression class.
  • QuantumKitchenSinks: The model’s hyperparameter search varies circuit size and the number of circuits kmax.The tested values are n_qfeatures ∈ [d, ⌊d/2⌋] and n_episodes ∈ [10, 100, 500, 2000].
  • Related convolutional models: The related quantum convolutional model applies random quantum circuits to thresholded image windows before a trainable convolutional network.Its quantum circuits output the number of ones in the most-probable sampled bitstring, with channels and kernel sizes treated as hyperparameters.
  • Classical baselines: The classical baselines include multilayer perceptron, support vector machine, and vanilla convolutional neural-network classifiers.The MLP uses stacked linear transformations and activations, the SVM uses an RBF kernel, and the CNN combines convolution, pooling, and fully connected layers.

17. SeparableVariationalClassifier

The SeparableVariationalClassifier uses independent single-qubit circuits, so it omits entanglement while retaining trainable rotations and product data encoding. Its predictions come from measured Pauli-Z observables mapped through a logistic function.

  • SeparableVariationalClassifier: The SeparableVariationalClassifier is a quantum neural network that does not use entanglement.Each feature is processed by a comparable single-qubit circuit.
  • Circuit construction: Each encoding layer applies trainable single-qubit rotations followed by product angle embedding through Pauli-Y rotations.An additional trainable single-qubit rotation layer follows the data encoding.
  • Prediction: Prediction uses the average of the single-qubit Z observables, O = 1/n(Z1 + ··· + Zn).The trainable parameters are those in the L circuit layers.
  • Training: Class probabilities are obtained with a logistic function and trained using binary cross entropy.The observable is multiplied by 6 so the sigmoid varies across the range [−6, 6].
  • Hyperparameters: The search varies learning rate and encoding depth, with L ∈ [1, 3, 5, 10].The separable kernel counterpart also varies encoding layers and SVM regularization.
  • Training convergence: Training convergence is assessed by comparing recent loss statistics, with the adopted criterion corresponding to an approximately flat loss curve over the last 200 steps.The procedure stores the previous 400 loss values and compares the first and last 200-step intervals.

2. Bars and stripes

The bars-and-stripes benchmark generates noisy grayscale images whose labels distinguish vertical bars from horizontal stripes. The surrounding dataset procedures also describe dimensionality reduction and synthetic manifold-based classification tasks.

  • Bars and stripes: Bars and stripes creates grayscale images of vertical bars or horizontal stripes on a d × d pixel grid.The task is intended as a simple benchmark for convolutional models.
  • Bars and stripes: Labels are sampled uniformly, and each image receives independent Gaussian noise with standard deviation σ and mean 0.For one class, columns are filled with random ±1 values; for the other, rows are filled similarly.
  • Bars and stripes: The benchmark varies image width d across 4, 8, 16, and 32.
  • MNIST variants: MNIST variants classify digits 3 and 5 after dimensionality reduction, using either coarse-grained pixels or principal components.The MNIST 3-versus-5 task is described as difficult to distinguish.
  • MNIST variants: The MNIST PCA dataset projects standardized images onto the d largest training-set principal components.The test set is transformed using parameters derived from the training set.
  • Synthetic manifold tasks: The hidden-manifold procedure generates balanced labels by applying a neural network to low-dimensional manifold vectors and thresholding after median subtraction.The resulting data is embedded in a higher-dimensional feature space.

5. Two curves

The two-curves benchmark embeds two classes along noisy Fourier-defined curves in a high-dimensional space. Its difficulty is controlled by curve geometry, while the accompanying hyperplanes task uses parity across multiple linear boundaries.

  • Two curves: The two-curves procedure represents the two classes as curves embedded in a d-dimensional space.Curve curvature and inter-curve distance are treated as variables influencing classification difficulty.
  • Two curves: A one-dimensional Fourier series with maximum degree D controls curve curvature in each dimension.Fourier coefficients are sampled uniformly from [0, 1].
  • Two curves: Random t values in [0, 1] determine positions of data points along the embedded curves.
  • Two curves: Class −1 uses the same Fourier embedding as class 1 but shifts every dimension by an offset Δ.A noise factor ε controls the variance of random spread around the curve trajectories.
  • Hyperplanes: The hyperplanes benchmark labels points by whether an even number of k linear predictions are positive.The feature vectors are linearly embedded into a higher-dimensional space before classification.
  • Detailed results: The appendix provides detailed ranking, accuracy, and separable-model results for benchmarks not shown in the main figures.
Loading 2403.07059v2…