Source-linked AI summary
SupermarQ: A Scalable Quantum Benchmark Suite
Teague Tomesh, Pranav Gokhale, Victory Omole, Gokul Subramanian Ravi, Kaitlin N. Smith, Joshua Viszlai, Xin-Chuan Wu, Nikos Hardavellas, Margaret R. Martonosi, Frederic T. Chong
TL;DR
Diverse quantum architectures make reliable performance comparison difficult, motivating SupermarQ’s scalable, hardware-agnostic benchmark suite. The paper applies classical benchmarking principles through feature vectors and scalable application workloads, finding broader coverage and architecture-specific performance tradeoffs.
Problem
Diverse quantum architectures and single-application or single-number benchmarks make reliable, representative performance comparison difficult.
Method
SupermarQ combines guiding principles, hardware-agnostic feature vectors, and eight scalable OpenQASM applications spanning realistic workloads.
Results
SupermarQ provides superior application-space coverage, correlates application features with system performance, and reveals architecture-specific tradeoffs.
Takeaways & Limitations
Quantum benchmarking can use evolving, application-level suites to compare full systems and identify strengths such as fidelity, gate speed, and connectivity.
Abstract
from arXiv · showhide
The emergence of quantum computers as a new computational paradigm has been accompanied by speculation concerning the scope and timeline of their anticipated revolutionary changes. While quantum computing is still in its infancy, the variety of different architectures used to implement quantum computations make it difficult to reliably measure and compare performance. This problem motivates our introduction of SupermarQ, a scalable, hardware-agnostic quantum benchmark suite which uses application-level metrics to measure performance. SupermarQ is the first attempt to systematically apply techniques from classical benchmarking methodology to the quantum domain. We define a set of feature vectors to quantify coverage, select applications from a variety of domains to ensure the suite is representative of real workloads, and collect benchmark results from the IBM, IonQ, and AQT@LBNL platforms. Looking forward, we envision that quantum benchmarking will encompass a large cross-community effort built on open source, constantly evolving benchmark suites. We introduce SupermarQ as an important step in this direction.
I. INTRODUCTION
SupermarQ addresses the difficulty of comparing diverse quantum architectures by adapting classical benchmarking principles into a scalable, application-level suite. It characterizes application coverage, includes scalable workloads, and evaluates performance across platforms.
- Motivation: New quantum architectures require tailored benchmark suites for meaningful performance measurement and comparison.The paper situates quantum benchmarking alongside benchmark suites created for supercomputers, workstations, chip multiprocessors, and machine learning.
- Motivation: Single-number metrics can misrepresent overall quantum-system performance because they target specific circuit classes or applications.The paper argues for suites spanning application domains rather than relying on one metric.
- Approach: Application-level benchmarks and feature vectors are used to represent realistic workloads and quantify coverage across quantum applications.Each application is described by a feature vector reflecting its resource requirements and position in the application space.
- Design requirements: The suite must span present NISQ devices and future fault-tolerant systems while adapting to advances in hardware, algorithms, and software.The paper emphasizes evaluating hardware and software together as quantum systems evolve.
- Contributions: SupermarQ contributes guiding principles, feature vectors, eight scalable OpenQASM benchmark applications, and cross-platform evaluation.The suite also correlates application features with observed system performance.
- Findings: Realistic benchmark suites provide better application-space coverage than existing single-application and feature-focused synthetic suites.This is presented as a discovery from the suite’s coverage analysis.
II. PRIOR WORK
Prior quantum benchmarks range from low-level characterization and synthetic protocols to application benchmarks, each addressing part of system evaluation. The paper argues that scalable, meaningful comparison requires application-level metrics combined across diverse workloads.
- Gate-Level Characterization: Low-level gate characterization measures native-operation error rates but does not directly capture performance on real-world applications.Randomized approaches scale polynomially with qubit count, while process tomography scales exponentially.
- Synthetic Benchmarks: Synthetic benchmarks such as quantum volume and quantum LINPACK use randomness but are neither representative of typical workloads nor scalable to verify.Their verification procedures can require classical computations that scale exponentially or unfavorably with qubit count.
- Application Benchmarks: VQE benchmarks molecular ground-state energy estimation, while QAOA and Q-score address optimization problems such as MaxCut.These examples illustrate application-specific benchmark workloads and metrics.
- Application Benchmarks: Application benchmarks use metrics such as ground-state energy or approximation ratio, supporting scalability and cross-platform comparison.These metrics avoid requiring direct comparison with ideal circuit outputs.
- Benchmark Suites: A single application cannot measure overall system performance because quantum workloads differ substantially.The paper therefore motivates benchmark suites containing multiple applications.
4) Benchmark Suites:
SupermarQ is designed as a scalable, meaningful, full-system, adaptive benchmark suite for diverse quantum architectures. It uses shared-abstraction applications and hardware-agnostic feature vectors to quantify workload coverage and resource demands.
- 4) Benchmark Suites:: Existing benchmark suites can be difficult to scale because their metrics compare experimental outputs with ideal circuit outputs, requiring exponential circuit simulation.This limitation affects QASMBench and PPL+2020.
- 4) Benchmark Suites:: Cross-platform comparisons have revealed effects from connectivity, native gates, and error rates, but often use too few applications to represent expected workloads.The paper identifies this as a limitation across photonic, trapped-ion, and superconducting architectures.
- Scalability: Scalable benchmarks should retain meaning from a few qubits to hundreds, thousands, and beyond while using efficiently scalable performance metrics.MaxCut is given as an example of an application defined on graphs of arbitrary size.
- Meaningful and Diverse: Meaningful and diverse suites should represent workloads from domains including chemistry, machine learning, cryptography, finance, physics, and database search.Domain diversity supplies performance points relevant to a wider range of users.
- Full-system evaluation: Full-system evaluation includes hardware–software interplay, allowing compilers and hardware-specific capabilities to affect measured performance.SupermarQ specifies applications at a shared abstraction such as OpenQASM rather than mandating one compilation toolflow.
- Adaptivity: Adaptivity requires benchmark applications to keep pace with changes in algorithms, compilation optimizations, and hardware.The suite is intended to reflect the current state of the art.
- Feature vectors: Feature vectors quantify coverage by indicating how strongly each benchmark stresses the processor and by capturing communication requirements through normalized interaction-graph degree.Sparse applications have values near zero, while dense programs approach one.
2) Critical-Depth:
Critical depth measures how much of a circuit’s two-qubit activity lies on its longest dependency path, contextualizing duration and accumulated error. The section also introduces features for entanglement, parallelism, and liveness.
- Critical-Depth: Critical depth is the proportion of two-qubit interactions on the longest path that sets circuit depth.It is calculated as D = n_ed/n_e, where n_ed counts interactions on that path and n_e counts all two-qubit interactions.
- Critical-Depth: Circuits with critical depth close to 1 are heavily serialized.A large fraction of two-qubit interactions therefore lies along the path determining circuit depth.
- Entanglement-Ratio: Entanglement-ratio estimates entanglement-related structure by dividing two-qubit interactions n_e by all gate operations n_g.The feature is motivated by entanglement’s importance to quantum advantage and by the difficulty of measuring entanglement exactly during execution.
- Parallelism: Parallelism captures susceptibility to cross-talk by relating qubit count, gate count, and circuit depth.Highly parallel applications fit many operations into relatively small depth and have a parallelism feature close to 1.
- Liveness: Liveness captures how frequently qubits are operated on rather than idle during circuit execution.Its matrix has one row per qubit and one column per circuit-depth time step; idling frequency is 1 − L.
6) Measurement:
The measurement feature characterizes mid-circuit measurement and reset activity, while the Mermin-Bell benchmark tests quantum behavior through GHZ-state preparation and operator measurements. The GHZ benchmark instead evaluates entangled-state preparation fidelity using a CNOT ladder.
- Measurement: The measurement feature counts circuit layers containing mid-circuit measurement and reset operations.For circuit depth d, it uses l_mcm, the number of layers containing those operations.
- GHZ: The GHZ benchmark prepares an entangled state with a Hadamard gate followed by a ladder of CNOTs.Performance is measured by Hellinger fidelity between experimental and ideal distributions concentrated equally on the all-zero and all-one states.
- GHZ: The CNOT-ladder GHZ construction is included because not all platforms support mid-circuit measurements.Other GHZ preparations can use mid-circuit measurements or parallel two-qubit gates with different gate-count and depth requirements.
- Mermin-Bell: The Mermin-Bell benchmark prepares a GHZ state and measures the expectation value of the Mermin operator.Its circuit rotates the state into the operator’s shared basis so each term can be measured simultaneously.
- Mermin-Bell: As Mermin-Bell size increases, basis-change gates begin to dominate state preparation.The benchmark score is computed as (⟨φ|M|φ⟩ + 2^n−1)/2^n.
C. Error Correction Subroutines
Because current NISQ error levels prevent full-scale error correction, SupermarQ uses proxy applications that reproduce circuit structures common to error-correcting codes without correcting errors.
- Error-Correction Context: Error-correcting codes trade off detectable errors, correctable errors, qubit requirements, and error thresholds for fault tolerance.Small experiments have demonstrated the feasibility of different schemes, but full-scale fault tolerance has not yet been observed.
- Proxy Applications: SupermarQ uses two proxy applications because current NISQ devices cannot implement full-scale error correction.The proxies reflect circuit structure common to many error-correcting codes but do not correct errors.
- Phase Code: The phase-code benchmark is a phase-flip repetition code parameterized by data-qubit count and error-correction rounds.It initializes data qubits in |+⟩ states, applies r correction rounds, and evaluates Hellinger fidelity against the known ideal output distribution.
- Bit Code: The bit-code benchmark is a bit-flip repetition code parameterized by data-qubit count and error-correction rounds.Its ideal final state is known a priori, so Hellinger fidelity measures agreement between experimental and ideal outcomes.
D. QAOA
The suite benchmarks QAOA and VQE using scalable, classically verifiable proxy applications, alongside Hamiltonian simulation of the one-dimensional transverse-field Ising model. These proxies replace full variational optimization with single-iteration hardware evaluations.
- QAOA: QAOA is benchmarked on weighted complete-graph MaxCut instances using Vanilla and ZZ-SWAP parameterized circuits.Vanilla QAOA requires all-to-all connectivity, while ZZ-SWAP uses a SWAP network for all pairwise interactions.
- QAOA: The QAOA proxy evaluates a single iteration because thousands of iterations make the full benchmark infeasible under cloud queue wait times.The level-one variant is chosen for scalable classical verification, and the score compares measured and ideal expectation values.
- VQE: VQE targets the ground-state energy of the one-dimensional transverse-field Ising model, which is exactly solvable classically.The benchmark classically optimizes parameters, then measures the resulting energy on the quantum computer for one proxy iteration.
- Hamiltonian Simulation: Hamiltonian simulation models the one-dimensional transverse-field Ising system with a time-varying magnetic field.The benchmark Trotterizes the Hamiltonian for selected system sizes and time steps, then measures final-state average magnetization against an exact value.
G. Coverage
SupermarQ measures benchmark-suite coverage using the volume of the convex hull spanned by application feature vectors. It also supports scalable, cross-platform evaluation through parameterizable applications and a shared OpenQASM-based benchmarking division.
- Coverage is computed as the volume of the convex hull formed by benchmark feature vectors in a six-dimensional feature space.
- Table I reports each benchmark suite’s coverage volume and the number of circuits used to compute it.
- Only SupermarQ and QASMBench exceed the synthetic benchmark suite’s coverage, and both include larger applications relevant to late NISQ and early FT devices.
- SupermarQ matches QASMBench’s benchmark-size range for the coverage computation while additionally generating arbitrarily sized benchmarks.
- SuperstaQ enables write-once-target-all execution, allowing one OpenQASM circuit specification to run across multiple backends.
- The Closed Division restricts optimizations to establish a lower bound on performance achievable by a typical quantum programmer.
VI. RESULTS
Benchmark results show that performance generally declines as circuit width and depth increase, while hardware–circuit connectivity and mid-circuit operations strongly shape platform performance. Correlation analyses indicate that error-correction benchmarks, especially their measurement and reset demands, substantially affect feature–performance relationships.
- As benchmark width and depth increase, hardware scores generally decrease, although additional gates can be more detrimental than additional qubits.
- Mermin-Bell benchmarks can surpass the classical limit, but few processors meet that limit for the 4-qubit instance.
- IonQ performs best on Mermin-Bell because its native all-to-all connectivity matches the benchmark’s communication structure despite higher two-qubit gate error rates.
- Connectivity mismatches can require SWAP operations that deteriorate performance, whereas matching circuit and hardware connectivity enables competitive superconducting-QPU performance.
- The Figure 3 correlations use linear regression to relate each application feature to system performance, with separate heatmaps including or excluding error-correction benchmarks.
- Mid-circuit measurements strongly affect current performance, and excluding error-correction benchmarks improves correlations for many feature–QPU pairs.
VII. DISCUSSION
The benchmark results expose tradeoffs among quantum systems, show the impact of mid-circuit measurements, and identify compilation and cloud access as evaluation constraints. SupermarQ is designed to evolve with quantum hardware, applications, and software.
- IonQ’s better connectivity offsets lower two-qubit gate fidelities, while superconducting systems remain competitive through higher-fidelity entangling gates.
- Correlation results support quantitative profiling of quantum programs and highlight the outsized performance impact of error-correction measurements.The discussion identifies mid-circuit measurements as critical components of quantum error-correcting codes.
- Compiler-selected qubit mappings and swap insertions introduce performance variability, especially when ansatz connectivity mismatches superconducting hardware.Even systems with superior gate fidelities can be hampered by sub-optimal compilation.
- Queue wait times make variational-algorithm evaluations with more than 10s of iterations impractical when classical and quantum processors are not tightly coupled.The adoption and availability of tightly coupled hybrid quantum-classical systems is described as crucial for benchmarking full variational algorithms.
- Repeated benchmark evaluation is costly, motivating suites that maximize application coverage with as few applications as possible.The reported IonQ evaluation was restricted in shots because of result-collection costs.
- SupermarQ is intended to scale with future qubit counts, reflect changing applications and software, and support community contributions through open sourcing.The suite is based on real-world applications and is described as providing broad application-space coverage.
APPENDIX
The artifact packages SupermarQ’s source code, benchmark experiments, and performance-score workflow for reproducible use with specified hardware and execution resources.
- The artifact contains source code for generating, evaluating, and computing the benchmark scores.
- The artifact uses quantum circuit simulation instead of proprietary-hardware evaluations requiring valid access tokens.
- The documented setup uses a Jupyter kernel and notebook workflow on a 6-Core Intel Core i7 system.
- The artifact records benchmark performance scores for SupermarQ benchmark applications.
- The artifact directory and Python virtual environment require approximately 1 GB, with preparation taking 10 minutes and experiments 30 minutes.
- The artifact is publicly available under an Apache 2.0 license and archived at DOI 10.5281/zenodo.5786391.
C. Description
The artifact is distributed through Zenodo with installation instructions, a Jupyter notebook, and simulation-based examples for generating benchmarks and computing scores.
- The source code and artifact notebook are packaged in supermarq_hpca_ae.tgz and archived on Zenodo.
- Because the paper’s results require cloud access to quantum computers, the artifact uses Cirq-based circuit simulation for broader accessibility.Any system capable of running Python programs should be able to evaluate the artifact.
- The SupermarQ Python package must be installed using dependencies listed in requirements.txt.
- HPCA_Artifact.ipynb provides an overview of the benchmarks and figures used in the paper.
- The notebook demonstrates benchmark generation and score computation from circuit-execution results obtained through simulation.Its simulations use increasing noise to reflect NISQ execution, with expected decreases in benchmark scores as noise increases.