PaperlayerView exact revision on arXiv

Source-linked AI summary

Application-Oriented Performance Benchmarks for Quantum Computing

Thomas Lubinski, Sonika Johri, Paul Varosy, Jeremiah Coleman, Luning Zhao, Jason Necaise, Charles H. Baldwin, Karl Mayer, Timothy Proctor

arXiv:2110.03137v3quant-ph

TL;DR

Current quantum computers require robust benchmarks and metrics to assess their capabilities amid complex errors. The paper introduces application-oriented volumetric benchmarks that measure result fidelity and quantum execution time across problem sizes, finding that quantum volume broadly predicts application performance on tested devices but with device-dependent reliability.

  • Problem

    Current quantum computers experience a wide range of complex errors, creating a need for robust performance benchmarks and metrics to understand near-term hardware capabilities.

  • Method

    The paper develops an application-oriented benchmark suite based on well-known quantum algorithms, mapping result quality across circuit width and depth while measuring result fidelity and quantum execution time.

  • Results

    Quantum volume was broadly predictive of application performance on tested devices, but the reliability of extrapolating it varied between devices.

  • Takeaways & Limitations

    The suite provides users with proxies for application quality and time to solution while supporting diverse benchmarks as quantum applications and algorithms evolve.

  • Takeaways & Limitations

    Average result fidelity can be misleading for some benchmarks, with device behavior differing from the normalized fidelity metric under depolarizing errors.

Abstract

from arXiv · show

In this work we introduce an open source suite of quantum application-oriented performance benchmarks that is designed to measure the effectiveness of quantum computing hardware at executing quantum applications. These benchmarks probe a quantum computer's performance on various algorithms and small applications as the problem size is varied, by mapping out the fidelity of the results as a function of circuit width and depth using the framework of volumetric benchmarking. In addition to estimating the fidelity of results generated by quantum execution, the suite is designed to benchmark certain aspects of the execution pipeline in order to provide end-users with a practical measure of both the quality of and the time to solution. Our methodology is constructed to anticipate advances in quantum computing hardware that are likely to emerge in the next five years. This benchmarking suite is designed to be readily accessible to a broad audience of users and provides benchmarks that correspond to many well-known quantum computing algorithms.

I. INTRODUCTION

Quantum hardware benchmarks need to reflect application performance because component-level metrics and single holistic metrics have limited predictive power. The paper introduces an extensible, open-source application-oriented suite that evaluates result quality and time to solution across scalable algorithmic circuits.

  • Motivation: Quantum volume summarizes device capability in one number, but it is not guaranteed to predict performance for all application circuit shapes.The cited limitation includes wide and shallow circuits as well as deep and narrow circuits.
  • Contribution: The paper introduces an extensible suite of application-oriented benchmarks based on diverse quantum algorithms and applications.Each benchmark specifies a scalable family of quantum circuits and complements other benchmarking methods based on small chemistry problems or basic circuits.
  • Accessibility: The benchmark suite is publicly open source, documented, implemented in several quantum programming languages, and deployable on major cloud quantum computers.Validated implementations are supplied in Qiskit, Cirq, Braket, and Q#.
  • Method: The suite measures proxies for application quality and time to solution using volumetric benchmarking across circuit width and computation length.A normalized result-quality measure is defined so results from different applications can be compared, alongside a normalized device-independent measure.
  • Scope: The first version is intended to evolve as weaknesses are identified and quantum hardware changes, following the historical evolution of classical benchmark suites.The authors explicitly highlight limitations of the initial suite and invite contributions from the research community.
  • Motivation: Component-level metrics expose hardware errors but often cannot accurately predict algorithm performance or be readily interpreted by non-specialists.Summary metrics may miss complex component interactions and errors such as crosstalk.

C. Volumetric Benchmarks

Volumetric benchmarking maps application performance across circuit width and depth, supporting diverse algorithm-derived benchmark families rather than only square random circuits. The suite defines circuits, selection procedures, and analysis methods for scalable, application-oriented evaluation.

  • Volumetric benchmarking: Volumetric benchmarking plots result quality for tested circuits as a function of circuit width and depth.The framework supports wide shallow, narrow deep, and square circuits.
  • Benchmark construction: Each benchmark begins with an algorithm or application and specifies circuits for each variable input problem size.The suite initially groups algorithms into tutorial, subroutine, and functional categories.
  • Benchmark construction: Benchmark definitions include circuit generation, circuit selection, problem-size variation, and data-analysis methods.These components turn an algorithm into an executable benchmark rather than treating the algorithm itself as a complete specification.
  • Scope and diversity: The suite includes subroutines because shared components such as the QFT can be valuable to benchmark even when isolated instances are classically simulable.The authors also emphasize diversity because future practical applications and algorithms remain uncertain.
  • Scope and limitations: Initial benchmarks trade off feasibility of implementation and analysis against application-representative behavior, and not all satisfy both criteria.For example, QFT(1) has efficiently computable outputs but does not test QFT action on entangled inputs, while other benchmarks require exponentially expensive classical simulations.

C. The Benchmarking Procedure

The procedure sweeps problem sizes, generates and compiles benchmark circuits, executes them on the target hardware, and measures result fidelity and execution time. Results are organized by circuit width and normalized depth, with volumetric backgrounds providing independent predictions for comparison.

  • Execution procedure: Each benchmark execution chooses problem sizes, generates selected circuits, executes them, and measures result fidelity and execution time.The execution loop nests problem-size, circuit, and parameter sweeps.
  • Volumetric analysis: Volumetric plots place each circuit family in a width–depth space and summarize fidelity across problem sizes.Figure 6 illustrates average fidelity, algorithmic depth, normalized depth, and the resulting volumetric footprint.
  • Volumetric analysis: A volumetric background predicts result fidelity independently of the application benchmarks, commonly using a binary success threshold of at least 1/2.The paper usually extrapolates this background from device quantum volume.
  • Volumetric analysis: Quantum-volume-based backgrounds use the heuristic success condition wd < (log2(VQ))2 for a circuit of width w and depth d.This prediction requires extrapolation and is not expected to accurately describe every circuit.
  • Circuit depth: Normalized depth is obtained by transpiling high-level circuits into a device-independent standard gate set before counting layers.The standard basis includes single-qubit rotations, CNOT, and measurement gates, while hardware execution may add SWAP gates and increase physical depth.

G. Quantifying Result Quality with Circuit Fidelity

The suite quantifies application output quality with normalized classical fidelity between ideal and observed probability distributions, typically averaged over circuit ensembles. This metric is useful for comparing degradation toward uniform randomness but requires carefully chosen circuits and efficient classical reference calculations.

  • Result fidelity: Result fidelity quantifies discrepancies between ideal and observed output probability distributions for each circuit.The metric compares Pideal with Poutput and is typically averaged over a circuit ensemble.
  • Metric design: Circuits with Fs(Pideal, Puni) ≈1 should be avoided because errors become difficult to observe when the ideal output is close to uniform.The QFT(1) benchmark follows QFT with inverse QFT so the error-free output is a computational-basis state.
  • Practical constraints: Direct fidelity estimation requires classical simulation of Pideal and Nshots hardware executions to estimate Poutput.Both simulation cost and the shots needed for low-uncertainty estimates can grow quickly with qubit number, motivating efficiently simulable benchmark circuits.
  • Metric choice: The suite uses normalized fidelity because standard fidelity remains nonzero for completely random output and depends on the ideal distribution’s sparsity.Normalized fidelity instead assigns F(Pideal, Puni) = 0.
  • Limitations: The normalized metric has weaknesses: its raw form can be negative, its maximum correction is ad hoc, and it does not replace process-level metrics.The authors anticipate adding alternative result-quality and process-fidelity techniques in future work.

H. Implementation Rules

The suite leaves implementation choices flexible while recommending practices for comparability and acknowledging unresolved risks from compilation and error mitigation. Simulator results show close agreement across application benchmarks under a simple error model, while more complex errors and limited connectivity can produce application-dependent performance differences.

  • Implementation parameters: Benchmark comparisons should use identical problem sizes n across processors, or a principled stated methodology for selecting n.One suggested strategy increases n until fidelity falls below 10% or the available qubit limit is reached.
  • Implementation parameters: At least Ncircs ≥10 circuits are recommended per problem size, although experiments used Ncircs = 3 because of hardware throughput limitations.The same circuits were run across tested devices using a single random seed for repeatability.
  • Compilation and mitigation: Logical-equivalent compilation is permitted to reflect end-to-end provider performance, but efficiently classically simulatable circuits can be compressed or offloaded to a classical co-processor.The authors identify compilation barriers as a promising future way to constrain such gaming without forbidding all compilation.
  • Compilation and mitigation: Error-mitigation strategies are difficult to constrain because rules must exclude unreasonable substitutions while preserving room for innovation.Replacing experimental data with perfect-computation simulation is described as infeasible for useful computations.
  • Simulator results: In simulator plots, application-benchmark fidelities closely agree with one another and with the volumetric background under simple errors and all-to-all connectivity.The two simulator cases use quantum volumes of 32 and 2048; the background heuristic predicts regions with fidelity above 1/2.
  • Simulator results: Complex errors and limited connectivity can make same-shaped circuits perform differently and cause application-dependent fidelity declines as circuit width increases.Transpilation to primitive gates can substantially increase depth, with the increase depending on the application and device connectivity.

B. Benchmark Results on Quantum Hardware

The suite is demonstrated across multiple quantum hardware platforms using application-oriented circuits, with fidelity generally declining as circuit width or depth increases. Results also show that mid-circuit measurement can improve fidelity by reducing circuit width, while volumetric backgrounds are broadly but not uniformly predictive.

  • Overview: Application-oriented benchmarks were executed on multiple providers’ hardware to demonstrate the suite rather than compare systems.The authors caution that hardware changes rapidly and cross-system comparisons could quickly become misleading.
  • Rigetti Aspen-9: On Rigetti Aspen-9, QFT(2) fidelity became negligible at around 5–6 qubits, motivating execution on only the first 6 qubits.The Hidden Shift benchmark used the first 8 qubits, while QFT(2) used the first 6.
  • Width and connectivity: Result fidelity generally decreases with circuit width, partly because limited connectivity requires non-connected controlled rotations to be decomposed into multiple two-qubit gates.The benchmark circuits are not tailored to each device’s best-connected qubits, approximating a user’s experience on qubits 1 through n.
  • Mid-circuit measurements: On IBM Q Lagos, Bernstein-Vazirani(1) fidelity dropped precipitously above width 5, whereas Bernstein-Vazirani(2) kept width 2 and achieved larger fidelity for the same problem instances.The second implementation reuses one qubit through mid-circuit measurement and reset, allowing fixed width for increasing input size.
  • Quantinuum H1.1: On Quantinuum H1.1, Bernstein-Vazirani(1), Hidden-Shift, and Hamiltonian Simulation achieved high fidelity (F̄ ≥0.77) through 12 qubits, while deeper benchmarks remained distinguishable from random outputs.The volumetric background extrapolated from quantum volume 1024 was broadly, but not entirely, predictive; the smallest QFT(1) fidelity was F̄ = 0.07 at n = 12.
  • IonQ hardware: IonQ’s next-generation hardware showed a significant performance increase over the previous generation, with Bernstein-Vazirani(1) reaching 70% fidelity at 21 qubits.The earlier cloud system reported 78% fidelity for its largest 11-qubit Bernstein-Vazirani(1) circuit, so the authors present the comparison alongside different widths.

C. Comparison to Generic Benchmarks

Generic benchmarks and quantum volume do not consistently predict application performance because algorithmic circuits differ in physical-gate requirements and device connectivity. Application-oriented results can therefore diverge substantially from generic volumetric backgrounds.

  • Randomized-mirror backgrounds can reasonably predict application performance on some devices, particularly those with all-to-all connectivity, but not universally.
  • Quantum-volume and randomized-mirror extrapolations are not consistently accurate predictors of diverse algorithmic benchmark performance.The paper attributes this to complex error interactions, connectivity, and the difficulty of predicting varied algorithms from a single generic metric.
  • Application benchmark fidelities decrease at different rates as circuit size increases, unlike the shared prediction implied by a generic background.
  • Randomized mirror circuits incur no CNOT transpilation overhead, making their predictions systematically over-optimistic as application widths grow.Their physical CNOT density can differ from algorithmic benchmarks and varies across algorithms.
  • Using physical depth rather than nominal depth could potentially reduce discrepancies between generic volumetric backgrounds and application results.

D. Impact of Compiler Optimization Techniques

Compiler and circuit-design optimizations improve benchmark fidelity by adapting layouts to device connectivity and reducing costly two-qubit interactions. The gains are especially visible for wide, shallow circuits and QFT benchmarks.

  • Compiler optimization significantly improves fidelity for wide, shallow circuits by optimizing circuit layout for device connectivity.The Guadalupe demonstration used Qiskit optimization_level = 3.
  • QFT benchmarks on Quantinuum H1.1 improve when arbitrary-angle two-qubit gates reduce interaction counts and rotation angles.
  • 3× improvement raised QFT(1) fidelity at n = 12 qubits from 0.07 without arbitrary-angle gates to 0.216 with them.

V. MEASURING EXECUTION TIME

The suite treats execution time as a multidimensional performance concern in hybrid quantum-classical applications, while its current measurements focus on provider-reported quantum execution time. Experiments show strong system- and simulator-dependent scaling with circuit width.

  • Total execution time: Total application time includes compilation, classical computation, quantum execution, and cloud queuing components.The suite currently reports only quantum execution time, while queuing can complicate reliable timing measurements.
  • Quantum execution time: Quantum execution time tquantum(N) is the time required for N shots, and experiments use N = 1000 without studying dependence on N.The same N should be used when comparing devices.
  • Total execution time: The VQE benchmark enables comparison between recompiling each parameterized circuit and updating one compiled circuit through low-level classical controls.Avoiding recompilation could potentially produce substantial compute-time savings.
  • Measurement caveats: Provider timing APIs may expose quantities analogous to quantum execution time whose precise meanings are unclear or differ between providers.
  • Quantum execution time: QFT(1) execution time increases exponentially with width on the noisy simulator but linearly on IBM Q Guadalupe.The simulator’s scaling reflects its simulation behavior and does not by itself establish classical hardness of the circuits.
  • System differences: IBM Q Casablanca shows an approximately width-independent minimum execution time of ∼7 seconds, whereas Rigetti Aspen-9 has a minimum of about 4 seconds and different scaling.

VI. SUMMARY AND CONCLUSIONS

The paper presents an open, application-oriented benchmark suite that measures result fidelity and quantum execution time across scalable algorithmic circuits. The suite is intended to clarify current hardware capabilities, track progress, and evolve as its limitations and new needs emerge.

  • The suite benchmarks well-known quantum algorithms across input sizes, measuring result fidelity and quantum execution time as proxies for quality and time to solution.
  • Some benchmarks require exponentially expensive classical computations and will be infeasible for O(100) qubits.Efficiently simulable alternatives must be designed carefully so benchmark performance remains indicative of the target algorithm.
  • Execution-time measurements are currently rudimentary because provider definitions vary and classical co-processing, such as compilation, is omitted.
  • Volumetric plots summarize result quality as a function of circuit width and depth, enabling comparisons across applications and performance metrics.
  • Quantum volume is broadly predictive of application performance on tested devices, but extrapolation reliability varies between devices.
  • Substantial advances are required for tested quantum hardware to outperform classical counterparts on useful tasks, while the suite can track progress toward useful quantum advantage.
  • The open-source suite is designed to help users assess cloud-accessible machines and reduce barriers to understanding their application capabilities.

Appendix A: Selected Algorithms and Applications

The appendix converts oracle and QFT algorithms into scalable benchmarks, using circuit width, depth, and output fidelity to characterize performance. The selected constructions include three oracle algorithms and two QFT-based benchmarks that can also cross-validate one another.

  • Shallow Oracle-Based Algorithms: The benchmark suite includes Deutsch-Jozsa, Bernstein-Vazirani, and Hidden Shift oracle algorithms.Each uses a secret integer or function encoded through an oracle operation.
  • Shallow Oracle-Based Algorithms: Oracle benchmarks initialize qubits in superposition, encode information through relative phases, and reveal the result after transforming back to the computational basis.The circuits are executed repeatedly with random oracle integers at widths from 2 to 12 qubits.
  • Shallow Oracle-Based Algorithms: The Hidden Shift circuit is nearly twice as deep as the Bernstein-Vazirani and Deutsch-Jozsa circuits because it applies two oracle functions sequentially.The two oracle applications differ by a hidden integer shift.
  • Quantum Fourier Transform: QFT(1) applies QFT, a Fourier-basis +1 modulo 2^n operation, and inverse QFT so that input x should produce x+1 modulo 2^n.The intermediate operation prevents straightforward compilation of the circuit into an identity.
  • Quantum Fourier Transform: QFT(2) prepares a Fourier-basis state directly and applies only inverse QFT, giving approximately half the depth of QFT(1) at fixed n.The two benchmarks can cross-validate performance when QFT and inverse-QFT errors are approximately equal.
  • Quantum Fourier Transform: Simulator results for QFT(1) and QFT(2) show the expected relationship between their fidelities under the simulated depolarizing-error model.

3. Grover’s Search Algorithm

The suite benchmarks Grover search, phase estimation, amplitude estimation, and Hamiltonian simulation as application-oriented workloads. Their simulator results expose how circuit depth, subroutine structure, and classical reference-cost requirements affect benchmark scaling and fidelity.

  • Grover’s Search Algorithm: Grover’s Search benchmark marks random bit strings with phase −1 and uses repeated oracle-based amplification to identify the marked item.Multiple random instances are constructed at each circuit width.
  • Grover’s Search Algorithm: Grover benchmark depth increases rapidly with circuit width, accompanied by a rapid drop in result fidelity.
  • Phase and Amplitude Estimation: Phase Estimation measures phases of a chosen unitary to precision 2^-k, using k register qubits and randomly selected compatible phase values.
  • Phase and Amplitude Estimation: Amplitude Estimation uses amplitude amplification followed by phase estimation, producing deeper circuits than Phase Estimation.A quantum volume of 32 is reported as insufficient for high-fidelity standard-form Amplitude Estimation even for tiny instances.
  • Phase and Amplitude Estimation: Phase Estimation depth scales similarly to the QFT-based benchmarks because QFT is its deepest subroutine.
  • Hamiltonian Simulation: The Hamiltonian Simulation benchmark evolves a disordered antiferromagnetic Heisenberg chain with a Trotterized circuit.Its circuit depth does not increase with width for the chosen Hamiltonian, while error-free reference outputs require exponentially scaling classical computation.

6. Monte Carlo Sampling

The appendix describes Monte Carlo Sampling and VQE benchmarks that translate estimation and electronic-structure applications into executable circuit families. The constructions vary oracle complexity, molecular problem size, and the quantum portion of the VQE workflow.

  • Monte Carlo Sampling: The Monte Carlo Sampling benchmark estimates the expected value of f(X) by encoding p(X) and f(i) into quantum amplitudes before applying amplitude estimation.
  • Monte Carlo Sampling: Two Monte Carlo variants trade generality for simpler circuits: one allows user-chosen f and p, while the other fixes uniform p and parity f.The specialized variant uses Hadamard gates for the distribution oracle and CNOT gates for the function oracle.
  • Variational Quantum Eigensolver: The VQE benchmarks target the NaH molecule using a unitary-CCSD ansatz mapped to Pauli operators through the Jordan-Wigner transformation.The Hamiltonian and ansatz are represented as weighted sums of Pauli words after transformation.
  • Variational Quantum Eigensolver: VQE iteratively evaluates and optimizes energy, but VQE(1) implements only the quantum portion by randomly sampling wavefunction parameters with a fixed measurement basis.
  • Variational Quantum Eigensolver: VQE(1) varies qubit count by changing the number of orbitals in NaH and is simulated from 4 to 10 qubits.The 4-qubit case represents 2 electrons and 4 spin orbitals.

8. Shor’s Order Finding

The Shor’s Order Finding benchmarks isolate the quantum period-finding subroutine underlying factoring. Two variants represent different circuit formulations, including one that uses mid-circuit measurements to reduce resource requirements.

  • Shor’s Order Finding: Shor’s factoring algorithm combines quantum order finding with a classical procedure that uses the order to determine factors in polynomial time.The quantum routine may need repetition if it produces an invalid order.
  • Shor’s Order Finding: The benchmark targets functions f(x) = ax mod N, where the order r satisfies a^r mod N = 1.The benchmark generates random r and N values and analytically calculates a corresponding base a.
  • Shor’s Order Finding: Shor’s Order Finding(1) follows the standard formulation and requires 4n + 2 qubits for an n-bit problem instance.
  • Shor’s Order Finding: Shor’s Order Finding(2) uses a Shor variant with mid-circuit measurements to reduce the benchmark’s resource requirements.

Appendix B: Limitations of the Average Result Fidelity

Average result fidelity can misrepresent application correctness because it may weight distinct inputs asymmetrically and obscure algorithm-specific behavior. The suite also faces implementation and evolution challenges across APIs and changing hardware capabilities.

  • Average Fidelity: QA and QB perform equally badly on Deutsch-Jozsa, yet their average fidelities are 0.5 and 0, respectively.QA always returns 0000..., while QB always returns 1000..., producing different fidelity averages despite matching correctness across the two oracle types.
  • Average Fidelity: Average result fidelity for one algorithm input does not directly equal the probability of obtaining the correct result.The paper recommends complementing result fidelity with an algorithm-specific metric for result quality.
  • Average Fidelity: IBM Q Guadalupe’s average fidelity decays quickly toward ∼0.5 with circuit width before decaying slowly, indicating asymmetric oracle performance.The asymmetry arises from differing behavior on balanced and constant Deutsch-Jozsa oracles and requires careful interpretation of averages.
  • Implementation Constraints: Complete consistency across quantum APIs is difficult because permitted gates, topology requirements, transpiled-circuit access, and execution-time metrics differ.The suite therefore uses Qiskit for all simulator and most hardware executions, with Rigetti experiments using Braket.
  • Future Evolution: The benchmark suite is intended to evolve as weaknesses emerge and quantum hardware advances over successive technology cycles.The authors explicitly frame the reported version as an initial suite whose limitations suggest future enhancements.

1. Mid-Circuit Measurements

Emerging hardware features can change how application benchmarks represent problem size, circuit structure, and execution time. Mid-circuit measurement reduces qubit requirements, while parameterized circuits, controlled gates, and tighter classical–quantum integration expose additional performance constraints.

  • Mid-Circuit Measurements: Mid-circuit measurement, qubit reset, and reuse enable algorithms to use fewer qubits while maintaining computation.The Bernstein-Vazirani implementation encodes a 12-bit secret string with two qubits instead of 13, with significantly greater measurement fidelity.
  • Mid-Circuit Measurements: The Bernstein-Vazirani(2) benchmark uses two qubits with mid-circuit measurements to implement the computation of a standard five-qubit circuit.This circuit is the basis for a qubit-equivalent benchmark instance.
  • Circuit Parameterization: Parameterized VQE and QAOA circuits require repeated executions with modified rotation angles, adding an iteration layer to application execution.The current VQE benchmarks create, compile, and execute a new circuit for each angle setting rather than using parameterized circuit representations.
  • Circuit Parameterization: Circuit preparation can take seconds to minutes for larger circuits, so repeated compilation and loading substantially affects applications requiring hundreds or thousands of iterations.Future control-system improvements could make latency between parameterized executions negligible.
  • Controlled Gates: n-qubit Toffoli gates cause rapidly increasing transpiled depth because current hardware does not implement multiply-controlled gates natively.This effect appears in the volumetric positioning profile for Grover’s Search as circuit width increases.
  • Classical–Quantum Integration: The benchmark suite does not yet exploit tightly integrated classical–quantum processing or Qiskit Runtime, and requires updates to test those capabilities fully.Execution-time reporting currently includes quantum-specific time, circuit creation, launch, and queue times.
Loading 2110.03137v3…