Source-linked AI summary
Benchmarking Quantum Computers and the Impact of Quantum Noise
Salonik Resch, Ulya R. Karpuzcu
TL;DR
Quantum benchmarking lacks a clear standard because hardware is diverse, applications remain uncertain, and quantum noise is difficult to model. The paper compares quantum benchmarking strategies under noise and finds that noise models and algorithm structure can drastically change measured performance, requiring cautious benchmark design.
Problem
Quantum benchmarking must evaluate diverse, immature systems despite uncertain useful applications and poorly understood noise.
Method
The paper quantitatively compares quantum strategies and benchmark circuits under multiple quantum-noise models.
Results
Noise model choice and algorithm structure drastically affect performance, with coherent noise damaging idle circuits immediately but being smeared out in random circuits.
Takeaways & Limitations
Benchmarking must jointly consider the noise model, target application, and performance metric.
Takeaways & Limitations
Quantum volume targets modern noisy systems without error correction and may need adaptation or replacement as error correction becomes central.
Abstract
from arXiv · showhide
Benchmarking is how the performance of a computing system is determined. Surprisingly, even for classical computers this is not a straightforward process. One must choose the appropriate benchmark and metrics to extract meaningful results. Different benchmarks test the system in different ways and each individual metric may or may not be of interest. Choosing the appropriate approach is tricky. The situation is even more open ended for quantum computers, where there is a wider range of hardware, fewer established guidelines, and additional complicating factors. Notably, quantum noise significantly impacts performance and is difficult to model accurately. Here, we discuss benchmarking of quantum computers from a computer architecture perspective and provide numerical simulations highlighting challenges which suggest caution.
1 Introduction
Quantum benchmarking inherits classical difficulties but adds hardware diversity, immature applications, limited system sizes, and difficult-to-model noise. The paper therefore presents quantitative comparisons to identify benchmarking pitfalls and argue for caution.
- Benchmark design: Benchmark choices can mislead because reduced programs introduce estimation error and fixed suites may encourage systems optimized for selected applications.The benchmark selection itself can influence what the community treats as important.
- Motivation: Quantum benchmarking is difficult because hardware is diverse, systems are small, applications are still emerging, and noise is difficult to characterize.These factors make metrics and benchmarks less portable and representative across quantum systems.
- Hardware diversity: Different physical platforms have distinct dynamics, advantages, and deficiencies, making direct comparison difficult.Examples include superconducting circuits, trapped ions, and silicon-embedded atoms.
- Scope: Exploratory quantum research is poorly captured by benchmarks because useful future applications remain unknown.The paper notes that developing a universal standard may be impossible or premature.
- Quantum noise: Noise affects quantum programs differently according to their length and structure, complicating performance quantification even for one-bit operations.Accurate noise modeling and operation-robustness assessment remain active research topics.
- Paper approach: The paper quantitatively compares quantum strategies under quantum noise to pinpoint benchmarking pitfalls and fallacies.It examines benchmarking from single- and two-qubit systems through near-term and fault-tolerant computing.
2 Quantum Primer
The primer introduces qubits, gates, measurements, mixed states, and fidelity as the concepts needed to understand quantum computation and its benchmarking. It emphasizes that quantum states and operations differ fundamentally from classical bits and irreversible measurements.
- Qubits: Qubits are two-level quantum systems that can occupy superpositions, while multiple qubits can become entangled.Qubits commonly use two states selected from a larger quantum system.
- Quantum states: Pure qubit states can be represented by complex-valued kets, with amplitude magnitudes determining measurement probabilities.A single qubit is visualized as a point on the Bloch sphere.
- Operations: Two-qubit gates use control and target qubits and can create entanglement.A controlled-NOT applies an X operation to the target when the control is in the |1⟩ state.
- Operations: Quantum gates are unitary matrix operations that reversibly transform quantum states, whereas measurements are non-unitary and irreversible.Single-qubit gates correspond to rotations on the Bloch sphere.
- Mixed states: Mixed states combine pure states with classical probabilities and are represented by density matrices.Density-matrix simulation tracks probabilities and errors but requires substantially more computational resources.
- Metrics: Quantum-state fidelity compares an actual state with a correct state, while average gate fidelity averages gate performance over possible input states.The Bloch-sphere representation illustrates why gate quality can depend on the input state.
3 Quantum Noise
Quantum noise is pervasive in practical quantum computing and must be considered when benchmarking or interpreting measured and simulated performance.
- Benchmarking implications: No benchmarking approach can be meaningful without considering quantum noise and its impact on measured or simulated results.Quantum noise is difficult to characterize accurately.
3.1 Physical Sources of Noise
Quantum noise arises from environmental interactions, unwanted qubit interactions, imperfect control, and leakage beyond the computational subspace. These sources produce different error characteristics and may be unitary or non-unitary.
- Noise sources: Quantum noise has multiple sources, including environmental interaction, qubit cross-talk, and imperfect control operations.The sources have different error characteristics and motivate different noise models.
- Interaction with the environment: Environmental interaction produces non-unitary noise and is characterized by coherence times such as T1 relaxation and T2 dephasing.Operations can increase environmental interaction because they require external driving.
- Interaction with the environment: External fields can cause unitary rotations of the quantum state.This is distinct from non-unitary environmental effects.
- Cross-talk: Accidental qubit interactions, called cross-talk, can correlate qubits and cause mixtures or decoherence.Cross-talk is particularly difficult to characterize.
- Control errors: Imperfect gate application can cause over- or under-rotations that coherently drive the state into an undesired state.These errors commonly result from imperfect calibration.
- Leakage: Leakage occurs when a system encoded as a qubit leaves the computational subspace of the larger quantum system.The paper categorizes physical noise sources by whether they are unitary or non-unitary.
3.2 Noise Models
Quantum noise models trade physical realism against simulation efficiency, and their assumptions can materially change benchmarking results. Stochastic Pauli noise is convenient but may be overly optimistic, while coherent noise can be especially damaging in repeated circuits.
- Modeling challenges: Accurate and efficient noise models are needed because quantum noise is difficult to model and publicly available systems have limited qubit counts.The modeling goal is to estimate how noise disrupts algorithm correctness during operation.
- Stochastic Pauli noise: Stochastic Pauli noise inserts X, Y, or Z errors probabilistically and is easy to simulate and correct.It is commonly used for environmental interactions modeled as effectively unintentional measurements.
- Stochastic Pauli noise: Pauli noise should be injected in every cycle because qubits can accumulate errors while idle, not only after gates.Studies find these models often produce inaccurate and overly optimistic results, though they can approximate logical-level errors under QEC.
- Model extensions: Clifford extensions add H, S, CNOT, and Pauli measurements to improve simulation accuracy while preserving efficient simulability.The enlarged gate set retains the random-insertion approach but captures more noise behavior.
- Coherent noise: Coherent noise can amplify across repeated gate sequences and has a higher worst-case error rate than stochastic noise.QEC can reduce coherence at the logical level, but using stochastic Pauli noise for physical noise can significantly underestimate logical error.
- Coherent noise: Randomized Compiling disrupts coherent noise and tailors it into stochastic Pauli noise, potentially without execution overhead.Modern quantum computers lack QEC, so this approach targets a setting where logical-level resilience is unavailable.
4 Metrics
Quantum metrics quantify different aspects of noisy operation quality, but no single gold-standard metric applies to every use case. Metrics range from output-distribution similarity to worst-case distinguishability of quantum processes.
- Metric selection: There is no unifying gold-standard quantum metric, so the preferred metric depends on the use case.Metric selection can mislead when it does not match the property being evaluated.
- State and process metrics: Process fidelity measures how similar a noisy output state is to the intended target and equals 1 for an error-free operation.Its physical interpretation is weaker when comparing two mixed states.
- Gate metrics: Average gate fidelity averages operation fidelity over possible input states, and equivalent averaging can use a finite input set.The metric accounts for the fact that fidelity may vary across the Bloch sphere.
- Distance metrics: Trace distance measures distinguishability between ideal and noisy output states, while diamond distance takes the maximum over inputs, including entangled ancillas.Diamond distance is used in rigorous fault-tolerance analyses.
- Distribution metrics: Hellinger fidelity compares a noisy operation’s measurement distribution with an ideal distribution using repeated Z-basis measurements.It is convenient experimentally when only output measurement probabilities matter.
5 Qubit Benchmarking
Qubit benchmarking seeks reliable operation-level estimates despite destructive measurements, SPAM errors, and rapidly growing tomography costs. Randomized Benchmarking provides a scalable, calibration-free alternative that estimates average gate error in computational contexts.
- Motivation: Average gate fidelity coarsely predicts how many gates can be applied before a quantum state becomes too corrupted.Operation reliability matters even in single- or two-qubit systems, where effective quantum error rates can exceed 1%.
- Measurement challenges: Destructive measurements require repeated state preparation or operation execution, while SPAM errors can obscure the operation’s error.These constraints complicate direct experimental determination of states and operation fidelity.
- Tomography: Quantum state tomography reconstructs a state from measurements in multiple bases, but its resources scale exponentially with system size.State reconstruction requires determining coefficients from X-, Y-, and Z-basis measurement probabilities.
- Tomography: Quantum process tomography identifies an operation by applying it to known inputs and tomographically analyzing the outputs, also requiring exponential resources.Gate set tomography is calibration-free but remains mainly a 1- and 2-qubit method because its resources grow exponentially.
- Fidelity estimation: Direct Fidelity Estimation extracts fidelity relative to an intended pure state without exponential overhead by estimating only the needed information.It avoids reconstructing all information about the state.
- Randomized Benchmarking: Randomized Benchmarking estimates error probability per gate in computational contexts and is calibration-free and insensitive to variation among gate types.It generates random gate sequences, appends an inverse operation, and averages survival probabilities across sequences of equal length.
- Randomized Benchmarking: Randomized Benchmarking scales polynomially with qubit count, but the commonly used Clifford gate set is not universal by itself.The fitted parameter r represents the average gate error, with A and B as fitting constants.
6 (Quantum) Computer Benchmarking
Near-term quantum benchmarking spans program performance, capability metrics, and operation-level characterization, but hardware diversity, scaling effects, unknown applications, and noise make results difficult to compare or extrapolate. The section therefore emphasizes representative, transparent benchmarks and cautions against universal metrics.
- 6.1 Program Benchmarks: Small benchmark programs may misestimate larger-system performance because scaling can increase noise susceptibility and introduce communication complexity.A small system’s success rate may differ significantly from that of a larger system, making extrapolation nontrivial.
- 6.1 Program Benchmarks: Program benchmarks measure a quantum system’s compute capability on selected circuits, but different architectures can excel on different programs.Higher connectivity favored the Toffoli circuit, while performance was more comparable on the Margolis circuit.
- 6.1 Program Benchmarks: Useful program benchmarks should reflect customer applications and disclose results across all programs rather than favoring selected outcomes.The text identifies invested interests as a concern when deciding which programs matter.
- 6.2 Quantifying Capability: Quantum capability benchmarking abstracts away architecture and specific algorithms to quantify general computational power, motivated by current machines’ limited useful applications.Cross-entropy benchmarking evaluates how closely sampled outputs match the predicted distribution of ideal random circuits.
- 6.2 Quantifying Capability: Cross-entropy difference α ranges from 0 for an uncorrelated distribution to 1 for a perfect distribution, and quantum supremacy requires a higher α than a classical computer.Direct evaluation becomes impractical at supremacy-scale circuits because the ideal probabilities require exponential classical simulation.
- 6.2 Quantifying Capability: Quantum volume uses random circuits, but differing error responses across circuit types and its potentially misleading results motivate more general volumetric benchmarks.Future benchmarking may need adaptation as error correction becomes relevant.
- 6.2 Quantifying Capability: Cycle Benchmarking characterizes the fidelity of operation cycles, allowing specific operations and arbitrary programs to be evaluated through finite cycle sets.This approach is similar to randomized benchmarking while targeting operations across all qubits.
7 Fault Tolerant (Quantum) Computer Benchmarking
Fault-tolerant quantum computers have not yet been built, so their benchmarking remains underdeveloped. Future evaluations are expected to resemble classical performance benchmarking while accounting for the substantial overhead of quantum error correction.
- 7 Fault Tolerant (Quantum) Computer Benchmarking: No fault-tolerant quantum computers have been built, leaving limited research on how to benchmark them.Near-term quantum benchmarking has focused mainly on research and scalability because current performance trails classical competitors.
- 7 Fault Tolerant (Quantum) Computer Benchmarking: Future fault-tolerant benchmarking will likely emphasize latency among processors with enough qubits to run the algorithm of interest.The section anticipates a shift toward performance-oriented, program-like benchmarks.
- 7 Fault Tolerant (Quantum) Computer Benchmarking: Quantum error correction imposes high overhead through additional qubits, repeated operations, and classical orchestration resources.Both the qubit chip and supporting classical architecture must be designed to support QEC.
8 Simulations
The simulations compare representative quantum algorithms and reference circuits under isolated noise models, using process fidelity or output accuracy to expose benchmarking sensitivities. They vary circuit structure, gate sets, noise assumptions, and randomized compiling under simplifying hardware assumptions.
- Metrics: Process fidelity is the default metric, while the QAOA experiment evaluates the expectation value of the output.The paper selects process fidelity because it is widely used in quantum information science, while noting that other metrics may also be suitable.
- Assumptions: The simulations assume all-to-all connectivity and full parallelism, eliminating movement overhead and allowing nonconflicting single-qubit gates to run simultaneously.Limited-connectivity machines would introduce additional complications from qubit movement.
- Noise mitigation: Randomized compiling is evaluated because it converts coherent noise into effectively stochastic Pauli noise and can mitigate coherent-noise effects.It is not intended to affect a Pauli noise model, and parameterized-rotation circuits cannot be randomly compiled.
- Noise models: Four isolated noise models are used to show how noise characteristics and modeling assumptions affect benchmark results.The models include equally likely Pauli errors, with noise inserted on every qubit in every cycle regardless of whether it is operated on.
9 Results
The simulations show that benchmark outcomes depend strongly on both the noise model and circuit structure. Randomized Compiling helps mainly with coherent noise, while other noise types and circuit choices can produce different results.
- Circuit dependence: Even slight coherent noise immediately damages Idle-circuit fidelity, whereas Randomized Circuits can smear out its effect into exponential decay.This difference is consistent with coherent noise affecting idle or cyclic circuits differently from randomized circuits.
- Noise-model dependence: Noise-model choice has a drastic impact on quantum-algorithm performance, making its physical assumptions critical for benchmarking.The paper also finds that algorithm structure determines how strongly noise affects measured performance.
- Randomized Compiling: Randomized Compiling significantly increases fidelity under coherent noise, but its benefit depends on the circuit and noise type.The addition, QFT, and related circuit results include cases where coherent noise is mitigated by randomized compilation.
- Randomized Compiling: Randomized Compiling provides no significant improvement under Pauli noise or Amplitude Damping.Pauli noise is already random, while Amplitude Damping models energy loss rather than coherent rotations.
- Randomized Compiling: Randomized Compiling converts coherent noise to stochastic Pauli noise but cannot serve as a generalized noise-mitigation technique.The conversion preserves the validity of proofs based on Pauli-noise assumptions.
- QAOA: For Clifford+T QAOA, increasing noise drives the expectation value toward the random-guess level, even at very low noise.Amplitude Damping can push performance below 50% of the maximum, while Randomized Compiling prevents that drop in the reported case.
- QAOA: Parameterized-rotation QAOA is shorter and less noise-affected, but its gate set is incompatible with Randomized Compiling and quantum error correction.The shorter circuit remains feasible for the small cases studied but is not scalable under the paper’s stated constraints.
10 Conclusion
The paper concludes that quantum benchmarking is complicated by noise models, target applications, and performance metrics. Simplifying assumptions can produce inaccurate results across the system stack.
- Conclusion: Quantum noise is more complex and difficult to model than often assumed, complicating benchmarking at higher system-stack levels.The paper emphasizes that quantum systems can defy abstraction and simplifying assumptions.
- Conclusion: Meaningful benchmarking requires careful consideration of the noise model, target application, and performance metric.