PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Capabilities of Quantum Computers

Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen, Robin Blume-Kohout

arXiv:2008.11294v2quant-ph

TL;DR

Quantum-processor errors make it difficult to predict which programs hardware can run successfully. This paper develops circuit-mirroring benchmarks and applies them across processors, finding that processor-specific layer-set choices complicate comparisons while instability varies substantially across devices. The results support benchmarking concrete circuits with suites designed around their contexts and structures.

  • Problem

    Existing processor comparisons are limited because layer-set choices can make performance comparisons unfair and processor instability can vary substantially across devices.

  • Method

    The paper develops mirroring transformations that convert user-specified circuits into benchmarking suites and analyzes them with statistically rigorous instability tests.

  • Results

    All eight analyzed processors showed statistically significant evidence of instability between two circuit passes, with instability magnitude varying dramatically across processors.

  • Takeaways & Limitations

    Mirror benchmarks can be adapted to probe concrete circuit performance, but processor comparisons require multiple benchmarks because no single layer-set choice is uniquely fair.

Abstract

from arXiv · show

A quantum computer has now solved a specialized problem believed to be intractable for supercomputers, suggesting that quantum processors may soon outperform supercomputers on scientifically important problems. But flaws in each quantum processor limit its capability by causing errors in quantum programs, and it is currently difficult to predict what programs a particular processor can successfully run. We introduce techniques that can efficiently test the capabilities of any programmable quantum computer, and we apply them to twelve processors. Our experiments show that current hardware suffers complex errors that cause structured programs to fail up to an order of magnitude earlier - as measured by program size - than disordered ones. As a result, standard error metrics inferred from random disordered program behavior do not accurately predict performance of useful programs. Our methods provide efficient, reliable, and scalable benchmarks that can be targeted to predict quantum computer performance on real-world problems.

B. Desiderata for benchmark circuits

A useful benchmark must evaluate representative quantum programs while remaining classically verifiable. The paper therefore specifies fully contextualized test suites whose outcomes can predict performance on related use cases.

  • The benchmark should measure concrete circuit execution rather than only distributions or unitary transformations.
  • Running an exemplar circuit alone is insufficient because its output may be impractical to verify classically.Interesting circuits may produce non-classically simulable results or solve problems outside NP.
  • Benchmark suites must fully specify initialization and measurement contexts around embedded subroutines.
  • Each benchmark circuit needs an easy-to-simulate target output so processor performance can be measured directly.
  • Success probabilities across the suite should represent how the target circuit performs in contexts where it may be used.

C. Mirror circuit benchmarks

Mirror benchmarking transforms user-specified circuits into efficiently analyzable suites while preserving their relevant structure and context. The framework supports randomized and periodic variants across circuit sizes, depths, and layer definitions.

  • Mirroring generates benchmark suites from user-specified circuits and supports randomized and periodic mirror circuits.
  • The framework is intended to apply broadly across quantum circuits, with methods described for general quantum-computer use.
  • The experiments used each processor’s published error rates for prediction and analyzed randomized mirror circuits in experiment #1.
  • A circuit consists of ordered logic layers from a layer set, with strict barriers preventing layers from being compiled together.
  • Circuit width is the number of qubits, depth counts non-overhead logic layers, size is wd, and shape is (w, d).
  • Benchmark depth excludes a benchmark-wide constant number of overhead layers from full depth.In the experiments, d = d0 − 5 because initialization, readout, and three extra logic layers are overhead.

B. Layer sets for benchmark circuits

A benchmark’s layer set determines both which circuits it can contain and how their depth is measured. Choosing standardized or processor-specific layers therefore changes what performance the benchmark probes.

  • A canonical layer set determines the circuits constructible in a benchmark and how their depth is calculated.
  • The layer-set choice is significant because circuit content affects the measured performance aspect, while depth rules affect its quantification.
  • Standardized layer sets allow broad connectivity but can penalize processors with lower connectivity through additional gate decomposition.
  • Native layer sets correlate with native-gate error rates but do not capture usefulness, connectivity limits, or parallelism constraints.
  • Different algorithmic circuits can incur different connectivity overheads, making standardized-layer calibration difficult.

C. The layer sets for experiments #1 and #2

The experiments use processor-aware layer sets and mirroring transformations to benchmark circuits under native connectivity while retaining a general transformation framework. This choice avoids connectivity complexity but limits direct fairness comparisons.

  • The experiments use each processor’s native entangling gate, selected single-qubit Cliffords, and directed connectivity constraints.
  • No parallelization constraint is imposed on benchmark layers, even when hardware executes their gates serially.
  • Processor-specific layer sets can make comparisons unfair because removing poorly performing connections can improve measured benchmark performance.
  • No single layer-set choice uniquely compares processors fairly, so multiple benchmarks are needed to explore their complex performance characteristics.
  • Mirroring transforms arbitrary circuits into related benchmarking suites, and the experimental constructions use self-inverse layer sets.

A. Circuit mirroring as used in our experiments

SCC mirroring transforms Clifford subroutines into efficiently verifiable benchmark ensembles that probe performance across randomized contexts. The resulting tests expose errors, estimate average performance, and cannot efficiently capture worst-case behavior across all contexts.

  • SCC mirroring: SCC mirroring maps a Clifford subroutine C to an ensemble S(C) of complete, efficiently verifiable benchmark circuits.The procedure is designed to probe C as a subroutine rather than only in one fixed initialization and readout context.
  • Efficient verification: Each benchmark circuit has an easily computed deterministic target because Clifford and Pauli operations reduce the central circuit to a Pauli operation.The target Pauli layer can be calculated efficiently on a conventional computer, yielding a unique target bit string.
  • Error sensitivity: For any nonidentity error map, at least one circuit in S(C) has success probability S < 1, so the ensemble can witness contextual implementation errors.The benchmark is therefore sensitive to errors that might not appear in every tested context.
  • Interpretation: The expected success probability over S(C) is closely related to the process fidelity and approximately probes performance on C in a uniformly random context.Its average performance is also closely related to the average fidelity with which the processor implements C.
  • Scope: Worst-case performance over all contexts cannot be extracted efficiently in w because the number of possible contexts is eO(w).SCC mirroring instead provides a natural and achievable representative-performance notion rather than an exhaustive worst-case guarantee.
  • Extensibility: The mirroring transformations can also generate benchmarks with different properties and extend beyond Clifford circuits, although the reported experiments use SCC mirroring exclusively.This establishes extensibility beyond the specific benchmark construction used in the main experiments.

B. Transformation 1: simple circuit mirroring

Simple circuit mirroring embeds a circuit and its inverse in one complete program with an efficiently simulable all-zeros target. However, using only one input state, measurement basis, and context makes it insensitive to several coherent and state-dependent errors.

  • Construction: Simple circuit mirroring maps a (w, d) QI/QO circuit into one FI/CO circuit of shape (w, 2d + 2).The construction appends initialization, the circuit, its reversed layerwise inverse, and readout.
  • Verification: An error-free mirror circuit deterministically returns the all-zeros bit string, making its target output efficiently simulable.The inverse circuit implements the inverse unitary of the original circuit.
  • Limitations: Simple mirroring can hide systematic coherent errors because errors in C may exactly cancel errors in C−1.In the example Λ(C) = V and Λ(C−1) = V−1, the success probability can reach S = 1 despite imperfect implementation of C.
  • Limitations: Because C always receives |0⟩w and is measured in the computational basis, simple mirroring misses errors that have no observable effect in that setting.Examples include errors that do not affect |0⟩w and dephasing or coherent z-axis errors after projection.
  • Extensions: Three additional circuit transformations place C in a wider range of contexts by altering and typically enlarging the simple-mirroring benchmark suite.These configurable transformations address limitations caused by running C in only one context.

C. Transformation 2: inserting a central subroutine

Central subroutine insertion places diverse, shallow, efficiently verifiable circuits between C and its quasi-inverse to prevent systematic error cancellation. Pauli subroutines additionally provide sensitivity to small errors, faithful infidelity estimates, calculable and unbiased targets, and low-depth implementation.

  • Motivation: Central subroutine insertion inserts QI/QO circuits between C and C−1, generating a larger suite that can be run exhaustively or sampled.Its purpose is to prevent systematic errors in C and C−1 from canceling.
  • Design criteria: The subroutine set must be diverse, efficiently verifiable, and shallow enough that the expanded circuits remain practical to run.These are the three stated competing criteria for choosing A.
  • Pauli subroutines: For Clifford circuits, choosing all w-qubit Pauli layers makes nearly all small errors detectable; only large, discrete, unlikely errors remain undetected.The Pauli group does not span full superoperator space, but its structure constrains undetected errors to Pauli-equivalent implementations.
  • Pauli subroutines: Uniform Pauli insertion reduces error maps to stochastic Pauli channels, making benchmark fidelity closely approximate the product of the fidelities of C and C−1 for small errors.This provides faithfulness in infidelity, not merely detectability.
  • Practical properties: Pauli subroutines yield efficiently calculable, uniformly random target bit strings and can be implemented with low-depth native circuits.Uniform target strings prevent readout bias from artificially changing average success probabilities.
  • Quasi-inversion: Quasi-inversion replaces C−1 with related inverse circuits to reduce correlations between layers, with transformations available for Clifford and non-Clifford circuits.The experiments use both the trivial inverse and Pauli-equivalent quasi-inverses for Clifford layers.

2. The circuit sampling of experiment #2

Experiment #2 samples randomized and periodic mirror circuits across widths, depths, connectivity patterns, and repeated germ structures. The randomized sampler controls expected two-qubit density, while periodic circuits repeat a germ and reverse it with layerwise inverses.

  • Randomized circuit sampling: The edge-grab sampler constructs candidate disjoint edges, then independently retains each with probability wξ̄/|E|.This produces sparse two-qubit layers compatible with arbitrary processor connectivity.
  • Randomized circuit sampling: The selected-edge count has expectation wξ̄, giving each sampled layer expected two-qubit density 2ξ̄ and each full randomized mirror circuit expected density ξ̄.In the experiments, ξ̄ = 1/8; individual circuit densities fluctuate around this value.
  • Periodic circuit sampling: Periodic mirror circuits repeat a low-depth germ circuit d/2d_g times, insert a random Pauli layer, and then reverse the repeated layers with each replaced by its inverse.They include initialization, randomized single-qubit Clifford, inverse, and computational-basis readout layers.
  • Periodic circuit sampling: Periodic circuits use even benchmark depth d and full depth d_0 = d + 5, because five constant layers are excluded from benchmark depth.If d/2d_g is nonintegral, the final germ repetition is partial.
  • Germ rationale: Repeated germs amplify some errors while suppressing others, so the experiment samples germs randomly rather than attempting to amplify every parameter of a general error model.The random procedure is biased toward parameters considered physically important.

1. The germ selection of experiment #2

Experiment #2 generates shallow random germ circuits, replaces selected single-qubit gates with two-qubit gates while enforcing ξ ≤ 1/8, and converts published error rates into predictive circuit-success models. The prediction framework standardizes gate and readout error information but corrects a naive model’s long-depth behavior.

  • Single-qubit germ construction: The germ sampler first chooses a depth-d_g circuit of single-qubit Clifford gates, constructs independent local germs, and combines them in parallel.Depths are sampled from a truncated exponentially decaying distribution, with local germs truncated to d_g.
  • Single-qubit germ construction: Depth-1 local germs have marginal probability 3/4, biasing the sampler toward coherent over- and under-rotation errors considered especially important.The paper motivates this bias by the error-amplifying behavior of shallow germs.
  • Two-qubit germ construction: Selected germ gates are replaced with two-qubit gates using edge samples, while repeating the germ first ensures the resulting density satisfies ξ ≤ 1/8.The procedure selects layer-index and edge pairs before replacement.
  • Two-qubit germ construction: Periodic circuits can slightly exceed ξ = 1/8 because the final germ repetition may be partial when the germ depth does not divide d/2.The generated germ itself obeys the bound, but partial repetition changes the periodic circuit density slightly.
  • Predictive error models: The standardized error-rate representation includes single- and two-qubit entanglement infidelities plus per-qubit readout error, with initialization error folded into readout.The readout quantity averages state-preparation and measurement error, and differs from average gate infidelity.
  • Predictive error models: A naive product-of-successes model predicts S → 0 as depth grows, so the analysis uses a corrected formula whose long-depth behavior approaches S → 1/2^w.The flaw is the implicit assumption that multiple errors cannot cancel.

A. Experimental details

Experiment #1 generated randomized mirror-circuit benchmarks for twelve processors using processor-specific native gates, connectivity constraints, and systematically chosen circuit shapes. The procedure sampled multiple connected qubit embeddings and circuits across widths and depths.

  • A. Experimental details: Benchmark #1 was designed for processors with fewer than approximately 20 qubits and was applied in Experiment #1 to twelve processors.The experiments used procedures that changed in minor ways over time because the methods were still under development.
  • A. Experimental details: Benchmark #1 generated mirror-circuit suites for generic gate-model processors using processor-specific single- and two-qubit gate sets.Single-qubit gates were assumed available on every qubit, while two-qubit gates followed the processor’s directed connectivity graph.
  • A. Experimental details: Circuit widths were powers of two, with the full processor width n additionally included, while benchmark depths were approximately exponentially spaced and constrained to multiples of four.Some depth values were excluded for six experiments, and the tested depth sets differed across processor groups.
  • A. Experimental details: For widths below n, the benchmark selected connected physical-qubit sets to respect native-gate connectivity constraints.Processors with at most five qubits used every possible connected set; larger processors used ceil(n/w) sets covering every qubit at each width.
  • A. Experimental details: Each processor, circuit shape, and selected qubit set received 40 sampled randomized mirror circuits, generated with the χ1 sampler and incorporated into pyGSTi.The circuits were explicitly assigned to physical qubits and built from gates allowed by each device’s connectivity.

6. Experimental details

The experiments benchmarked circuits across widths, depths, and qubit embeddings, then summarized performance using polarization and best-, mean-, and worst-case frontiers. These frontiers identify where performance crosses a 1/e threshold under a monotonic pass/fail rule.

  • 6. Experimental details: Each device was probed at exponentially spaced circuit widths and depths, with several physical-qubit embeddings tested for each width.For every width, depth, and embedding, the experiments ran 40 randomized mirror circuits.
  • 6. Experimental details: Polarization P rescales success probability S to correct for few-qubit effects, with P = (S − 1/2^w)/(1 − 1/2^w).The rescaling compares observed success against the uniformly random-bit baseline for a width-w circuit.
  • 6. Experimental details: For each shape, the analysis retained the best-performing w-qubit embedding, then plotted maximum, mean, and minimum polarization across its 40 circuits.The selected embedding maximized the depth at which mean polarization fell below 1/e, with ties resolved by larger mean polarization at that depth.
  • 6. Experimental details: The frontier calculation labeled each circuit shape pass or fail and used the border of the largest monotonic region of passing shapes.The 1/e threshold was chosen because polarization decays exponentially with depth under a simple uniform depolarizing-noise model.

5. Accounting for finite sample fluctuations

Finite-sample analysis used different frontier procedures for mean, maximum, and minimum polarization. The maximum–minimum procedure was designed so that unequal frontiers require statistically significant evidence of variation.

  • 5. Accounting for finite sample fluctuations: The mean frontier used a raw pass/fail rule: a shape passed when its mean observed polarization exceeded 1/e.This procedure was described as an unbiased estimate of whether mean polarization exceeded the threshold.
  • 5. Accounting for finite sample fluctuations: Maximum and minimum frontiers were constructed with statistical hypothesis tests because their discrepancy was interpreted as evidence of structured errors.The testing procedure began from the assumption that the two frontiers were equal and separated them only when the data rejected that assumption.
  • 5. Accounting for finite sample fluctuations: The tests evaluated whether all 40 circuits at a shape had polarization above or below the 1/e threshold, using 5% significance.The hypotheses concerned the sampled circuits that were run, not the broader circuit distributions from which they were sampled.
  • 5. Accounting for finite sample fluctuations: When both directional hypotheses were rejected, the maximum polarization was assigned pass and the minimum polarization fail; otherwise both received a shared designation based on their distance from 1/e.The procedure used K = 40 log-likelihood-ratio tests and controlled their significance with the Benjamini–Hochberg procedure.
  • 5. Accounting for finite sample fluctuations: Alternative methods for generating frontiers may be preferable in other contexts.The frontier construction is therefore a context-dependent analysis choice rather than a universally required procedure.

A. Experimental details

Experiment #2 compared randomized and periodic mirror circuits across eight processors using standardized single-qubit gates and processor-specific two-qubit gates. Circuit shapes, qubit subsets, and sampling were chosen to compare ordered and disordered circuits under similar properties.

  • A. Experimental details: Experiment #2 benchmarked eight processors with randomized and periodic mirror circuits to compare disordered and ordered circuit performance.Both circuit families were designed with similar properties because the experiment targeted the role of circuit order and disorder in failure rates.
  • A. Experimental details: Benchmark #2 used all 24 single-qubit Clifford gates for every processor while retaining each processor’s native two-qubit gate and directed connectivity graph.This standardized the single-qubit gate set across devices while preserving processor-specific two-qubit hardware constraints.
  • A. Experimental details: The tested widths were every integer from 1 through n, and the candidate depths were 0, 4, 8, 16, 32, 64, 128, 256, and 512.For larger widths, the largest depths were excluded when error rates implied that all circuits would almost certainly fail.
  • A. Experimental details: For each width, the experiment selected one connected qubit subset using a heuristic based on published error rates.The heuristic maximized a predicted 1/e polarization depth, but the authors did not claim that this choice was optimal.
  • A. Experimental details: Forty randomized and 40 periodic mirror circuits were sampled for each circuit shape on the selected qubit subset.Randomized circuits used expected two-qubit gate density 1/8, while periodic circuits had density approximately bounded by 1/8.
  • A. Experimental details: The periodic circuits were also randomly sampled, so the experiment referred to the randomized circuits as disordered mirror circuits for the remainder of the analysis.The sampling code was incorporated into the open-source pyGSTi software package.

5. Experimental details

Experiment #2 benchmarked eight processors, with one processor tested as an 8-qubit device because half its qubits were unavailable.

  • Eight processors were benchmarked in experiment #2 using IBM Q and Rigetti online access services.Four processors from experiment #1 were unavailable when benchmark #2 was designed.
  • Rigetti Aspen 4 was tested as an 8-qubit processor because 8 of its 16 qubits were not functioning.

B. Data analysis

The analysis compares first- and second-pass circuit outcomes to detect temporal instability, then uses the first pass after finding instability across all eight processors.

  • Instability analysis: The analysis tests whether each circuit’s success probability differs between the two passes through the benchmarking circuits.Figure 8 plots first-pass failure rates against second-pass failure rates for periodic and disordered circuits.
  • Instability analysis: All eight processors showed statistically significant evidence of instability between the two passes.The magnitude varied by processor: differences were small on IBM Q Yorktown and large for many circuits on IBM Q Melbourne and Rigetti Aspen 4.
  • Data selection: The analysis discarded second-pass data and used only first-pass data thereafter.The decision followed the detected instability between passes.
  • Figure 8: Figure 8 distinguishes statistically significant failure-rate differences from nonsignificant differences using solid circles and translucent stars.

2. Worst-case volumetric benchmarks

Worst-case volumetric benchmarks summarize minimum observed polarization across circuit shapes and compare it with error-rate-based predictions, while accounting for finite-sampling bias through bootstrapping.

  • Observed benchmarks: For each processor and circuit shape, observed polarization is minimized over all tested circuits with width and depth no greater than that shape.The calculation uses only first-pass data and compares periodic with disordered circuits.
  • Sampling limitation: The observed minimum polarization is a biased estimate of the true minimum over a circuit ensemble.The analysis made no finite-sampling adjustment because instability undermined the stability assumption required for a two-pass correction.
  • Predicted performance: Predictions from published processor error rates are compared with observed failure probabilities for every tested circuit.
  • Predicted performance: 1000 bootstrap datasets, each based on 1024 simulated repetitions per circuit, estimate predicted worst-case performance using the same analysis as the experimental data.The predicted statistic is the mean of the 1000 bootstrapped values.
  • Performance regions: Success, indeterminate, and fail regions classify circuit shapes where all, some, or none of 80 test circuits succeed, with success defined by P ≥1/e.The region analysis uses first-pass data and statistical hypothesis testing.
Loading 2008.11294v2…