Source-linked AI summary
A volumetric framework for quantum computer benchmarks
Robin Blume-Kohout, Kevin C. Young
TL;DR
Quantum volume tests square circuits, while important quantum-computing performance questions involve broader circuit shapes and benchmark families. The paper proposes volumetric benchmarks that map width-depth pairs to circuit ensembles and a graphical reporting framework; it also notes that application-circuit examples remain speculative and underspecified.
Problem
Quantum volume tests square circuits, but relevant processor performance properties remain unclear and circuit shapes vary with applications.
Source: §1 · p. 3Method
The paper proposes volumetric benchmarks that map width-depth pairs to structured circuit ensembles and adds a reporting and visualization framework.
Source: §1 · pp. 3–4, §3 · p. 6Results
The framework accommodates varied circuit structures and benchmark families, while the proposed examples include random, periodic, and algorithm-inspired circuits.
Source: §1 · p. 3, §8.3 · p. 19Takeaways & Limitations
A diverse pool of benchmarks, rather than reliance on random or scrambling circuits alone, is argued to provide robustness against unknown and unexpected effects.
Source: §2 · p. 4The suggested Grover-step and Trotterized-simulation benchmarks are speculative, and their construction details are not settled.
Source: §8.3 · p. 19
Abstract
from arXiv · showhide
We propose a very large family of benchmarks for probing the performance of quantum computers. We call them volumetric benchmarks (VBs) because they generalize IBM's benchmark for measuring quantum volume \cite{Cross18}. The quantum volume benchmark defines a family of square circuits whose depth $d$ and width $w$ are the same. A volumetric benchmark defines a family of rectangular quantum circuits, for which $d$ and $w$ are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes -- $(w,d)$ pairs -- to test suites $\mathcal{C}(w,d)$. A test suite is an ensemble of test circuits that share a common structure. The test suite $\mathcal{C}$ for a given circuit shape may be a single circuit $C$, a specific list of circuits $\{C_1\ldots C_N\}$ that must all be run, or a large set of possible circuits equipped with a distribution $Pr(C)$. The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have "passed" a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the $d$ vs $w$ trade-off for the processor being benchmarked. [1] A. Cross, et al., Phys. Rev. A, 100, 032328, September 2019.
1 Introduction
Quantum volume summarizes square-circuit performance, but quantum-computing benchmarks need to probe a wider range of processor behavior and circuit shapes. The paper proposes volumetric benchmarks and a reporting framework that generalizes quantum volume.
1 Introduction
- Processors need holistic benchmarks that capture overall performance and support comparison across processors and architectures.The motivation grows with the emergence of larger processors.Source: §1 · pp. 1–2
- Quantum volume tests square circuits, although circuit width and depth can vary independently with the target application.The paper describes probing circuit shapes as a device diagnostic and performance metric.Source: §1 · p. 2
- Benchmarks may define synthetic metrics rather than measure a pre-existing computer property; even established metrics can vary in meaning across contexts.The paper uses LINPACK as an example of a benchmark that effectively defines a new synthetic metric.Source: §1 · p. 3
- Because relevant performance properties remain unclear, the authors advocate a broad set of candidate benchmarks that can be tested and compared.They expect the most useful benchmarks to persist.Source: §1 · p. 3
- Volumetric benchmarks generalize quantum volume with flexible circuit families whose different structures can probe different noise and error properties.The authors also propose a reporting and visualization framework.Source: §1 · pp. 3–4
- Quantum volume is a prominent precedent, and the authors describe their contribution as integrating prior approaches and adding a reporting and visualization framework.They explicitly intend to generalize quantum volume.Source: §1 · pp. 3–4
2 Motivations for generalizing (why not just quantum volume?)
Quantum volume is useful but cannot capture all relevant processor performance because it tests square circuits and circuit structure affects how errors appear. The authors argue for diverse benchmarks, including structured, periodic, and algorithm-inspired circuits.
2 Motivations for generalizing (why not just quantum volume?)
- Quantum volume tests square circuits, but algorithms can require different width-depth trade-offs, such as O(n) qubits and O(n^3) depth for Shor's algorithm [26].Other algorithms can be reformulated for polylog(n) depth at the cost of much larger width.Source: §2 · p. 4
- Accessing the full Hilbert space is not a complete proxy for useful computation: Grover's algorithm operates in a two-dimensional subspace, while algorithms may need long paths through Hilbert space.The paper notes that one scrambling is not usually enough for algorithms.Source: §2 · p. 4
- Users and designers need to understand width-depth trade-offs because processors with different widths and gate-layer depths may suit different tasks.The paper contrasts 100 qubits with 1,000 layers against 1,000 qubits with 100 layers.Source: §2 · p. 4
- Different errors affect circuit families differently, so random or scrambling circuits may not reliably proxy algorithm performance.Coherent errors tend to be smeared into effective depolarizing errors in random circuits but can be amplified in periodic circuits; effects on structured application circuits remain incompletely understood.Source: §2 · p. 4
- Because hardware and success metrics may evolve, the authors advocate a flexible framework with enough shared structure to communicate performance while encouraging new benchmarks.They state that examples show many existing benchmarks and test suites can fit the framework.Source: §2 · p. 5
3 Framework for volumetric benchmarks
A volumetric benchmark maps width-depth shapes to circuit ensembles and specifies how circuits are implemented and judged. The framework allows different ensemble types and experimental designs, while emphasizing precise reporting for reproducibility and comparison.
3 Framework for volumetric benchmarks
- Benchmark definitions also specify compilation constraints and measures of success for individual circuits and ensembles.A success measure can return a pass/fail bit or a score; ensemble measures condense circuit-level results.Source: §3 · p. 5
- A benchmark may specify an experimental design, including repetition counts or circuit order, when the task requires it.Circuit ordering can matter for testing stability over time.Source: §3 · pp. 5, 7
- Precise benchmark specifications and reporting of variations support repeatability and cross-platform comparison.The authors recommend clear, unambiguous descriptions of benchmark properties.Source: §3 · pp. 5–6
- A volumetric benchmark maps each width-depth pair to a circuit ensemble, the framework's defining property.For a given shape, the suite may be a single circuit, a list, or circuits drawn from a distribution.Source: §3 · pp. 5–6
- Circuit ensembles should share a structure that scales naturally with width and depth, with larger shapes expected to make tests harder or no easier.Circuit implementation rules help prevent optimizers from exploiting large-scale structure to simplify a benchmark, as in randomized benchmarking circuits composing to identity.Source: §3 · p. 6
4 Volumetric benchmarking circuits
Volumetric benchmarks organize layered circuits by width and depth, allowing those dimensions to vary independently. The framework accommodates varied circuit families, but results from different benchmarks are generally not directly comparable.
4 Volumetric benchmarking circuits
- Circuit width is the number of qubits and depth is the number of central layers; prefix and postfix operations may prepare states or measurements without counting toward depth.The permissible central layers define depth and can range from high-level subroutines to canonical or native gates.Source: §4 · pp. 7–8
- Different volumetric benchmarks generally cannot be compared directly because their layer structures can produce different depth limits and error accumulation.Even benchmarks using the same layer set may differ: random and periodic circuits can accumulate error differently.Source: §4 · p. 8
4.1 Classes of circuits for volumetric benchmarks
- The framework is a non-exhaustive container for circuit classes, not a proposal for one preferred benchmark.Its examples are intended to illustrate the framework’s potential and encourage creative benchmark designs.Source: §4.1 · pp. 8–9
4.1.1 Random circuits
- Random-circuit benchmarks include RB variants and scrambling circuits, and RB circuits are repurposed here as benchmarks rather than analyzed conventionally for intrinsic error rates.Scrambling circuits can explore every corner of the Hilbert space in principle, whereas RB circuits typically explore a discrete subset spanning the state space.Source: §4.1.1 · p. 9
4.1.2 Periodic circuits
- Periodic circuits repeat a subroutine O(d) times, emphasizing different performance aspects from random circuits and potentially amplifying coherent errors.Rabi and Ramsey sequences, robust phase estimation, and gate set tomography are examples; periodic structures also occur in algorithms such as Grover search and phase estimation.Source: §4.1.2 · pp. 9–10
4.1.3 Application circuits
- Application circuits aim to represent real-world usage, but full algorithms may not scale smoothly across width and depth, so representative subroutines can be used instead.Constructing application-based suites will require significant additional research; the framework is intended to help guide their development.Source: §4.1.3 · p. 10
5 Rules for compiling volumetric benchmarking circuits
A benchmark must specify how much compilation and scheduling freedom it permits, since that choice affects portability and what performance is measured. Barriers can encode some compiler constraints, but hardware implementation details may require particular compilation freedoms.
5 Rules for compiling volumetric benchmarking circuits
- Barriers can permit local optimization between adjacent circuit segments while preventing the compiler from using information across those segments.Barriers may reflect the circuit’s layer structure, but need not do so.Source: §5 · p. 11, Figure 3 · p. 11
- Compiler freedom ranges from full-circuit recompilation to compiling individual layers or accommodating only hardware parallelism constraints.Random circuit sampling permits re-compilation even to classical simulation; direct randomized benchmarking allows only adjustments for parallelism, while crosstalk tests may forbid serialization.Source: Figure 3 · p. 11, §5 · pp. 11–12
- Quantum-volume circuits specify target unitaries that may be recompiled freely, whereas randomized benchmarking forbids joint compilation across its Clifford layers.For quantum volume, approximating the target unitary is encouraged when the approximation error is offset by reduced hardware errors.Source: §5 · pp. 11–12
- Extensive compilation makes benchmarks easier to adapt across hardware and reflects both quantum and compiler performance, while restricted compilation gives more direct information about low-level hardware.The appropriate freedom depends on benchmark purpose and the gates used to specify circuits.Source: §5 · pp. 10–12
- Some benchmarks may be incompatible with processors unless their required compilation is allowed; virtual Z rotations illustrate how low-level implementation details can complicate constraints.Repeated-Z-heavy circuits are an example where bridging multiple operations for virtual Z gates may matter.Source: §5 · p. 12
6 Success criteria for individual circuits
Individual-circuit success can be defined in several ways, including output-probability, distribution-distance, and coarse-grained criteria. Each criterion has a scope and possible prerequisites, and benchmark designers should explain why the chosen criterion indicates success.
6 Success criteria for individual circuits
- Because weak or meaningless criteria can inflate performance claims, benchmarks should specify the criterion precisely and justify it against better-studied alternatives.The authors call this risk “benchmarketing.”Source: §6 · p. 13
- A correct-output probability threshold applies only to circuits with a unique expected outcome, and the probability must be estimated statistically from observed outcomes.A threshold such as 2/3 is conventional but arbitrary, and the best choice may depend on the problem.Source: §6 · pp. 13–14
- Heavy output probability compares the probability mass on outputs above the median with a threshold, but requires computing all ideal outcome probabilities and does not suit uniform-output circuits.This is the criterion used for quantum volume [1].Source: §6 · pp. 13–14
- Distribution-distance criteria compare observed and ideal output distributions, but require ideal probabilities and more data as distribution entropy increases.Total variation distance is one possible metric.Source: §6 · p. 13
- A Hamming-distance criterion can count outputs near the correct string as successful, reflecting that some errors may be less severe than others.The passage notes that few bit flips may suggest suitability for quantum error correction.Source: §6 · p. 13
- Coarse-graining compares distributions over fewer outcome bins; keeping the bin count constant as width grows can make distribution comparisons manageable.Heavy output probability is one coarse-graining, alongside local marginals and grouping expected strings against all others.Source: §6 · pp. 13–14
7 Success critera for circuit ensembles
For each circuit shape, a volumetric benchmark can specify one circuit, a finite list, or a distribution over circuits. The benchmark must also define how success across the selected ensemble is judged.
7 Success critera for circuit ensembles
- For each width-depth shape, an ensemble may be a single circuit, a finite list, or a distribution over a possibly infinite circuit set.The benchmark’s test suite is denoted C(w,d).Source: §7 · p. 14
- A list-based test may require success on every circuit, a specified fraction, or an average success metric above a threshold.For a single-circuit test, passing means succeeding at that circuit.Source: §7 · pp. 14–15
- Requiring success on every same-sized circuit is challenging but provides a stronger performance guarantee and avoids users encountering an unrunable exception.The authors argue that a processor able to run some or most circuits of a given size can reasonably be expected to run all of them.Source: §7 · p. 15
8 Examples of volumetric benchmarks
The volumetric-benchmark framework adapts existing evaluation protocols while allowing circuit families and analysis choices to vary. Its examples span random, periodic, and application-inspired circuits, though the application examples remain speculative.
8 Examples of volumetric benchmarks
- The framework generalizes existing processor-evaluation techniques, but adapting a protocol can change its analysis and relax its circuit restrictions.For example, GST-derived circuits need not retain the standard restriction to depths that are powers of two.Source: §8 · p. 15
- Circuit structure defines the benchmark class: changing it can create a new, incomparable VB, while changing the success metric may have little effect.The examples are organized by circuit type and are not intended to be exhaustive.Source: §8 · p. 15
8.1 Random circuits
- Randomized benchmarking, quantum-volume scrambling, and random quantum-supremacy circuits need little adaptation; their VB analysis identifies shapes whose failure probability falls below a threshold.Randomized benchmarking's decay-rate estimate is closely correlated with this threshold-based analysis.Source: §8.1 · pp. 16–17
8.2 Periodic circuits
- Periodic protocols such as Rabi and Ramsey oscillations can use local-outcome total-variation-distance thresholds to probe rotation errors and decoherence or dephasing.These protocols are commonly analyzed to estimate parameters rather than treated as benchmarks.Source: §8.2 · pp. 17–18
- Idle tomography, robust phase estimation, and long-sequence GST can require every listed circuit to pass local-marginal error thresholds, probing worst-case idle or gate errors.Long-sequence GST permits various success metrics, with local-marginal total variation distance given as a simple option.Source: §8.2 · p. 18
8.3 Application circuits
- Application-circuit VBs based on Grover iterations or Trotterized Hamiltonian simulation are speculative, and their construction details remain unsettled.Their physical circuit depth depends strongly on architecture and on the selected oracle or Hamiltonian.Source: §8.3 · p. 19
9 Reporting and interpreting VB results
The section presents a consistent visual format for showing which circuit shapes a device can run, and recommends Pareto frontiers for clearer comparison across devices. It also shows how model predictions and quantum-volume results can be displayed alongside volumetric benchmark data.
9 Reporting and interpreting VB results
- Circuit shapes are arranged with depth on the x-axis and width on the y-axis; figures should specify the circuit family and success metric for reproducibility.Linear axes suit small processors, while logarithmic axes can display larger width or depth ranges; the suggested width scaling uses powers of 1.2 and depth scaling powers of 2.Source: §9 · p. 20
- Binary benchmarks can mark successes with filled squares and failures with hollow squares, while real-valued metrics use marker shading to show measured values.Hollow squares distinguish failed tests from shapes that were not tested.Source: §9 · p. 20
- Pareto frontiers provide a simpler, cleaner summary for comparing devices, and a slope change across generations is described as a hallmark of reduced crosstalk.The plots should use the same benchmark and success metric for comparisons.Source: §9 · p. 21, Figure 5 · p. 22
- Comparing observed performance with model predictions can expose discrepancies that suggest failure mechanisms, including width deficiencies associated with bad qubits or crosstalk.For width-1 circuits, depth deficiencies can indicate non-Markovianity or coherent errors when the benchmark is sensitive to them.Source: §9 · p. 21
- For the illustrative two-qubit example, single-qubit circuits reach depth ∼2000 reliably, while two-qubit circuits are limited to depth ∼8.The plot also shows that two-qubit circuits perform significantly worse than single-qubit circuits; its data are illustrative, not from a real device or rigorous simulation.Source: §9 · pp. 20, 22
- Across three illustrative device generations, the plots report significant increases in qubit performance and available qubits.The figure caption identifies the Pareto frontiers as the comparison, and notes that shading successful experiments by value can clutter the display.Source: §9 · p. 22, Figure 5 · p. 22
- In the illustrative 16-qubit example, the model is built from one- and two-qubit experiments, and qubit 16 is defective.The plots use a heavy-output-probability pass criterion greater than 2/3.Source: §9 · p. 23
- A quantum-volume example with log2(VQ) = 8 highlights the diagonal experiments and implied-success region, while showing that shallower circuits can run with fewer qubits.Logarithmic axes accommodate a wider range of circuit shapes.Source: §9 · pp. 21, 24, Figure 8 · p. 24