Quality, Speed, and Scale: three key attributes to measure the performance of near-term quantum computers

I. INTRODUCTION

• Benchmarks should reflect typical user workloads, guide system selection and developer improvements, and include quantum-classical interactions relevant to meaningful applications.The paper connects this goal to progress toward quantum advantage, where a quantum program finds a solution faster, cheaper, or more accurately than classical computing alone.Source: §I · p. 1
• Quantum-computing performance is useful work per unit time, so application metrics must holistically account for the QPU and its supporting classical runtime.The runtime converts circuits into a form consumable by the QPU and retrieves results for further processing.Source: §I · p. 1
• The paper identifies scale (number of qubits), quality (faithfully executable circuit size), and speed (primitive circuits executed per unit time) as key performance factors.It introduces a benchmark for measuring speed in section III C.Source: §I · p. 1

A. Quantum Circuits and Programs

• A quantum circuit is an ordered sequence of gates, measurements, and resets on qubits, coupled with concurrent real-time classical computation [2].Classical computation can use measurement results, and quantum operations can be conditioned on or parameterized by its data.Source: A. Quantum Circuits and Programs · p. 1
• Circuit-level classical computation is real-time only within qubit coherence time; longer near-time computations are excluded from the circuit definition.Extended quantum circuits can be completely described by OpenQASM 3 [2].Source: A. Quantum Circuits and Programs · p. 1
• A quantum program expresses a task or algorithm that executes or samples multiple circuits within a larger classical program.Variational algorithms are an example, executing circuits within a classical optimization loop.Source: A. Quantum Circuits and Programs · p. 1
• For workloads such as variational algorithms, performance increases substantially when runtime architecture hosts near-time classical computation with low-latency hardware access.The classical computation coordinates circuit execution in a larger optimization loop.Source: A. Quantum Circuits and Programs · p. 1
• Communication time for classical requests to execute one or more circuits plays a large role in system performance and must be captured by a speed metric.This matters because quantum-classical interactions occur through those requests.Source: A. Quantum Circuits and Programs · p. 1
• Unlike offline quantum-volume compilation, CLOPS updates circuit parameters during runtime, making its measurement heavily dependent on runtime architecture and compilation.Figure 1 contrasts the offline compilation of quantum-volume circuits with runtime parameter updates in CLOPS.Source: Figure 1 · p. 2

B. Compiling quantum programs: offline & runtime compilation

• Quantum Volume circuits allow substantial offline compilation: permutation layers can become data-movement networks, while random two-qubit unitaries can be compiled to the target gate set [4].Circuit quality depends substantially on compiler performance.Source: B. Compiling quantum programs: offline… · p. 2
• The runtime compiler binds parameters available only during execution, generates new binaries, and feeds them to QPU control electronics; repeated quantum-classical interactions make this work critical to measured speed.The benchmark separates compilation phases to measure speed for interactive QPU use.Source: B. Compiling quantum programs: offline… · p. 2
• The compilation-phase separation is not exact: runtime parameters or recent calibration data can enable approximation or noise adaptivity, and using them includes their effects in the benchmark measurement.Source: B. Compiling quantum programs: offline… · p. 2

III. PERFORMANCE METRICS

• Many design choices affect quality, speed, and scale, so device-level parameters alone do not accurately represent overall performance.Holistic benchmarks capture interactions among system parameters, though with less specificity.Source: §III · p. 2
• Quantum Volume and Circuit Layer Operations Per Second are the holistic metrics used to capture ingredients influencing quality and speed.The section presents speed as an additional face of the benchmarking pyramid.Source: §III · p. 2, Figure 2 · p. 3
• The benchmarking pyramid organizes benchmarks by complexity: higher-level benchmarks capture more complexity but less specificity.The figure extends the pyramid with speed as another benchmarking dimension.Source: §III · p. 2, Figure 2 · p. 3

A. Number of qubits

• Qubit count caps the size of solvable problems and is therefore a key measure of quantum-computer scale.In chemistry simulations, it sets the basis-set size used to represent each electron wavefunction and thus the size of molecules that can be simulated.Source: A. Number of qubits · p. 3
• Extra qubits can improve quality and speed: auxiliary qubits can reduce circuit depth and increase fidelity, while multiprogramming QPUs can increase circuit processing speed.Source: A. Number of qubits · p. 3
• Scaling quantum hardware requires technologies that maintain processor coherence, and developing them takes significant investment and time.Platforms face scaling challenges; superconducting qubits are making fast progress beyond 100 qubits.Source: A. Number of qubits · p. 3

B. Quantum volume

• QV is 2^N for the largest N-qubit square random circuit that produces more than 2/3 heavy outputs against ideal simulation results.Each QV layer combines a qubit permutation with pair-wise random SU(4) two-qubit gates; compilation to a QPU’s native gates typically increases circuit depth.Source: B. Quantum volume · p. 3
• QV depends on coherence, gate and measurement fidelity, connectivity, and compilers [4], so improving it requires synergistic improvement across the system.The metric has been widely adopted in research and industry and reported for ion-trap and superconducting quantum computers.Source: B. Quantum volume · p. 3

• The holistic benchmark includes run-time interaction because quantum data cannot persist across invocations, making data transfer a prominent part of execution.The benchmark times multiple executions of parameterized circuits whose parameters are deferred until run time.Source: C. Circuit Layer Operations per Second… · pp. 3–4
• QV circuits provide a representative random-circuit family while preserving a rigorous quality criterion, and the authors identify other circuit families as a possible extension.The benchmark uses the same qubits, gate length, and inter-circuit delays as when establishing the device's QV.Source: C. Circuit Layer Operations per Second… · p. 4
• CLOPS counts QV layers executed per second, dividing the total layers by execution time that includes parameter updates, job submission, QPU execution, and returned results.Each QV circuit has D = log2 QV layers.Source: C. Circuit Layer Operations per Second… · p. 4
• The benchmark uses 100 fully parameterized QV circuit templates, each run 10 times with different parameter sets and 100 shots per execution.Parameters for later runs are seeded from the preceding circuit's output, requiring that output before the next parameter set can be generated; this models iterative algorithms such as VQE and quantum kernel alignment.Source: C. Circuit Layer Operations per Second… · p. 4, Figure 3 · p. 4
• CLOPS stresses the execution pipeline, including circuit and result transfer, run-time compilation, control-electronics loading and initialization, gates, measurement, reset, delays, result processing, and parameter updates.Physical architecture and software orchestration can both affect the measured speed; physical qubit technologies can differ substantially in repetition, gate, reset, and measurement times.Source: C. Circuit Layer Operations per Second… · p. 5

IV. MEASUREMENT OF PERFORMANCE METRICS OF IBM QUANTUM COMPUTING SYSTEMS

• The 65-qubit system measured 753 layers per second, versus 1419 for the 5-qubit system, despite both having quantum volume 32 and 250-microsecond repetition delays.The benchmarked systems ranged from 5 to 65 qubits.Source: §IV · p. 5
• Circuit delay is the idle time between circuits and was the same across systems; it can matter more with larger shot counts and as compilation and data-transfer overheads fall.The systems used the same default delay, circuit count, and shot count in this benchmark.Source: §IV · pp. 5–6
• In Fig. 5, ibmq brooklyn’s circuit execution appeared fastest but its overall CLOPS was slowest: circuit delay was an order of magnitude above gate time, and compilation and data transfer were two orders larger.The breakdown identifies non-circuit execution overheads as barriers to improving speed.Source: §IV · p. 5, Figure 5 · p. 6
• Run-time compilation and data transfer increase with machine size as larger control-electronics complexes take longer to initialize and load; software-stack changes and an OpenQASM3 compiler are expected to reduce these costs.Parameterized updates throughout the stack are also expected to help reduce the overhead.Source: §IV · p. 6

V. SUMMARY

• CLOPS offers a holistic benchmark of application-relevant performance and helps identify system bottlenecks, unlike low-level single-dimension benchmarks.The benchmark is intended to capture necessary system components without being overly cumbersome to execute.Source: §V · p. 6

VI. APPENDIX: DEPTH-1 CIRCUIT PERFORMANCE

• Additional instrumentation on CLOPS can compare a system’s speed relative to its own basis gate set and help extrapolate effects on other algorithms.Source: §VI · p. 7

A. Circuit depth

• Circuit depth affects result quality, while speed metrics require a consistent way to count QPU operations.Source: A. Circuit depth · p. 7
• Before depth is calculated, non-native gates are decomposed; gates are assumed fully parallel across qubits, with two-qubit gates synchronizing both qubits.Depth is the longest sequence of gate boxes across the qubits.Source: A. Circuit depth · p. 7
• Native one-qubit gates of non-zero duration, measurements, resets, and native two-qubit gates each have depth 1; two-qubit gates impose an implicit barrier.Source: A. Circuit depth · p. 7
• Barriers have depth 0 but synchronize applied qubits to their current maximum depth, and circuit depth is the maximum depth across all qubits.Source: A. Circuit depth · p. 7
• A depth-1 circuit executes across a set of qubits in one unit of time, so a depth-d circuit is an ordered sequence of d such circuits.This definition lets system speed be expressed as the execution speed of depth-1 circuits.Source: A. Circuit depth · p. 7

B. Measurement of depth-1 circuit performance

• Depth-1 circuits per second scaled linearly from CLOPS across the machines because their measured depths and benchmark inputs were the same.Figure 6 presents CLOPS results with depth-1 circuits per second.Source: B. Measurement of depth-1 circuit… · p. 7, Figure 6 · p. 8
• The instrumented benchmark reported the average depth of each of its 100 random circuit templates.Source: B. Measurement of depth-1 circuit… · p. 7
• All machines had an average depth of around 99, reflecting a similar straight-line topology of 5 qubits; depth-1 circuits per second was calculated from depth, circuit and shot counts, and total run time.Source: B. Measurement of depth-1 circuit… · p. 7