Benchmarking gate-based quantum computers

1. Introduction

• The central question is whether processor hardware performs the desired computation according to its mathematical circuit model.From a user's perspective, identifying the physical cause of a malfunction is immaterial.Source: §1 · p. 2
• A useful processor should return the correct output for each fixed algorithm and input, and keep that output stationary over time.Correct outputs can be checked against the exact mathematical model, including with a simulator.Source: §1 · pp. 2–3
• The paper explores simple, decisive gate sequences as user-facing tests of whether hardware meets those requirements.Public access to small processors allows independent parties to validate and benchmark them.Source: §1 · pp. 1, 3
• The authors report that the IBM-QE does not meet the two requirements for a useful computing device.The paper examines singlet-state preparation, a modulo-4 adder, identity operations, and error-correction schemes.Source: §1 · p. 3

2. The IBM quantum experience

• The IBM-QE provides public access to a five-qubit processor and a simulator that follows the idealized mathematical model.The device supports single-qubit operations on all qubits and CNOT operations on specified pairs.Source: §2 · p. 4
• An experiment specifies a circuit, checks it on the simulator, then executes it on hardware for repeated shots, with each shot initializing the processor.Readout returns counts of the bit strings produced across shots.Source: §2 · p. 4

2.1. Device characteristics

• For the device version described, X and Hadamard gates took 130 ns each, CNOT took 650 ns, and single-qubit coherence time was about 100 µs.Estimated gate errors were in the 10^-2–10^-3 range, and parameters varied between calibrations.Source: §2.1 · p. 4

2.2. Data analysis

• Output bit patterns are treated as basis states, with subscripts identifying IBM-QE qubits when logical and physical qubits may be permuted.The paper uses 8192 shots and treats the most frequent state, or states, as the correct result when applicable.Source: §2.2 · p. 5
• Under the quantum-theory assumption, measurements are independent trials, and the standard error for 8192 shots is bounded by 0.012.Frequencies within five standard errors, 0.06, are treated as the same.Source: §2.2 · p. 5
• Experimental variations were much larger than predictions from probabilistic error models based on estimated single-gate errors.The authors say this supports concerns that randomized-benchmarking estimates have deficiencies.Source: §2.2 · pp. 5–6

3. Entanglement

• Agreement of repeated measurement frequencies with the singlet probability distribution characterizes the process statistically; it does not prove the device is actually in the singlet state after preparation.The singlet describes measurement statistics, not the internal state at a particular instant.Source: §3 · pp. 6–7
• The singlet predicts zero averages for each spin and a joint correlation of −a·b, or −cosθ for measurement directions separated by θ.Here a and b are three-dimensional unit vectors.Source: §3 · p. 7
• The test prepares a singlet with two X gates, a Hadamard, and a CNOT, then uses rotated measurement bases and 8192 shots to estimate outcome frequencies and observables.The measured averages and correlation are compared with the quantum-theory expectations for the singlet state.Source: §3 · p. 8
• For a = b, quantum theory predicts correlations in [0.945,1], but IBM-QE gives F(θ1,θ1) ≈ 0.80; single-qubit averages are zero within 6 SE.The authors also note that entanglement experiments reproduce the predicted correlation ⟨Ψ|σ1·a σ2·a|Ψ⟩ = −1 to no more than two digits of accuracy.Source: §3 · pp. 8–9
• The authors note that entanglement experiments with photons, neutrons, and ions also reproduce the quantum-theory correlation with no more than two digits of accuracy, unlike the cited atomic-system example.They give roughly four digits of accuracy for quantum predictions of hydrogen Balmer-line wavelength ratios.Source: §3 · p. 9

4. Two-qubit + two-qubit adder

• The adder uses a quantum Fourier transform and adds integers no larger than 3; its implementation was checked on a simulator against arithmetic modulo 4.Integer addition is used because the correctness of its results is straightforward to check.Source: §4 · p. 9
• IBM-QE returns sometimes correct and sometimes wrong results for the listed addition cases, with no detected systematics; a result correct on one date may be wrong on another.The reported cases are collected in Table 1.Source: §4 · pp. 9–10
• For all shown inputs that are superpositions of integer-number states, IBM-QE returns the expected answer.These results are presented in Table 2.Source: §4 · p. 10

5. Identity operations

• The test uses even numbers of CNOT gates, which theoretically act in pairs as identity operations; representative sequences contain 8 or 12 gates.Some sequences add X and H gates to change the input or output state; 12 successive CNOTs take about 8 µs, within the stated 100 µs coherence time.Source: §5 · p. 10
• Except in the last two Table 3 rows, the most frequent output states are repeatable, yet comparing the second and third rows shows that the device is not guaranteed to operate properly.The authors report errors that are hard to gauge and no systematic way to identify when blatantly wrong results occur.Source: §5 · p. 10
• The most frequent states remain unchanged on repetition in most tested cases, but this repeatability does not establish that their outputs are correct.The paper gives the second-versus-third-row comparison as evidence that proper operation is not assured.Source: §5 · p. 10

6. Error correction

• The study tests a distance-two surface code [[5,1,2]] and a perfect distance-three code [[5,1,3]] on the five-qubit processor.The distance-three code is the smallest code theoretically able to correct any single-qubit error.Source: §6 · p. 10, §6.2 · p. 13

6.1. Distance-two surface code

• For the distance-two code, postselected results deviate from the ideal rotation more than single-qubit results, and encoding errors also undermine a test with rotation after encoding.Postselection reduces 8192 shots to about 2000–2300 valid outcomes; the second test’s error-corrected results are only slightly better than random zeros or ones.Source: §6.1 · p. 12, Figure 4 · p. 13

6.2. Distance-three 5-qubit code

• The distance-three encoding circuit is adapted to IBM-QE-supported gates and connectivity, and the ideal simulator produces the expected codewords for both logical inputs.The full circuit takes about 33 µs to run; simulator sampling uses 8192 shots.Source: §6.2 · p. 14, Figure 5 · p. 14
• The IBM-QE produces nearly the same output for either encoded input, preventing users from distinguishing logical |0⟩L and |1⟩L; decoding could not be tested because the device allowed at most 80 gates.The simulator correctly produces the respective codewords, unlike the hardware processor.Source: §6.2 · p. 15, Figure 6 · p. 15

• Only some two- and four-qubit systems showed qualitative agreement with quantum theory.Source: §7 · p. 16
• Users could neither identify nor correct errors, and the errors could not be attributed to specified gate errors.Source: §7 · p. 16
• Results varied strongly between device calibrations.Source: §7 · p. 16
• CNOT-built identity-operation sequences are simple, scalable, sensitive to malfunctioning hardware, and proposed as a minimal benchmark.Source: §7 · pp. 15–16
• The current IBM-QE does not meet the paper’s two elementary requirements for a computing device.Source: §7 · p. 16
• Whether the two-state qubit model explains failures remains unresolved, and the authors propose realistic coupled-transmon simulations and tests on other hardware.Source: §7 · pp. 16–17

• The appendix supplies the .qasm files for circuits used in the reported experiments and directs readers to Ref. for language details.Source: Appendix A · p. 21

Appendix A.1. Singlet state

• The listing uses IBMQASM 2.0 and includes the qelib1 library.Source: Appendix A.1 · p. 21
• It declares five-qubit and five-bit classical registers.Source: Appendix A.1 · p. 21
• On q1 and q2, the listing applies X, Hadamard, CNOT, zero-angle u1, and further Hadamard operations before measurement.Source: Appendix A.1 · p. 21

Appendix A.2. Adder using qubits 0 – 3

• The listing uses IBMQASM 2.0, includes qelib1, and declares five-qubit and five-bit classical registers.Source: Appendix A.2 · p. 21
• The circuit applies X gates to qubits 1–3, then uses T, TDG, S, SDG, Hadamard, and CNOT operations on qubits 0–3 in the shown sequence.Source: Appendix A.2 · pp. 21–22
• Additional code lists ID, TDG, T, Hadamard, and CNOT operations on q2 and q3, then measures q3, q2, q1, and q0 into matching classical bits.Source: Appendix A.2 · pp. 22–23

Appendix A.3. Adder using qubits 1 – 4

• The five-qubit register is initialized by applying X gates to qubits 2 and 3, followed by a Hadamard gate on qubit 2.Source: Appendix A.3 · p. 23
• The circuit measures qubits 1–4 after applying a sequence of single- and two-qubit gates.The measurement instructions map q[1] through q[4] to the corresponding classical bits.Source: Appendix A.3 · pp. 23–24

Appendix A.4. Identity operation

• The identity-operation circuit applies eight consecutive CNOT gates to qubits 0 and 1, then measures both qubits.Source: Appendix A.4 · p. 24

Appendix A.5. Error correction: distance-two surface code

• The excerpted code comments that these logical X gates may be repeated 0−8 times, and shows Z gates on q[0] and q[1] in the same listing.The passages show measurements of q[0] through q[4] into corresponding c bits.Source: Appendix A.5 · pp. 27–28

Appendix A.6. Error correction: distance-three 5-qubit code

• The gate listing contains repeated h and cx operations, with sdg operations on q[4] and q[1].Source: Appendix A.6 · pp. 28, 31