PaperlayerView exact revision on arXiv

Source-linked AI summary

QASMBench: A Low-level QASM Benchmark Suite for NISQ Evaluation and Simulation

Ang Li, Samuel Stein, Sriram Krishnamoorthy, James Ang

arXiv:2005.13018v3quant-ph

TL;DR

Quantum-computing evaluation needs low-level benchmarks and metrics that characterize diverse circuits, NISQ execution, and related software and simulators. QASMBench addresses this need with OpenQASM-based benchmark circuits and four circuit metrics, then evaluates fidelity across IBM-Q, IonQ, and Rigetti systems. The evaluations show substantial device- and circuit-dependent fidelity variation, while density-matrix tomography limits practical evaluation to small circuits.

  • Problem

    The paper addresses the need for low-level benchmark suites and insightful metrics to characterize NISQ devices, quantum software tools, and classical quantum-system simulators.

  • Method

    QASMBench combines diverse benchmark circuits with four metrics—gate density, retention lifespan, measurement density, and entanglement variance—and evaluates executions across multiple NISQ platforms.

  • Results

    Fidelity varies substantially by circuit and device: IonQ generally performs better for shallow circuits with smaller retention lifespan, but its performance drops significantly for deeper circuits.

  • Takeaways & Limitations

    QASMBench provides a low-level, easy-to-use basis for comparing circuit characteristics and execution fidelity across IBM-Q, IonQ, and Rigetti NISQ devices.

  • Takeaways & Limitations

    Density-matrix tomography requires 3^n circuit evaluations and consecutive measurements within one calibration period, so it is feasible only for small circuits.

Abstract

from arXiv · show

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, etc., trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, which comprises 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench.

1 BACKGROUND: QUANTUM COMPUTING IN THE NISQ ERA

NISQ systems use hardware-defined basis gates and constrained qubit topologies, making transpilation necessary before execution. OpenQASM provides a low-level, sequential representation that supports static analysis, simulation, and validation across quantum software and hardware.

  • 1.1 NISQ Devices: NISQ devices require qubits to remain coherent throughout gate operations, while gate and measurement operations introduce error risks.The supplied passages describe coherence times, gate error, and measurement-induced state collapse as execution constraints.
  • 1.1 NISQ Devices: NISQ devices use hardware-defined basis gates and physical-qubit topologies that constrain which two-qubit operations can execute directly.Quantum transpilation translates user circuits into basis gates and inserts SWAP operations when gates target remote qubits.
  • 1.2 OpenQASM Ecosystem: OpenQASM defines basic, standard, and composition gates, with standard and composition gates converted into basic gates during machine-specific assembly and mapping.Table 2 contains 5 basic gates, 11 standard gates, and 18 composition gates.
  • 1.2 OpenQASM Ecosystem: OpenQASM is a sequential low-level intermediate representation without loops, branches, or jumps, making it convenient for static analysis and classical simulation.The representation is positioned between quantum software and hardware and can be emitted by several programming frameworks.
  • 1.2 OpenQASM Ecosystem: Qiskit supports simulation, hardware verification, and application development, while Rigetti and Microsoft Azure provide Qiskit interfaces for accessing additional systems.The supplied passages identify qiskit-aer, qiskit-ignis, and qiskit-aqua as component tools.

Qiskit:

Cirq and Scaffold are quantum software platforms that connect different programming environments to OpenQASM generation and compilation.

  • Qiskit:: Cirq is a Python-based quantum software platform with local simulators and a to_qasm() function for exporting circuits to OpenQASM.The passage also mentions a simulator for Google’s native gateset.
  • Qiskit:: Scaffold embeds quantum programming in C/C++ and uses the LLVM compiler toolchain to support algorithm development and advanced optimization.Scaffold can generate complex OpenQASM code, including VQE examples, through its compiler.

Scaffold:

Scaffold-related material situates QASMBench within a broader quantum-computing stack and describes how Scaffold produces OpenQASM for complex routines.

  • Scaffold:: QASMBench’s categorization extends a quantum-computing stack with quantum arithmetic, quantum machine learning, and quantum communication.The supplied figure caption states that the categorization is built based on prior work.
  • Scaffold:: Scaffold generates complex OpenQASM programs, such as VQE examples, by compiling C/C++-embedded quantum programs through Scaffcc.The compiler uses the “-b” option to produce native OpenQASM code.
  • Scaffold:: ProjectQ similarly provides a mechanism to generate OpenQASM so programs can be verified on IBM testbeds through IBMBackend.get_qasm().ProjectQ relies on classical simulation rather than providing its own dedicated real quantum backend.

2 QASMBENCH

QASMBench is an end-to-end OpenQASM benchmark package spanning diverse quantum routines and circuit scales, with benchmarks organized by qubit count and linked to representative applications and platforms.

  • Suite organization: QASMBench combines diverse benchmark circuits, performance evaluation, and circuit-metric characterization for pre- and post-transpilation analysis.Its routines are drawn from existing open-source quantum-computing packages and local development.
  • Suite organization: The suite partitions benchmarks into small-scale, medium-scale, and large-scale categories according to qubit count.Small-scale uses 2–5 qubits, medium-scale 6–15, and large-scale more than 15 qubits.
  • Suite organization: The largest circuit, adder_n127, contains 127 qubits and 3991 standard OpenQASM gates.The large-scale category corresponds to machines such as IBM-Q Washington with 127 physical qubits.
  • Benchmark applications: The benchmark collection covers chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and other quantum routines.Examples include adders, chemistry circuits, Bell and cat states, Deutsch, Fredkin, QEC, QFT, QAOA, quantum walks, W states, BB84, SAT, and Simon’s algorithm.
  • Benchmark applications: The package includes representative figures for a 2-qubit multiplier, a 3-qubit swap test, and measurement results from a swap-test circuit using 7 qubits.The multiplier is shown with preparation, execution, and measurement phases.
  • Benchmark applications: Individual routines target distinct circuit or device properties, including CNOT stability, entangling capability, topology, decay, quantum error correction, and optimization workloads.The GHZ routine uses CNOT gates across available qubits, while other routines exercise swap interactions, deep chemistry circuits, or redundant physical qubits.

3 QASMBENCH CHARACTERIZATION METRICS

QASMBench uses circuit metrics to characterize quantum applications and estimate execution demands, error sensitivity, and optimization opportunities on NISQ hardware. The metrics cover spatial and temporal requirements, gate occupancy, measurement importance, retention, and entanglement balance.

  • The metrics indicate how quantum circuits stress NISQ hardware and support software-hardware co-design.
  • Circuit Width: Circuit width counts qubits entering superposition and represents the spatial capacity required to run a circuit.Qubits measured and later re-entering superposition count once.
  • Circuit Depth: Circuit depth is the minimum number of time-evolution steps after decomposing OpenQASM into standard gates.The calculation uses a qubit-by-time matrix to determine the required evolution steps.
  • Gate Density: Gate density measures gate-slot occupancy, with lower occupancy indicating idle slots caused by gate dependencies.It is computed from 1-qubit and 2-qubit gate counts normalized by circuit depth and width.
  • Retention Lifespan: Retention lifespan measures the longest qubit lifespan and relates circuit size to susceptibility to T1/T2-driven information loss.The metric uses the maximum qubit lifespan and applies a logarithm to shrink the scale.
  • Measurement Density: Measurement density assesses measurement importance, which increases when fewer measurements summarize a wider or deeper circuit.The SWAP test illustrates how one measurement can be highly important for a large circuit.
  • Entanglement Variance: Entanglement variance measures how evenly two-qubit entangling operations are distributed across circuit qubits.Higher variance identifies more strongly connected qubits and can inform their placement on device topologies.

4 EVALUATION

The evaluation first profiles QASMBench circuits with the proposed metrics and then benchmarks a subset on real NISQ devices.

  • QASMBench evaluation proceeds from circuit-metric profiling to benchmarking a circuit subset on real NISQ devices.

4.1 Circuit Metrics

QASMBench circuits span broad ranges of width, depth, gate density, retention lifespan, measurement density, and entanglement variance. These metrics expose execution occupancy, error sensitivity, measurement importance, and topology-relevant entanglement patterns.

  • Circuit Width: QASMBench partitions circuits into small, medium, and large categories by qubit count.The evaluation describes width distributions across these categories.
  • Circuit Depth: Circuit depth ranges from 11 in Linearsolver_n-3 to 14,281 in QFT_n-85.DNN, VQE, and QFT are among the significantly deeper circuits.
  • Gate Density: Larger circuits generally have sparser gate density, while Ising_n-26 reaches 0.8462 and BB84_n-8 reaches 0.675.Lower density can indicate internal operation dependencies and potential gains from circuit reformulation.
  • Retention Lifespan: Retention lifespan ranges from 1.3863 for Deutsch_n-2 to 5.6168 for Square-Root_n-18.The metric accounts for dependency and applies to a particular qubit rather than the overall system.
  • Measurement Density: SWAP-test, multiplier, DNN, and QFT have significantly higher measurement density than the other circuits.For QFT, the single final measurement is more important because the circuit is very deep.
  • Entanglement Variance: QASMBench includes both evenly and unevenly entangled circuits, with Basis-trotter concentrating entanglement mainly between qubits 1 and 2.Mapping those qubits to central device vertices can require fewer SWAP gates on ibmq_belem.

4.2 Evaluating QASMBench on Real NISQ Devices

The evaluation uses fidelity from density-matrix tomography to compare QASMBench circuits across IBM-Q and other NISQ devices, revealing circuit- and device-dependent execution quality.

  • 4.2.1 Evaluation Metric: Fidelity compares noisy device states with ideal classical-simulation states reconstructed through density matrix tomography.The metric uses the mixed state from the device and the pure state from a noiseless simulator.
  • 4.2.1 Evaluation Metric: Density matrix tomography requires 3^n circuit evaluations, making fidelity evaluation practical only for circuits with few qubits.A 10-qubit circuit requires 59,049 circuit evaluations before counting shots.
  • 4.2.2 IBM-Q Evaluation: 24 small-category circuits were evaluated on 12 IBM-Q machines using density matrix tomography.The machines ranged from 7 to 127 qubits and had quantum volumes from 16 to 128.
  • 4.2.2 IBM-Q Evaluation: Fidelity generally correlates with circuit depth and width, but qubit count and quantum volume weakly predict machine-level fidelity.For example, qnn n-2 performed worse on Washington than on the smaller-QV Hanoi machine.
  • 4.2.2 IBM-Q Evaluation: Noise substantially affects execution: qft n-4 had fidelity below 0.6 on most machines despite only 36 gates, except Hanoi.The evaluation also reports higher fidelity for a relatively deep dnn n-2 circuit than for adder n-4 because it uses fewer qubits.
  • 4.2.3 Evaluations of NISQ Devices: IonQ generally achieved higher fidelity for shallow circuits with smaller retention lifespan, but its performance dropped significantly for deeper circuits.The paper attributes this pattern to lower single-gate fidelity despite IonQ’s higher T1 time.

5 RELATED WORK

Prior work includes random benchmarks, limited application suites, quantum-volume-style metrics, and unified programming abstractions. QASMBench addresses this landscape with broader application coverage and low-level benchmarking.

  • Benchmarking NISQ Devices: Existing random benchmarks evaluate gate-operation error rates for particular NISQ devices.These approaches use randomly generated benchmark circuits.
  • Benchmarking NISQ Devices: Patel et al. evaluated several IBM-Q machines with seven benchmarks, but included limited applications and proposed no new metrics.Their work focused on extracting and validating new insights about error and execution time.
  • QC Metrics: Quantum Volume, Quantum Linpack, and qBAS provide broader evaluation metrics but are described as less intuitive and partly dependent on synthetic randomness.Random inputs or circuits require offline classical simulation, whose cost grows exponentially with increased circuit size.
  • QC Programming Environments: XACC provides a low-level intermediate representation and compiler front-end for evaluating different quantum hardware and programming environments.Related work also compared programming environments in library, hardware, compiler, and simulator performance.

6 CONCLUSION

QASMBench is presented as a lightweight OpenQASM benchmark suite combining broad quantum workloads, circuit metrics, and cross-platform fidelity evaluation for NISQ characterization.

  • 6 CONCLUSION: QASMBench provides a lightweight, low-level, easy-to-use benchmark suite based on OpenQASM and spanning quantum algorithms and routines from varied domains.Its workload collection is intended for quantum-computing characterization and evaluation.
  • 6 CONCLUSION: Four metrics—gate density, retention lifespan, measurement density, and entanglement variance—assess execution efficiency, NISQ-error susceptibility, and optimization potential.The metrics complement conventional circuit width and depth measures.
  • 6 CONCLUSION: The study evaluates QASMBench fidelity on IBM-Q, IonQ, and Rigetti devices through density matrix state tomography and derives new observations.The suite, metrics, and evaluations support characterization of NISQ devices, quantum transpiler optimization potential, and classical simulator performance.
Loading 2005.13018v3…