Source-linked AI summary
MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing
Nils Quetschlich, Lukas Burgholzer, Robert Wille
TL;DR
Quantum software tools need practical benchmarks for evaluation across abstraction levels, but existing suites remain fragmented and lack consolidated benchmark choices. MQT Bench addresses this gap with a single, accessible and extensible suite spanning four levels and more than 70,000 circuits, aiming to improve evaluation comparability, reproducibility, and transparency.
Problem
Existing benchmark suites focus on specific abstraction levels, with no consolidated benchmark selection for quantum-software evaluations.
Method
MQT Bench provides the same benchmark algorithms across four abstraction levels, with accessible interfaces, broad algorithm coverage, and extendable components.
Results
MQT Bench comprises more than 70,000 benchmark circuits ranging from 2 to 130 qubits across four abstraction levels.
Takeaways & Limitations
MQT Bench presents a first step toward increasing comparability, reproducibility, and transparency through a single benchmark suite for different abstraction levels.
Takeaways & Limitations
The current four-level design is expected to expand as pulse-level programming and higher-level abstractions emerge.
Abstract
from arXiv · showhide
Quantum software tools for a wide variety of design tasks on and across different levels of abstraction are crucial in order to eventually realize useful quantum applications. This requires practical and relevant benchmarks for new software tools to be empirically evaluated and compared to the current state of the art. Although benchmarks for specific design tasks are commonly available, the demand for an overarching cross-level benchmark suite has not yet been fully met and there is no mutual consolidation in how quantum software tools are evaluated thus far. In this work, we propose the MQT Bench benchmark suite (as part of the Munich Quantum Toolkit, MQT) based on four core traits: (1) cross-level support for different abstraction levels, (2) accessibility via an easy-to-use web interface (https://www.cda.cit.tum.de/mqtbench) and a Python package, (3) provision of a broad selection of benchmarks to facilitate generalizability, as well as (4) extendability to future algorithms, gate-sets, and hardware architectures. By comprising more than 70,000 benchmark circuits ranging from 2 to 130 qubits on four abstraction levels, MQT Bench presents a first step towards benchmarking different abstraction levels with a single benchmark suite to increase comparability, reproducibility, and transparency.
1 Introduction
MQT Bench addresses fragmented quantum-software evaluation by providing a single, cross-level benchmark suite designed for comparability, reproducibility, transparency, accessibility, generalizability, and extension.
- Motivation: Existing quantum-software benchmarks typically target specific abstraction levels, leaving cross-level evaluation and benchmark-selection consolidation insufficient.This fragmentation lowers comparability, reproducibility, and transparency of empirical results.
- Contribution: MQT Bench proposes a single suite that uses the same benchmark algorithms across the quantum software stack.The suite is intended as a first step toward benchmarking the whole stack across abstraction levels.
- Core traits: MQT Bench provides benchmarks on four abstraction levels to support cross-level benchmarking.
- Core traits: The suite combines a web interface, a Python package, broad algorithm coverage, and extendability to algorithms, gate-sets, and hardware architectures.Its algorithms range from building blocks such as QFT to applications such as Grover’s algorithm.
- Expected impact: MQT Bench aims to improve the comparability, reproducibility, and transparency of empirical evaluations across the quantum software stack.
2 Background
Quantum compilation transforms hardware-agnostic algorithms into device-executable circuits across abstraction levels, while quantum software supports compilation, simulation, and verification under substantial complexity and hardware constraints.
- Compilation flow: Quantum algorithms begin as hardware-agnostic circuits composed of high-level building blocks at the algorithmic level.Compilation is required before execution on an actual device.
- Compilation flow: The target-independent level synthesizes high-level blocks and applies transformations such as loop unrolling, constant folding, propagation, and gate simplification.
- Compilation flow: Variational quantum algorithm parameters can be propagated into a concrete circuit once determined by the hybrid quantum-classical workflow.The example assigns θ_i = π for i = 0, ..., 5 before propagation.
- Compilation flow: Target-dependent native-gate compilation translates circuits to device-supported gates, while connectivity constraints require mapping logical qubits to physical qubits.The ibmq_manila example converts Ry gates into Rz and X sequences and applies non-trivial mapping for unsupported connectivity.
- Quantum software stack: Quantum software tools support simulation, compilation, and verification across abstraction levels, but these tasks involve high complexity and trade-offs between resource demands and result quality.Simulation is important because quantum devices are scarce and simulators expose quantum-state amplitudes during execution.
3 Benchmarking
Existing quantum benchmark suites generally target individual abstraction levels, leaving cross-level evaluation and common benchmark selection unresolved. MQT Bench addresses this gap with a shared, accessible, broad, and extendable suite intended to improve evaluation comparability, reproducibility, and transparency.
- Current state of the art: High-level suites include 13 algorithms or eight algorithms, while lower-level suites provide target-independent or reversible circuits in intermediate representations.Application-Oriented Performance Benchmarks provides 13 algorithms; SupermarQ provides eight algorithms and six feature vectors; QASMbench and RevLib target lower abstraction levels.
- Current state of the art: Existing suites each target a specific abstraction level, so cross-level benchmarking and mutual consolidation of evaluation benchmarks remain unmet.
- MQT Bench motivation: The suite provides four core traits: cross-level benchmarking, accessibility, broad algorithm selection, and extendability.
- Accessibility: Users can filter pre-generated benchmarks through a no-coding-required web interface or generate benchmarks on demand with the Python package.
- MQT Bench motivation: MQT Bench aims to use the same benchmarks across the whole quantum circuit compilation flow instead of adapting different suites for each level.
4 MQT Bench
MQT Bench supplies benchmarks across four abstraction levels, with configurable representations, compilers, gate-sets, devices, and architectures. It combines broad algorithm coverage with public generation tools and extension mechanisms for evolving quantum computing workflows.
- 4.1 Cross-Level Benchmarking: MQT Bench’s main contribution is providing benchmarks on four abstraction levels spanning the quantum software stack.
- 4.1 Cross-Level Benchmarking: The levels progress from algorithmic descriptions through target-independent and native-gate representations to mapped circuits satisfying device connectivity constraints.
- 4.2 Accessibility: Benchmark generation is transparent through publicly available scripts, while the Python package supports on-demand generation and the website supports filtering.
- 4.3 Algorithm Selection: The benchmark collection spans building blocks such as QFT and GHZ through higher-level algorithms including Grover’s and Shor’s algorithms.
- 4.3 Algorithm Selection: Application benchmarks cover optimization, machine learning, finance, and nature, including travelling salesman, QNN, portfolio optimization, option pricing, and ground state estimation.
- 4.4 Extendability: MQT Bench can incorporate new algorithms, native gate-sets, and hardware architectures, with other abstraction levels envisioned for future use.
5 Evaluation
The evaluation characterizes the generated benchmarks by qubit count, target-device distribution, and circuit properties. Benchmark availability decreases at larger qubit counts, while target-device and native-gate-set distributions follow distinct patterns.
- Benchmark frequency decreases as the number of qubits increases, partly because fewer devices are available and generation time limits are exceeded more often.
- TKET generation generally takes longer than Qiskit, producing fewer benchmarks, especially at the target-dependent mapped level.
- Larger target devices receive more target-dependent mapped-level benchmarks.
- The five native gate-sets have an equal distribution of benchmarks at the target-dependent native-gates level.
- Six characteristics are evaluated across pre-generated benchmarks, including five proposed feature measures and the percentage of multi-qubit gates.
6 Conclusion
MQT Bench combines varied algorithms, compilers, gate-sets, and target devices into more than 70,000 circuits across four abstraction levels. Its web interface, Python package, and open-source repository support filtered access and on-demand generation for comparable, reproducible, and transparent evaluation.
- MQT Bench comprises more than 70,000 circuits spanning 2 to 130 qubits across four abstraction levels.
- The suite includes different algorithms, compilers, native gate-sets, and target devices.
- A web interface lets users filter benchmarks, while a Python package supports on-demand generation and local web-interface deployment.
- MQT Bench is available through an open-source GitHub repository and is intended to facilitate comparable, reproducible, and transparent evaluations.