Source-linked AI summary
QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
Ali Slim, Haydar Hamieh, Jawad Kotaich, Yehya Ghosn, Mahdi Chehimi, Ammar Mohanna, Hasan Abed Al Kader Hammoud, Bernard Ghanem
TL;DR
Quantum-code generation is difficult to evaluate across frameworks because existing evidence often cannot separate quantum reasoning from framework familiarity. QuanBench+ addresses this with aligned multi-framework tasks and executable evaluation, finding progress but persistent framework-dependent reliability and recoverable yet unresolved semantic errors.
Problem
Existing quantum-code evaluations are often tied to single frameworks or fail to hold task intent fixed, limiting separation of quantum reasoning from framework familiarity.
Method
QuanBench+ evaluates aligned quantum-code tasks across Qiskit, PennyLane, and Cirq using executable functional tests, Pass@k metrics, KL-divergence-based acceptance, and feedback-based repair.
Results
Performance remains strongly framework-dependent: Qiskit is strongest, PennyLane hardest, and Cirq typically intermediate, while feedback separates recoverable interface failures from deeper semantic mistakes.
Takeaways & Limitations
Modern models can produce plausible quantum code, but reliable multi-framework correctness remains out of reach.
Takeaways & Limitations
The 42-task benchmark covers only Qiskit, PennyLane, and Cirq and omits robustness to prompt variation, longer repair horizons, and tool-augmented workflows.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
1 INTRODUCTION
QuanBench+ addresses the challenge of evaluating whether LLMs generate correct quantum programs across frameworks with different abstractions and APIs. It standardizes aligned tasks and executable evaluation to separate framework familiarity from quantum reasoning and studies prompting and feedback-based repair.
- Quantum code generation requires correctness checks based on probabilistic measurement statistics, output distributions, measurement schemes, and execution settings.
- QuanBench+ holds task intent constant while varying the target framework, enabling clearer attribution of conceptual versus framework-specific errors.
- Current models show progress, but cross-framework reliability remains low and strongly framework-dependent.
- Prefill mainly reduces interface friction and boilerplate mistakes, while feedback-based repair recovers many first-attempt failures without eliminating deeper reasoning errors.
- The benchmark spans Qiskit, PennyLane, and Cirq with 42 framework-aligned tasks and automated grading.
- Evaluation standardizes executable Pass@k testing, KL-divergence-based acceptance for probabilistic outputs, and Pass@1 after feedback-based repair.
2 RELATED WORK
Prior quantum-code benchmarks cover several frameworks and task types, but many remain tied to a single ecosystem or do not hold task intent fixed. QuanBench+ extends this work by comparing aligned objectives across Qiskit, PennyLane, and Cirq.
- Classical executable evaluation established Pass@k-style testing, but deterministic task designs do not transfer cleanly to probabilistic quantum programs.
- Existing quantum-code benchmarks include Qiskit HumanEval, QHackBench, QCircuitBench, and QuanBench, covering framework proficiency, challenges, circuit generation, algorithms, state preparation, and decomposition.
- Related assistants and training resources remain ecosystem-specific, including Qiskit-focused, PennyLane-focused, and tool-augmented quantum-generation efforts.
- QuanBench+ holds task objectives fixed while varying the target framework, testing whether performance reflects portable quantum reasoning or framework-convention recall.
3 EVALUATING QUANTUM CODE GENERATION
QuanBench+ evaluates quantum code through executable functional correctness, using Pass@k for generated programs and KL-divergence thresholds for probabilistic outputs. It prioritizes task-level measurement behavior over global circuit similarity because functionally equivalent circuits can differ substantially in unitary fidelity.
- Functional correctness: Generated programs are considered correct when they execute and satisfy deterministic or distributional task-specific criteria under a fixed harness.The benchmark checks required measurement statistics or output probability distributions for prompt-specified inputs and measurement schemes.
- Correctness metrics: Pass@k measures the probability that at least one of the top-k generated solutions is correct.The benchmark reports Pass@1 and Pass@5; n denotes generated samples and c denotes correct samples.
- Probabilistic outputs: Probabilistic outputs are accepted using KL divergence between canonical distribution P and generated distribution Q, after smoothing and renormalization.Acceptance requires the resulting divergence to be below the global threshold 0.05.
- Why fidelity is excluded: Task-level functional correctness can accept circuits with low unitary-overlap fidelity when they produce the required measurement statistics.Basis-dependent phase transformations can preserve computational-basis probabilities while substantially changing fidelity.
- Why fidelity is excluded: Fidelity may produce false negatives because compilation and optimization can create syntactically different circuits that remain functionally equivalent.The benchmark therefore prioritizes executable functional evaluation and distributional comparison for prompt-level correctness across frameworks.
4 QU A NBE N C H+ BENCHMARK
QuanBench+ standardizes cross-framework benchmarking by preserving task objectives while adapting prompts and canonical solutions to Qiskit, PennyLane, and Cirq. Its edits support consistent automated grading across the three ecosystems.
- Benchmarking workflow: The benchmarking workflow defines objectives, aligns metrics with task outputs, controls execution, constructs paired prompts and canonical solutions, and uses one automated harness.This workflow is applied before comparing models and frameworks.
- Benchmark construction: The final benchmark contains 42 tasks adapted from QuanBench across Qiskit, PennyLane, and Cirq while preserving task objectives.Two original tasks were removed because they lacked reliable cross-framework grading.
- Canonical solutions: Canonical solutions are unified across models and frameworks, while each task uses the same prompt structure, inputs, and standardized probability-array outputs.Random non-trivial inputs are generated once and reused across models and frameworks.
- Prompt modifications: Prompts import the correct framework libraries and require code-only responses to improve execution efficiency.These modifications are part of the benchmark-level changes needed for consistent grading.
5 EXPERIMENTAL SETUP
The experiments evaluate diverse LLMs through a unified API and controlled framework environments. They measure one-shot and sampled generation, then test feedback-based repair after runtime errors or wrong answers.
- Generation settings: Pass@1 uses greedy decoding at temperature 0.0 with one completion per task, while Pass@5 samples five completions at temperature 0.8.The evaluated models include frontier and open-weight systems accessed through a unified API router.
- Execution environment: All generated solutions are executed in a controlled Python environment using Python 3.10, Qiskit v0.46.0, Cirq v1.6.1, and PennyLane v0.43.1.These fixed versions facilitate comparison with prior results.
- Execution and grading: Each completion is parsed, executed in its target framework, and graded with deterministic checks or a distributional threshold.The same evaluation procedure is applied to every model completion.
- Feedback-based repair: Feedback-based repair is triggered by runtime exceptions or incorrect outputs and reports Pass@1 (FB) after up to five repair chances.The corrected response is requested as code only using the original prompt and feedback context.
6 RESULTS
Results show persistent framework asymmetry: Qiskit is easiest, PennyLane hardest, and feedback repair improves all three without producing fully reliable generation. One-shot and repaired scores both reveal progress alongside continuing semantic failures.
- Cross-framework results: Qiskit is consistently easiest, PennyLane hardest, and feedback repair recovers many first-attempt failures without eliminating semantic mistakes.These patterns hold across the three frameworks and are broad rather than limited to one model.
- One-shot correctness: 59.5% Qiskit, 54.8% Cirq, and 42.9% PennyLane are the strongest one-shot Pass@1 scores.The values demonstrate progress but do not support dependable cross-framework generation.
- One-shot correctness: Framework asymmetry outweighs universal model dominance: Gemini 3 Pro leads average one-shot ranking, while GPT-5.1 achieves the best one-shot PennyLane score.Qiskit is highest and PennyLane lowest across nearly every model, with framework familiarity explaining meaningful variance.
- Prefill effects: Prefill mainly reduces interface friction and boilerplate mistakes rather than resolving semantic program-construction errors.Its largest gains tend to occur for smaller and mid-tier models when imports, signatures, and setup are easy to get wrong.
- Feedback-based repair: 83.3% Qiskit, 76.2% Cirq, and 66.7% PennyLane are the strongest Pass@1 (FB) scores after feedback repair.The middle of the ranking also improves sharply when runtime traces or wrong-answer signals are returned to models.
7 DISCUSSION
Performance remains strongly framework-dependent: Qiskit performs best, while framework-specific exposure and API familiarity still appear to matter. Feedback-based repair recovers many failures, but residual errors increasingly reflect deeper semantic mistakes; evaluation also faces canonical-solution, category-balance, and coverage limitations.
- Qiskit consistently yields the strongest outcomes, while PennyLane remains harder and Cirq typically falls between them.
- Current systems still appear to rely heavily on framework-specific exposure and API familiarity rather than portable quantum programming competence.
- Feedback-based repair recovers many first-attempt failures, but residual failures increasingly concentrate in deeper semantic mistakes.The remaining errors are distinct from runtime and interface failures, which are often recoverable.
- Threats to validity: Evaluation depends on correct and complete canonical solutions, while cross-framework adaptation can create subtle mismatches between prompts and reference implementations.The benchmark mitigates this risk by excluding ambiguously graded tasks and reviewing framework-specific canonical code for functional equivalence.
- Threats to validity: Category imbalance may amplify quantum-algorithm tasks in aggregate metrics, and framework versioning can cause execution failures when models reproduce stale APIs.
- Limitations and future work: With 42 tasks and three covered frameworks, QuanBench+ does not capture the full long tail of quantum development or broader evaluation settings.The reported metrics omit prompt variation, longer repair horizons, and tool-augmented workflows; additional frameworks remain future work.
8 CONCLUSION
QuanBench+ provides a unified, executable multi-framework evaluation of LLM quantum code generation and measures how iterative repair recovers failures. The conclusion is that reliable cross-framework correctness remains out of reach.
- 8 CONCLUSION: QuanBench+ adapts one task set across Qiskit, PennyLane, and Cirq and grades generated programs with executable functional tests.The benchmark also evaluates how much iterative repair can recover.
- 8 CONCLUSION: Reliable multi-framework quantum code generation remains unsolved despite modern models often producing plausible quantum code.
- 8 CONCLUSION: The benchmark’s comparative focus is supported by exact one-shot and feedback-repair scores reported in the core narrative.
- 8 CONCLUSION: Probabilistic-task acceptance is calibrated from repeated canonical executions using a global KL-divergence threshold of τ = 0.05.The pooled null distribution places the 99.7th percentile at 0.048, motivating the threshold.
D TASK CATEGORIES AND EXAMPLES
QuanBench+ organizes 42 tasks across quantum algorithms, gate decomposition, and state preparation, then compares one-shot and multi-sample generation across frameworks and prompting conditions. Pass@5 recovers unstable failures, while prefill reduces setup friction but does not remove framework differences.
- D TASK CATEGORIES AND EXAMPLES: 42 tasks span quantum algorithms, gate decomposition, and state preparation, with quantum algorithms dominating the benchmark mix.
- D TASK CATEGORIES AND EXAMPLES: Pass@1 measures top-1 correctness, whereas Pass@5 measures correctness across five generated solutions.
- D TASK CATEGORIES AND EXAMPLES: Large Pass@1-to-Pass@5 gaps indicate that models often generate a correct solution among several samples even when one-shot decoding misses it.
- D TASK CATEGORIES AND EXAMPLES: Pass@1 heatmaps expose framework-specific task difficulty, with persistent white columns marking tasks that remain difficult for almost everyone.
- D TASK CATEGORIES AND EXAMPLES: Prefill supplies imports, a function signature, and minimal boilerplate, while no-prefill requires generating the full solution from scratch.
- D TASK CATEGORIES AND EXAMPLES: Larger prefill gaps indicate stronger dependence on scaffolding for executable framework code, with setup sensitivity visible across PennyLane, Cirq, and Qiskit.
H ERROR DISTRIBUTIONS
Most first-attempt failures are semantic rather than syntactic or implementation-related. Wrong answers and logic errors dominate the error budget across frameworks.
- H ERROR DISTRIBUTIONS: 46.7% of Pass@1 failures are wrong answers and 25.0% are logic errors, together dominating the error budget.
- H ERROR DISTRIBUTIONS: The distribution indicates that implementation problems are secondary to semantic mistakes in first-attempt generation.
- H ERROR DISTRIBUTIONS: 11.8% of failures involve missing methods or gates, 8.0% shape mismatches, 4.7% syntax errors, and 3.9% qubit specification errors.
I FEEDBACK-LOOP RESULTS
Feedback-based repair improves functional correctness across all three frameworks, with most gains arriving early and diminishing thereafter. Repair reduces surface-level failures, but deeper semantic mistakes remain the main residual problem.
- I FEEDBACK-LOOP RESULTS: Up to 5 repair attempts were applied through feedback loops.
- I FEEDBACK-LOOP RESULTS: Feedback densifies success coverage across Qiskit, PennyLane, and Cirq, especially for stronger and middle-ranked models.
- I FEEDBACK-LOOP RESULTS: Performance increases monotonically with feedback, with most gains arriving from attempts 1→2 and diminishing returns after roughly three attempts.
- I FEEDBACK-LOOP RESULTS: Qiskit saturates earlier for the strongest models, while PennyLane and Cirq often improve more gradually through attempts 4–5.
- I FEEDBACK-LOOP RESULTS: Feedback compresses the spread among stronger models, but the weakest systems plateau quickly.
- I FEEDBACK-LOOP RESULTS: Wrong tasks decrease from 977 to 665 after feedback, while residual failures become more concentrated in wrong answers, logic errors, and shape mismatches.The reported residual shares are 53.4%, 22.0%, and 12.8%, respectively.