PaperlayerView exact revision on arXiv

Source-linked AI summary

QuSema: Detecting Silent Bugs in Quantum Libraries via Quantum-knowledge-enhanced Agents

Yujin Song, Kaining Zhang, Qixin Zhang, Shuai Wang, Pingchuan Ma, Yuxuan Du

arXiv:2610.10258v1cs.SEcs.AIquant-ph

TL;DR

Black-box testing may lack expected execution-result relations for some library functionality, while generic test generation can struggle to reach bugs requiring specific semantic conditions. QuSema uses quantum semantics and documentation as source-level constraints on implementation logic, then checks whether candidate defects can be reproduced through documented APIs. On a benchmark of 20 historical silent bugs, QuSema substantially outperformed Claude Code and Codex in localizing known defects and uncovered 40 developer-confirmed bugs, including 30 silent bugs.

  • Problem

    Black-box testing may lack expected execution-result relations for some library functionality, while generic test generation can struggle to reach bugs requiring specific semantic conditions.

    Source: §1 · p. 2, §3.1 · pp. 5–6

  • Method

    QuSema uses quantum semantics and documentation as source-level constraints on implementation logic, then checks whether candidate defects can be reproduced through documented APIs.

    Source: §1 · p. 2, §4.1 · p. 9, §9 · p. 18

  • Results

    On a benchmark of 20 historical silent bugs, QuSema substantially outperformed Claude Code and Codex in localizing known defects and uncovered 40 developer-confirmed bugs, including 30 silent bugs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    QuSema's source-level semantic approach can identify silent bugs beyond the scope of existing differential and metamorphic testing.

    Source: §1 · p. 2, §9 · p. 19

    Black-box approaches may lack suitable execution-result relations, and generic testing may need substantial exploration when defects require specific semantic conditions.

    Source: §3.1 · pp. 5–6

Abstract

from arXiv · show

Quantum libraries are now critical infrastructure for quantum algorithm development, yet their correctness remains difficult to test. Existing testing techniques mainly rely on failure-based or comparison-based oracles, exposing bugs only when executions fail, violate runtime checks, or disagree with another implementation. Their applicability is limited when suitable execution-based oracles are unavailable, leaving some silent bugs undetected. Such missed bugs can produce incorrect results that propagate into experimental conclusions, simulation studies, and algorithmic designs. Here we present QuSema, an autonomous testing agent for finding silent bugs in quantum libraries. QuSema uses constraints from quantum semantics and documentation as a source-level semantic oracle to assess whether implementation logic can produce invalid outputs from valid inputs. It operates through an agentic loop that repeatedly inspects library API documentation and source code, reasons about the intended behavior of quantum operations, identifies potential semantic deviations, and validates them by generating executable tests through library APIs. Guided by quantum-domain reasoning, QuSema turns high-level behavioral mismatches into concrete, user-triggerable bug reports, enabling it to uncover non-crash defects. We implement QuSema for Qiskit and PennyLane. On a benchmark of 20 historical silent bugs, QuSema achieves higher mean bug relocation counts than Claude Code and Codex, with the DeepSeek configuration costing less than Claude Code. QuSema also discovers 40 previously unknown bugs confirmed by the developers, including 30 silent bugs.

1 Introduction

QuSema targets silent quantum-library bugs that black-box tests may miss when execution-result relations are unavailable or semantic trigger conditions are hard to reach. It reasons from source code, quantum semantics, and documentation, then tests whether defects can be triggered through library APIs.

1 Introduction

  • Quantum libraries such as Qiskit and PennyLane have become critical infrastructure for developing, simulating, and executing quantum algorithms, while remaining rapidly evolving software with bugs.Silent incorrect results can propagate into algorithm design, hardware execution, and resource use.Source: §1 · pp. 1–2
  • Black-box differential and metamorphic testing can lack suitable expected-result relations, while generic generation may struggle to reach bugs requiring specific semantic conditions.These limits leave some quantum-library functionality difficult to test and can make triggering inefficient.Source: §1 · p. 2
  • QuSema uses LLMs to compare implementation logic with quantum semantics and documentation, then validates candidate defects through execution and documented library APIs.This source-level semantic oracle extends detection beyond cases covered by existing execution-based relations.Source: §1 · p. 2
  • On 20 historical silent bugs, QuSema substantially outperformed Claude Code and Codex in localizing known defects; on Qiskit and PennyLane, it uncovered 40 developer-confirmed unknown bugs, including 30 silent bugs.The bugs span quantum-program execution, semantic analysis, and library infrastructure; five silent bugs affect functionality beyond existing differential and metamorphic testing.Source: §1 · pp. 2–3

2 Background

The background introduces quantum states, gates, and circuits; describes quantum-library functionality and APIs; and reviews existing silent-bug testing approaches and their oracle constraints.

2.1 Basic Concepts in Quantum Computing

  • An n-qubit circuit applies gates to quantum states, and different circuit structures can implement the same computation while differing in depth and gate count.The background defines quantum states as superpositions of basis states and quantum gates as unitary operations.Source: §2.1 · p. 3

2.2 Functionality and Software Structure of Quantum Libraries

  • Quantum libraries support programming abstractions, compilation, and execution environments, exposing functionality through documented APIs implemented by source-code functions or methods.The APIs specify intended usage, accepted inputs, expected outputs, and relevant behavioral constraints.Source: §2.2 · pp. 3–4

2.3 Existing Approaches to Bug Detection in Quantum Libraries

  • Differential testing compares corresponding executions across libraries, while metamorphic testing checks relations expected to hold after semantics-preserving transformations.Both depend on black-box oracles, limiting coverage when comparable executions or suitable transformations are unavailable; QuSema instead derives semantic oracles from implementation logic.Source: §2.3 · p. 4

3 Current Limitations and Semantic Evidence for Quantum Silent Bug Detection

The section identifies oracle availability and inefficient triggering as limits of black-box testing, then motivates API-local analysis using source-level semantic evidence. It frames quantum-semantic reasoning and valid-API triggerability as the central challenges for reliable detection.

3.1 Two Limitations of Existing Approaches

  • Black-box differential and metamorphic testing can lack expected execution relations and may require extensive exploration to trigger defects requiring specific quantum-semantic conditions.Resource estimation illustrates both gaps: equivalent circuits can yield different gate counts, and PennyLane's functionality lacks a Qiskit counterpart; a CU-gate defect also requires specific parameter conditions.Source: §3.1 · pp. 5–6

3.2 Motivating Study: Source-Level Semantic Evidence

  • The authors define silent bugs as invalid outputs from valid inputs without explicit failure and localize their origin to the first API that introduces the deviation.Screening 3,184 Qiskit and PennyLane reports found no silent-bug counterexample to this API-localization criterion; the Qiskit HoareOptimizer example shows source logic conflicting with circuit semantics and documented node replacement.Source: §3.2 · pp. 6–7, Figure 2 · p. 7
  • Source-level semantic evidence can reveal defects without predefined execution-result relations and can help identify triggering conditions for tests.The motivating Qiskit case illustrates both uses: implementation and documentation expose the deviation, and its semantic conditions can guide input construction.Source: §3.2 · pp. 7–8

3.3 Implications and Challenges

  • Reliable detection must incorporate quantum semantics into source-level reasoning and determine whether an internal defect is reachable through a documented API with valid use.Defects unreachable through documented APIs are outside the work's scope.Source: §3.3 · p. 8

4 QuSema: Semantics-Driven Detection of Silent Bugs in Quantum Libraries

QuSema uses LLMs to assess implementation logic against quantum semantics and documentation, organizing analysis around focused code segments and relevant context. Its workflow verifies suspected defects and then checks whether documented APIs can expose them through valid inputs.

4.1 Overview of QuSema

  • QuSema's three-stage workflow prepares contextual units, detects semantic defects, and searches for API-level triggers before classifying reproduced cases.The method uses quantum semantics and documentation as constraints on implementation behavior.Source: §4 · p. 8, §4.1 · p. 9

4.2 Contextual Unit Preparation

  • Contextual units pair a target code segment with relevant documentation and call relations, keeping initial analysis focused while retaining access to dependencies.The design addresses the risk that whole files contain irrelevant context while isolated functions omit necessary implementation logic.Source: §4 · p. 8, §4.2 · p. 9, Figure 3 · p. 10

4.3 Semantic Defect Detection

  • Semantic defect detection treats quantum semantics and documented behavior as constraints, forms testable hypotheses for valid inputs, and retains candidates only when execution reproduces the predicted violation.The LLM can follow call relations and revise a hypothesis when observed behavior does not match its prediction.Source: §4.3 · p. 11

4.4 Library API Triggering and Bug Validation

  • API triggering searches for documented entry points and valid inputs that reach a verified defect; qualification checks reproduction and classifies cases by whether documentation permits the input.Inputs valid but not established as supported by API documentation are retained as potential bugs for developer assessment.Source: §4.1 · p. 9, §4.4 · p. 12

5 Experimental Setup

The evaluation tests QuSema against coding agents and examines its components on 20 historical silent bugs, then applies it to full Qiskit and PennyLane repositories. Three research questions cover comparative performance, component contributions, and discovery of previously unknown bugs.

Research questions and subjects

  • RQ1 compares coding agents, RQ2 tests QuSema components, and RQ3 examines discovery in Qiskit 2.4.1 and PennyLane 0.45.0.RQ3 uses the complete source code of both library versions; Table 2 reports library statistics, including documented API counts and code-size measures.Source: §5 · p. 12, Table 2 · p. 13

Experimental subjects and benchmark

  • The historical benchmark contains 20 maintainer-confirmed bugs, 10 from each library; 10 lack comparable cross-library implementations, limiting differential testing.Metamorphic testing is also insufficient for those 10 cases.Source: §5 · p. 12, Table 1 · p. 13

Configurations

  • RQ1 compares QuSema using DeepSeek-V4.1-Flash or Opus 5 with Claude Code and Codex, while RQ2 controls the semantic oracle, contextual preparation, and hypothesis verification.The benchmark, environment, execution budget, and other settings remain fixed within each RQ2 comparison.Source: §5 · p. 13

Procedure and metrics

  • For historical relocation findings, Table 4 summarizes deduplicated candidate outcomes; RQ3 reports each unique finding to maintainers and records its status.Reported statuses are fixed, confirmed, pending, or rejected.Source: Table 4 · p. 14, §5 · p. 14
  • Three independent runs assess each historical-bug configuration; defect identification requires a matching ground-truth defect and correct semantic explanation, while relocation additionally requires reproducible valid-API evidence.RQ1 reports relocation counts and total cost; RQ2 reports identification and relocation counts for its two comparisons.Source: §5 · p. 14, Table 3 · p. 14

6 Evaluation Results

The evaluation compares QuSema with coding agents on historical silent bugs, tests component contributions, and reports previously unknown bugs confirmed in Qiskit and PennyLane.

6.1 RQ1: Historical Bug Relocation

  • Each run yielded 56–73 candidates, with 54–72 proceeding after validation; Stages II–III excluded 0–4 non-reproducible or invalid-use cases, and non-target reports included 15 independently reported issues.Other manually reviewed reports were consistent with the silent-bug criteria but had not yet been submitted for maintainer validation.Source: §6.1 · p. 15
  • 16.00 (Opus 5) and 15.67 (DeepSeek-V4.1-Flash) exceed Claude Code’s 14.67 and Codex’s 13.33 mean relocations; DeepSeek costs $162.27 across three runs, about one-tenth of Opus and less than Claude Code.QuSema with DeepSeek also achieves higher mean relocation counts than Claude Code at lower cost and than Codex at a modest cost increase.Source: §6.1 · pp. 14–15

6.2 RQ2: Component Contributions

  • Defect identification averaged 13.67 on full source files, 15.00 with Contextual Unit Preparation, and 15.67 after adding the Source-level Semantic Oracle.Context preparation focuses analysis on relevant implementation context; the oracle supplies evidence for distinguishing semantic defects from permissible implementation choices.Source: §6.2 · p. 15, Figure 4 · p. 16
  • Hypothesis verification raised mean relocations from 15.33 to 15.67 and corrected an invalid universal-gate-set assumption in Qiskit #11974.The figure reports relocation counts for runs with and without verification in Stages II and III.Source: §6.2 · p. 15, Figure 4 · p. 16

6.3 RQ3: Previously Unknown Bugs

  • QuSema reported 40 previously unknown, maintainer-confirmed bugs: 30 silent bugs and 10 crashes or exceptions on valid inputs.The evaluation analyzed Qiskit 2.4.1 and PennyLane 0.45.0 using DeepSeek-V4.1-Flash (Max).Source: §6.3 · pp. 16–17
  • Five silent bugs affect PennyLane functionalities without equivalent evaluated Qiskit APIs or applicable circuit-based metamorphic relations.Examples include Bosonic-operator multiplication and measurement-condition comparison.Source: §6.3 · p. 16
  • Examples expose distinct silent mechanisms: insertion-order mismatch exchanges bosonic operators, while wrong nested-control order changes control state from 1 to 2 without an error.QuSema’s root-cause information can guide developers’ subsequent diagnosis and repair.Source: §6.3 · pp. 16–17, Figure 5 · p. 16

7 Discussion and Threats to Validity

QuSema’s semantic judgments depend on LLM knowledge and documentation, and its evaluation has threats involving repeatability, manual assessment, and generalizability. The authors note that results may not extend beyond the evaluated libraries and API-local silent bugs.

Construct Validity

  • QuSema’s LLM-based judgments may be affected by incomplete knowledge or hallucinations, while incorrect documentation may also affect bug assessment.Execution-based hypothesis verification cannot guarantee reliable detection, and documentation errors require maintainer review.Source: §7 · p. 18

Internal Validity

  • LLM outputs vary across runs and versions, candidate reports receive subjective manual review, and configuration-level comparisons cannot isolate agent-design effects.The authors repeat each configuration three times; public benchmark issues may have appeared in model training data, though RQ3 evaluates previously unknown bugs.Source: §7 · p. 18

External Validity

  • Evaluation covers 20 historical bugs from Qiskit and PennyLane, while practical discovery uses Qiskit 2.4.1 and PennyLane 0.45.0.Results may not generalize to closed-source or poorly documented libraries, other languages, or different binding systems.Source: §7 · p. 18
  • QuSema targets API-local silent bugs and does not directly address bugs arising solely from interactions among individually correct APIs.Applying it elsewhere may require changes to code segmentation, call-relation analysis, and documentation association.Source: §7 · p. 18

8 Related Work

Related work spans quantum-software correctness and testing, as well as LLM-based software analysis. QuSema distinguishes itself by auditing quantum-library logic with quantum semantics and documentation, then validating defects through documented APIs.

Quantum software correctness and testing

  • QuSema uses quantum semantics and documentation as source-level correctness evidence for library logic without requiring predefined execution relations.Prior work includes language and type-system support, formal verification, static analysis, differential and metamorphic testing, and codebase-guided test generation.Source: §8 · p. 18

LLM-based software analysis and semantic auditing

  • QuSema combines documentation with quantum semantics for source-level auditing and validates candidate defects through documented library APIs.Related LLM approaches include repository issue resolution, test and oracle generation, execution-feedback test generation, and specification-based auditing.Source: §8 · p. 18

9 Conclusion

QuSema audits quantum-library implementation logic against quantum semantics and documentation, then uses library APIs to test whether candidate defects yield invalid outputs. Applied to Qiskit and PennyLane, it discovers 40 previously unknown bugs.

9 Conclusion

  • QuSema analyzes implementation logic against quantum semantics and documentation, then tests candidate defects through library APIs.The approach does not depend on cross-library counterparts or predefined metamorphic relations.Source: §9 · pp. 18–19
  • 40 previously unknown bugs were discovered by QuSema when applied to Qiskit and PennyLane.Source: §9 · p. 19
Loading 2610.10258v1…