PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Testing the Wrong Quantum Bugs? A Study of Fault Representation Gaps in Quantum Software Testing

Asif Mohammed Saad, Ahsanul Ameen Sabit, Jonah Brandenburgh, Jiyuan Wang, Qian Zhang, Soneya Binta Hossain

arXiv:2610.22089v1cs.SEcs.PL

TL;DR

Quantum testing artifacts are widely used to evaluate techniques, but it is unclear how well their fault mechanisms represent those found in modern quantum software. The study compares four mutation tools, three curated benchmarks, and 410 verified repository bugs; 354 bugs (86.3%) fall into families absent from both artifact sources. The findings point toward testing compiler transformations, representation boundaries, and software-component interactions.

  • Problem

    Quantum testing techniques are often evaluated with mutation-generated faults and curated bug benchmarks, but the extent to which these artifacts represent faults in current quantum software is unclear.

    Source: §1 · p. 2

  • Method

    The study audits four mutation tools and three curated benchmarks, then classifies their faults alongside 410 verified bugs from ten repositories using a common taxonomy of 56 families.

    Source: §3.2 · p. 9, §6 · p. 20

  • Results

    354 of 410 recent bugs (86.3%) belong to families absent from both mutation tools and curated benchmarks.

    Source: §6 · p. 20

  • Takeaways & Limitations

    Testing quantum software should extend beyond circuit changes to compiler transformations, representation boundaries, and execution contexts needed to reproduce faults.

    Source: §6 · p. 20

    The results depend on the selected four mutation tools, three curated datasets, and ten repositories, with issues collected from July 24, 2024 through July 23, 2026.

    Source: §4.4.2 · p. 18

Abstract

from arXiv · show

Quantum software relies on compilers, intermediate representations, and execution frameworks that transform programs before execution. Faults in these components can change the behavior of otherwise correct quantum programs. Yet quantum testing techniques are often evaluated using mutation-generated faults and curated bug benchmarks. We investigate how well the faults represented by these artifacts reflect those encountered in modern quantum software. We audit four quantum mutation-testing tools and three curated bug benchmarks, and collect 410 verified bugs from ten quantum software repositories over two years. We classify faults from all three sources into a common taxonomy of 56 bug families and compare their representation across mutation operators, curated benchmarks, and recent real-world bugs. The mutation tools support 20 bug families, and curated benchmarks add 15 more. Yet only 56 of the 410 real-world bugs (13.7%) belong to families represented by either source; the remaining 354 (86.3%) belong to families absent from both. The largest unrepresented families concern compiler transformations (66 bugs), type handling (53), and input validation (47). Unlike local gate or statement changes, these faults often depend on intermediate representations, compiler pipelines, input types, or dependency versions. Existing mutation operators primarily modify circuits and program constructs, whereas many real-world faults arise in the compilers, representations, and framework interfaces that process them. Addressing this gap requires looking beyond circuit-level faults toward testing semantic preservation across compiler transformations, correctness at representation boundaries, and interactions between software components. Our study provides empirical evidence to guide fault models and benchmarks that capture these mechanisms and preserve the contexts needed to reproduce them.

1 Introduction

Quantum testing evaluates techniques with mutation-generated faults and curated bug benchmarks, but their coverage of fault mechanisms in current quantum software is uncertain. Comparing these artifacts with recent repository bugs reveals a large gap, motivating testing beyond circuit-level changes.

1 Introduction

  • Quantum faults can arise in compilers, intermediate representations, optimization passes, execution frameworks, input handling, APIs, and dependencies—not only in developer-written circuits.These components transform or interpret programs, so their faults can alter behavior even when the original circuit is correct.Source: §1 · p. 2
  • Existing quantum testing artifacts are useful for reproducible evaluation, but conclusions about testing depend on which fault mechanisms those artifacts represent.Mutation tools provide controlled faults, while curated benchmarks preserve faults collected from existing projects.Source: §1 · p. 2
  • The study compares four mutation-testing tools, three curated bug sources, and 410 verified bugs from ten repositories using a common taxonomy of 56 families.Two authors independently reviewed candidate reports, resolved disagreements through consensus, and retained the verified bugs.Source: §1 · p. 3
  • Only 56 of 410 recent bugs (13.7%) belong to families represented by mutation tools or curated benchmarks; 354 (86.3%) are absent from both.The largest unrepresented families are compiler transformation errors (66 bugs), type-handling errors (53), and missing input validation (47).Source: §1 · p. 3
  • The authors motivate fault models and benchmarks that extend beyond circuits to test compiler semantics, representation boundaries, and component interactions.Some mechanisms may suit new operators; others require regression cases preserving triggering inputs, compiler configurations, or dependency environments.Source: §1 · p. 3

2 Background & Related Work

Quantum software testing draws on empirical bug studies, mutation tools, and curated benchmarks, each representing different evidence about faults. This study connects these lines by mapping their mechanisms and recent repository bugs into a common family-level comparison.

2.1 The Quantum Software Stack and Its Bug Challenges

  • Quantum software correctness depends on both developer-written circuits and the representations and transformations used to process them.Prior work found faults in components that represent, compile, and optimize quantum abstractions, while this study asks whether testing artifacts represent those mechanisms.Source: §2.1 · p. 4

2.2 Quantum Mutation Testing

  • Existing mutation tools generate controlled program or circuit changes, and this study compares their supported mechanisms with real bug families rather than measuring mutation scores or mutant survival.Tools vary in scope, from circuit gate edits to Python program mutations and trained quantum neural network changes.Source: §2.2 · p. 5

2.3 Curated Quantum Bug Benchmarks

  • Curated benchmarks preserve real faults for reproducible testing, and the study examines which mechanisms they represent and which recent fault families they omit.The selected sources include bug reports, code changes, tests, and fault descriptions; the study treats them as existing evidence rather than introducing another collection.Source: §2.3 · p. 5

2.1 The Quantum Software Stack and Its Bug Challenges

  • Prior studies characterize faults across quantum platforms, hybrid architectures, and bug lifecycles, while mutation studies examine generated mutants and their usefulness for evaluating tests.This study instead compares whether fault mechanisms in recent software are represented by testing artifacts.Source: §2.4 · p. 6

2.4 Positioning of This Work

  • The study maps mutation operators, curated bug cases, and recent repository bugs into the same 56 families to identify what earlier testing artifacts represent or omit.Family membership does not establish that an operator can reproduce every bug in that family or that a test will detect it.Source: §2.4 · p. 6
  • Examples include a dependency-version interaction in PennyLane and a QPY serialization failure when reconstructing a circuit representation.Neither dependency incompatibility nor serialization error is represented by the selected mutation operators or curated benchmarks.Source: §2.4 · p. 6

3 Our Approach

The study compares fault families supported by mutation tools and curated benchmarks with verified bugs from recent quantum software repositories. It builds a common baseline, screens and reviews repository issues, then measures which families overlap across these sources.

3 Our Approach

  • The three-stage study builds a 35-family baseline, classifies recent repository issues, and compares the reviewed bugs with that baseline.Two authors reviewed 412 candidates and retained 410 bugs for comparison.Source: §3 · p. 6, Figure 1 · p. 7

3.1 Data Collection

  • The study audited four mutation tools and three curated benchmarks, then collected issues from ten repositories spanning the quantum software stack.The issue search covered July 24, 2024, through July 23, 2026; Qiskit and PennyLane accounted for 725 of 1,019 issues (71.1%).Source: §3.1 · pp. 7–8, Figure 2 · p. 9

3.2 Constructing the Bug-Family Baseline

  • The baseline groups faults by underlying cause: 20 families come from mutation operators, and mapping curated bugs adds 15 more.Mutation operators are mapped to their primary fault, and curated cases reuse a family when its definition matches or add one otherwise.Source: §3.2 · pp. 9–10

3.3 Screening and Classifying In-the-Wild Bug Issues

  • Issues were tiered by evidence of bug status; an agent assessed Tier 1–2 issues using issue records and linked fixes, confirming 529 of 606 as bugs.The agent proposed families from the baseline or suggested new ones, while 110 described bugs were set aside when evidence did not establish what and where the fault was.Source: §3.3 · pp. 10, 12
  • Two authors independently reviewed 412 candidates, retained 410 bugs, and revised proposed family labels for 64 (15.6%).The retained bugs yielded 431 family assignments across 36 observed families; agreement on family labels had Cohen’s κ of 0.78.Source: §3.3 · p. 13, Figure 3 · p. 13

3.4 Cross-Source Comparison

  • A repository bug counts as represented when at least one assigned family occurs in the mutation-tool or curated-benchmark baseline; family counts are reported separately from bug counts.A bug is classified as in-the-wild only when none of its assigned families occurs in that baseline.Source: §3.4 · pp. 13–14

4 Results

The results compare four mutation tools and three curated bug sources with 410 verified bugs, classifying faults across 56 families. Most recent bugs fall into families absent from both earlier sources, while the results also identify limits in coding and corpus scope.

4.1 RQ1: Mutation-Supported Bug Families

  • The four mutation tools support 20 families—11 quantum-specific and 9 inherited classical families—with gate edits receiving the broadest support.None targets compiler passes, serialization, dependency versions, or component interactions; only Wrong constant and Wrong decorator among the classical families occur in the 410-bug corpus.Source: §4.1 · pp. 14–15

4.2 RQ2: Curated Real Bug Families

  • The curated sources contain 310 bugs and patterns across 35 families, adding 15 beyond the mutation-supported families; their cases are concentrated in program-level framework use.Most cases come from Qiskit and 2020 or earlier; 13 families account for 80.6% of curated bugs.Source: §4.2 · p. 15

4.3 RQ3: Comparison with In-the-Wild Bugs

  • 354 of 410 recent bugs (86.3%) belong to families absent from both earlier sources, with in-the-wild-only families spanning 361 bugs.At the family level, 21 of the 36 observed families are absent from both sources.Source: §4.3 · pp. 15–16, Figure 5 · p. 16
  • The five largest in-the-wild-only families are compiler transformation error (66), type-handling error (53), missing input validation (47), circuit construction error (29), and dependency incompatibility (28).These faults can depend on pass sequences, input types or shapes, and specific library versions.Source: §4.3 · pp. 16–17
  • Mutation tools match in-the-wild bugs mainly on local edits, while curated sources add framework-use faults; changing platform versions also leave earlier benchmarks without later faults.The curated-only families appearing in 34 bugs concern API contracts, object fields, method calls, and control logic.Source: §4.3 · pp. 15–16
  • 271 of 410 bugs (66.1%) are quantum-specific, and quantum-specific bugs are the majority in 26 of 36 observed families.Nineteen bugs span multiple families, including a CUDA-Q example combining classical control and compiler transformation errors.Source: §4.3 · pp. 17–18

4.4.1 Internal and Construct Validity.

  • Classification involves judgment: review changed 64 retained classifications, and the reported 84.4% agreement may be inflated because both reviewers saw the supplied proposal.Issue records may omit patches, regression tests, or full root-cause analyses, so some bugs may be excluded or classified at a broader mechanism level.Source: §4.4.1 · p. 18

4.4.2 External Validity.

  • The corpus covers selected tools, datasets, and ten repositories during a two-year window, and its observed distribution may not generalize to other quantum projects or periods.Qiskit and PennyLane contribute 310 of the 410 bugs; the authors frame the reported shares as evidence about the selected corpus, not all quantum software.Source: §4.4.2 · pp. 18–19

5 Discussion and Lessons

Most recent bugs fall outside the families represented by existing mutation tools and curated benchmarks, and many arise in compilers, representations, or software-component interactions. The discussion calls for fault models and benchmarks that capture those mechanisms and preserve the conditions needed to reproduce failures.

Overall findings

  • 354 bugs (86.3%) belonged to families absent from both mutation tools and curated benchmarks, while 23 (5.6%) belonged to mutation-supported families.These figures measure fault representation, not the detection capability of individual testing techniques.Source: §5 · p. 19

Lesson 1: Correct circuits do not guarantee correct compilation

  • Compiler transformation errors (66 bugs) were the largest uncovered family; other representation-boundary findings included 53 type-handling, 47 input-validation, 20 format-conversion, and 18 serialization errors.The findings motivate checks for contracts between representations, including accepted types, intermediate-representation invariants, conversions, and serialization round trips.Source: §5 · p. 19

Lesson 3: Fault models must capture context, not just local edits

  • Uncovered faults can depend on compiler pipelines or library versions; 28 involved dependency incompatibilities, and 19 bugs had multiple family assignments.Fault models may need to preserve particular inputs, configurations, and dependency environments rather than only make local edits.Source: §5 · p. 20

Lesson 4: Benchmarks must preserve failure conditions and evolve with the software

  • Twenty-one recent bug families were absent from both earlier sources, and benchmarks need to retain failure conditions such as compiler pipelines, input representations, and dependency versions.The discussion recommends retaining these conditions alongside the buggy version, fix, and regression test as platforms evolve.Source: §5 · p. 20

Methodological implication

  • Nineteen bugs involved multiple families, and independent review changed 64 proposed classifications, motivating auditable mappings that distinguish bug counts from family assignments.The discussion identifies compiler transformations, representation boundaries, and execution context as targets for future testing techniques, fault models, and benchmarks.Source: §5 · p. 20

6 Conclusion

The study compared four mutation tools, three curated benchmarks, and 410 verified bugs from ten repositories using a common taxonomy. It found that most recent bugs belonged to families absent from both sources, pointing to compiler transformations, representation boundaries, and execution context as targets for future testing.

Results

  • 354 of 410 bugs (86.3%) belonged to families absent from both mutation tools and curated benchmarks; 23 (5.6%) belonged to mutation-supported families.The comparison used 56 bug families across four mutation tools, three curated benchmarks, and ten repositories.Source: §6 · p. 20

Implications

  • Future fault models and benchmarks should account for compiler semantics, representation boundaries, and the conditions needed to reproduce real faults.Many real faults arise in compiler transformations, type handling, input validation, and component interactions that isolated program or circuit mutations cannot reproduce.Source: §6 · p. 20
Loading 2610.22089v1…