Identifying Flakiness in Quantum Programs

I. INTRODUCTION

• The study investigates 14 quantum software repositories and reports flaky tests in 12, with 46 distinct reports and lower-bound issue percentages of 0.26%–1.85%.The authors also identify eight cause groups and seven common fixes.Source: §I · p. 1

II. EMPIRICAL STUDY: DATA GATHERING

• The study examines four popular quantum platforms and selects 14 active repositories with bug reports, while noting that this is not an exhaustive set.The platforms are IBM Qiskit, Microsoft Quantum Development Kit, TensorFlow Quantum, and NetKet.Source: §II · p. 1
• The authors search closed GitHub issues using ten flakiness-related keywords because closed reports have been verified by developers.The search is followed by manual review of issues and associated pull requests and commits.Source: §II · pp. 1–2
• At least two authors cross-examine candidate reports and related changes; their review labels 46 of 253 keyword-search results as flaky and retains 12 repositories.The paper uses “flaky test reports” for four observed report-and-fix cases, including issue reports with fixes, pull-request-only fixes, and closed reports without an associated pull request.Source: §II · p. 2

A. RQ1: How prevalent are flaky tests in quantum programs?

• 46 flaky test reports occur across 12 repositories; reported lower-bound percentages range from 0.26% in qiskit-machine-learning to 1.85% in Microsoft/Quantum.qiskit-terra has the most reports, with 25; keyword limits mean the reported percentages may underestimate prevalence.Source: A. RQ1: How prevalent are flaky tests… · p. 2
• Direct frequency comparisons between quantum and classical programs are not possible because the studies report different denominators, such as flaky reports versus test cases or failures.The authors plan to map reports to test cases for a future comparison.Source: A. RQ1: How prevalent are flaky tests… · p. 2

B. RQ2: What causes flakiness in quantum software?

• Five cause categories overlap with classical-software findings: concurrency, network, randomness, floating-point operations, and unordered collections.The paper notes that the leading causes in the cited classical-software study were async waits and concurrency, unlike the quantum-study leaders.Source: B. RQ2: What causes flakiness in… · p. 2
• Randomness accounts for 21% of reports and software environments for 15%, making them the leading reported causes in the quantum programs studied.The analysis identifies eight cause categories, with “others” and “unknown” as special cases; one report is counted in both randomness and floating-point categories, yielding 47 category assignments for 46 reports.Source: B. RQ2: What causes flakiness in… · p. 2
• The prominence of randomness resembles findings in probabilistic programming and machine learning, where randomness may cause 60% of flaky tests; quantum simulators’ PRNG use is cited as relevant context.The paper contrasts this with generic classical-software studies, where randomness is reported as a non-major cause.Source: B. RQ2: What causes flakiness in… · p. 2, 1) Randomness · p. 3

1) Randomness:

• A two-qubit decomposer has a nondeterministic component that can generate random results when producing controlled versions of the same unitary gate.The example’s equality operation can consequently return either true or false at random.Source: 1) Randomness · p. 3
• The decomposer example demonstrates that repeated equality checks on the same unitary-gate construction can return different results.The reported equality operation returns “True” or “False” at random.Source: 1) Randomness · p. 3
• The test_append_circuit logic is correct, but random_circuit’s default randomly selected seed can make the test fail occasionally.The cited test checks whether appending quantum circuits works properly.Source: 1) Randomness · p. 3

2) Software Environment:

• A dependency-related test failed on GitHub Actions with Python 3.10 but was not reproduced locally, and a common fix is to alter the software environment.The cited pull request describes simplifying the test to avoid the failure.Source: 2) Software Environment · p. 3

3) Multi-Threading:

• Parallel documentation builds caused address collisions when multiple sphinx-build jobs competed, illustrating a multithreading-related flakiness cause.The example describes parallelization as improving throughput while also producing an error.Source: 3) Multi-Threading · p. 3

4) Floating Point Operations:

• A hard-coded 10−5 tolerance caused intermittent failures, addressed by changing assertion tolerance, rounding actual values, or removing the test.Developers also adjusted the software environment to address dependency-related failures.Source: 4) Floating Point Operations · pp. 3–4, 2) Software Environment · p. 3

5) Visualization:

• The three visualization reports had no recurring fix pattern, and one involved out-of-date reference indexes in qiskit-terra’s test-manager visualizer.The visualizer schedules tests sequentially and lets them communicate with one another.Source: 5) Visualization · p. 4

6) Unhandled Exception:

• Unhandled exceptions caused intermittent test failures when a number fell outside the expected range.The example is from Microsoft/QuantumLibraries.Source: 6) Unhandled Exception · p. 4

7) Network:

• Network-related failures included timeouts and socket problems when jobs completed before a websocket connection could be established.The example is issue #584 in qiskit-ibm-runtime.Source: 7) Network · p. 4

8) Unordered Collection:

• Python dictionary ordering was implicated when a test compared dictionaries using insertion order, creating nondeterministic outcomes for order-dependent tests.The passage describes dictionaries as unordered collections from Python 3.3 to 3.7.Source: 8) Unordered Collection · p. 4

9) Others:

• Other examples include a typo that injected a random value into a test and local Jupyter Notebook outputs that kept flaky tests out of the CI pipeline.The typo example is from Microsoft/Quantum; the notebook example is from TensorFlow/quantum.Source: 9) Others · p. 4
• When QC configuration changed, hard-coded expected values caused intermittent failures; testers resolved this by reading configuration parameters from files.Source: 9) Others · p. 4
• Some reports had an unknown cause because the available issue details or associated commits were insufficient to identify it.One qiskit-machine-learning report was closed with insufficient information.Source: 10) Unknown · p. 4

C. RQ3: How do quantum programmers fix flaky tests?

• Seven recurring fix patterns cover 49% of fix reports; the remaining 51% are one-offs, and visualization has no recurring pattern.Visualization-related flaky tests were observed only in qiskit-terra.Source: C. RQ3: How do quantum programmers fix… · p. 4

1) Fix Seed:

• 9 of 10 randomness-related reports (19% of all reports) use constant seeds, sometimes with a reduced convergence threshold.One report instead replaces the PRNG with a deterministic formula.Source: 1) Fix Seed · pp. 4–5
• Fixed PRNG seeds can limit execution paths that expose real bugs, and a changed PRNG algorithm may reintroduce flakiness.The paper also reports no observed robust distribution-based fixes; possible barriers include execution cost and the effort of building or finding an existing framework.Source: 1) Fix Seed · p. 5
• Four of seven software-environment reports (9% of all reports) were fixed by upgrading or changing dependencies.The example removes dependence on a specific NumPy version.Source: 2) Alter Software Environment · p. 5

2) Alter Software Environment:

• The remaining three reports were addressed by changing CI configuration, simplifying a test, or closing the report after dependency changes.Source: 2) Alter Software Environment · p. 5

3) Make Single Thread:

• 2 of 6 multi-threading-related reports (4% of all reports) were resolved by limiting execution to one thread, though performance overhead may result.The examples include timeout issues and disabling parallelization.Source: 3) Make Single Thread · p. 5

• Three of five floating-point-related reports (6% of all reports) were fixed by adjusting tolerance, but loose hard-coded tolerances may increase false negatives.One example doubles the tolerance value.Source: 4) Adjust Tolerance · p. 5

4) Adjust Tolerance:

• Dynamic tolerance is presented as a more robust adjustment than a fixed tolerance for a reported floating-point issue.Source: 4) Adjust Tolerance · p. 5
• Developers also round actual values or remove a flaky test case when it is deemed non-essential.Source: 4) Adjust Tolerance · p. 5

5) Add Exception Handler:

• All three reports (6% of all reports) involving unhandled exceptions were fixed by adding an exception handler or removing the exception.One example adds a condition to remove unexpected negative coefficients.Source: 5) Add Exception Handler · p. 5

6) Synchronize:

• One network-related report (2% of all reports) was addressed by increasing callback-test iterations, though the count is hard-coded; synchronization with socket-test completion is described as ideal.Source: 6) Synchronize · pp. 5–6

7) Use Keys for Order:

• For one dictionary-ordering report (2% of all reports), developers can order by keys or upgrade to Python 3.8+, though upgrades may cause dependency issues.Source: 7) Use Keys for Order · p. 6

IV. THREATS TO VALIDITY

• Manual data collection and ten search keywords may miss reports, while findings from 14 open-source projects may not generalize to other projects.The authors state that keyword false negatives lead to underestimation of flaky-test reports.Source: §IV · p. 6

V. RELATED WORK

• Quantum testing work applies software-engineering principles, while four causes in a 10-cause classical flaky-test taxonomy overlap with this study’s categories.Simulator debugging is available but limited to small problems,.Source: §V · p. 6

VI. CONCLUSIONS AND FUTURE WORK

• 46 flaky-test reports across 12 of 14 quantum programs correspond to at least 0.26%–1.85% of issue reports; the study identifies eight cause groups and seven common fixes.The issue-report range applies to the 12 programs with identified flaky tests.Source: §VI · p. 6
• Future work will expand repository and keyword searches, compare affected-test fractions across quantum and classical programs, and automate flaky-test detection and fixes.The authors also hope the findings will support better bug-report detection, mitigation strategies, and flaky-test datasets.Source: §VI · p. 6