Source-linked AI summary
HyQDB: LLM-Assisted Debugging for Hybrid Quantum Workflows
Charlie Campbell, Hao Mark Chen, Shuang Liang, Hongxiang Fan
TL;DR
Silent hybrid-program faults are poorly represented in public-data evaluations, despite their prevalence in expert reports. HyQDB combines deterministic fault evidence with intent reconstruction, raising held-out repair accuracy from 45±1.7% to 75±5.8%.
Problem
Public-data evaluations underrepresent expert-reported hybrid-program faults, including silent failures that existing methods do not diagnose by fault type.
Source: §I · p. 1Method
HyQDB injects evidence from deterministic analyzers into LLM repair and escalates evidence-free cases to intent reconstruction.
Source: §I · p. 1, B. Intent Reconstruction · p. 4Results
45±1.7% to 75±5.8% repair accuracy on held-out human-authored programs, from the base model to HyQDB with intent reconstruction.
Source: A. Key Results · p. 5Takeaways & Limitations
Detector evidence improves mechanical-fault repair, while conceptual faults improve only after intent reconstruction is added.
Source: A. Key Results · p. 5, §VIII · p. 8QFaultBench has only 62 tasks, some fault families have few examples, and detector thresholds and pattern coverage do not transfer perfectly to held-out programs.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Hybrid quantum program failures frequently occur silently, yet existing debugging tools provide limited support for detecting and repairing them. These faults dominate the failures reported by domain experts, yet existing tools evaluate on public data that under-represents this failure mode. Our key insight is that faults divide into two classes that require different strategies: mechanical faults, which allow deterministic analysis, and conceptual faults, which require reconstructing the program's intent. To address this challenge, we present HyQDB, a tiered agent that injects deterministic hardware, physics and optimization evidence into the LLM repair process. When no evidence is detected, the agent treats this silence as a signal to escalate to a second intent-reconstruction tier, that infers the program's behaviour and reconciles it with the implementation. To evaluate HyQDB, we introduce QFaultBench, a benchmark built from an expert-derived fault taxonomy that represents the true failure modes of hybrid programs. On a held-out set of human-authored programs, HyQDB raises repair success over a standard LLM from 45% to 75%. We show that the escalation gate is crucial, giving a 20% improvement in mechanical fault repair accuracy.
I. INTRODUCTION
Hybrid quantum programs can fail silently at component boundaries, and expert-reported faults are poorly represented in public-data evaluations. HyQDB separates mechanical and conceptual faults, while QFaultBench evaluates repairs against an expert-derived taxonomy.
I. INTRODUCTION
- Silent faults can corrupt scientific conclusions without crashing, and they dominate expert-reported faults.Hybrid programs interleave a quantum circuit with a classical optimization loop, creating fault opportunities at their boundaries.Source: §I · p. 1
- Public-data benchmarks reflect reported failures rather than the failure modes experts experience.Only 35% of GitHub faults produced a wrong output, versus over 75% of expert-reported faults.Source: §I · p. 1
- Existing tools address only parts of the fault space: static analyzers miss execution semantics, and prior systems cover limited fault types or data.QBugLM focuses on syntactic and structural circuit faults, while MetaMorphQ does not address faults manifesting during training.Source: §I · p. 1
- HyQDB classifies faults as mechanical or conceptual, using deterministic analysis for the former and intent reconstruction for the latter.When deterministic analysis finds no evidence, the framework routes the repair to its intent-reconstruction tier.Source: §I · p. 1, Figure 1 · p. 1
- QFaultBench contains 62 taxonomy-derived faults and uses hidden execution-checked asserts, including a human-authored held-out set.The held-out set is used to demonstrate performance generalization.Source: §I · p. 2
II. BACKGROUND AND MOTIVATION
Hybrid programs combine quantum circuit execution with classical parameter updates, but existing debugging tools leave the classical loop and its interface unsupported. Prior evidence also indicates that public fault collections do not reflect the silent failures developers report.
II. BACKGROUND AND MOTIVATION
- A hybrid program executes a parameterized circuit, uses its measurement in a classical program, and repeats after updating circuit parameters.Existing tools focus on the circuit, leaving the classical loop and its interface—where most faults are concentrated—unsupported.Source: §II · p. 2
A. Studies of Hybrid Quantum Software Faults
- Only 35% of GitHub faults produce a wrong output without a crash, compared with 75% in domain-expert interviews.An independent survey of 26 developers likewise found incorrect outputs without crashes to be the most common bug type; 69% used no dedicated debugging tool.Source: A. Studies of Hybrid Quantum Software… · p. 2
- Publicly collected bugs can be incomplete or unrepresentative: Bugs4Q had 41% silent-profile bugs, and many bugs were not reproducible.The reported reasons included missing test cases and failure to run in the latest environment.Source: A. Studies of Hybrid Quantum Software… · p. 2
- Current manual debugging uses optimization-first gradient checks or problem-first checks of the Hamiltonian and qubit mapping.Neither procedure is automated in current tooling, so debugging requires manual tracing and domain expertise.Source: A. Studies of Hybrid Quantum Software… · p. 2
B. Current Debugging Tools
Existing quantum debugging tools often focus on circuits rather than hybrid programs and have limitations in execution awareness, repair, or evaluation scope. These gaps motivate grounding diagnosis in deterministic evidence collected before and during optimization.
B. Current Debugging Tools
- Existing tools are designed for standard quantum circuits, not hybrid quantum programs.The comparison is framed around capabilities needed to address silent hybrid faults.Source: B. Current Debugging Tools · p. 2
- LLM-based linters improve error detection precision from 62% to 70%, but perform poorly on complex data flow and cannot repair detected errors.LintQ-LLM achieves 30% on complex data-flow problems and fails 10% of tasks due to a limited context window.Source: B. Current Debugging Tools · p. 2
- QBugLM reports that a single retry raises repair rate from 25% to 80%, but its evaluation uses a limited set of scripts with GitHub-derived faults.The passage notes that this taxonomy is not representative of the true faults quantum developers face.Source: B. Current Debugging Tools · p. 2
- MetaMorphQ tests physics-based invariants before optimization, but its evaluation on LLM-generated scripts leaves human-authored performance and broader generalizability unknown.Its described framework targets variational quantum eigensolvers.Source: B. Current Debugging Tools · pp. 2–3
C. Key Insights
- Existing tools lack deterministic, execution-aware evidence, motivating the use of data collected both before and during optimization.Linters do not execute programs, while QBugLM decisions rely on unaugmented LLM judgment.Source: 1) Insight #1 · p. 3
2) Insight #2:
HyQDB uses intent reconstruction when mechanical evidence cannot explain a fault, and its design injects collected analyzer evidence directly into the repair process. The benchmark is motivated by the mismatch between public data and expert-reported fault patterns.
2) Insight #2:
- Because circuit checks alone cannot determine whether a program solves its intended problem, HyQDB adds intent reconstruction when mechanical evidence cannot explain a fault.The intent step is motivated by the need to distinguish conceptual faults from mechanical ones.Source: 2) Insight #2 · p. 3
3) Insight #3:
- The benchmark is built from an expert-derived taxonomy and held-out human-authored programs because prior evaluations use GitHub data or LLM-generated circuits.Those evaluation sources do not follow the fault taxonomy reported by experts.Source: 3) Insight #3 · p. 3
III. HYQDB DESIGN
- HyQDB switched from a ReAct tool-calling loop to pre-computed evidence injection after the model often called the wrong analyzer or hallucinated results.The passage also notes QBugLM's finding that single-shot prompting outperformed ReAct.Source: §III · p. 3
A. Deterministic Analyzers
- Three analyzers collect potential fault evidence as structured JSON, which is passed to an LLM repair call when findings are present.The analyzers automate two manual debugging procedures reported in the cited study.Source: §III · p. 3, A. Deterministic Analyzers · p. 3
1) Hardware Detector:
HyQDB combines source-level hardware checks, static and runtime optimization analysis, and domain-specific physics checks. These detectors identify concrete patterns, while an undetected basis-intent mismatch illustrates a fault requiring LLM reasoning.
1) Hardware Detector:
- The hardware analyzer scans source code for target device, gate-set, and entangling-gate properties, flagging native-gate mismatches and mixed gate sets.A trapped-ion target using CNOT gates is one example of a native-gate mismatch.Source: 1) Hardware Detector · p. 3
2) Optimization Detector:
- The optimization analyzer pairs AST checks with runtime profiling of QNode calls during training.Static checks flag zero-initialized trainable parameters, learning rates outside 10^-2–2.0, and ansatz depth above a development-set-calibrated threshold.Source: 2) Optimization Detector · p. 3
- Runtime trajectory variance indicates frozen optimization when minimal and divergence when large, though either can have multiple possible causes.Zero variance can expose a parameter-update fault that static checks cannot consistently identify.Source: 2) Optimization Detector · pp. 3–4
- A basis-measurement intent mismatch can evade detectable faults and instead be identified and resolved by the LLM.The example contrasts implemented Z-basis measurement with intended X-basis behavior.Source: Figure 4 · p. 4
3) Physics Detector:
- The physics analyzer infers the problem domain and checks known invariants, currently focusing on quantum chemistry and general computation checks.It checks for operators that can break particle-number or spin symmetry, plus missing measurements and low shot counts; detected-domain information may aid intent reconstruction.Source: 3) Physics Detector · p. 4
B. Intent Reconstruction
HyQDB escalates when fixed-pattern detectors find no fault, using intent reconstruction to address conceptual mismatches. QFaultBench evaluates silent injected faults, including human-authored held-out tasks.
B. Intent Reconstruction
- The detectors cover fixed patterns, so no finding does not establish correctness; conceptual faults are mismatches between defined and intended behavior.This category has no instances in public GitHub data but is the largest category raised in expert interviews.Source: B. Intent Reconstruction · p. 4
- When detectors find no fault, HyQDB uses a bounded two-call intent-reconstruction tier before falling back to standard repair if intent cannot be inferred.The first call infers algorithm class, expected outcome, and implementation discrepancies; the second reconciles the implementation with that specification.Source: B. Intent Reconstruction · p. 4
- The example uses a function name to infer intended X-basis measurement from Z-basis code, a mismatch static detectors cannot resolve.The reconstructed intent guides the code change to the intended measurement behavior.Source: B. Intent Reconstruction · p. 4
IV. QFAULTBENCH
- QFaultBench contains 62 silent-fault repair tasks across ten PennyLane workloads, with variants admitted only after failing a hidden assert.Development faults were LLM-injected and held-out faults were human-authored; scripts were cleaned to remove comments and hints.Source: §IV · p. 4, A. Construction · pp. 4–5
B. Fault Taxonomy
QFaultBench organizes hybrid-program faults into taxonomy-based families, emphasizing conceptual and interface faults and adding safeguards against benchmark shortcuts. On the held-out set, full HyQDB reaches 75% repair accuracy versus 45% for the baseline.
B. Fault Taxonomy
- The taxonomy groups 13 fault types from into five families and was fixed before framework design; the benchmark emphasizes conceptual and interface faults.The 62 tasks use short programs with PennyLane-autograd or JAX, and include limited examples from other areas.Source: B. Fault Taxonomy · p. 5
C. Benchmark Fairness
- Comment stripping, exclusion of exact-value oracles, and a human-authored held-out split guard against passing through hints, answer derivation, or memorized base scripts.The held-out programs are authored by a human expert.Source: C. Benchmark Fairness · p. 5
A. Key Results
- Mechanical-fault accuracy rises from 48% to 63% on held-out and 53% to 69% on development with detector evidence; conceptual faults rise only after intent reconstruction.This pattern supports escalating after detectors find no evidence rather than always running both tiers.Source: Figure 6 · p. 5, A. Key Results · p. 5
- Across three base LLMs, HyQDB improves performance and reaches similar final repair accuracy, suggesting the framework—not the model—is the current limitation.The authors motivate additional detectors and intent-reconstruction tools.Source: Figure 7 · p. 5, A. Key Results · pp. 5–6
B. Detailed Fault Analysis
Fault-category results support different repair strategies: detector evidence helps detectable faults, while conceptual faults generally improve only with intent reconstruction. Repair remains uneven for difficult held-out cases and faults requiring missing structure.
B. Detailed Fault Analysis
- Detectable optimization and interface faults mostly improve with detector evidence, with little additional gain from intent reconstruction.The category breakdown shows evidence injection improves most such faults, even when the evidence tier does not itself resolve them.Source: B. Detailed Fault Analysis · p. 6, Figure 8 · p. 6
- Conceptual faults generally improve only after intent reconstruction, especially on the more complex held-out set.Detector evidence alone does not improve these tasks, indicating that resolving them requires semantic understanding rather than static or runtime analysis.Source: B. Detailed Fault Analysis · p. 6, Figure 9 · p. 6
- Barren-plateau repair succeeds on development but not held-out tasks; depth bugs resolve only with intent reconstruction, likely after detector failure.The authors identify the depth-bug behavior as an open problem because the framework is not designed to solve hardware-category faults.Source: B. Detailed Fault Analysis · p. 6
- Intent reconstruction handles present-but-wrong components more reliably than missing components, making restoration of absent structure the harder repair task.An incorrect measurement basis is an example of a localized edit; the data re-uploading case requires adding a missing component.Source: B. Detailed Fault Analysis · p. 6
C. Routing Mechanism
The routing mechanism uses detector findings to guide mechanical repairs and sends detector-silent faults to intent reconstruction. Its gate preserves mechanical-fault accuracy, although detector thresholds transfer imperfectly to human-authored held-out tasks.
C. Routing Mechanism
- On the held-out set, 88% of conceptual faults are correctly routed to the intent tier.On development, evidence is found for 75% of mechanical faults and 80% of conceptual faults are correctly routed to intent reconstruction.Source: C. Routing Mechanism · p. 7, Figure 10 · p. 7
- Held-out detectors are silent for 60% of mechanical faults, partly because thresholds tuned on development do not transfer to human-authored tasks.For example, a learning rate of 1.5 may impair convergence but be classified as correct under the detector's observed-value thresholds.Source: C. Routing Mechanism · p. 7
- Detector-tier accuracy is 85% versus 78% across development and held-out sets, while intent-tier accuracy is 57% versus 74%, respectively.The authors attribute the higher held-out intent-tier repair rate partly to more mechanical faults being routed there after detector silence.Source: C. Routing Mechanism · p. 7, Figure 11 · p. 7
- Removing the escalation gate reduces mechanical-fault repair accuracy by 19% on both benchmark sets.Without the gate, intent reconstruction discards detector localization and makes the mechanical repair more complex; conceptual-fault performance is unchanged.Source: C. Routing Mechanism · p. 7, Table III · p. 7
D. Token Usage
Token use varies by fault class: detector-localized mechanical repairs cost less than an unaugmented baseline, while conceptual repairs require an additional intent-reconstruction call.
D. Token Usage
- Conceptual faults use 2–4× the tokens of detector faults because intent reconstruction adds a model call before repair.Mechanical faults use fewer tokens than an unaugmented baseline because localized diagnostics can enable a direct solution, whereas baseline repairs may require retries.Source: D. Token Usage · p. 7
- Token counts also vary by model design; deepseek-v4-flash often emits long reasoning chains that exceed generation limits and require retries.The paper presents this as an example of why token counts differ between models.Source: D. Token Usage · p. 8
VI. LIMITATIONS & DISCUSSION
The discussion identifies limits in benchmark size, detector calibration, and detector coverage, then outlines extensions to repair missing structures and broaden benchmark coverage. The conclusion reports improved held-out repair accuracy with HyQDB.
VI. LIMITATIONS & DISCUSSION
- QFaultBench has only 62 tasks, some fault families have few examples, and per-category results are therefore only indicative.Detector thresholds calibrated on development do not transfer perfectly, and bounded detector coverage sends some mechanical faults to intent reconstruction.Source: §VI · p. 8
VII. FUTURE WORK
- Future work should extend intent reconstruction to propose and verify missing structures and replace fixed thresholds with algorithm-aware calibration.The authors also call for a compilation-aware analyzer for hardware faults and broader QFaultBench coverage across algorithms and fault-tolerant workflows.Source: §VII · p. 8
VIII. CONCLUSION
- On human-authored held-out programs, HyQDB raises repair accuracy from 45% to 75% over a base LLM.The conclusion describes QFaultBench as built from an expert-derived taxonomy and validated with hidden asserts.Source: §VIII · p. 8
- The conclusion frames HyQDB as combining deterministic evidence for mechanical faults with intent reconstruction for conceptual faults.The paper presents this two-strategy design as addressing silent faults that existing tooling leaves unresolved.Source: §VIII · p. 8