Source-linked AI summary
Evaluating Verified Autonomy in Quantum Engineering
Naixu Guo, Changhao Li, Siyu Cheng, Qicheng Tang, Binzhao Luo, Bikun Li, Yuxuan Du, Shihao Ru, Jiaqi Cai
TL;DR
As quantum platforms grow, manual coordination becomes harder, while reliable completion of full quantum-engineering workflows by current agents remains unclear. The paper develops Quantum-Harbor and QIQCBench to evaluate execution-supported outcomes across quantum-engineering tasks, finding meaningful but uneven capability and continued reliability challenges.
Problem
Whether current agents can reliably complete full quantum-engineering workflows and support their conclusions with execution-generated evidence remains unclear.
Source: INTRODUCTION · p. 1Method
The paper develops Quantum-Harbor to measure whether outputs are supported by execution evidence and applies it through QIQCBench across cross-layer quantum-engineering tasks.
Source: INTRODUCTION · p. 1, DISCUSSION · p. 7Results
Current agents show meaningful but uneven capability, with large performance gaps between systems and substantial variability across repeated attempts.
Source: DISCUSSION · p. 7Takeaways & Limitations
Quantum-Harbor and QIQCBench provide a framework for measuring verified autonomy and reveal system differences not apparent from domain-knowledge and coding benchmarks alone.
Source: DISCUSSION · p. 7The results are limited to simulated environments and complete agentic systems using different harnesses under fixed resource budgets.
Source: DISCUSSION · p. 7
Abstract
from arXiv · showhide
Reliable quantum engineering is essential for turning quantum phenomena into practical technologies. As quantum platforms grow in scale and complexity, their characterization and operation require increasing human effort and coordination. Scientific artificial intelligence agents, which can plan experiments, operate instruments, and analyze observations, offer a promising route towards autonomous quantum engineering. Yet whether current agents can perform reliably in this setting has not been systematically established. To fill this gap, we developed Quantum-Harbor, a virtual laboratory that provides a controlled execution environment for agents to interact with quantum systems. This design enables direct verification of both the actions taken and the conclusions drawn. Building on this framework, we introduce QIQCBench, a benchmark of $49$ expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking. Across $17$ frontier agentic systems, QIQCBench reveals wide variation in verified performance. These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering.
INTRODUCTION
As quantum platforms grow, coordinating their components and workflows makes manual diagnosis increasingly impractical, while reliable completion of full quantum-engineering workflows by current agents remains unclear. This paper introduces Quantum-Harbor and a 49-task benchmark to evaluate whether agents complete tasks and support their conclusions with execution-generated evidence.
Motivation
- Manual diagnosis becomes increasingly impractical as quantum platforms grow in components, parameters, and workflows, motivating agents that can plan, use tools, and adapt to feedback.Quantum engineering integrates platforms, control methods, and software into reliable systems.Source: INTRODUCTION · p. 1
Research gap
- Whether current agents can reliably complete full quantum-engineering workflows remains unclear, and existing benchmarks do not systematically check whether results are supported by execution records.Verified autonomy requires completing a task and supporting conclusions with execution-generated evidence; calibrations and designs can be retested under fresh hidden conditions.Source: INTRODUCTION · p. 1
Contribution
- Quantum-Harbor records agent actions and data to verify results, and QIQCBench uses it to evaluate 49 expert-designed tasks across quantum-engineering layers.The grader recomputes numerical results from raw data or retests calibration and design outputs under fresh hidden conditions.Source: INTRODUCTION · pp. 1–2
Representative quantum research tasks
The representative tasks require agents to gather information during execution and adapt their work to distinct quantum-engineering challenges. Strong decoder calibration was demonstrated, while consistent end-to-end completion in surrogate modeling remains challenging.
Shared workflow demands
- Both tasks require adaptive workflows in which agents interpret newly acquired data and revise plans, with execution records supporting verification of their outputs.The decoder task tests noise emerging in the merged configuration; surrogate modeling tests generalization to unseen parameter settings.Source: Representative quantum research tasks · pp. 2–3
Lattice-surgery decoder calibration
- The strongest lattice-surgery runs cleared both fresh-syndrome criteria, reaching 6.65×10−4 distance-5 memory error per cycle and 0.131 mean logical-CNOT error.Agents that measured the merged configuration directly could detect seam-correlated errors not visible in earlier separate-patch measurements.Source: Representative quantum research tasks · p. 3
Surrogate modeling
- Surrogate modeling requires learning from a fixed measurement budget to predict 1,770 pairwise correlations on 40 hidden inputs, and consistent end-to-end completion remains challenging.Runs can fail after spending resources because sampling is uninformative, validation unreliable, or execution incomplete.Source: Representative quantum research tasks · p. 3
The Quantum-Harbor framework
Quantum-Harbor separates the agent-facing environment from a hidden sidecar that hosts hardware implementations and records experiment evidence. Its grader binds submitted claims to that evidence, recomputing results or testing configurations under fresh conditions.
Agent environment
- Agents use a public interface to consult specifications and notebooks, submit asynchronous experiments, retrieve raw records, and provide final answers, while a separate sidecar remains hidden.The architecture is shown in Fig. 2.Source: The Quantum-Harbor framework · p. 3, Figure 2 · p. 4
Hardware layer
- Qtypes abstract different quantum platforms, and simulator, live-hardware, and replay backends can be switched without changing task definitions or public surfaces.Supported qtypes include platforms such as transmons, trapped ions, and neutral atoms.Source: The Quantum-Harbor framework · pp. 3–4
Evidence-bound grading
- The grader uses the sidecar evidence log to recompute numerical claims from raw data or replay calibration submissions on fresh hidden draws.This evidence-bound grading is intended to prevent rewards for results the agent did not measure.Source: The Quantum-Harbor framework · p. 4, Figure 2 · p. 4
The QIQCBench benchmark
QIQCBench comprises expert-contributed tasks spanning quantum-engineering categories and layers, with varied evaluation modes and deliberately designed, realistic sources of difficulty.
Task composition and coverage
- QIQCBench contains 49 expert-contributed tasks built on 36 qtypes, spanning six categories and five quantum-computing-stack layers.The task map shows that individual tasks can connect multiple layers.Source: The QIQCBench benchmark · p. 4, Figure 3 · p. 5
Evaluation design
- The suite uses pass–fail or Elo-ranked racing evaluations, while three infrastructure tests are excluded from science scores and task thresholds remain hidden.The task map distinguishes 39 pass–fail tasks, 7 racing tasks, and 3 infrastructure tests.Source: The QIQCBench benchmark · p. 5, Figure 3 · p. 5
Design principles
- Benchmark tasks target end-to-end results, derive difficulty from nonobvious physical inference rather than obscurity, and use only realistic, clearly labeled traps.The principles require explicit budgets and labeled stale notebooks, and do not punish agents for missing infrastructure information.Source: The QIQCBench benchmark · p. 5
Benchmark results
Across 17 systems, QIQCBench showed broad variation in verified performance, with pass–fail, racing, failure-category, and task-level analyses revealing distinct reliability patterns.
Benchmark results
- Pass rates on pass–fail tasks ranged from 78% to 3% across 17 systems, declining steadily without clear groups.The systems ran each task three times independently.Source: Benchmark results · p. 5, Figure 4 · p. 6
- 937 of 1,510 failures were no-submission trials; seven systems exhausted their budgets before submitting in more than 60% of trials.Timed-out agents were often still running experiments.Source: Benchmark results · p. 6, Figure 4 · p. 6
- Among remaining failures, 54% were science failures, while many failures involved stale priors, mismanaged budgets, or discarded results at submission.The remainder were contract failures.Source: Benchmark results · pp. 6–7
- 125 of 357 racing trials cleared their gates; ratings ranged from 1694 to 582 around a 1000 anchor, with the same top three as pass–fail rankings.Racing produced wider margins by incorporating design quality alongside reliability.Source: Figure 4 · p. 6, Benchmark results · p. 7
- At least one system solved 37 of 39 tasks, while six tasks were solved only by the three highest-ranked systems.Expert reviewers confirmed intended solution paths for the two tasks no system solved.Source: Figure 4 · p. 6, Benchmark results · p. 7
- Simulation and many-body physics was hardest (15% pass rate); the top three passed 64% of trials versus 4% for the other fourteen, with no significant stack-depth correlation.Only 34% of 249 system-task pairs that passed at least once passed all three attempts.Source: Benchmark results · p. 7
DISCUSSION
Current agents show meaningful but uneven capability on demanding quantum-engineering tasks, with substantial variability indicating that reliable completion remains a challenge. Quantum-Harbor and QIQCBench make verified autonomy measurable, while the results are limited to simulated environments and fixed resource budgets.
DISCUSSION
- Current agents show meaningful but uneven capability, and large system gaps and repeated-attempt variability leave reliable completion a central challenge.Failure patterns highlight experimental judgment, including choosing informative measurements, revising stale assumptions, and deciding when to stop under finite resources.Source: DISCUSSION · p. 7
- Failures are not explained by scientific knowledge or laboratory execution alone; agent-harness improvements could substantially increase reliability.Source: DISCUSSION · p. 7
- Quantum-Harbor evaluates whether outputs are supported by information acquired during execution, and QIQCBench reveals system differences not apparent from domain-knowledge and coding benchmarks alone.The framework examines evidence gathering, hypothesis revision, resource allocation, and verifiable conclusions across cross-layer quantum-engineering tasks.Source: DISCUSSION · p. 7
- Quantum-Harbor could support controlled studies that vary task conditions, instructions, and resource constraints to disentangle model capability, harness design, and scientific decision-making.Such studies could track progress beyond aggregate pass rates and guide reliability and efficiency improvements.Source: DISCUSSION · p. 7
- The present results cover simulated environments, complete agentic systems using different harnesses, and fixed resource budgets; transfer to experimental conditions and physical devices remains for future work.Future work could also test whether increasingly capable agents reduce measurement cost and human effort.Source: DISCUSSION · p. 7
METHODS
The methods describe expert task construction and review, trial execution and failure classification, and distinct procedures for racing-task ratings and pass-rate uncertainty. All tasks ran on the simulation backend, with trial records audited and confirmed infrastructure faults rerun.
METHODS
- Each task begins with an expert brief, is implemented as separate public materials, hidden configuration, and grader artifacts, then receives a second expert review and end-to-end verification.Coding agents also audit selected agents’ outcomes as a sanity check.Source: METHODS · p. 7
- Trials run for up to 3 hours on the simulation backend with internet access, and coding agents audit complete action records; trials with confirmed infrastructure faults are rerun.Source: METHODS · pp. 7–8
- Pass–fail task failures are classified as science failures, contract failures, or no-submission trials based on the scorer, termination records, or expert transcript audit.Science failures fail scientific checks; contract failures are rejected at an admissibility gate for schema, evidence, or submission-channel violations.Source: METHODS · p. 8
- Racing tasks compare feasible trials by recomputed race metric, while Bradley–Terry ratings model pairwise wins and losses with a ridge penalty of λ = 0.5.Task ratings have a mean of 1000; a 400-point advantage corresponds to model-implied 10:1 win odds, and aggregate ratings weight tasks by retained comparisons.Source: METHODS · p. 8
- Pass rates use k/n over 117 trials from 39 tasks repeated three times, and Fig. 4a error bars are Wilson score intervals with z = 1.The intervals use a pooled binomial approximation, correspond to approximately 68% nominal confidence, and do not quantify uncertainty from task choice.Source: METHODS · p. 8