Source-linked AI summary
Experimental Deep Reinforcement Learning for Error-Robust Gateset Design on a Superconducting Quantum Computer
Yuval Baum, Mirko Amico, Sean Howell, Michael Hush, Maggie Liuzzi, Pranav Mundada, Thomas Merkh, Andre R. R. Carvalho, Michael J. Biercuk
TL;DR
Robust gate design depends on detailed hardware models, but unknown and changing system effects make complete characterization unwieldy. This paper uses model-free deep reinforcement learning to design quantum gates on superconducting hardware, reporting faster single-qubit gates and higher-fidelity entangling gates than hardware defaults.
Problem
Robust control design relies on accurate physical models, yet unknown and time-varying hardware effects make complete Hamiltonian characterization unwieldy as systems grow.
Source: §I · p. 1Method
Deep reinforcement learning learns gate controls from measured hardware responses without prior knowledge of the device model or its underlying error processes.
Source: §I · p. 1, §IV · p. 8Results
Single-qubit gates reduce duration by up to 3×, while ZX gates achieve up to ≈2.38× higher fidelity than defaults and FZX > 99.5%.
Source: §I · p. 2, §IV · p. 8Takeaways & Limitations
The results demonstrate useful novel controls despite restricted measurement access and without onerous Hamiltonian tomography.
Source: §IV · p. 8
Abstract
from arXiv · showhide
Quantum computers promise tremendous impact across applications -- and have shown great strides in hardware engineering -- but remain notoriously error prone. Careful design of low-level controls has been shown to compensate for the processes which induce hardware errors, leveraging techniques from optimal and robust control. However, these techniques rely heavily on the availability of highly accurate and detailed physical models which generally only achieve sufficient representative fidelity for the most simple operations and generic noise modes. In this work, we use deep reinforcement learning to design a universal set of error-robust quantum logic gates on a superconducting quantum computer, without requiring knowledge of a specific Hamiltonian model of the system, its controls, or its underlying error processes. We experimentally demonstrate that a fully autonomous deep reinforcement learning agent can design single qubit gates up to $3\times$ faster than default DRAG operations without additional leakage error, and exhibiting robustness against calibration drifts over weeks. We then show that $ZX(-\pi/2)$ operations implemented using the cross-resonance interaction can outperform hardware default gates by over $2\times$ and equivalently exhibit superior calibration-free performance up to 25 days post optimization using various metrics. We benchmark the performance of deep reinforcement learning derived gates against other black box optimization techniques, showing that deep reinforcement learning can achieve comparable or marginally superior performance, even with limited hardware access.
I. INTRODUCTION
Reliable quantum computation is limited by noisy, imperfect hardware, while designing robust controls is difficult when the system’s Hamiltonian and changing errors are not accurately known. The paper uses model-free deep reinforcement learning (DRL) to design experimental gates and reports improvements over hardware defaults.
I. INTRODUCTION
- Although quantum hardware and cloud access have advanced, demonstrating practical computational advantage remains an outstanding challenge.The introduction notes demonstrations of quantum advantage by Google and the Chinese Academy of Sciences.Source: §I · p. 1
- Unknown, transient Hamiltonian terms, nonlinear control effects, signal distortion, crosstalk, and time-varying noise make full hardware characterization unwieldy as system size grows.Even small simplifying approximations can cause numerically optimized controls to fail because system evolution is sensitive to how it is steered through Hilbert space.Source: §I · p. 1
- DRL designs an error-robust universal gateset without a prior hardware model, learning from measured responses while exploring piecewise-constant controls for RX(π/2) and ZX(−π/2) operations.The agent incorporates targeted responses and undesired effects including cross-couplings, nonlinearities, and signal distortions.Source: §I · p. 1
- Single-qubit gates reduce duration by up to 3×, to ≈12.4 ns, without evident leakage increases; ZX gates reach ≈2.38× higher fidelity and FZX > 99.5%, remaining above 99.3% for up to 25 days without recalibration.DRL-defined gates also produced a SWAP with 1.45× lower error than calibrated defaults, while two-qubit gates performed comparably to simulated annealing.Source: §I · p. 2
II. OPTIMIZED QUANTUM LOGIC DESIGN WITH DEEP REINFORCEMENT LEARNING
The method addresses quantum-gate optimization without relying on a fully characterized system Hamiltonian. It combines sequential pulse construction, intermediate state measurements, fidelity-based rewards, and a policy-gradient agent to learn controls on hardware.
II. OPTIMIZED QUANTUM LOGIC DESIGN WITH DEEP REINFORCEMENT LEARNING
- Accurate Hamiltonian characterization is generally intractable for large interacting systems, and simplifying approximations can cause numerically optimized controls to fail.Iterative interactive learning offers an alternative that does not require an accurate representative physical model; DRL uses intermediate observations as well as final rewards to refine an internal model.Source: §II · p. 2
- Earlier simulation-based quantum-control studies could assume zero noise or deterministic errors, perfect instantaneous controls, or complete state observability; this method is designed for experimental constraints.The paper identifies effective-Hamiltonian representation, limited state observability, and efficient learning under constrained measurements as three challenges.Source: §II · p. 3
- The agent builds a pulse from piecewise-constant segments, observing the state after each segment and reinitializing and replaying earlier actions before selecting the next one.This sequential procedure produces a complete candidate gate over Nseg state-observation cycles per episode.Source: §II · p. 3
- Tomography estimates states and gate fidelities; the episode reward uses weighted fidelity estimates across gate repetitions to amplify errors beyond hardware SPAM noise.Single-qubit optimization measures three Pauli bases and leakage; the two-qubit protocol collects nine measurements to evaluate Pauli-string expectations.Source: §II · p. 4
- The agent uses a policy-gradient algorithm to maximize long-term reward, representing its policy as a neural network updated during learning.The policy maps system states to action probabilities; the algorithm choice followed a simulated comparison of DRL methods.Source: §II · p. 4
III. EXPERIMENTAL DEEP REINFORCEMENT LEARNING ON SUPERCONDUCTING QUANTUM COMPUTERS
The experiments optimize single-qubit Rx(π/2), cross-resonance ZX(−π/2), and SWAP gates on IBM superconducting hardware. Benchmarks assess gate performance and whether optimized gates retain their advantages without recalibration.
Experimental constraints
- With limited hardware access, convergence typically took 10–20 experimental batches, or 0.5–1 wall-clock hours, with hardware API and queue times dominating.Typical total experimental execution was under a minute, and agent calculations consumed negligible cloud-server time.Source: §III · p. 4
Single-qubit robustness
- Fixed DRL single-qubit pulses remained within natural hardware fluctuations near the T1 limit for up to two weeks, while daily performance tracked measured T1 fluctuations.The reported comparison is against daily calibrated defaults; the DRL pulses were not recalibrated between measurements.Source: Figure 2 · p. 5, §III · p. 7
Entangling-gate optimization
- On IBM Casablanca, DRL and simulated-annealing optimized ZX(−π/2) pulses outperformed the default, yielding up to 2.38× reductions in error-per-gate and gate fidelity above 99.5%.A typical DRL optimization step was approximately 2× longer than a simulated-annealing step, but both methods converged in roughly the same number of iterations. The optimized gates did not use the default gate’s cancellation tone on channel d1.Source: Figure 3 · p. 6, §III · p. 7
Single-qubit gates
- The 28.4 ns Rx(π/2) pulse reached EPG 3.7×10^-4, about 25% below default; the 12.4 ns pulse was comparable to default despite being 3× faster.The 12.4 ns result indicates leakage errors can be suppressed through the DRL reward function; the text leaves the cause of its lack of further improvement unresolved.Source: Figure 2 · p. 5, §III · p. 6
Entangling-gate optimization
- The cross-resonance optimization searched 10-segment waveforms over 20 amplitude-and-phase variables without an additional crosstalk cancellation tone.The default implementation used a simultaneous cancellation tone on the target-qubit resonant drive.Source: §III · p. 7
ZX(−π/2) and CNOT results
- Up to 2.38× lower ZX(−π/2) error-per-gate than defaults; optimized gates still outperform after 25 days, and a derived CNOT exceeds 99% fidelity.The 2.38× comparison is reported for optimized pulses; the >99% CNOT fidelity is reported for an example tested more than 25 days after calibration.Source: §III · pp. 7–8
SWAP optimization
- Direct DRL optimization of SWAP achieved up to 1.45× improvement in interleaved randomized benchmarking fidelity.The SWAP retains a decomposition into three CNOTs built from ZX(−π/2) gates and single-qubit unitaries.Source: §III · p. 8
IV. CONCLUSIONS AND OUTLOOK
The paper demonstrates autonomous DRL design of error-robust superconducting gates from measured hardware responses, without prior device-model knowledge. Its gates outperform competitive alternatives for multiqubit entangling operations, and the authors identify further in-situ circuit-level optimization as a future direction.
IV. CONCLUSIONS AND OUTLOOK
- DRL-derived multiqubit entangling gates were validated to outperform the best competitive alternatives.Source: §IV · p. 8
- Up to 3× shorter single-qubit Rx(π/2) gates and ≈2.38× higher fidelity for ZX(−π/2), with FZX > 99.5%, support a universal gateset that outperforms hardware defaults weeks after calibration.The gates use measured system responses without prior knowledge of the device model or underlying error processes.Source: §IV · p. 8
- The authors expect richer measurement access may enable DRL to construct gate implementations beyond simpler cost-function minimization.Source: §IV · p. 8
- In-situ optimization of frequently used circuit elements may capture transient and context-dependent errors beyond individual gates; the authors suggest extending DRL toward algorithmic advantages.Source: §IV · p. 8
Appendix A: Methods
The appendix describes how gate performance was evaluated using tomography, randomized benchmarking, interleaved randomized benchmarking, and repetition-based experiments. These procedures estimate state fidelity, leakage, and effective gate error.
Appendix A: Methods
- For single-qubit Rx(π/2), the reward combines state fidelities across variable gate repetitions with a leakage penalty based on population in the transmon’s second level.The fidelity is obtained by computational-space tomography, and the reward is a weighted mean over repetitions.Source: Appendix A · p. 8
- Single-qubit RB uses 18 sequence lengths up to 2280 Clifford gates, with 20 random sequences and 1024 repetitions per length, then fits survival probability to estimate error per gate.The error per gate is derived from the fitted α parameter as 6r/7, where r = (1 − α)/2.Source: Appendix A · pp. 8–9
- Two-qubit error is evaluated with RB/IRB or repetition-based experiments, the latter fitting mean infidelity versus gate repetitions to extract an effective error per gate.The repetition protocol averages five experiments per repetition count and uses full state tomography; its low-error-limit estimate is cross-validated with IRB.Source: Appendix A · p. 9
Appendix B: Reinforcement Learning Algorithm
The gate-optimization algorithm uses an on-policy policy-gradient method to update a stochastic policy from measured trajectories and episodic rewards. Its actions construct piecewise-constant control pulses, and the approach does not require a model of the learning environment.
Appendix B: Reinforcement Learning Algorithm
- The DRL method is an on-policy REINFORCE variant that estimates the gradient of expected discounted episodic return and updates policy parameters with Adam.The optimizer was selected for effectiveness on non-convex, slowly changing objective landscapes.Source: Appendix B · p. 9
- The learning process estimates the policy-objective gradient by averaging over episode batches and is described as efficient enough for gate optimization without forming an environment model.Source: Appendix B · p. 9
- Actions specify pulse-segment amplitude and phase, while a feedforward network with a tanh hidden layer and softmax output samples from a discrete action space.The hidden layer has size 10 for single-qubit gates and 18 for two-qubit gates.Source: Appendix B · p. 9
- Each trajectory contains applied pulse segments, tomographic state measurements, and the fidelity-based reward received when pulse construction ends.The policy is updated at episode end using a batch of trajectories collected under its current policy.Source: Appendix B · p. 9
Appendix C: Fast Simulated Annealing
The appendix reports additional simulated-annealing optimizations and describes a search that perturbs default pulse amplitudes and phases. Candidate costs are evaluated with the same weighted state-fidelity reward used for DRL.
Appendix C: Fast Simulated Annealing
- The simulated-annealing search starts from the device-default gate, evaluates full gate performance with the DRL weighted-state-fidelity reward, and collects no intermediate information.Source: Appendix C · p. 10
- The search draws amplitude and phase perturbation scales from Cauchy distributions, shifts candidate pulse parameters, and accepts candidates based on cost with probabilistic acceptance for non-improvements.The three search temperatures are updated during the process.Source: Appendix C · p. 10