Source-linked AI summary
Optimizing Quantum Error Correction Codes with Reinforcement Learning
Hendrik Poulsen Nautrup, Nicolas Delfosse, Vedran Dunjko, Hans J. Briegel, Nicolai Friis
TL;DR
The paper tackles the difficulty of choosing economical QEC encodings for unknown or changing noise. It uses reinforcement learning to fault-tolerantly adapt surface codes, finding effective solutions across noise models and transferring experience between settings.
Problem
Choosing resource-efficient QEC codes for arbitrary, correlated, unknown, or changing noise is challenging despite scarce physical-qubit resources.
Method
A reinforcement-learning agent modifies a surface-code quantum memory through fault-tolerant deformations, using logical-error-rate feedback in simulations.
Results
Agents learn to protect surface-code memories from i.i.d., correlated, and non-isotropic noise and transfer experience between seemingly different noise models or tasks.
Takeaways & Limitations
Training on simulations can bootstrap adaptive optimization of physical quantum memories in on-line hardware settings.
Takeaways & Limitations
The simulations assume arbitrary qubit connectivity, while larger systems may face unfavorable learning-complexity scaling and hardware-specific interaction constraints.
Abstract
from arXiv · showhide
Quantum error correction is widely thought to be the key to fault-tolerant quantum computation. However, determining the most suited encoding for unknown error channels or specific laboratory setups is highly challenging. Here, we present a reinforcement learning framework for optimizing and fault-tolerantly adapting quantum error correction codes. We consider a reinforcement learning agent tasked with modifying a family of surface code quantum memories until a desired logical error rate is reached. Using efficient simulations with about 70 data qubits with arbitrary connectivity, we demonstrate that such a reinforcement learning agent can determine near-optimal solutions, in terms of the number of data qubits, for various error models of interest. Moreover, we show that agents trained on one setting are able to successfully transfer their experience to different settings. This ability for transfer learning showcases the inherent strengths of reinforcement learning and the applicability of our approach for optimization from off-line simulations to on-line laboratory settings.
1 Introduction
The paper addresses the challenge of selecting resource-efficient QEC codes for arbitrary, changing, or correlated noise in near-term quantum devices. It proposes reinforcement learning to adapt the quantum system’s code structure rather than only its decoder.
- Physical qubits are scarce, making resource-efficient QEC codes desirable for a targeted logical error rate.
- Selecting a suitable QEC strategy is difficult when noise may be correlated, non-identically distributed, unknown, time-varying, or qubit-dependent.
- Decoder optimization alone is limited by the underlying code structure, motivating changes to the quantum system itself.
- The proposed reinforcement-learning framework adapts and optimizes surface-code QEC codes for realistic noise models through adaptive code selection.
2 Framework & Overview
The framework trains an agent to modify a surface-code quantum memory through fault-tolerant code deformations, using logical-error-rate feedback to select codes under varied noise models. Simulations show optimization and transfer-learning capabilities across settings.
- Framework: The agent observes the current code structure and performs fault-tolerant local deformations until the logical error rate falls below a desired threshold.
- Framework: The environment combines a noisy topological quantum memory with classical control that guides QEC and estimates logical error rates.
- Implementation: The implementation uses an 18-qubit surface code extendable by up to 50 additional data qubits, with arbitrary connectivity in simulation.
- Generality: The framework is designed to accommodate arbitrary topological QEC codes, decoders, noise models, optimizers, and hardware platforms.
- Results: Simulations demonstrate agent ability to determine optimal QEC codes for standard, non-isotropic, and changing noise settings.
- Results: Agents trained with simplified error models can transfer beneficial strategies to modified requirements and more realistic noise models.
- Caveat: The decoder is optimal only for the simplified approximate error model used to estimate logical error rates more efficiently.
- Caveat: On-line optimization can be constrained by hardware requirements such as nearest-neighbour interactions, although off-line optimization remains viable.
3 Adaptable Quantum Memory
Surface codes encode information through lattice-defined stabilizers and logical operators, while lattice deformations provide a fault-tolerant way to explore codes with different performance and hardware connectivity.
- 3.1 Surface Code Quantum Memory: Surface codes place data qubits on lattice edges, with vertices and plaquettes defining stabilizers whose common +1 eigenspace is the codespace.Nontrivial logical operators are Pauli strings along noncontractible torus cycles.
- 3.1 Surface Code Quantum Memory: The code distance is the shortest noncontractible-loop length, determining the minimum physical errors required to realize a logical operator.Fewer than d/2 errors can always be corrected in the stabilizer formalism.
- 3.2 Adaptable Surface Code: Changing the lattice changes its stabilizer group and can substantially alter performance, including adaptation to biased noise and loss tolerance.This flexibility motivates searching over lattice configurations rather than using only the standard square lattice.
- 3.2 Adaptable Surface Code: The two basic deformation moves connect vertices across plaquettes or plaquettes across vertices, adding one data qubit while modifying the stabilizers.The dual-lattice interpretation makes the two moves complementary, and both map surface codes to surface codes fault-tolerantly.
- 3.3 Hardware constraints: Hardware connectivity constrains which deformations are available, while higher connectivity may trade adaptability against lower fidelities from greater technical demands.The framework therefore restricts actions to fault-tolerant local deformations despite platform differences.
4 Reinforcement Learning and Projective Simulation
The paper frames code adaptation as reinforcement learning in which an agent modifies a noisy surface-code memory using percepts, actions, and rewards. Its Projective Simulation implementation learns transition probabilities in a clip network while remaining ignorant of the noise model.
- 4 Reinforcement Learning: The reinforcement-learning environment is a noisy surface-code memory and control system, and the agent receives code information as percepts before selecting code-deformation actions.The agent is rewarded when the logical error rate falls below the specified target.
- 4.1 Projective Simulation: Projective Simulation represents percepts and actions as clips in a two-layer bipartite network, with edges encoding possible actions for each perceived code.Not every percept connects to every action because available deformations depend on the lattice.
- 4.1 Projective Simulation: Each percept–action edge has a time-dependent h-value, and its transition probability is computed with a softmax function controlled by β > 0.The h-values initially equal 1, producing uniform random behavior for a newly triggered percept.
- 4.1 Projective Simulation: Environmental rewards update the h-matrix to increase the future probability of actions associated with higher reward, while additional updates support exploration.After learning, nonuniform transition probabilities encode a policy that balances systematic decisions with exploratory behavior.
5 Optimizing Quantum Memories - a reinforcement learning problem
The paper formulates resource-efficient quantum-memory design as reinforcement learning, with an agent modifying surface codes until a target logical error rate is reached. Across independent, depolarizing, correlated, and changing-threshold settings, the agent learns resource-efficient code adaptations and can reuse prior knowledge.
- RL formulation: The PS agent interacts with a noisy surface-code memory, modifying its code through fault-tolerant local deformations and receiving rewards when the logical error rate meets the target.The agent operates without access to the environment’s detailed physical identity or noise source.
- RL formulation: The simulations start from an 18-qubit distance-three surface code and permit up to 50 additional data qubits, with the initial code and action set adaptable to hardware constraints.A single trial may simulate the performance of up to 50 candidate codes.
- i.i.d. error channels: 5 additional data qubits on average reduced the logical error rate to P rew L = 0.001 for the single-qubit Z-error channel, compared with 20 additional qubits for random strategies.The learned policy gradually shortens rewarded action sequences and approaches the best known strategy for the available actions.
- i.i.d. error channels: 4 additional qubits were sufficient for the best agents in the Eq. (4) task, while learned constructions increased X-stabilizers to protect against logical Z-errors.The resulting low-connectivity surface codes match the best known strategy for this error channel.
- i.i.d. error channels: For equal X- and Z-error probabilities pX = pZ = 0.09, the agent again improved the logical error rate while optimizing the number of required qubits.This task changes the error model while retaining the optimization objective.
- Correlated error channels: Under spatially dependent noise, the agent protected the surface code by increasing nearby Z-stabilizers, reducing affected-plaquette connectivity, and moving flawed plaquettes apart.The learned lattice also balanced connectivity and produced an X-to-Z stabilizer ratio of 14/15.
- Changing requirements: After learning one target, the agent further reduced the logical error rate to a quarter of its initial value by drawing on knowledge from the first stage.This demonstrates adaptation when the rewarded threshold becomes stricter.
6 Simulation vs. Experiment
The study evaluates whether reinforcement-learning agents can optimize quantum memories offline and transfer learned strategies to changed noise settings. Simulations support offline optimization and show substantial transfer benefits despite limited second-stage trials and altered error models.
- Simulation-to-experiment transition: RL agents can adapt and optimize surface-code memories for varied noise models and requirements, supporting potential online optimization.The authors report successful adaptation in simulations and suggest future laboratory applicability.
- Simulation-to-experiment transition: Efficient simulations are valuable because logical-error-rate estimation requires many samples and remains computationally expensive.The simulation components were selected to make offline optimization practical for near-term devices.
- Simulation-to-experiment transition: The approach compares logical-error estimates across surface-code search spaces and decoder or error-model choices to assess model agreement.Figure 11 contrasts SQUAB with Union Find and uses codes within distance 3 of the 2 × 2 root code.
- Transfer learning: In a higher-noise task, agents without initial training behaved randomly because random search could not find a suitable code within 6,000 trials.At p = 0.16, the initial code required many modifications before reaching the target logical error rate.
- Transfer learning: 60 pretrained agents substantially benefited from prior simulations when transferred to a markedly different error channel, while untrained agents barely learned under fewer than 10% as many trials.The second-stage restriction models the higher cost of experimental optimization.
- Transfer learning: Transfer remained substantial when spatial correlations were replaced by doubled base X-error rates, with pZ = 0.14 and pX = 0.04.The comparison used the same pretrained agent in the new setting.
7 Discussion
The paper presents reinforcement learning for fault-tolerantly adapting surface-code memories by modifying their code structure to reduce logical errors. The agents protect memories across diverse noise models, transfer experience between different tasks, and may bootstrap optimization on hardware, while larger systems remain challenging to scale.
- Framework: The framework uses an RL agent that interacts with a quantum memory and classical control to request fault-tolerant code modifications lowering logical error rates.The simulations use surface codes extended by adding qubits through fault-tolerant deformations.
- Results: Agents learn to protect surface-code memories against i.i.d., correlated, and non-isotropically distributed errors.The reported scope includes both simple and more complicated noise patterns.
- Transfer learning: Experience transfer between seemingly different noise models or tasks supports using RL for adaptive error correction.The authors connect this capability to switching from offline optimization toward online adaptation.
- Transfer learning: A trained agent can bootstrap optimization on actual hardware after initial training on simulations based on a reasonable guess of expected errors.This is presented as a route from offline optimization to online adaptive error correction.
- Limitations and scaling: The simulations start with 18 qubits and allow up to 50 additional qubits, while larger systems may face unfavorable learning-complexity scaling.Parallel agents operating on fixed-size surface-code patches are proposed as a possible way to address this bottleneck.
- Limitations and scaling: Machine-learning optimization requires a suitably narrowed search problem because exploring all possible QEC strategies can become intractable as the space grows.The paper contrasts its resource-efficient code optimization with approaches searching broad quantum-circuit spaces.
- Framework: The method’s flexibility comes from treating errors and decoding as a black-box environment, allowing extensions beyond the specific RL paradigm, code family, and noise models studied.The framework is described as applicable to other RL methods and QEC codes.
A Learning in Projective Simulation
Projective simulation learns by updating transition-governing h-values using current rewards and fading traces of past experiences. Forgetting and deletion mechanisms encourage exploration while limiting memory use.
- The h-values governing transition probabilities are updated from time step t to t+1 as learning manifests in the projective simulation model.
- The glow matrix redistributes rewards to past experiences, with older experiences receiving a decreasing fraction of the reward.Glow values encode how strongly past experiences are remembered and influence future behavior.
- The glow matrix provides long-term memory that is especially important when rewards are delayed rather than immediate.
- A forgetting mechanism decreases h-values during updates to reinforce exploratory behavior, while unused percept clips can be deleted below an outgoing-value threshold.The deletion mechanism is introduced to save memory.
B Details of the environment
The environment uses SQUAB to simulate error correction on arbitrary surface-code graphs and represents code structures and local deformations explicitly. The agent receives graph-based percepts and selects actions that split vertices on the primal or dual lattice.
- SQUAB simulates error correction on arbitrary surface codes, while its erasure-channel implementation produces rates that differ slightly from corresponding Pauli-channel rates.The use of erasure channels is described as well motivated elsewhere in the paper.
- A percept represents the current surface-code structure as ordered adjacency matrices for the primal and dual graphs, including spatial labels for vertices and plaquettes.
- An action is a = (d, v, p1, p2), where d selects the dual lattice and v identifies a vertex split using two non-neighboring adjacent plaquettes.
C Searching the space of surface codes
The surface-code search space grows rapidly because each local deformation adds possible successor codes, making exhaustive neighborhood exploration intractable. Random exploration samples this tree and estimates logical error rates for the visited codes.
- A search tree starts from an initial topological code, with successors formed by adding edges while preserving the number of logical qubits.For a 3 × 3 square lattice, one extra qubit yields 36 more topological codes, while two yield 1440 new codes.
- 10,000 decoding trials per 20-qubit code take roughly 0.018 seconds in SQUAB, which estimates maximum-likelihood decoder performance over the quantum erasure channel.The exploration focuses on residual Z-errors and ignores the X component.
- Figure 13 colors nodes by logical error rate for random branches within radius 8 of the 3 × 3 torus code, estimated with 10,000 SQUAB trials at pZ = 0.15 while ignoring X errors.
- Random search is used because exploring a sufficiently large complete neighborhood is hopelessly difficult; it evaluates selected branches after ensuring all distance-one codes are included.
- 1,230 randomly selected codes are obtained within a few minutes at exploration probability pexpl = 0.03, revealing that error-increasing moves are common and later decreases remain possible.
D Parameters
The reported results use a projective-simulation agent and SQUAB with tunable parameters that may also be changed between task settings.
- The experiments use a projective-simulation agent with parameters selected for the tasks, and parameters can be tuned online or changed between settings.