Source-linked AI summary
Quantum error correction for the toric code using deep reinforcement learning
Philip Andreasson, Joel Johansson, Simon Liljestrand, Mats Granath
TL;DR
The paper studies whether deep reinforcement learning can decode the toric code, where quantum error correction must infer recovery operations from syndromes that do not uniquely identify physical errors. It trains an agent to find short correction paths and reports performance close to MWPM, while noting that scaling beyond the tested code distances requires parallelized training.
Problem
A decoder must choose recovery operations from syndromes that do not uniquely determine the physical errors, a challenge for protecting fragile quantum information.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The results demonstrate that deep reinforcement learning can perform on par with a hand-made decoder on this minimal model.
Source: §1 · p. 2, §5 · p. 8Because the state-action space grows exponentially, going much beyond the presented code distances requires parallelized training.
Source: §5 · p. 9
Abstract
from arXiv · showhide
We implement a quantum error correction algorithm for bit-flip errors on the topological toric code using deep reinforcement learning. An action-value Q-function encodes the discounted value of moving a defect to a neighboring site on the square grid (the action) depending on the full set of defects on the torus (the syndrome or state). The Q-function is represented by a deep convolutional neural network. Using the translational invariance on the torus allows for viewing each defect from a central perspective which significantly simplifies the state space representation independently of the number of defect pairs. The training is done using experience replay, where data from the algorithm being played out is stored and used for mini-batch upgrade of the Q-network. We find performance which is close to, and for small error rates asymptotically equivalent to, that achieved by the Minimum Weight Perfect Matching algorithm for code distances up to $d=7$. Our results show that it is possible for a self-trained agent without supervision or support algorithms to find a decoding scheme that performs on par with hand-made algorithms, opening up for future machine engineered decoders for more general error models and error correcting codes.
1 Introduction
The paper tests deep reinforcement learning as a decoder for the toric code, focusing on perfect syndromes and bit-flip errors. The resulting agent performs comparably to MWPM, though it does not outperform state-of-the-art decoders.
1 Introduction
- Quantum error correction is needed because quantum information is fragile, and a decoder must infer recovery operations from syndromes that do not uniquely identify physical errors.Imperfect stabilizer measurements may also require the decoder to account for noisy syndromes.Source: §1 · pp. 1–2
- The decoder treats the syndrome as the state, defect moves as actions, and the correction outcome as the basis for reward.Deep learning addresses the challenge of selecting actions across a large state-action space.Source: §1 · p. 2
- The trained agent achieves performance comparable to MWPM for the paper’s toric-code setting, without outperforming that decoder.The study focuses on perfect syndromes and bit-flip errors; it presents the benchmark as a minimal model.Source: §1 · p. 2
- Unlike concurrent work using a referee decoder and addressing imperfect syndromes and depolarizing noise, this study uses perfect syndromes and bit-flip errors on the toric code.The torus’s translational invariance makes only relative defect positions relevant, reducing the state-space complexity.Source: §1 · p. 2
- The paper’s value is demonstrating that reinforcement learning can perform excellently on a minimal model, not producing a better-performing decoder.The authors frame this as a prospect for self-trained decoders adaptable to other error scenarios and code architectures.Source: §1 · p. 2
2 Background
The toric code places physical qubits on a periodic square lattice, where local stabilizers diagnose errors but winding logical loops encode distinct states. Error correction must use the syndrome to choose recovery operations without creating a logical error; Q-learning frames this as finding a high-return action policy, and deep Q-learning approximates action values with a neural network.
2.1 Toric code
- A toric code uses spin-1/2 qubits on the bonds of a periodic square lattice, with local stabilizers and winding logical loops distinguishing four ground states.Products of vertex operators generate topologically trivial loops, whereas logical operators wind the torus and are not products of stabilizers.Source: §2.1 · pp. 2–3
2.1.1 Error correction
- Bit-flip errors create plaquette defects, but a syndrome does not uniquely identify the errors, so a recovery chain can erase defects yet still cause a logical failure.For uncorrelated noise, Minimum Weight Perfect Matching (MWPM) is a close-to-optimal benchmark that seeks the shortest total chain length.Source: §2.1.1 · pp. 3–4
2.2 Q-learning
- Q-learning estimates action values from rewards and state transitions, using an ε-greedy policy to balance choosing the best-known action with exploration.A constant reward of −1 per step makes the optimal policy minimize the number of actions; the learning-rate update balances current estimates and new experience.Source: §2.2 · p. 5
2.2.1 Deep Q-learning
- Because the state-action space is too large to store directly, deep Q-learning represents the action-value function with a neural network trained in a scheme similar to one-step Q-learning.For d = 7, the network has approximately 10^6 parameters; the cited state-space estimate is approximately 10^13 for p ≈ 10% and d = 7.Source: §2.2.1 · p. 5
3 RL Algorithm
The decoder uses a deep Q-network to choose defect moves from toric-code syndromes, with the explicit training objective of clearing the syndrome in as few steps as possible. Translational invariance lets it represent each defect through a centralized perspective, while experience replay and a target network support training.
3 RL Algorithm
- The agent receives a reward of −1 at every step until the syndrome is empty, so its explicit training target is to clear defects in as few steps as possible.Whether the resulting correction causes logical failure is used only to evaluate performance, not as a training signal.Source: §3 · p. 6
- Training directly from hidden-state success or failure was difficult because a syndrome can correspond to different error strings, some of which MWPM corrects and others it does not.The authors suggest exploring this signal for future tasks with biased noise or spatial variations.Source: §3 · p. 6
3.1 State-space formulation
- Each defect’s perspective records the relative positions of all other defects, and the set of perspectives forms an observation equivalent in information to the syndrome.Periodic boundary conditions allow any plaquette to serve as the center for a defect’s perspective.Source: §3.1 · p. 6, Figure 4 · p. 6
- The network evaluates four possible moves for each defect from its perspective, and a greedy policy selects the defect and action before the process repeats until no defects remain.Sending perspectives individually to the Q-network keeps its output size constant regardless of the number of defects.Source: §3.1 · p. 6
3.2 Training the neural network
- Training uses experience replay: stored transitions are randomly sampled in mini-batches to update the Q-network, while a periodically synchronized target network supplies update targets.The target network changes more slowly, and the training procedure also uses an ϵ-greedy acting policy.Source: §3.2 · pp. 6–7, Figure 5 · p. 7
4 Result
The trained agents perform close to MWPM for code distances up to d = 7, and match its asymptotic behavior at small error rates. The same Q-network can guide syndromes with varying numbers of defects, though performance at d = 9 is only partial.
4 Result
- Training data use a fixed 10% error rate, and each syndrome is decoded until no errors remain.Success is checked using the hidden physical-qubit configuration and compared with MWPM on the same syndromes.Source: §4 · p. 7
- The converged agents achieve performance close to MWPM for d = 3, 5, and 7.The comparison is shown across bit-flip error rates; the d = 30 MWPM curve is included as a large-distance reference.Source: §4 · p. 7, Figure 6 · p. 8
- At small error rates, the Q-network agent matches MWPM’s asymptotic fail-rate behavior.The agent was checked explicitly on the dominant error strings underlying that asymptotic behavior.Source: §4 · p. 8, Figure 7 · p. 8
- At d = 9, performance is good at small error rates but below MWPM at larger error rates.The paper relates the difficulty to exponential state-space growth and anticipates larger distances with more hardware and a larger manageable Q-network.Source: §4 · p. 8
- In a demonstrated syndrome, the network’s best equivalent moves have discounted return −3.57, close to the optimal-policy value −3.62 for annihilating defects in four steps (reward −1, discount rate 0.95).Source: §4 · p. 8
- One Q-network suggests good actions regardless of how many defects are in the syndrome.The authors illustrate this with a d = 7 syndrome that may begin with about 20 defects and is reduced pairwise using the same network.Source: §4 · p. 8
5 Conclusions
The paper implements deep reinforcement learning for toric-code error correction under uncorrelated bit-flip or phase-flip noise. Its decoder performs close to MWPM on moderate-size systems, while extending the formalism to broader settings remains future work.
5 Conclusions
- Deep Q-learning achieves accuracy close to MWPM for moderate-size toric-code systems under uncorrelated bit-flip or phase-flip noise.The construction uses translational invariance on the torus to reduce the state-space representation.Source: §5 · p. 8
- Figure 8 shows the Q-network action-value function for two syndromes at code distance d=7; the largest arrow(s) mark optimal next moves, with (a) consistent with reward −1 per step and γ=0.95.Arrow magnitude indicates the expected return from taking that next step and then following the optimal policy.Source: Figure 8 · p. 9
- The authors identify imperfect measurements, advanced noise models, and other topological codes as settings for future generalization.They state that going substantially beyond the presented code distances will require parallelized training and very large networks on state-of-the-art hardware.Source: §5 · p. 9
- The paper presents deep reinforcement learning as a potential route to flexible, machine-engineered quantum decoders.The authors frame this as a longer-term prospect tied to future developments in AI.Source: §5 · p. 9
A Small error rate
The small-error-rate analysis derives a benchmark for MWPM from the minimal bit-flip patterns that cause logical failure, with separate cases for odd and even code distances.
A Small error rate
- The small-p limit supplies a benchmark for the RL decoder by allowing an exact MWPM logical-failure expression to be derived for toric-code bit-flip errors.The analysis follows the small-error-rate operating regime discussed for surface codes by Fowler et al. [47].Source: A Small error rate · p. 11
- For odd d, the leading failure cases place ⌈d/2⌉ bit flips along one of 2d shortest non-trivial loops, and MWPM fails on each such case.This benchmark assumes bit-flip probability p ≪ 1 and considers the limit p → 0.Source: A Small error rate · p. 11
- For even d, the minimal failure scenario has d/2 errors on a length-d loop, and MWPM has a 50% chance of choosing a non-trivial loop.The corresponding asymptotic failure-rate expression is introduced, but its full formula is not legible in the supplied passage.Source: A Small error rate · p. 11
B Network architecture and training parameters
The decoder approximates action values with a convolutional network applied to each defect’s lattice perspective. Training uses experience replay, and convergence examples and training durations are reported for several code distances.
Network architecture
- The Q-network maps each d × d binary defect-perspective input through a convolutional layer and fully connected layers to four action values.The four outputs represent the four possible actions; hidden layers use ReLU and the output layer uses linear activation.Source: B Network architecture and training… · p. 12
- The architecture is summarized for d=5 and d=7, while the authors note that parameterizing the network greatly reduces the representation burden relative to tabulating Q values.The supplied passages do not include the parameter counts or the tabulated-Q-function order.Source: B Network architecture and training… · p. 12, Table 1 · p. 12, Table 2 · p. 12
Training
- Figure 9 shows success rate Ps versus iterations for lattice sizes d=3, 5, and 7; each iteration solves one syndrome and runs corresponding mini-batch training sessions, while testing syndromes remain fixed so fluctuations reflect agent performance.The very early success rate below 1/4 is attributed to a max-count cap on validation error-correction steps.Source: Figure 9 · p. 12, B Network architecture and training… · p. 12
- Training on a single desktop converged in hours for d=3 and days for d=7.Q-learning, experience-replay, and network-training hyperparameters are listed in Table 3.Source: B Network architecture and training… · p. 12, Table 3 · p. 12