Reinforcement Learning with Neural Networks for Quantum Feedback

Thomas Fösel; Petru Tighineanu; Talitha Weiss; Florian Marquardt

Source-linked AI summary

Reinforcement Learning with Neural Networks for Quantum Feedback

Thomas Fösel, Petru Tighineanu, Talitha Weiss, Florian Marquardt

arXiv:1802.05267v3Feb 14, 2018quant-ph

TL;DR

The paper addresses how to discover quantum-error-correction strategies autonomously when useful sequences are rare and the environment may be only partially observed. It combines two-stage teacher–student learning with a reward based on recoverable quantum information, and finds adaptive strategies across noise and hardware settings. The approach is demonstrated for few-qubit systems, while modest qubit numbers and computational effort remain important limitations.

Problem
Useful quantum-error-correction sequences are difficult to discover because good strategies are extraordinarily rare and separated from trivial strategies by a wide barrier.
Method
The approach trains a state-aware network with reinforcement learning, distills its behavior into an experimentally applicable recurrent network, and rewards recoverable quantum information.
Results
The agent discovers nontrivial adaptive detection and recovery strategies from scratch for diverse noise models and hardware settings using only the available gate set.
Takeaways & Limitations
The trained networks can in principle control experimental quantum devices, and the approach can extend to further feedback-control settings in quantum and classical physics.
Takeaways & Limitations
The numerical method is limited to modest qubit numbers, with further improvements potentially reaching about 10 qubits, and full-scale quantum-computer simulation is out of scope.

Abstract

from arXiv · show

Machine learning with artificial neural networks is revolutionizing science. The most advanced challenges require discovering answers autonomously. This is the domain of reinforcement learning, where control strategies are improved according to a reward function. The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes, such as playing Go, but its benefits for physics are yet to be demonstrated. Here, we show how a network-based "agent" can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise. These strategies require feedback adapted to measurement outcomes. Finding them from scratch, without human guidance, tailored to different hardware resources, is a formidable challenge due to the combinatorially large search space. To solve this, we develop two ideas: two-stage learning with teacher/student networks and a reward quantifying the capability to recover the quantum information stored in a multi-qubit system. Beyond its immediate impact on quantum computation, our work more generally demonstrates the promise of neural-network-based reinforcement learning in physics.

Reinforcement Learning

The agent selects quantum gates and measurements through a probabilistic policy updated by reinforcement learning. The paper uses neural networks to map quantum-environment information or observations to action probabilities.

The reinforcement-learning agent chooses quantum gates and measurements to maximize an expected return composed of rewards.
At each time step, a policy πθ assigns probabilities to discrete actions conditioned on the environment state.The parameters θ comprise the neural network’s weights and biases.
The state-aware network receives a representation of the quantum evolution map and outputs probabilities for available gates.Example actions include bit-flips, CNOTs, and measurements, yielding roughly 10–20 actions.
Policy-gradient learning updates network parameters using a learning rate and expectations over gate sequences and measurement outcomes.The implementation also uses a baseline, natural policy gradient, and entropy regularization.
When the environment is partially observed, the policy depends on observations rather than the full underlying state.The paper considers both fully observed and partially observed Markov settings.

Memory

The paper frames quantum-error correction as preserving arbitrary logical states in a constrained, stochastic quantum memory. It combines two-stage learning with a recoverable-information reward so an agent can discover adaptive protection strategies.

Memory: The agent must preserve an arbitrary logical qubit using available gates and measurements despite hardware connectivity constraints and decoherence.The target is retrieval of the stored state with optimum fidelity after a specified time span.
Memory: Naive reinforcement learning usually fails because useful sequences are extraordinarily rare, incomplete sequences can accelerate decay, and trivial and corrective strategies are separated by a wide barrier.In the described scenarios, randomly selecting a good sequence has probability much less than 10^-12.
Memory: The solution combines two-stage learning, in which an RL-trained network teaches a second network, with recoverable quantum information as the reward measure.The first network has expanded state information, while the experimentally applicable network learns from measurement results.
Memory: All possible logical input states are evolved simultaneously so the state-aware network cannot memorize one initial state and later reconstruct it.The shared gate sequence must preserve every possible logical qubit state.
Memory: The first learning stage observes the full space of completely positive maps, whereas the experimental setting provides only partial measurement information and requires a recurrent network.Measurements make the reinforcement-learning environment stochastic.
Memory: Recoverable quantum information measures worst-case remaining distinguishability over the Bloch sphere and therefore evaluates guaranteed preservation of unknown logical states.It can provide an intermediate reward and remains 1 when errors are theoretically perfectly recoverable.
Memory: In a 4-qubit example, training eventually exceeds encoding with periodic parity checks because adaptive recovery responds to unexpected measurement outcomes.The figure evaluates RQ after a 200-step gate sequence and reports approximately a 15% decoherence-time increase over a non-adaptive scheme.

Results

The agent discovers adaptive quantum-error-correction strategies across noise models and hardware constraints, using measurements and gates to preserve quantum information. A two-stage approach makes the learned feedback experimentally applicable through a recurrent network.

Bit-flip error correction: The network learns encoding and repeated parity measurements that detect bit-flip errors and prevent undetected two-bit-flip states.It first discovers an entangled encoding related to the 3-qubit repetition code, then measures parity repeatedly.
Adaptive recovery: Unexpected measurement outcomes trigger nontrivial recovery sequences that resolve ambiguities through additional measurements before returning to detection.The network distinguishes among multiple possible error locations or an erroneous ancilla outcome.
Hardware adaptation: Feedback strategies adapt to hardware connectivity, with different gate sequences discovered for fixed-measurement chains, freely measurable chains, and other layouts.Enhanced connectivity is exploited for greater decoherence-time improvement, while recovery sequences are discovered from scratch.
Hardware adaptation: For unreliable measurements, the network increases measurement redundancy, with the number of measurements rising from 1 per cycle to about 6.This behavior is shown as measurement error increases.
Adaptive phase estimation: Under spatially correlated dephasing, the same reinforcement-learning program tracks field fluctuations through nearby ancillas and discovers adaptive measurement strategies.The method applies without situation-specific input beyond the available gates and physical simulation.
Two-stage learning: A state-aware network is trained first, then a recurrent LSTM network is supervised to imitate it using measurement results and prior gate choices.The recurrent controller can then select gate sequences during experimental runs, where the quantum state is inaccessible.

Conclusions

The paper concludes that neural-network-based reinforcement learning can discover nontrivial quantum-error-correction strategies from scratch and may extend to broader feedback-control problems.

Networks discover a-priori unknown nontrivial detection/recovery sequences for diverse settings using only the available gate set.
The approach is limited to modest qubit numbers, potentially increasing to about 10 with further optimizations.
The authors identify discovering strategies on more qubits, full protection against all noise sources, and multiple logical qubits as future challenges.
Other reinforcement-learning schemes, continuous controls, and recurrent networks could replace or extend the adopted approach.
The two-stage learning approach could also be applied to other reinforcement-learning scenarios using expanded state information.

A. Physical time evolution

The physical evolution tracks an arbitrary logical qubit state by decomposing its density matrix into a reference state and perturbations, then evolving these components stepwise.

The initial logical state is represented by a Bloch vector and a density-matrix decomposition involving a reference state and three perturbation operators.
The reference state and perturbations are evolved separately for each time interval according to an update rule.
Without measurements, the evolution map is completely positive; with measurements, it is unnormalized before explicit renormalization.
The reference state and perturbations provide the density matrix for every logical qubit state at any time.

B. Physical scenarios

The simulations initialize a logical qubit in one physical qubit and model discrete-time evolution under unitary or measurement operations with Markovian noise.

The logical qubit starts in one physical qubit while all other qubits are prepared in the down state |1⟩.
Evolution is divided into uniform time steps, with maps applying dissipative dynamics after unitary operations or projection operators.
Measurement outcomes are sampled stochastically according to their respective probabilities.
The examples use bit-flip and correlated-noise error models.
The coupling parameter μ_q describes how qubit q interacts with the noise.

C. Recoverable quantum information

Recoverable quantum information is evaluated through trace-distance expressions for logical-state perturbations, including measurement-branch averaging when outcomes differ.

The recoverable-information quantity R_Q is defined from the trace norm of a linear combination of perturbation operators weighted by the logical Bloch vector.
The relevant trace-norm expression can depend nontrivially on the logical qubit state, making its minimum difficult to find.
For CHZ quantum circuits, anticommutation restricts the minimum to one of the coordinate axes.
Under correlated noise, the trace-distance expression is symmetric around the z-axis and minimized at the equator.
The averaged quantity R̄_Q combines post-measurement values across branches using their outcome probabilities.

D. Protection reward

The protection reward trains the agent to preserve the ability to recover the target quantum state at the end of a simulation. It combines changes in recoverability with penalties for measurements that reveal the logical state and uses discounted returns for policy gradients.

The protection reward aims to maximize RQ at the end of the simulation, representing the ability to recover the target state in principle.
The immediate reward includes a term based on the change in RQ between successive time steps.
Measurements that reveal the logical qubit state receive a punishment when recoverability is lost.
The return discounts the reward sequence with rate γ for computing the policy gradient.

E. Recovery reward

The recovery reward supplements protection when the agent should decode the quantum state. Decoding is rewarded only after Tsignal, with the objective of maximizing the minimum overlap between target and recovered states on the target qubit.

The protection reward does not encourage the network to decode the quantum state, so suitable reward terms are added when decoding is desired.
Decoding is rewarded only after Tsignal, while the decoding term Dt is set to zero for t ≤ Tsignal.
The recovery criterion tests whether the logical qubit state is encoded in a specific target qubit, although this condition is not necessary.
At final time T, CT equals 1 only when the logical state is encoded solely in the target qubit with the prescribed, non-flipped polarization.
The recovery reward uses the same return discount rate γ as the protection reward.
The reward optimizes OQ, the minimum overlap between pure target states and final states reduced to the target qubit by tracing out all other qubits.

F. Input of the state-aware network

The state-aware network receives compressed representations of density matrices describing arbitrary logical-qubit evolution, along with measurement-safety, action-history, and decoding-time information. These inputs expose relevant state information while reducing dimensionality and accelerating learning.

The network input includes four density matrices representing the completely positive evolution map for arbitrary logical qubit states.The matrices are ρ0, ρ1, ρ2, and ρ3, with the latter three formed by perturbing ρ0 along x, y, and z.
Principal component analysis compresses the density matrices by selecting eigenstates with the largest eigenvalues.
The selected states are scaled by the square roots of their eigenvalues and sorted by eigenvalue before entering the network.
The simulations retain six largest components, producing 768 input neurons from four matrices, 16-dimensional states, and real and imaginary parts.
Boolean features indicate whether each possible measurement projector would reveal quantum information, speeding training by supplying a deduction available from the density matrix.
The latest measurement result is unnecessary because the updated density matrix already conditions future decisions on that outcome, although the previous action is supplied to accelerate learning.
One-hot encoded time information is added for the final steps after Tsignal so the network can learn when decoding should occur.

G. Layout of the state-aware network

The state-aware networks use feedforward multilayer perceptrons with dense connections, ReLU hidden-layer activations, and softmax action outputs. The output therefore represents a probability distribution over available actions.

The state-aware network uses a feedforward architecture with one output neuron per action and two or three hidden layers.
All neighboring layers are densely connected, and hidden units use the rectified linear unit activation function.
A softmax function converts the output activations into a probability distribution over actions.

H. Reinforcement learning of the state-aware network

The state-aware network is trained with policy gradients, using return baselines and natural-gradient information to shape updates. Entropy regularization is applied only to the state-aware network, with Adam used for optimization.

H. Reinforcement learning of the state-aware network: Policy-gradient learning updates the network parameters θ according to the learning gradient.The full gradient expression specifies how θ changes during training.
H. Reinforcement learning of the state-aware network: The discounted return R_t is corrected by an explicitly time-dependent baseline formed as an exponentially decaying average of returns.For epoch N, the baseline uses prior-epoch return information with decay parameter κ.
H. Reinforcement learning of the state-aware network: The natural gradient is computed by multiplying the policy-gradient update by F^-1, the Moore–Penrose inverse of the Fisher information matrix.This incorporates the Fisher-information geometry into the update.
H. Reinforcement learning of the state-aware network: Entropy regularization is used only to train the state-aware network, and parameter updates use Adam without bias correction.The entropy term is not applied to the recurrent network.

I. Layout of the recurrent network

The recurrent network is designed for operation in real-world experiments, where quantum-state information is unavailable directly. It therefore receives measurement outcomes and the previous action as its input.

I. Layout of the recurrent network: The recurrent network is designed to operate in principle in a real-world experiment.Its design excludes direct access to the quantum state or evolution map.
I. Layout of the recurrent network: Measurements are the recurrent network’s only means of obtaining information about the quantum system.Its input contains the present measurement result and the previous action rather than the quantum state itself.
I. Layout of the recurrent network: The input uses one-hot encoding for the previous action and distinguishes measurement outcomes when measurements occur.This encoding supplies the network with recent control and observation history.

J. Supervised learning of the recurrent network

The recurrent network is trained by supervised learning on trajectories generated from a reinforcement-learned state-aware network. LSTM-based training preserves the full temporal sequence, with dropout and Adam used during optimization.

J. Supervised learning of the recurrent network: Training data comes from inference trajectories generated by a state-aware network trained with reinforcement learning.Each time step stores the network input and policy probabilities for all actions.
J. Supervised learning of the recurrent network: The recurrent network is trained on saved inputs and action-probability policies from those trajectories.The data can be generated in sufficient quantity that overfitting is not a concern in the cited example.
J. Supervised learning of the recurrent network: Because of the LSTM layers, training uses complete trajectories in their true time sequence rather than individual actions.Dropout provides regularization during this sequence-based training.
J. Supervised learning of the recurrent network: The supervised training update uses adaptive moment estimation with Adam.The same update rule is specified alongside dropout regularization.

K. Physical parameters and hyperparameters

The paper characterizes recoverable quantum information under measurements and decoherence, and shows how learned feedback strategies encode, monitor, and recover quantum states.

Measurement effects: Parity measurements can temporarily increase RQ when they indicate no bit flip, whereas alternating outcomes can drive RQ to zero.Measurement outcomes therefore produce trajectory-dependent behavior rather than a single deterministic curve.
Measurement effects: Repeated parity measurements delay future losses in RQ, although they do not improve RQ immediately on average.This makes error detection useful for preventing subsequent degradation.
Measurement effects: Measurements that reveal the logical qubit state destroy superpositions, with complete destruction reducing RQ to zero.The learning procedure treats such measurements separately because their effects are exceptionally strong and destabilize training.
Recoverable quantum information: RQ quantifies whether the original quantum information can be recovered from a noisy multi-qubit state without explicitly performing the recovery steps.The learning objective is to maximize RQ at the final time.
Learned feedback: The learned network first distributes information through entangling gates, then uses parity measurements whose outcomes determine later feedback actions.After encoding, decoherence slows; successful measurement outcomes can produce recoverability revivals, while detected errors trigger recovery cycles.
Network behavior: The recurrent network’s hidden representation forms action-specific clusters that track measurement states and a six-fold periodic detection sequence.The representation distinguishes post-measurement coherence, CNOT-induced entanglement, and ancilla states.

b. Modified hyperparameter sets

The analysis tests modified hyperparameter sets across several physical scenarios and finds that learning rate dominates training behavior, while other settings are generally less decisive.

Scenario dependence: The common hyperparameter set succeeds across the tested scenarios, but the triangle setup requires multiple independent attempts and selection of the best run.The triangle scenario’s limited success rate also appears with the hyperparameters used in fig. 5a.
Scenario dependence: A modified ANN layout failure for chain (1 msmt) is treated as an outlier because most retrained runs under the same conditions performed much better.The exact standard learning rate used in simulation is 0.0001.
Other hyperparameters: All other hyperparameters generally work reliably within reasonable bounds, reaching similar final performance although convergence speeds differ.The network architecture appears to play only a minor role.
Learning-rate influence: Learning rate has the largest impact on training progress: values that are too small slow convergence, while overly high values cause instabilities and collapse.A smooth learning curve indicates that the learning rate need not be decreased.
Other hyperparameters: Larger batch sizes seem favorable because they improve the estimate of the Fisher information matrix used by the natural gradient.More data points reduce statistical noise in that estimate.