PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for Quantum Technology

Marin Bukov, Florian Marquardt

arXiv:2601.18953v1quant-phcond-mat.quant-gascond-mat.stat-mechcs.AIcs.LG

TL;DR

Quantum constraints make direct transfer of existing machine-learning ideas inappropriate or insufficient for quantum physics. This review surveys reinforcement-learning applications and reports experimental demonstrations, including entangling-gate pulses with fidelities an order of magnitude higher than state-of-the-art pulses.

  • Problem

    Quantum constraints make direct transfer of existing machine-learning ideas inappropriate or insufficient, prompting a quantum-focused account of reinforcement-learning applications.

    Source: §I · p. 4

  • Method

    The review surveys reinforcement-learning applications in quantum physics, distilling successful approaches and identifying outstanding challenges.

    Source: §I · p. 4

  • Results

    An order of magnitude higher fidelities than state-of-the-art pulses were reported for optimized entangling gates, alongside spin-squeezed-state preparation and trapped-ion circuit construction.

    Source: §I · p. 4

  • Takeaways & Limitations

    The review describes reinforcement learning as increasingly applied in real-world quantum-device experiments.

    Source: §I · p. 4

    A key open challenge is learning efficiently from sparse quantum measurements, since simulation-trained policies may need modification before experimental use.

    Source: §XII · p. 50

Abstract

from arXiv · show

Many challenges arising in Quantum Technology can be successfully addressed using a set of machine learning algorithms collectively known as reinforcement learning (RL), based on adaptive decision-making through interaction with the quantum device. After a concise and intuitive introduction to RL aimed at a broad physics readership, we discuss the key ideas and core concepts in reinforcement learning with a particular focus on quantum systems. We then survey recent progress in RL in all relevant areas. We discuss state preparation in few- and many-body quantum systems, the design and optimization of high-fidelity quantum gates, and the automated construction of quantum circuits, including applications to variational quantum eigensolvers and architecture search. We further highlight the interactive capabilities of RL agents, emphasizing recent progress in quantum feedback control and quantum error correction, and briefly discuss quantum reinforcement learning as well as applications to quantum metrology. The review concludes with a discussion of open challenges -- such as scalability, interpretability, and integration with experimental platforms -- and outlines promising directions for future research. Throughout, we highlight experimental implementations that exemplify the increasing role of reinforcement learning in shaping the development of quantum technologies.

I. INTRODUCTION · II. REINFORCEMENT LEARNING IN A NUTSHELL

This section frames reinforcement learning as an interactive, model-free approach to quantum-technology challenges and introduces how agents learn reward-driven policies. It also outlines Actor-Critic methods and presents PPO as a robust starting point.

I. INTRODUCTION

  • Quantum technology faces challenges in state preparation, gate pulses, circuit compilation, hardware-specific transpilation, and feedback; RL offers model-free, adaptive, interactive, and autonomous control.The review also notes that RL can address stochastic optimization problems and mitigate noise online.Source: §I · pp. 2–3
  • Quantum measurement stochasticity, unknown noise or dynamics, and weak-measurement constraints make RL a natural fit, while abundant quantum data and midcircuit feedback motivate data-efficient methods.RL agents can learn from partial observations and may help identify effective macroscopic degrees of freedom in interacting quantum matter.Source: §I · pp. 3–4
  • The review surveys quantum state preparation, gate optimization, circuit synthesis, feedback control, error correction, quantum RL, and metrology, alongside experimental applications and quantum-specific challenges.It also identifies open challenges for future research.Source: §I · p. 4

II. REINFORCEMENT LEARNING IN A NUTSHELL

  • In RL, an agent observes an environment state, selects actions, receives rewards as the environment changes, and learns a policy to maximize expected cumulative reward over an episode.Episodes proceed from an initial state until a terminal condition, after which the environment resets.Source: §II · p. 5
  • Reward design assigns a task-specific measure of success and should distinguish good from unwanted behavior; the reward function is fixed before training and has no universal design recipe.The text recommends using dimensionless physical quantities to construct rewards.Source: §II · pp. 5–6
  • RL algorithms include policy-gradient methods that optimize parameterized policies and value-function methods that evaluate states or actions; actor-critic methods learn both.Q-learning uses a self-consistency relation to select actions with the highest Q-value.Source: §II · pp. 6–7, Figure 4 · p. 7
  • Actor-Critic methods combine value functions and policy gradients; examples include TRPO, PPO, and DDPG, with PPO recommended as a robust, generic starting algorithm.The passage also directs readers to tutorials on reinforcement learning for quantum technology (Duncan et al., 2025).Source: §II · p. 7

III. REINFORCEMENT LEARNING FOR QUANTUM SYSTEMS · A. Reinforcement Learning Framework · 1. Environment, States, Observations, Actions, and Rewards

The section formalizes how an RL agent interacts with a quantum environment, where measurement limits access to the state and can itself change the system. It explains how states, observations, actions, and rewards shape that interaction.

III. REINFORCEMENT LEARNING FOR QUANTUM SYSTEMS

  • The section formalizes RL concepts for quantum systems, then turns to deep RL algorithms and contrasts model-free, model-based, and optimal-control approaches.Source: §III · p. 7

A. Reinforcement Learning Framework

  • Quantum RL frameworks must account for inaccessible wavefunctions, costly full-state tomography, uncertain and state-disturbing measurements, and discrete probabilistic outcomes.Measurements of local observables reveal information, but uncertainty remains and projective measurements collapse the state.Source: A. Reinforcement Learning Framework · p. 7

1. Environment, States, Observations, Actions, and Rewards

  • The environment is a physical system with states and action-dependent transition probabilities; quantum states are often continuous wavefunctions or density matrices, with deterministic or stochastic dynamics.An effective state may describe only relevant degrees of freedom, and RL states are properties of the environment rather than the agent.Source: §1 · p. 8
  • Observations are the state information available to the agent, so partial observability makes learning harder; quantum-control observations may be expectation values, while error-correction observations may be syndrome measurements.Different states can yield the same observation, as different errors can produce the same syndrome.Source: §1 · p. 8
  • Adding task parameters to observations can let one agent learn a family of tasks, generalize across related tasks, and operate across that range without retraining.Source: §1 · p. 8
  • Quantum actions are controls such as fields, gates, and measurements; action spaces may be discrete, continuous, mixed, multidimensional, or state-dependent.Because measurement changes the system state, choosing to measure is an action rather than a passive observation.Source: §1 · p. 8
  • The agent learns through episodes to maximize expected long-term return, using rewards that may arrive only at the final measurement or at each step, as in quantum error correction.Reward design should make task completion maximize return; optimizing return can favor actions with delayed benefits rather than the highest immediate reward.Source: §1 · pp. 8–9

2. Policy, Value, and Action-Value (Q-) Functions · 3. Markov Decision Processes and the Reinforcement Learning Objective · 4. Simulated versus Real Environments

Reinforcement learning selects actions through policies and evaluates them using value functions, optimizing expected return in a Markov decision process. Quantum applications must also account for model mismatch, partial observation, measurement backaction, and the practical constraints of interacting with real devices.

2. Policy, Value, and Action-Value (Q-) Functions

  • Stochastic policies balance exploration and exploitation, support gradient-based learning with discrete actions, and can outperform deterministic policies in stochastic environments.Their action probabilities also provide a natural framework for manipulating quantum systems, whose dynamics are inherently stochastic.Source: §2 · p. 9
  • Value functions estimate expected return from a state, while Q-functions resolve actions and can yield a deterministic policy at least as good as the original.Both are defined through Bellman equations; under certain conditions, an optimal Q-function can be used to reconstruct an optimal policy.Source: §2 · p. 9

3. Markov Decision Processes and the Reinforcement Learning Objective

  • An MDP assumes each next state depends only on the current state and action; the agent optimizes its policy while the environment’s transition probabilities remain fixed.In partially observable quantum applications, the policy may instead condition on observations rather than the full state.Source: §3 · p. 10
  • MDP trajectories run from an initial state to a terminal state in finite-horizon episodic tasks, whereas infinite-horizon tasks have T →∞.At the end of an episode, the environment resets to its initial state.Source: §3 · p. 10
  • Optimizing expected return supports stochastic optimization and discrete rewards, and its time-nonlocal objective lets agents plan across state-action pairs within an episode.This capacity to capture long-range correlations does not violate the MDP’s Markovianity.Source: §3 · p. 10

4. Simulated versus Real Environments

  • Simulation enables detailed observations, sophisticated rewards, and faster data generation, but deployment can suffer when real-device parameters differ from the assumed model.Model-free agents treat the simulator as a black box; deployment with less information may require two-stage training or learning from demonstration (Li et al., 2025).Source: §4 · pp. 10–11
  • Real-device control can be latency-limited: superconducting gates take 10ns, while a sub-microsecond-latency FPGA neural network enabled RL for a single superconducting qubit.Agent computation and physical data transfer can be slower than the quantum system’s response.Source: §4 · p. 11
  • Quantum environments are partially observable, measurements cause backaction and can halt evolution through the quantum Zeno effect, and wave-function collapse typically restricts rewards to the final step.Repeated measurements can yield expectation-value observations when agent computation is slower than system response, while comparable timescales favor single-shot data and sufficiently large training batches. Ancilla measurements provide partial information but perturb the qubit; stronger information causes stronger collapse, and the setup formalizes a quantum partially observable MDP.Source: §4 · p. 11

B. Deep Reinforcement Learning · C. Overview of Reinforcement Learning Algorithms · 1. Policy Gradient Methods

Deep RL uses variational approximations to handle large or continuous state spaces, while policy-gradient methods optimize parameterized policies from sampled trajectories. The section also describes alternating policy evaluation and improvement, along with practical refinements to policy learning.

B. Deep Reinforcement Learning

  • Large or continuous state spaces make exhaustive exploration infeasible, so variational policy or value-function approximations can interpolate or extrapolate to unseen states.The choice of ansatz affects learning; deep neural networks are common, while tensor networks have been suggested for Q-functions in quantum spin-chain control.Source: B. Deep Reinforcement Learning · p. 12
  • For discrete actions, policy networks can output log-probabilities; for continuous actions, they can parameterize a state-dependent Gaussian, with bounded actions also supported by Beta or sigmoid-Gaussian distributions.A logsoftmax layer enforces the probability constraint, and learning log-probabilities can improve resolution among low-probability actions.Source: B. Deep Reinforcement Learning · pp. 12–13
  • Neural architectures can exploit local input structure and known symmetries, using invariant value functions and equivariant policies or Q-functions; constructing quantum-state embeddings from binary observations remains open.A permutation-equivariant transformer policy was used to disentangle multiqubit states.Source: B. Deep Reinforcement Learning · p. 13

C. Overview of Reinforcement Learning Algorithms

  • RL algorithms alternate policy evaluation, which collects states and rewards, with policy improvement, while function approximation can compromise convergence guarantees established for tabular or linear methods.Modern approaches often combine policy-gradient and value-function ideas rather than fitting a strict category.Source: C. Overview of Reinforcement Learning… · p. 13

1. Policy Gradient Methods

  • REINFORCE parameterizes a policy and uses Monte Carlo trajectory samples to estimate its objective gradient for ascent, avoiding enumeration of exponentially many trajectories.The gradient estimate follows from expressing the trajectory-probability derivative as an expectation.Source: §1 · pp. 13–14
  • REINFORCE variance can be reduced by using only current and future rewards and subtracting an action-independent baseline; a learned value-function baseline yields actor-critic updates.A state-dependent baseline can be approximated by a separate value model trained on return data.Source: §1 · pp. 14–15
  • PPO reuses interaction data while limiting policy changes (Schulman et al., 2017), whereas TRPO’s Fisher-informed updates can be costly; MinSR motivates an open efficiency question for TRPO.The cost arises from estimating the inverse Hessian of the KL divergence for policies with many parameters.Source: §1 · p. 15
  • Entropy regularization adds policy entropy to the reward to discourage convergence to a delta-function policy when multiple actions should remain equally likely (Levine, 2018).The tradeoff resembles competition between return and policy entropy.Source: §1 · p. 15

2. Value Function Methods

Value-function methods seek an optimal Q-function and derive actions from it, using transition-based Q-learning and deep-learning variants to learn value estimates. The section also describes exploration strategies and modifications for training stability, estimation bias, and continuous actions.

2. Value Function Methods

  • Policies derived from Q-functions can use ε-greedy exploration, Boltzmann action probabilities, temporally correlated noise, or neural-network parameter noise.A deterministic policy selects the action with the largest Q-value; softening it enables exploration (Lillicrap et al., 2015; Fortunato et al., 2017).Source: §2 · pp. 15–16
  • Tabular Q-learning updates Q-values from sampled transitions and converges to the optimal value function if every state can be visited arbitrarily often.The algorithm was proposed by Watkins and is known as Q-learning.Source: §2 · p. 16
  • DQN approximates the Q-function with a neural network and trains it by regression on sampled batches of transitions.Learning from time-local transitions rather than complete trajectories can help with tasks that have excessively long episodes (Mnih et al., 2013, 2015).Source: §2 · p. 16
  • A fixed auxiliary Q-network supplies training targets, with soft parameter updates and gradient clipping used to stabilize DQN training.The auxiliary parameters move toward the target network parameters incrementally, while the auxiliary network is held fixed during updates.Source: §2 · p. 16
  • Double Q-learning addresses bias from the max operation by using one Q-network to select an action and a second to estimate its value.The passage identifies inaccurate learned Q-values as a common issue with Q-learning.Source: §2 · p. 16
  • DDPG and TD3 are deep Q-learning algorithms that can handle continuous actions (Lillicrap et al., 2015).Source: §2 · p. 17
  • Using strictly negative rewards in Q-learning can produce shorter protocols because the agent seeks to minimize its penalty.Source: §2 · p. 17

3. Other Related Algorithms · D. Model-Free versus Model-Based Reinforcement Learning · E. Choosing the Appropriate Algorithm

The section distinguishes evolutionary, search, and projective-simulation approaches from RL, then explains how model availability and data, runtime, and sample-efficiency trade-offs guide algorithm choice.

3. Other Related Algorithms

  • Genetic algorithms search with evolving populations, typically run faster and consume more data than RL, whose agents select actions based on environmental observations.Their processes include mutation, crossover, and selection.Source: §3 · p. 17
  • Search algorithms explore large state spaces and support RL planning; they can also pretrain policies and Q-functions through imitation learning, as in AlphaZero’s MCTS.Examples include Monte Carlo Tree Search, Beam Search, and A∗-Search.Source: §3 · p. 17
  • Projective Simulation models agent cognition as a stochastic network of episodic memory clips, with reinforcement-driven plasticity updating transitions from reward feedback (Briegel and De las Cuevas, 2012).Its clips encode percepts and actions; the framework also describes adaptive forgetting, compositional learning, and generalization through shared pathways.Source: §3 · p. 17

D. Model-Free versus Model-Based Reinforcement Learning

  • Model-free RL suits black-box systems, while model-based methods can differentiate through a faithful, modern-framework simulation to reduce training samples; “model-based” has broader uses elsewhere.The broader literature also applies the term to learned differentiable models and simulations learned in advance (Sch¨afer et al., 2020; Abdelhafez et al., 2019; Sch¨afer et al., 2021; Khalid et al., 2023; Porotti et al., 2023).Source: D. Model-Free versus Model-Based… · pp. 17–18

E. Choosing the Appropriate Algorithm

  • On-policy methods such as REINFORCE train on fresh trajectories from the current policy, whereas off-policy Q-learning reuses old and new transitions and supports asynchronous updates.Reusing data can improve data efficiency, and decoupled collection and policy improvement are useful when data are scarce or slow to collect.Source: E. Choosing the Appropriate Algorithm · p. 18
  • Offline methods such as REINFORCE and DQN wait until an episode ends to improve the policy, while online Q-learning improves it at each episode step.The section notes that learning from already available information may be beneficial.Source: E. Choosing the Appropriate Algorithm · p. 18
  • For quantum experiments, off-policy Deep Q-Learning can suit data-collection bottlenecks, while Policy Gradient and Actor-Critic are starting points for faster simulators.The recommended choice depends on whether obtaining environment data or training the agent is the bottleneck.Source: E. Choosing the Appropriate Algorithm · p. 18
  • Figure 6’s heuristic places more sample-efficient methods toward slower runtimes, with gradient-free and evolutionary methods often faster than model-based RL.Shallow value iteration is sample-efficient and fast for small tabular problems but does not scale to high-dimensional observations without function approximation; deep model-free methods need many environment steps but can parallelize, while online actor-critic methods are generally more sample-efficient than high-variance REINFORCE.Source: E. Choosing the Appropriate Algorithm · pp. 18–19, Figure 6 · p. 19

IV. REINFORCEMENT LEARNING & QUANTUM OPTIMAL CONTROL · V. QUANTUM STATE PREPARATION

RL offers quantum control methods that can operate without detailed dynamics models and adapt to noisy, partially observed systems, though they require more training data and iterations than conventional optimal control. In state preparation, RL has produced fast protocols, including experimental preparation of a Dicke state with high fidelity.

IV. REINFORCEMENT LEARNING & QUANTUM OPTIMAL CONTROL

  • RL’s gradient-free methods do not require derivatives of the cost with respect to controls, so model-free algorithms can work without detailed dynamics models and with discontinuous rewards such as binary measurement outcomes.This flexibility comes with a computational tradeoff: RL generally needs more iterations and a steady supply of training data, and may minimize cost less efficiently than gradient-based control.Source: §IV · p. 19
  • RL agents learn in expectation, handle stochastic dynamics and reward signals, and can produce policies resilient to noise and perturbations; after training, they may generalize to unseen initial states and self-correct errors.Unlike optimal control, which requires re-optimization when the environment changes, an RL policy or Q-function can exploit learned knowledge without further optimization.Source: §IV · p. 19
  • RL’s advantages include operating without prior system knowledge, transferring learned knowledge between similar systems, enabling feedback control, and autonomously developing complex protocols.These capabilities are relevant to unknown noise, incompletely known Hamiltonians, measurement backaction, and experimental manipulation without human intervention.Source: §IV · p. 19

V. QUANTUM STATE PREPARATION

  • State-preparation RL maps control parameters to actions and fidelity-related rewards, but inaccessible fidelity may require approximations, projective-measurement estimates, or energy minimization using a known parent Hamiltonian.Because quantum states are not directly measurable, observations may rely on tomography for small systems or a partially observable MDP built from observables and expectation values.Source: §V · p. 20
  • For Dicke-state preparation, well-defined large-N observations supported pre-training on smaller systems, while an actor-critic PPO pipeline progressed from dissipationless simulation to dissipative trajectories and experiment.The agent’s observations included condensate density, its fluctuations, and a first-order quantum correction; its bounded continuous actions set the normalized quadratic Zeeman shift.Source: §V · pp. 21–22
  • For Dicke-state preparation in a spin-1 Bose-Einstein condensate, F ≈0.99 was reached at |c2|τ = 15.5, versus ≈600 for a linear adiabatic sweep and ≈350 for a locally optimized nonlinear sweep at the same fidelity (Guo et al., 2021).Source: §V · p. 21, Figure 7 · p. 21
  • The Dicke-state protocol crosses the critical point twice, using controlled excitation dynamics to prepare a Gaussian-like wavepacket and rapidly translate it toward the target state (Guo et al., 2021).The first crossing creates a prepared superposition of low-lying excitations; the second stage sweeps adiabatically through the critical point before the final rapid translation.Source: §V · pp. 21–22
  • RL state-preparation applications span tweezer steering, laser-pulse shaping (Capuano et al., 2025), many-body excitation transfer (Sgroi et al., 2024), and prescribed current states.The review organizes subsequent work by system size, from few-body qubits and Λ-systems to many-body spin-squeezed states, spin chains, and Hubbard-type models.Source: §V · p. 22

A. Few-Body Systems

Reinforcement learning is surveyed across few-body quantum control, from state preparation and measurement-based feedback to quantum-dot characterization, arbitrary-target control, thermodynamic cycles, and energy storage. The studies illustrate both practical successes and constraints, including fidelity and robustness gains, inaccessible or sparse rewards, and trade-offs among performance objectives.

Control of Λ-systems

  • RL agents find robust and flexible control strategies across dissipative few-body systems, with physics-informed PPO outperforming the state of the art in fidelity and robustness (Ernst et al., 2025).Other studies report resilience to unmodeled perturbations in three-level feedback control, efficient coherent population-transfer protocols competitive with adiabatic schemes, and resilience under dissipative dynamics and restricted controls (An et al., 2021). Earlier work applied tabular Q-learning to spin-1/2 and Λ systems, while PPO found fast, flexible nonadiabatic integer and fractional STIRAP transfer protocols.Source: A. Few-Body Systems · pp. 22–23

Quantum dots control

  • In semiconductor quantum dots, RL supports robust state preparation, automated device characterization, improved adiabatic passage, and measurement-based spin control.DQN outperformed gradient-based optimization and found protocols robust to charge and nuclear noise; dueling DQN automatically located bias triangles. TRPO found a sequence outperforming the counterintuitive control sequence, while DQN learned nuclear-spin entangled-state preparation sequences that generalized previously found human solutions.Source: A. Few-Body Systems · p. 23

Learning from projective measurements

  • Because experimental agents often cannot access state fidelity directly, studies use measurable projective rewards, but end-of-episode sparse rewards impede learning over long episodes (Bukov, 2018; Sivak et al., 2022).Bukov used binary rewards and action histories, which avoid unmeasurable quantum-state inputs but make generalization to different initial states difficult; Sivak et al. used tomographically complete POVMs and probabilistic reward measurements to estimate target fidelity.Source: A. Few-Body Systems · p. 23

Arbitrary target state control

  • To generalize across targets, RL agents have generated arbitrary NV-center superpositions, mapped arbitrary state pairs to control trajectories, and prepared a family of targets on an experimental circuit (Wang and Wang, 2024).Haug et al. used a network to group protocols into similar types and identify general strategies; Wang and Wang directly encoded target-state information. Hutin et al. trained on simulation, then tested learned pulse shapes on an experimental superconducting circuit.Source: A. Few-Body Systems · pp. 23–24

Quantum thermodynamics

  • Thermodynamic RL has also reduced entropy production and energy dissipation in closed systems, while energy-optimized deep RL exposed a process-fidelity versus energetic-cost trade-off but performed worse than gradient-based methods (Fauquenot et al., 2024).Fauquenot et al. used quantum optimal-control techniques to optimize gate process fidelity and energetic cost.Source: A. Few-Body Systems · pp. 24–25
  • RL has identified quantum-engine protocols with Pareto trade-offs among power, efficiency, and fluctuations, and asymmetric nonadiabatic Szilard engines with efficiency matching the traditional adiabatic engine (Sørdal and Bergli, 2019; Erdman et al., 2023; Erdman and Noé, 2023).The Pareto-cycle frameworks target quantum dots, a superconducting-qubit refrigerator, and a quantum-harmonic-oscillator heat engine.Source: A. Few-Body Systems · p. 24
  • For Dicke quantum batteries, adaptive RL charging improved stored energy and charging precision over constant-parameter schemes while preserving collective speedup and mitigating detrimental quantum-chaos effects.The learned controls varied atom-cavity coupling strength and system-cavity detuning.Source: A. Few-Body Systems · p. 25

Miscellaneous

  • In cavity optomechanics, DDPG found driving pulses that suppress dissipation-induced transitions and prepare high-fidelity phononic Fock-state superpositions and two-mode entangled states (Liu et al., 2025).The demonstrations covered single- and two-resonator systems.Source: A. Few-Body Systems · p. 25

B. Many-Body Systems

Many-body state preparation is difficult because state and configuration spaces grow exponentially, and fidelity generally decreases with system size. The section surveys RL approaches spanning cooling, entangled and ground-state preparation, quantum circuits, and learning from limited measurements.

Cooling

  • RL cooling agents control cold-atom traps, coupled oscillators, and mechanical resonators, using feedback or force pulses to reduce atom-cloud counts, oscillator occupation, or vibrational-mode energy (Reinschmidt et al., 2023; Kalita and Sarma, 2024).The resonator agent used partial observations of collective mode quadratures to damp multiple modes concurrently.Source: B. Many-Body Systems · p. 25

Spin-squeezed states

Quantum spin systems

  • RL prepared twin-Fock states that outperformed adiabatic dynamics in a many-body spin model, while spin-chain agents also found local optima and faster-than-known high-fidelity transfer protocols (Chen and Xue, 2019; Bukov et al., 2018; Zhang et al., 2018).The transfer task sought the shortest high-fidelity protocols; Q-learning ground-state preparation could become trapped in local optima.Source: B. Many-Body Systems · p. 26
  • A comparative study found adaptive RL algorithms reduced control steps to find optimal state-preparation solutions, whereas SGD and Krotov used a fixed number of steps.The comparison included tabular Q-learning, deep Q-learning, policy gradient, stochastic gradient descent, and Krotov.Source: B. Many-Body Systems · p. 26
  • RL also supports spin-Hamiltonian ground-state search by simulated annealing and robust control of nonergodicity in a one-dimensional Fermi-Hubbard model (Mills et al., 2020; Ye and Lai, 2024).The Fermi-Hubbard PPO protocols control tilt field and interaction strength; the annealing reward is sparse and based on minimum negative energy.Source: B. Many-Body Systems · p. 26

Quantum circuit architecture

  • RL circuit optimization reduced CNOT gate counts by two orders of magnitude for thermal-state preparation in SYK systems with N ≥12, versus traditional methods, including noisy hardware.DDQN jointly optimizes circuit architecture and parameters using free energy as the reward.Source: B. Many-Body Systems · p. 27

Limited-measurement control

  • With limited measurements, representation-guided PPO efficiently controlled uncharacterized intermediate-scale systems, preparing XXZ-chain ground states across phases and continuous-variable cat states (Zhu et al., 2024).The method constructs an internal state representation from a small amount of measurement data and compares it with the target representation.Source: B. Many-Body Systems · p. 27

Meta-learning

  • Meta-RL learned variational-circuit training schedules, reduced training data needs relative to quantum state tomography, and generalized from 3 to 5 qubits (Jae et al., 2024).The agent optimized the sampling-distribution variance schedule and gradient-descent step size.Source: B. Many-Body Systems · p. 27

VI. PULSE-SHAPE ENGINEERING FOR QUANTUM GATE IMPLEMENTATION · A. Single-Qubit Gates

The section surveys reinforcement learning for quantum-gate synthesis, highlighting experimental pulse optimization and approaches to single-qubit gates. It describes model-free control that can learn hardware imperfections, alongside methods for general unitary synthesis, faster low-leakage pulses, noise-robust control, and measurement-based learning.

VI. PULSE-SHAPE ENGINEERING FOR QUANTUM GATE IMPLEMENTATION

  • RL can compensate for miscalibration without accurate noise or Hamiltonian models, learn platform-specific errors from measurements, and account for leakage and parameter drift.The conventional model-based approach can lose gate fidelity when its assumptions or approximations are inaccurate.Source: §VI · pp. 27–28
  • Infidelity fell from (1.05 ± 0.07)×10−2 for the theory ZX(−π/2) pulse to (4.40±0.58)×10−3 for RL; the optimized gate remained resilient 25 days later.Randomized benchmarking also showed the RL gate outperforming the theory pulse.Source: Figure 8 · p. 28, §VI · p. 29
  • Baum et al. (2021) report RL RX(π/2) rotations up to 3 times faster than DRAG and ZX(−π/2) gates up to 2 times faster, with FZX > 99.5%.They designed an error-robust gate set on superconducting hardware and also discussed direct in-situ SWAP-gate optimization.Source: §VI · pp. 28–29
  • Baum et al. (2021) addressed limited quantum-state observability with repeated-measurement tomography during episodes and a weighted reward, then evaluated gates using Clifford randomized benchmarking.They repeated intermediate measurements 256 times and final-state measurements 1024 times; training reinitialized the qubits and replayed prior actions before applying each new action.Source: §VI · p. 28

A. Single-Qubit Gates

  • Moro et al. (2021) trained a PPO agent on Haar-random gates to approximate arbitrary single-qubit unitaries, producing shorter execution times and improved tradeoffs between protocol length and execution time.The agent used target-relative observations and π/128 rotations about ±x, ±y, and ±z.Source: A. Single-Qubit Gates · p. 29
  • Wright and De Sousa (2023) used separate RL agents for in-phase and out-of-phase pulses; experiments produced X gates up to twice as fast while maintaining equivalent fidelity and leakage rates.Their approach used measured in-phase and quadrature signals directly as observations, avoiding quantum-state classification.Source: A. Single-Qubit Gates · p. 29
  • Ding et al. (2021) combined PPO with shortcuts to adiabaticity for bounded, noise-robust digital control, and Ai et al. (2022) experimentally validated fast protocols on a trapped 171Yb+ ion without prior error knowledge.The ion experiment found robustness to over-rotations and off-resonance errors, with operation times consistent with STA estimates.Source: A. Single-Qubit Gates · p. 29
  • Ding et al. (2023) used weak ancilla measurements and PPO to learn single-qubit gates under detuning, dephasing, and relaxation, with transfer learning adapting to new noise patterns without retraining from scratch.The agent used measurement outputs and qubit-state information to choose normalized Rabi-frequency actions.Source: A. Single-Qubit Gates · p. 30

B. Two-Qubit Gates · VII. QUANTUM CIRCUIT DESIGN

RL offers a route to synthesizing entangling two-qubit gates where exact analytical solutions are typically unavailable, and to constructing quantum circuits through sequential decisions. In a trapped-ion demonstration, an RL agent disentangled four-qubit states using state-dependent circuits despite experimental noise.

B. Two-Qubit Gates

  • TRPO reduced average gate error by two orders of magnitude versus stochastic gradient descent and reduced gate time by up to an order of magnitude versus optimal synthesis baselines.The framework jointly optimized speed and fidelity against leakage and stochastic control errors in two-state and three-state transmon models.Source: B. Two-Qubit Gates · p. 30
  • DDPG discovered cross-resonance ZX and CNOT control solutions without prior system knowledge, surpassing direct and echoed schemes in both fidelity and execution time.The approach used quantum states as observations and continuous actions on the real and imaginary components of transmon control channels.Source: B. Two-Qubit Gates · p. 30
  • Dueling double DQN synthesized CNOT and Hadamard gates, while Shindi et al. (2023) extended the approach to Hadamard, CNOT, and T gates using only end-of-episode measurements.The latter framework enabled policy discovery without quantum-state access during learning.Source: B. Two-Qubit Gates · p. 30
  • TD3 with step penalties discovered shorter control sequences than gradient-based and evolutionary algorithms while maintaining high accuracy for single-qubit and CNOT gates.The reward combines an affine function of log-resolved gate fidelity with step penalties.Source: B. Two-Qubit Gates · p. 30
  • PPO was used to design RYY(π/4) entangling gates, and the study concluded that effective implementation requires no physics insights such as Hamiltonian details.The work also analyzed the need for human expert guidance.Source: B. Two-Qubit Gates · p. 30

VII. QUANTUM CIRCUIT DESIGN

  • The agent builds circuits gate-by-gate, using reduced two-qubit states to select qubit pairs with RL, analytic gate parameters, and an entropy-threshold reward that penalizes circuit length.A state is considered disentangled when average single-qubit entanglement falls below 10^-3.Source: §VII · p. 31
  • For 4-qubit disentangling, the trained agent uses at most five state-dependent two-qubit gates and successfully disentangles entangled inputs despite trapped-ion experimental noise.The agent was trained in simulation and then tested experimentally; the figure describes circuits for arbitrary 4-qubit states.Source: Figure 9 · p. 32, §VII · p. 32
  • PPO training uses permutation-equivariant attention, and agents outperform full-state deterministic methods on structured entanglement while demonstrating robustness to noise with local observations.The authors estimate two-qubit reduced density matrices, which scale as ∝L2, rather than measuring exact states; attention layers adjust actions when qubits are permuted.Source: §VII · pp. 31–32

A. Entanglement Control · B. Variational Quantum Eigensolvers

Reinforcement learning is applied to entanglement generation, stabilization, and disentanglement, as well as to VQE parameter optimization and circuit discovery. Across these tasks, agents use varied observations, actions, and rewards to control quantum systems and construct circuits under practical constraints.

A. Entanglement Control

  • Continuous-control agents optimize entanglement under dissipation and in distributed architectures, while tabular Q-learning finds optimal gate sequences for representative four-qubit entangled states.Li et al. (2024) use a DQN to control two-photon driving; Almanakly et al. (2024) target remote four-qubit |W⟩ states; Giordano and Martin-Delgado (2022) use tabular Q-learning.Source: A. Entanglement Control · p. 33
  • Bao et al. (2024) found that fewer measurements can disentangle random Clifford circuits than predicted by measurement-induced phase-transition studies.Their PPO agent searches measurement configurations by balancing final-state entropy against measurement costs.Source: A. Entanglement Control · p. 34
  • Cemin et al. (2025) showed feedback control can shift Clifford-circuit steady states from equilibrium to non-equilibrium and trained an agent to control entanglement across up to 128 qubits.The agent chooses where to place disentangling gates using masked Clifford-tableau observations and a reward based on negative chain entanglement.Source: A. Entanglement Control · p. 34
  • Melnikov et al. (2018) showed projective simulation could autonomously rediscover advanced photonic techniques and propose experimental setups researchers initially deemed impossible.The agent assembles optical components to generate target classes of high-dimensional entangled multiphoton states.Source: A. Entanglement Control · p. 34

B. Variational Quantum Eigensolvers

  • These frameworks outperformed existing optimization algorithms for quantum state preparation under Hamiltonian noise or measurement uncertainty, while policy-gradient methods with correlated multivariate Gaussian policies were more competitive than gradient-free methods on noisy and noiseless circuits.The frameworks used binary rewards determined by target-state fidelity. The correlated-Gaussian approach used wave-function amplitudes as states, continuous gate angles as actions, and circuit-fidelity rewards averaged over random initial states.Source: B. Variational Quantum Eigensolvers · p. 35
  • Patel et al. (2024) reported state-of-the-art performance on quantum chemistry VQA tasks, while Kundu et al. (2024) found curriculum DQN with Kolmogorov-Arnold networks superior to multilayer perceptrons.Source: B. Variational Quantum Eigensolvers · pp. 35–36
  • Garcia-Saez and Riu (2019) achieved optimal Max-Cut results up to N = 21 qubits, while Khairy et al. (2019) generalized from small instances and outperformed other optimizers on superconducting NISQ devices.Source: B. Variational Quantum Eigensolvers · p. 36
  • Gadget RL discovered compact hardware-suitable circuits for transverse-field Ising ground states, reducing errors by a few orders of magnitude compared with pure RL approaches.Kundu and Sarra (2024) use Double Deep Q-learning to learn composite gates that make searching the gate space more efficient.Source: B. Variational Quantum Eigensolvers · p. 36

C. Quantum Circuit Compilation and Architecture Search

Quantum circuit transpilation and synthesis use RL to search sequentially for gate decompositions that reduce depth or gate counts, while architecture search discovers circuits tailored to target states and hardware. The studies span Clifford+T synthesis, hardware-aware and graph-based optimization, fault-tolerant compilation, and specialized compilation tasks.

Quantum Circuit Compilation and Architecture Search

  • RL discovered variational ansatz circuits that diagonalize random 2-qubit density matrices using fidelity rewards incorporating conditional entropy and concurrence.The Double DQN agent appended gates to ansatz circuits, with entanglement-guided incentives weighted against circuit depth and gate count.Source: C. Quantum Circuit Compilation and… · p. 37

Clifford+T Gate Set Synthesis

  • Clifford+T RL achieved minimal Bell-state circuits [K¨olle et al., 2024b], exact synthesis up to five qubits outperforming state-of-the-art methods [Rietsch et al., 2024], and up to 16.8% fewer T gates [Weiden et al., 2024].AlphaTensor-Quantum outperformed state-of-the-art T-count optimization on arithmetic benchmarks, finding the best human-designed solutions for computations used in Shor’s algorithm and quantum chemistry simulations [Ruiz et al., 2024].Source: C. Quantum Circuit Compilation and… · p. 37

Hardware-Aware Circuit Optimization

  • Hardware-aware RL found unit-fidelity 3-qubit QFT circuits using seven CZ gates on a 9-qubit processor, while reducing 12-qubit random-circuit depth by 27% and gate count by 15% on average [F¨osel et al., 2021].Model-based RL reduced SWAP usage by up to 62.5% under nearest-neighbor constraints; the convolutional agent also demonstrated application to 50-qubit circuits [F¨osel et al., 2021].Source: C. Quantum Circuit Compilation and… · pp. 37–38

Graph-Theoretic and Tree-Search Approaches

Advanced Circuit Synthesis and Compilation

  • Policy- and value-network-guided weighted A∗ search produced near-optimal, ancilla-free Fibonacci-anyon braiding sequences [Zhang et al., 2020], while Double Q-learning found exact circuit compilations with fewer gates than state-of-the-art methods.PPO with curriculum learning provided near-optimal synthesis for Linear Function, Clifford, and Permutation circuits, reducing two-qubit gate depth and count [Kremer et al., 2024].Source: C. Quantum Circuit Compilation and… · p. 38

Fault-Tolerant and Multi-Core Compilation

  • Transfer-learning PPO discovered fault-tolerant logical-state circuits with fewer gates and ancillary qubits than state-of-the-art methods, for up to 15 physical qubits.For multi-core compilation, policy-gradient agents modeled core allocation and communication costs, while maskable PPO reduced nonlocal communication and outperformed state-of-the-art algorithms.Source: C. Quantum Circuit Compilation and… · pp. 38–39

Specialized Compilation Tasks

  • PPO-generated photonic circuits violated the CHSH inequality [Lanore et al., 2024], while hybrid discrete-continuous RL reduced circuit sizes for trapped-ion devices, including for unknown processes.RL-based methods also extended quantum compilation to Pauli network synthesis [Dubal et al., 2025].Source: C. Quantum Circuit Compilation and… · p. 39
  • RL routing outperformed state-of-the-art procedures on random and realistic circuits up to 50 qubits, and topology-aware compilation-flow design outperformed state-of-the-art compilers on expected fidelity and critical depth.Distributed compilers jointly managed EPR generation, remote-operation scheduling, routing, and SWAP insertion to reduce expected execution time.Source: C. Quantum Circuit Compilation and… · p. 39

VIII. QUANTUM FEEDBACK CONTROL

The section examines how reinforcement learning turns quantum measurements into feedback actions, from real-time experimental control to learned strategies for cooling, state preparation, and error correction.

Experimental real-time feedback

  • Under 1 microsecond response enabled real-time feedback for ground-state initialization of thermally excited qubits and qutrits; training took three minutes after about 1,000 experimental actions.The FPGA implementation overlapped network evaluation with incoming measurement data, adding only 50 nanoseconds of latency.Source: Figure 10 · p. 40, §VIII · p. 40

Feedback challenges

  • Because a non-Markovian policy may require actions for each of M^N measurement histories, feedback agents must map observations to actions quickly, unlike fixed controls; active calibration can tolerate longer waits.Physical feedback observations are real measurements, and qubit feedback may need to arrive much faster than the coherence time.Source: §VIII · pp. 39–40

Open-system feedback

  • Erdman et al. (2024b) used soft actor-critic to discover interpretable feedback strategies balancing measurement-powered cooling and efficiency across weak or projective measurement regimes and timescale hierarchies.The agent acted on fully observable conditioned density matrices using continuous controls and discrete thermalization, measurement, or unitary-feedback choices.Source: §VIII · p. 41

Oscillator control

  • Sivak et al. (2022) trained a PPO agent to prepare oscillator states through binary ancilla measurements and adaptive feedback, outperforming state-of-the-art model-free methods in sample efficiency without tomography or fidelity estimation.The framework uses a POMDP with continuous gate parameters in a fixed circuit architecture.Source: §VIII · p. 41

Cavity feedback

  • Porotti et al. (2022) used PPO to prepare and stabilize high-fidelity cavity Fock states without prior dynamics knowledge, reaching superpositions when measurement rates could be controlled individually.Their reward used the eighth power of target-state fidelity at every episode time step.Source: §VIII · p. 41

Model-based RL

  • Feedback GRAPE combines Monte Carlo sampling, policy-gradient methods, and automatic differentiation to handle stochastic discrete measurements, enabling robust preparation and stabilization and an enhanced GKP error-correction strategy.The approach was developed by Porotti et al. (2023) and later applied to a non-Markovian feedback version of the GKP protocol.Source: §VIII · p. 41

IX. QUANTUM ERROR CORRECTION · A. Quantum Decoders · B. Discovery of Error Correcting Codes

Reinforcement learning supports quantum error correction by adapting decoding and code design to device-specific noise and control constraints. The section surveys experimental QEC, decoder optimization, and discovery of error-correcting codes.

IX. QUANTUM ERROR CORRECTION

  • The experimental GKP QEC controller increased logical-qubit lifetime by 2.27±0.07 through in-situ learning of circuit parameters on the device.The logical qubit resides in a cavity oscillator controlled and measured through an auxiliary transmon; the agent adapts to real error channels and control imperfections.Source: §IX · pp. 42–43, Figure 11 · p. 42

A. Quantum Decoders

  • Sivak et al. (2025) reduced the surface-code logical error rate by 2.4 (3.5 with decoder steering) and demonstrated scalability to d = 15 with 40,000 gate controls.Their surrogate objective uses error-detection-event counts to train without knowing the logical state; Sivak et al. (2024) also report improved repetition- and surface-code decoding accuracy over the state of the art.Source: A. Quantum Decoders · p. 43
  • RL decoders also address varied settings: Matekole et al. (2022) adapt to drifting noise, Freire et al. (2025) optimize erasure-channel hypergraph product codes, and Park et al. (2024) find effective measurement-free protocols.Park et al. report improved logical-qubit lifetime over the state of the art for the 2D classical Ising model and 4D toric code.Source: A. Quantum Decoders · pp. 43–44
  • For toric codes, Andreasson et al. (2019) approached MWPM performance through d = 7, while Fitzek et al. (2020) surpassed MWPM under depolarizing noise through d = 9.The Fitzek decoder performed close to optimally for uncorrelated noise but suboptimally for biased noise.Source: A. Quantum Decoders · p. 44

B. Discovery of Error Correcting Codes

  • F¨osel et al. (2018) used a teacher-student learning procedure and a recoverable-information reward to discover feedback-based QEC strategies, including approaches beyond stabilizer encoding.QEC encodes a logical qubit across physical qubits, measures syndromes to detect changes, and applies corrections to extend logical-qubit lifetime.Source: §IX · p. 41, B. Discovery of Error Correcting Codes · p. 44
  • Nautrup et al. (2019) used transfer learning to adapt surface-code memories to unknown error channels or laboratory setups, finding near-optimal solutions for up to 70 data qubits.The agent grows the code lattice until its error rate falls below a target threshold.Source: B. Discovery of Error Correcting Codes · p. 44
  • Olle et al. (2024) discovered QEC codes and encoding circuits from specified hardware constraints, scaling to 20 physical qubits and d = 5 codes.A noise-aware meta-agent produces strategies across noise models; a follow-up study introduced composite Clifford gates (Olle et al., 2025).Source: B. Discovery of Error Correcting Codes · pp. 44–45
  • He and Liu (2025) discovered low-weight stabilizer codes that outperform the state of the art and reduce physical-qubit overhead by an order of magnitude for weight-6 codes.The agent minimizes Tanner-graph node degrees while preserving code distance.Source: B. Discovery of Error Correcting Codes · p. 45

X. QUANTUM REINFORCEMENT LEARNING

Quantum reinforcement learning (QRL) uses quantum policies or value functions in classical or quantum environments; experimentally, a hybrid quantum-classical agent combined quantum search with classical learning to learn faster.

QRL framework

  • QRL makes the agent’s policy or value function quantum, spanning classical-agent/quantum-environment work (CQ) and quantum-agent settings with classical (QC) or quantum (QQ) environments.Quantum agents may offer a more natural way to interact with quantum environments and avoid reliance on full-state tomography, but practical advantage over classical RL remains unknown.Source: §X · p. 45

Featured experimental application

  • The hybrid quantum-classical agent outperformed purely quantum and purely classical strategies, demonstrating a quadratic speedup in learning time.On a programmable nanophotonic processor, the quantum strategy’s average reward peaked and then declined; switching to classical learning when it began decreasing at η≈0.39 produced the stronger hybrid strategy. Quantum episodes use amplitude amplification, while classical episodes supply rewards for policy updates.Source: §X · pp. 45, 47, Figure 12 · p. 46

Quantum Advantage for Classical RL

Data Encoding and NISQ Constraints

  • QRL work tackles data-encoding and NISQ constraints with Quantum-Train policy networks, tensor-network variational circuits, data re-uploading, and metaheuristic optimization (Liu et al., 2024; Chen et al., 2022c; Kölle et al., 2024d; Kölle et al., 2024c).Quantum-Train reduces policy-network parameters and needs only classical inference; tensor-network architectures handle input dimensions beyond available qubits, while data re-uploading and exponential learning-rate decay improve stability and performance.Source: §X · p. 48

Circuit Architecture Optimization

  • QRL circuit-search studies optimize variational architectures for MNIST, combine metaheuristics with multi-agent optimization, and use model-based RL to reduce circuit evaluations or select problem-specific encoding circuits (Wang et al., 2024a; Kölle et al., 2024a; Rapp et al., 2024; Eisenmann et al., 2024).The MNIST agent selects target qubits and gates, with rewards based on validation accuracy.Source: §X · p. 48

Photonic QRL Platforms

  • Photonic QRL studies use orbital angular momentum for scalable multiarmed bandits and examine how non-ideal agents’ decisions deviate from Born’s rule (Konaka et al., 2025; Piera et al., 2024).The bandit study reports performance exceeding existing techniques; the Born’s-rule study simulates agents betting on informationally complete measurements using heralded single photons.Source: §X · p. 48

Quantum Communication

  • In quantum communication, RL agents identified improved protocols for teleportation, entanglement purification, and quantum repeaters, with solutions for long-distance communication beyond the state of the art (Wallnöfer et al., 2020).Source: §X · p. 48

Quantum Speedups in Active Learning

  • Quantum methods have demonstrated quadratic speedups in active RL, including proof-of-principle two-qubit projective-simulation experiments with radiofrequency-driven trapped ions.The trapped-ion experiments reported quadratic improvements in agent deliberation time over classical RL agents.Source: §X · pp. 48–49

XI. QUANTUM METROLOGY: PARAMETER ESTIMATION & SENSING

RL can optimize quantum probe states and adaptive measurement strategies for parameter estimation and sensing, but compelling experimental demonstrations beyond the state of the art remain unrealized.

Parameter Estimation

  • Quantum metrology seeks precision beyond the SQL scaling N^-1/2 toward the Heisenberg limit N^-1, using probe states and measurements whose QFI bounds precision through the Cramér–Rao inequality.Entangled probes include squeezed, GHZ, and critical ground states; adaptive measurement settings updated from prior outcomes make RL a natural optimization framework, while adiabatic preparation becomes inefficient near criticality. Compelling experimental RL demonstrations beyond the state of the art remain unrealized, despite theoretical evidence of useful adaptive strategies and the transferability of methods from other quantum tasks.Source: §XI · pp. 49–50
  • Belliardo et al. (2024a, 2024b) develop model-aware Bayesian metrology with particle filtering and automatic differentiation, applying it to fields, hyperfine interactions, photonic phases, and diamond-spin decoherence times.Crosta et al. (2024) demonstrate model-free RL recalibration of drifting device parameters, while Xu et al. (2019) use RL to identify controls for precision and study generalization.Source: §XI · p. 49
  • Fallani et al. (2022) find that RL feedback improves long-time frequency estimation over control-free and standard open-loop policies; Qiu et al. (2022) obtain Heisenberg-scaling protocols for one-axis-twisting entanglement generation.Xiao et al. (2022) incorporate noise bounds and a physics-inspired linear time-correlated control ansatz, validating time-dependent and time-independent estimation in noisy and noise-free settings.Source: §XI · p. 49

Quantum Sensing

  • Schuff et al. (2020) use RL-optimized nonlinear control pulses to mitigate decoherence and improve quantum-chaotic sensor precision, outperforming periodic protocols.The cross-entropy method optimizes pulse strength and position.Source: §XI · pp. 49–50
  • Chih et al. (2022) find an optical-lattice gyroscope protocol that differs from a conventional Mach–Zehnder-type interferometer and improves sensitivity 20-fold over traditional Bragg interferometry.The agent generates end-to-end lattice-shaking protocols optimized for rotational signals.Source: §XI · p. 50
  • Xu et al. (2025) report that a quantum RL agent finds protocols saturating the finite quantum speed limit, generalizing to large systems and achieving Heisenberg and super-Heisenberg scaling.The protocols target precise state preparation for quantum-critical sensing.Source: §XI · p. 50

XII. OPEN CHALLENGES & FUTURE DIRECTIONS

The review identifies open challenges in quantum RL, from physics-informed learning and scalable architectures to real-time hardware deployment, scientific discovery, and integration with broader AI advances.

Design of Reward Functions and Learning Objectives

  • Quantum RL needs physics-informed rewards, sample-efficient learning from sparse measurements, rapid adaptation, and multi-objective control; meta-learning is a proposed route.Simulation-trained policies may not transfer directly to experiments, especially for active feedback, so agents must learn from sparse quantum measurements. Adaptation to new Hamiltonians, target states, or experimental conditions should avoid extensive retraining while balancing fidelity, decoherence resilience, and runtime.Source: §XII · p. 50

Next-Generation RL Architectures

  • Quantum RL architectures must handle non-Markovian histories and exploit model-based planning or coordinated multi-agent control, whose quantum-specific benefits remain untested.Candidate memory mechanisms include transformer attention, recurrent networks, and graph neural networks; predictive models combined with tree search and quantum-specific heuristics are also proposed.Source: §XII · p. 50

Scaling to Many-Body Quantum Systems

  • Scaling RL to many-body systems requires addressing exponential state-space growth, partial observations, expanding action spaces, and emergent phenomena.Symmetry-respecting equivariant networks, tensor networks, quantum simulators, or direct training on the target system may mitigate simulation and representation costs; learning must also account for sparse, invasive measurements of incompatible observables. Proposed directions include parallel or compound actions, reusable hierarchical “quantum gadgets,” exploiting emergent degrees of freedom such as order parameters, and understanding transfer of policies across system sizes.Source: §XII · p. 51

Real-Time Implementation on Quantum Hardware

  • Real-time quantum-hardware RL must overcome communication latency, limited feedback bandwidth, incomplete measurements, and costly sampling.Quantum control can operate on nanosecond-to-microsecond timescales while classical computation and data transfer may add microseconds to milliseconds; low-latency architectures and edge computing are proposed. Shadow tomography, randomized measurements, Bayesian uncertainty estimation, and hybrid model-based/model-free learning may reduce measurement overhead and sample requirements.Source: §XII · p. 51

Physics Discovery through Reinforcement Learning

  • Interpretable quantum-control policies could turn RL from an optimization tool into a means of discovering physical principles, protocols, and hidden conservation laws.Proposed interpretability tools include visualizing critical policy components and using symbolic regression to distill learned behavior into analytical forms; whether RL can reveal new dynamical principles remains open.Source: §XII · pp. 51–52

Integration with the Broader AI Ecosystem

  • LLMs could support quantum-protocol reasoning, experimental design, and RL development, while RLHF could incorporate physicists’ expertise (Bommasani, 2021).Foundation models may enable natural-language interfaces, literature synthesis, and protocol suggestions, and their attention and training approaches are being adapted for quantum science (Zaklama et al., 2025). Coding assistance may accelerate RL projects, while RL with verifiable rewards could support quantum-science reasoning agents; however, leading LLMs may be closed-source and costly, and RLHF requires substantial human time.Source: §XII · p. 52
  • Combining RL with computer vision could enable automated experimental setup optimization and real-time quantum-device diagnostics.Visual feedback paired with quantum measurements may reveal links between system performance and environmental factors that escape human observation; broader convergence between quantum technologies and AI may open new approaches to discovery and experimental design.Source: §XII · p. 52

Related papers

Loading 2601.18953v1…