PaperlayerView exact revision on arXiv

Source-linked AI summary

When does reinforcement learning stand out in quantum control? A comparative study on state preparation

Xiao-Ming Zhang, Zezhu Wei, Raza Asad, Xu-Chen Yang, Xin Wang

arXiv:1902.02157v2quant-ph

TL;DR

It remains unclear which quantum-control algorithm is most suitable under specific control constraints. The paper compares three reinforcement-learning methods with SGD and Krotov for quantum state preparation, finding that PG and DQL generally perform best, especially for discrete or scaled-up problems.

  • Problem

    It remains unclear which algorithm is most suitable for quantum control under different constraints.

  • Method

    The paper compares TQL, DQL, and PG with SGD and Krotov on quantum state preparation, including a multi-qubit extension.

  • Results

    PG and DQL generally perform best, with DQL outperforming the other methods in the scaled-up cases considered.

  • Takeaways & Limitations

    Reinforcement learning is particularly effective for discrete controls and can adaptively reduce the number of control pieces.

Abstract

from arXiv · show

Reinforcement learning has been widely used in many problems, including quantum control of qubits. However, such problems can, at the same time, be solved by traditional, non-machine-learning methods, such as stochastic gradient descent and Krotov algorithms, and it remains unclear which one is most suitable when the control has specific constraints. In this work, we perform a comparative study on the efficacy of three reinforcement learning algorithms: tabular Q-learning, deep Q-learning, and policy gradient, as well as two non-machine-learning methods: stochastic gradient descent and Krotov algorithms, in the problem of preparing a desired quantum state. We found that overall, the deep Q-learning and policy gradient algorithms outperform others when the problem is discretized, e.g. allowing discrete values of control, and when the problem scales up. The reinforcement learning algorithms can also adaptively reduce the complexity of the control sequences, shortening the operation time and improving the fidelity. Our comparison provides insights into the suitability of reinforcement learning in quantum control problems.

I. INTRODUCTION

The paper asks when reinforcement learning is more suitable than conventional optimization for constrained quantum control. It compares three reinforcement-learning algorithms with SGD and Krotov using quantum state preparation as a common test problem.

  • I. INTRODUCTION: Quantum control typically requires numerical optimization, motivating comparison between reinforcement learning and methods such as SGD and Krotov.
  • I. INTRODUCTION: The central question is which algorithm works best under different constraints on qubit state preparation.
  • I. INTRODUCTION: The study compares TQL, DQL, and PG with SGD and Krotov for steering a qubit toward a target state.
  • I. INTRODUCTION: SGD updates controls along randomly selected parameter-space directions using the state-target mismatch as its cost function.
  • I. INTRODUCTION: Krotov propagates the state forward and a mismatch-encoding co-state backward while updating the control fields.
  • I. INTRODUCTION: Q-learning selects actions from states using rewards, while PG uses a neural-network policy that outputs action probabilities and supports continuous input states.

II. RESULTS

The single-qubit study evaluates state preparation under piecewise-constant, bounded, and discrete controls using averaged fidelity. Across these settings, reinforcement-learning methods can adapt control complexity, while PG and DQL generally achieve the strongest performance.

  • A. Single-qubit case: PG remains best as N increases, although its fidelity decreases slightly; fidelities from the other methods also generally decrease except for Krotov at large N.
  • A. Single-qubit case: TQL, DQL, and PG naturally handle discrete control values, whereas Krotov favors continuous controls and SGD is relatively insensitive to discretization.
  • A. Single-qubit case: For N < 30, DQL gives the highest fidelity apart from PG, while for N > 30 Krotov outperforms DQL, whose fidelity remains close to 1 at F = 0.9988.
  • A. Single-qubit case: TQL, DQL, and PG adaptively reduce the number of control pieces, whereas SGD and Krotov use a fixed N and can miss optimal solutions.

B. Multi-qubit case

The multi-qubit task transfers a single spin across a chain under algorithm-specific control-field restrictions. As the number of spins increases, DQL and PG retain higher fidelity than Krotov, while SGD fails for the illustrated K = 8 case.

  • Multi-qubit state preparation: The task transfers an excitation from the leftmost spin to the rightmost spin over K spins using 20 equal time steps.The system starts with only the leftmost spin up and targets a state with only the rightmost spin up.
  • Multi-qubit state preparation: The control field is continuous in [0, 40] for SGD and Krotov but restricted to {0, 40} for the reinforcement-learning algorithms.The multi-qubit Hamiltonian uses constant nearest-neighbor coupling and time-dependent local magnetic fields for control.
  • Multi-qubit state preparation: At K = 8, fidelity is 0.0989 for Krotov, 0.4214 for PG, and 0.5433 for DQL, with DQL highest among these methods.All algorithms deteriorate as the number of spins increases; for K < 4, Krotov, DQL, and PG have comparable performance.
  • Multi-qubit state preparation: DQL and PG have better average performance than Krotov across 100 runs, although their best individual results are comparable.At K = 8, amplitudes measure overlap with states having each spin up, with perfect transfer requiring amplitude 1 only at the rightmost spin.

III. DISCUSSION

The comparison identifies conditions under which particular algorithms are better suited to quantum state preparation. PG and DQL generally perform best, especially for restricted or scaled-up control problems, while Krotov favors continuous controls.

  • Maximum control-sequence length: PG has the best overall performance as the maximum number of control pieces N varies, although its fidelity decreases slightly as N increases.All methods except Krotov show decreasing fidelity with increasing N; Krotov improves when N is large.
  • Adaptive segmentation: TQL, DQL, and PG can adaptively reduce the number of control pieces, whereas fixed-N SGD and Krotov can miss the optimum.Adaptive segmentation can reduce control-sequence complexity during optimization.
  • Restricted control-field strength: TQL, DQL, and PG perform well with restricted control-field strengths, while restrictions severely compromise Krotov because it makes significant updates.The restriction has a more moderate effect on SGD because its updates are essentially local.
  • Discrete control values: TQL, DQL, and PG naturally support discrete control values, but their fidelity decreases as the allowed values become more continuous.The paper suggests Actor-Critic and deep deterministic policy gradient methods as possible ways to address this issue.
  • Scaled-up problems: DQL generally outperforms the other algorithms for multi-qubit problems with K ≤8, whereas SGD is inefficient and TQL does not straightforwardly scale.Except for TQL, the other algorithms can be generalized to more than one qubit.
  • III. DISCUSSION: PG and DQL generally achieve the best performance among the five algorithms, highlighting reinforcement learning with neural networks for complex optimization.The paper summarizes algorithm behavior across control-sequence length, adaptive segmentation, field-strength restrictions, discrete controls, and multiple qubits.
  • III. DISCUSSION: The direct comparison is intended to clarify algorithm suitability under different quantum-control constraints and suggest directions for improving the methods.The authors specifically suggest incorporating Krotov gradients into Q-learning procedures.

Methods

The implementation section introduces tabular Q-learning, deep Q-learning, and policy-gradient methods used in the comparison. Full procedures for all algorithms are provided in the supplementary methods.

  • Methods: The study implements TQL, DQL, and PG, while the complete algorithms for all methods appear in Supplementary Method 1.The five compared methods are SGD, Krotov, TQL, DQL, and PG.

A. TQL

TQL represents qubit control as a discrete-state, discrete-action reinforcement-learning problem, with rewards assigned from fidelity to the target state.

  • A. TQL: TQL discretizes qubit states and assigns each control value to an action in the allowed action set.After each evolution step, states are mapped to the closest member of the discrete state set.
  • A. TQL: The reward increases sharply as the resulting state approaches the target, using fidelity-based reward tiers.The reward assigns 10, 100, or 5000 according to fidelity ranges from 0.5 to 1.
  • A. TQL: An epsilon-greedy policy usually selects the action with the largest Q-value but occasionally explores randomly to avoid poor local minima.The Q-table is updated using a learning rate and a discount factor for reward stability.

B. DQL

DQL replaces tabular action-value storage with a neural network, allowing continuous qubit-state representations while retaining discrete control actions and rewards.

  • B. DQL: DQL encodes the qubit state as real and imaginary amplitudes and uses a neural network to output Q-values for every action.This removes the need to discretize states on the Bloch sphere.
  • B. DQL: DQL trains separate evaluation and target networks using randomly sampled stored experiences.Each experience contains the preceding state, action, reward, and resulting state.

C. PG

PG uses a neural network to produce action probabilities and updates its parameters to increase discounted episode rewards, but its fidelity can oscillate across training.

  • C. PG: PG outputs a probability distribution over actions from the current state and samples each control action from that distribution.The network parameters are updated using the gradient of log action probability weighted by total discounted reward.
  • C. PG: PG optimizes network parameters by increasing the total discounted reward accumulated during each episode.The return at time t is defined as the discounted sum of subsequent rewards.
  • C. PG: Unlike SGD and Krotov, PG and the other reinforcement-learning methods can produce oscillating fidelities and cannot guarantee optimal solutions in every trial.Higher-fidelity outputs must therefore be selected as the learning outcome.

Supplementary Discussion 1: Effects of bounds and discrete control field for SGD and Krotov methods

Control bounds and discretization affect Krotov much more strongly than SGD, while larger control sequences or finer control values can recover Krotov performance.

  • Supplementary Discussion 1: Effects of bounds and discrete control field for SGD and Krotov methods: For N < 30, restricting Krotov controls to Ji ∈[0, 1] substantially lowers average fidelity, whereas SGD is largely unaffected by the same bound.Krotov uses large control updates, while SGD changes controls locally; at N = 6, the restricted Krotov result is higher but both fidelities remain below 0.6.
  • Supplementary Discussion 1: Effects of bounds and discrete control field for SGD and Krotov methods: For N > 30, unrestricted Krotov approaches 1−F < 10^-7, while the restricted result remains close, reaching F = 0.9822 at N = 30.The comparison indicates that more control pieces can compensate for limited control-field strength.
  • Supplementary Discussion 1: Effects of bounds and discrete control field for SGD and Krotov methods: At N = 20, increasing Krotov's bound Jmax from 0 to 20 raises averaged fidelity from 0.4 to above 0.8.This directly demonstrates the sensitivity of Krotov performance to the allowed control range.
  • Supplementary Discussion 1: Effects of bounds and discrete control field for SGD and Krotov methods: SGD fidelity quickly saturates as the number of discrete control values increases, whereas Krotov improves more strongly and exceeds SGD near M + 1 = 15 for N = 20.The result indicates that Krotov depends more critically on continuity in both sequence length and allowed control values.

Supplementary Discussion 2: Improving the fidelity

Allowing more iterations improves fidelity across algorithms, with the largest improvement reported for SGD. At N = 50, Krotov reaches unit fidelity rapidly, while DQL improves only slightly after 50 iterations.

  • Increasing the iteration budget can improve all algorithms’ fidelities, beyond the fixed N iter = 500 comparison used in the main text.
  • The supplementary comparison plots average fidelity against iteration count for maximum control-piece counts N = 20 and N = 50, averaging each value over 100 runs.
  • SGD’s fidelity improves most strongly with additional iterations, rising from around 0.6 at N iter = 500 to 1 at N iter ≳500.
  • Krotov reaches fidelity 1 at approximately N iter ∼20 for N = 50, whereas DQL improves slightly after N iter = 50.

Supplementary Discussion 3: Target state

The supplementary analysis examines how average fidelity varies with the target state using equatorial Bloch-sphere states parameterized by ϕ. Krotov remains stable across these targets, while the other methods vary substantially.

  • The target states lie on the Bloch-sphere equator and are parameterized solely by ϕ, enabling average fidelity to be plotted against ϕ.
  • Krotov maintains an average fidelity of about 0.8 across the examined target states, unlike the other methods whose outputs vary substantially.
  • DQL reaches average fidelity 1 at ϕ = 1.5π but falls below 75% at ϕ = 1.25π.

Supplementary Discussion 4: Noise effect

The supplementary noise analysis evaluates robustness by perturbing selected control fields and averaging fidelity over repeated noise realizations. It also lists representative high-fidelity pulse profiles for the 8-qubit state-transfer problem.

  • Noise robustness is tested by adding uniformly distributed control errors δJ_i ∈ [−ϵ, ϵ] and evaluating 100 noisy realizations for each selected sequence.
  • Supplementary Figure S5 plots average fidelity against noise level for the target state |1⟩ with N = 20, averaging each point over 100 runs.
  • The supplementary material provides pulse profiles for the 8-qubit state-transfer problem from Krotov, SGD, DQL, and PG runs.
Loading 1902.02157v2…