PaperlayerView exact revision on arXiv

Source-linked AI summary

Fidelity-based Probabilistic Q-learning for Control of Quantum Systems

Chunlin Chen, Daoyi Dong, Han-Xiong Li, Jian Chu, Tzyh-Jong Tarn

arXiv:1806.03145v1cs.LGcs.SYeess.SYstat.ML

TL;DR

Q-learning must balance exploration and exploitation, while quantum-control learning may lack directional feedback until the agent reaches the target state. FPQL uses probabilistic action selection updated during learning and fidelity information from system structure or behavior to guide quantum-control learning. FPQL found the optimal control sequence after less than 50 episodes, while PQL needed about 150 episodes and QL more than 200 episodes; FPQL outperformed basic Q-learning in convergence speed.

Abstract

from arXiv · show

The balance between exploration and exploitation is a key problem for reinforcement learning methods, especially for Q-learning. In this paper, a fidelity-based probabilistic Q-learning (FPQL) approach is presented to naturally solve this problem and applied for learning control of quantum systems. In this approach, fidelity is adopted to help direct the learning process and the probability of each action to be selected at a certain state is updated iteratively along with the learning process, which leads to a natural exploration strategy instead of a pointed one with configured parameters. A probabilistic Q-learning (PQL) algorithm is first presented to demonstrate the basic idea of probabilistic action selection. Then the FPQL algorithm is presented for learning control of quantum systems. Two examples (a spin- 1/2 system and a lamda-type atomic system) are demonstrated to test the performance of the FPQL algorithm. The results show that FPQL algorithms attain a better balance between exploration and exploitation, and can also avoid local optimal policies and accelerate the learning process.

I. INTRODUCTION

The introduction frames exploration–exploitation balance and delayed feedback as obstacles to reinforcement learning for quantum control. It proposes probabilistic action selection and fidelity-based Q-learning as approaches to improve learning in that setting.

I. INTRODUCTION

  • Existing exploration strategies can struggle to balance exploration and exploitation and often require difficult parameter choices.The introduction also notes that excessive exploration can reduce learning performance, while greedy exploitation may lead to locally optimal policies.Source: §I · p. 1
  • The paper proposes probabilistic action selection to dynamically balance exploration and exploitation, and presents FPQL for learning control of quantum systems.The authors focus on the probabilistic aspect of decision-making in Q-learning, with a fidelity-directed exploration strategy.Source: §I · p. 1
  • Delayed feedback can slow quantum-control learning because the agent may receive no directional information until it reaches the target state.The paper presents fidelity as a global direction signal that can be fed back to the learning system.Source: §I · p. 2
  • The stated evaluation applies FPQL to a spin-1/2 system and a Λ-type atomic system.The introduction says the algorithms are analyzed with the aim of speeding up learning.Source: §I · p. 2

II. Q-LEARNING AND EXPLORATION STRATEGY

This section reviews Q-learning and existing exploration strategies before introducing probabilistic action selection as an alternative. It contrasts fixed or parameter-driven strategies with action probabilities that can change during learning and be directed by fidelity.

II. Q-LEARNING AND EXPLORATION STRATEGY

  • Q-learning learns a policy by choosing actions from observed states, receiving rewards, and updating state-action values toward maximizing discounted reward.Its delayed-reward formulation does not require prior knowledge of the environment.Source: §II · p. 2
  • ε-greedy selects the greedy action most of the time but explores uniformly among other actions, making ε difficult to tune for the desired balance.The section notes that this equal weighting does not distinguish the worst action from the next-best action.Source: §II · p. 2
  • Softmax weights actions according to estimated Q-values, but its temperature τ is difficult to set and does not ensure re-exploration when reduced.When action values are close, Softmax may also fail to separate the best action from the others.Source: §II · p. 2
  • The proposed probabilistic approach updates action-selection probabilities during learning, with fidelity directing the distribution in FPQL.Unlike fixed ε-greedy probabilities, this distribution changes as learning proceeds rather than being computed only from Q-values and a temperature parameter.Source: §II · p. 3, Figure 1 · p. 3

A. Probabilistic Action Selection and Reinforcement Strategy

PQL represents a policy as a probability distribution over actions for each state, samples actions from that distribution, and updates the distribution alongside Q-values. The resulting probabilities can adapt during learning and support re-exploration when the environment changes.

A. Probabilistic Action Selection and Reinforcement Strategy

  • PQL represents the policy with a probability distribution over permitted actions for each state and selects actions according to those probabilities.The distribution is defined over the state-action space and normalized for each state.Source: A. Probabilistic Action Selection and… · pp. 3–4
  • PQL keeps the Q-value update used by Q-learning and also updates the selected action’s probability using the immediate reward and next-state Q estimate.The action probabilities for that state are normalized after each update; the experiments use step size k = 0.01.Source: A. Probabilistic Action Selection and… · p. 4
  • Action probabilities evolve from an initially uniform distribution toward a learned policy and can change online when the environment changes, triggering relearning.The section contrasts this behavior with ε-greedy, whose action distribution remains constant for a given state.Source: A. Probabilistic Action Selection and… · p. 4
  • The authors say probabilistic exploration eases parameter setting and naturally supports re-exploration, at the cost of some additional memory for probability updates.They identify the step size k as the only parameter to set and say its variation over a relatively large range has only slight effect after normalization.Source: A. Probabilistic Action Selection and… · p. 4

B. Probabilistic Q-learning Algorithms

PQL uses a state-specific probability distribution to select actions, updating that distribution alongside Q-values. This gives the agent an ongoing chance to explore while balancing exploration and exploitation.

B. Probabilistic Q-learning Algorithms

  • At each state, PQL selects an action according to its probability and normalizes the updated action probabilities.Its update uses the immediate reward and the estimated next-state action value.Source: B. Probabilistic Q-learning Algorithms · pp. 4–5
  • PQL dynamically updates action-selection probabilities alongside Q-values, rather than relying only on the policy learned so far.The agent samples an action according to its probability at the current state, then updates that probability based on the reward and estimated next-state Q-value.Source: B. Probabilistic Q-learning Algorithms · p. 4
  • Fig. 3 illustrates how the action-selection probability distribution evolves over the state-action space during learning and re-exploration.Source: Figure 3 · p. 5

C. Fidelity-based PQL Algorithm

FPQL extends probabilistic action selection by using fidelity to direct updates to action probabilities for quantum-control learning. The paper relates this guidance to exploration, convergence conditions, and changes in action-selection uncertainty as learning proceeds.

C. Fidelity-based PQL Algorithm

  • FPQL updates action-selection probabilities using reward, the estimated next-state Q-value, and fidelity between the next state and target state.The fidelity definition is task-dependent; this study uses fidelity between quantum pure states for quantum-control learning.Source: C. Fidelity-based PQL Algorithm · p. 5
  • Under the stated constraints, FPQL's Q-values converge to the optimal state-action value function.The constraints include bounded rewards, a discount factor in [0, 1), and conditions on the nonnegative learning rate.Source: C. Fidelity-based PQL Algorithm · pp. 5–6
  • The authors describe probabilistic exploration as quantum-tunnelling-like, with fidelity strengthening the effect that can step over local optima.They contrast this with thermal-fluctuation-type exploration in ε-greedy, Softmax, and simulated annealing methods.Source: C. Fidelity-based PQL Algorithm · p. 6
  • FPQL can also apply to RL problems with a clear quantitative fidelity measure of distance from the current state to the target.Fidelity regulates the action-probability distribution and, the authors state, does not deteriorate PQL's exploration strategy.Source: C. Fidelity-based PQL Algorithm · p. 6
  • Uniform action probabilities give maximum state exploration entropy, log2 m; entropy starts at this maximum and tends toward its minimum when learning converges to an optimal policy.Source: C. Fidelity-based PQL Algorithm · p. 6

A. Learning control of quantum systems

The section formulates learning control as a model-free sequential decision problem for N-level quantum pure-state transitions. A finite set of control fields supplies the actions used to drive the system from an initial state to a target state.

A. Learning control of quantum systems

  • The paper studies quantum pure-state transitions in N-level systems, with control laws learned through iterative search for an optimal strategy.Source: A. Learning control of quantum systems · p. 6
  • The controlled state evolves under a Schrödinger equation with a time-independent interaction Hamiltonian and a control field.The initial state is expanded in the eigenstates of the free Hamiltonian, and the resulting dynamics are finite-dimensional.Source: A. Learning control of quantum systems · pp. 6–7
  • A given finite control set maps each control field to a unitary operator; the task is to find a sequence that drives the system from its initial state to the target state.Source: A. Learning control of quantum systems · p. 7
  • The target problem class has a limited set of control fields and is formulated as a model-free sequential Markov decision process.The paper motivates this setting by assuming very limited control resources.Source: A. Learning control of quantum systems · p. 7

B. Quantum controlled transition landscapes

This section frames quantum control as maximizing a state-transition objective over control landscapes and explains how landscape properties inform learning-control design. It states conditions under which the landscapes are trap-free and connects those ideas to examples using FPQL.

B. Quantum controlled transition landscapes

  • Quantum control landscapes map time-dependent control Hamiltonians to performance values, with many control problems framed as maximizing transition probability from an initial to a desired final state.The section names the state-transition objective as the quantum controlled transition landscape.Source: B. Quantum controlled transition… · p. 7
  • Trap-free landscapes are guaranteed for the kinematic case under complete controllability, and for the dynamic case when complete controllability and a full-rank control-map Jacobian both hold.The section identifies these conditions as properties of the solution sets of the respective landscapes.Source: B. Quantum controlled transition… · p. 8
  • Landscape traps matter for learning because exploitation can settle on locally optimal policies, while too much exploration can worsen learning performance.The paper presents probabilistic action selection and fidelity-based design as ways to balance exploration and exploitation.Source: B. Quantum controlled transition… · p. 8
  • The section introduces spin-1/2 and Λ-type atomic control examples as demonstrations of FPQL as an alternative approach to quantum control design.The paper says these examples study learning control with FPQL.Source: B. Quantum controlled transition… · p. 8

C. Example 1: learning control of a spin- 1

The spin-1/2 example tests learning control toward a target state while minimizing control steps. FPQL reaches the optimal sequence sooner than PQL and standard QL in the reported comparison.

C. Example 1: learning control of a spin- 1

  • For each state, the permitted controls at every step are U1 (no control input), U2 (a positive pulse control), and U3 (a negative pulse control).Source: C. Example 1: learning control of a… · p. 8
  • Under 50 episodes, FPQL finds the optimal sequence, compared with about 150 for PQL and more than 200 for standard QL.The paper also reports that QL and PQL need more steps per episode to find the target state.Source: C. Example 1: learning control of a… · p. 9
  • The learned optimal sequence is used to demonstrate control of the spin-1/2 system from its initial state to the target state.The paper presents the corresponding control result in Fig. 12.Source: C. Example 1: learning control of a… · p. 9

D. Example 2: learning control of a Λ-type quantum system

The Λ-type example tests control from the first basis state to the third using a fixed-length pulse sequence. FPQL converges in fewer episodes than PQL and QL, and its learned sequence is shown controlling the system to the target state.

D. Example 2: learning control of a Λ-type quantum system

V. CONCLUSIONS

FPQL uses fidelity information to guide probabilistic action selection in quantum-system control. In discrete spin-1/2 and Λ-type atomic examples, it converges faster than basic Q-learning; comparisons with other methods on more general quantum-control problems are future work.

V. CONCLUSIONS

  • FPQL is reported to converge faster than basic Q-learning in control examples using spin-1/2 and Λ-type atomic systems.The considered cases are discrete examples.Source: §V · p. 12
  • FPQL extracts fidelity information from system structure or behavior to guide probabilistic action selection and improve the exploration–exploitation tradeoff.The stated aim also includes speeding up learning.Source: §V · p. 12
  • The authors identify comparison with genetic algorithms, gradient-based methods, and neural networks on more general quantum-control problems as future work.The examples considered in this study are discrete, though the method is described as extendable to other reinforcement-learning algorithms using function approximation and continuous probability distributions.Source: §V · p. 12
Loading 1806.03145v1…