PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantum Architecture Search via Deep Reinforcement Learning

En-Jui Kuo, Yao-Lung L. Fang, Samuel Yen-Chi Chen

arXiv:2104.07715v1quant-phcs.AIcs.LGcs.NE

TL;DR

Designing quantum circuits for particular states requires expertise, motivating automated architecture search. The paper applies DRL to generate gate sequences from state information, and reports that current DRL algorithms can discover near-optimal sequences with limited physics knowledge.

  • Problem

    Designing a quantum circuit for a specific task is non-trivial because it demands domain knowledge and sometimes extraordinary insights.

    Source: §I · p. 2

  • Method

    The paper presents a DRL-powered quantum architecture search framework and demonstrates step-by-step circuit construction without physics knowledge encoded in the agent.

    Source: §I · pp. 2–3

  • Results

    The authors report that current DRL algorithms can discover near-optimal quantum gate sequences with very limited physics knowledge.

    Source: §VI · p. 19

  • Takeaways & Limitations

    The framework generates quantum gate sequences from the density matrix only, supporting circuit search with limited physics knowledge.

    Source: §VI · p. 19

Abstract

from arXiv · show

Recent advances in quantum computing have drawn considerable attention to building realistic application for and using quantum computers. However, designing a suitable quantum circuit architecture requires expert knowledge. For example, it is non-trivial to design a quantum gate sequence for generating a particular quantum state with as fewer gates as possible. We propose a quantum architecture search framework with the power of deep reinforcement learning (DRL) to address this challenge. In the proposed framework, the DRL agent can only access the Pauli-$X$, $Y$, $Z$ expectation values and a predefined set of quantum operations for learning the target quantum state, and is optimized by the advantage actor-critic (A2C) and proximal policy optimization (PPO) algorithms. We demonstrate a successful generation of quantum gate sequences for multi-qubit GHZ states without encoding any knowledge of quantum physics in the agent. The design of our framework is rather general and can be employed with other DRL architectures or optimization methods to study gate synthesis and compilation for many quantum states.

I. INTRODUCTION

Designing a quantum circuit for a specific task requires domain knowledge. This section introduces a DRL framework that searches for circuits to generate desired quantum states, including step-by-step gate construction without physics knowledge encoded in the agent.

I. INTRODUCTION

  • The framework trains an RL agent to search for quantum circuit architectures that generate a desired state, addressing the expertise required for circuit design.The paper identifies quantum circuit design as non-trivial because it demands domain knowledge and sometimes extraordinary insights.Source: §I · p. 2
  • At each step, the agent selects a candidate gate or operation, the circuit is evaluated by fidelity, and a fidelity-based reward is returned for iterative training.The agent interacts with a quantum computer or simulator; Pauli-X, Y, and Z expectation values are also fed back to it.Source: §I · pp. 2–3
  • The authors demonstrate step-by-step circuit construction via DRL without encoding physics knowledge in the agent.They present the framework as a general approach for studying quantum architecture search.Source: §I · p. 3

II. REINFORCEMENT LEARNING

The section introduces reinforcement learning as sequential decision-making through interaction with an environment, then describes policy-gradient methods and the A2C baseline approach.

II. REINFORCEMENT LEARNING

  • An RL agent observes a state, selects an action under a policy, and receives a next state and reward until termination or a stopping condition.An episode begins from a selected initial state and continues until a terminal state or predefined stopping criterion.Source: §II · p. 4

A. Policy Gradient

  • Policy-gradient methods optimize a parameterized policy by gradient ascent on expected return, while a state-dependent baseline can reduce gradient-estimate variance.The standard REINFORCE update uses the return; subtracting a baseline preserves an unbiased estimate while reducing variance.Source: A. Policy Gradient · p. 5

B. Advantage Actor-Critic (A2C)

  • A2C uses the policy as actor and a learned value function as critic, with the return-minus-value baseline forming an advantage estimate for an action.The advantage compares an action's value with the average value of its state.Source: B. Advantage Actor-Critic (A2C) · p. 5

C. Proximal Policy Optimization (PPO)

PPO limits policy updates using a clipped objective, while the problem setup frames circuit synthesis as sequential gate selection guided by state observations and fidelity-based feedback. Training uses a simulator because running tens of thousands of episodes on a cloud quantum device is not yet practical.

C. Proximal Policy Optimization (PPO)

  • PPO addresses instability from policy updates that are too small, which slow training, or too large, which raise variance, by limiting the policy-probability ratio to 0.8 to 1.2.Source: C. Proximal Policy Optimization (PPO) · p. 6

III. PROBLEM SETUP

  • The synthesis task is to transform an initial state into a target state within a specified error tolerance using a circuit of one- and two-qubit operations.The agent chooses operations from the available action set.Source: §III · p. 6
  • The experiments use a quantum simulator because training tens of thousands of episodes on a cloud-based quantum device is not yet practical.The simulator is used with and without noise.Source: §III · p. 6
  • The environment returns Pauli measurements as observations, and fidelity determines whether the agent receives a large positive reward or a small negative reward.For an n-qubit system, the observation dimension is 3n; the episode ends at the fidelity threshold or maximum allowed steps.Source: §III · p. 7

A. Mathematical formulation of the problem

The problem is to find a circuit from the n-qubit zero state to a target state within a fidelity tolerance, using gates from a predefined set. Bell and three-qubit GHZ states serve as test targets for the architecture-search approach.

A. Mathematical formulation of the problem

  • The objective is to find a circuit mapping |0⟩^⊗n to |ψ⟩ with fidelity at least 1 − ϵ, using gates from G.The paper uses fidelity as the distance measure between the target and generated states.Source: A. Mathematical formulation of the… · p. 7

B. Multi-qubit entangled states as target

IV. EXPERIMENTS AND RESULTS

The experiments specify the optimizer, simulator noise models, and density-matrix representation used in evaluating quantum states.

1. Optimizer

  • Adam is used to train both the A2C and PPO agents, with adaptive learning rates for each parameter.The optimizer adjusts model parameters using gradient information.Source: §1 · p. 9

2. Quantum Noise in Quantum Simulator

  • The simulator tests gate errors and measurement errors, modeling them as depolarizing noise and random measurement flips, respectively.Gate noise replaces a qubit state with a random state at probability p_gate; measurement noise flips 0 and 1 at probability p_meas.Source: §2 · pp. 9–10

3. Density Matrix of Quantum States

  • The density matrix represents the state as a probability-weighted mixture of pure states and is used to calculate fidelity.The probabilities over the pure states sum to one.Source: §3 · p. 10

4. Quantum State Tomography

In the Bell-state experiment, both A2C and PPO learn to synthesize the target, with PPO reaching optimal results faster and more stably under the same network architecture. The section also describes finite-shot tomography, the agent environment, and the shared network structure.

4. Quantum State Tomography

  • Finite-shot measurements estimate Pauli-basis expectation values with limited accuracy, and full density-matrix determination requires exponentially many measurements as qubit number grows.The passage states that 4^N − 1 linearly independent projective measurements determine the density matrix.Source: §4 · pp. 10–11

5. Customized OpenAI Gym Environment

  • The customized environment provides Pauli-X, Y, and Z expectation values for each qubit and asks the agent to select an available gate.For n qubits, the observation contains 3 × n values.Source: §5 · p. 11
  • The environment gives a −0.01 reward per step before success and a terminal reward of F − 0.01 upon reaching the goal.The per-step penalty is intended to encourage the shortest path.Source: §5 · p. 11

6. Hyperparameters

  • The A2C and PPO agents use the same neural-network structure, and the stated PPO settings include learning rate 0.002, discount factor 0.99, and clip parameter 0.2.The table caption identifies the network as the one used for both algorithms.Source: §6 · p. 11, Table I · p. 12

1. 2-qubit Bell state

  • Both A2C and PPO successfully train agents to synthesize a Bell state, while PPO reaches optimal results faster and more stably with the same network architecture.The generated PPO circuit is shown in Figure 4.Source: Figure 4 · p. 12, §1 · p. 12

2. 3-qubit GHZ state

In noise-free three-qubit GHZ synthesis, both A2C and PPO succeed, with PPO converging faster and more stably. Noisy two-qubit results favor a 0.95 fidelity threshold over 0.99 for score stability, while higher error rates yield slightly lower final fidelity.

2. 3-qubit GHZ state

  • With the same network architecture, PPO converges faster and more stably than A2C in the noise-free two-qubit setting.The reported comparison is based on training scores and their standard deviation.Source: Figure 5 · p. 13, §2 · p. 13
  • PPO reaches optimal three-qubit GHZ results faster and more stably than A2C, with a larger advantage than in the two-qubit case.Both methods successfully train an agent to synthesize the GHZ state.Source: §2 · p. 12, Figure 7 · p. 14

C. Results: Noisy Environments

  • PPO performs significantly better than A2C in convergence speed and stability for noise-free three-qubit GHZ synthesis, consistent with the two-qubit case.The three-qubit training ran for 10,000 episodes.Source: §2 · p. 13, C. Results: Noisy Environments · p. 14
  • The threshold marks whether the agent reaches a minimum goal; training still maximizes overall return, and final fidelity is not capped by that threshold.The threshold comparison uses PPO in the noisy setting rather than vanilla A2C.Source: C. Results: Noisy Environments · pp. 13, 15
  • At fidelity threshold 0.95, single-qubit error rates 0.001 and 0.005 both converge quickly, but the higher error rate yields a slightly lower final converged fidelity.Source: C. Results: Noisy Environments · p. 15

V. DISCUSSION

The method generates circuits from scratch and incrementally rather than optimizing an existing circuit or sampling batches. Its general-size validity does not remove computational difficulty, and topology-aware search remains future work.

A. Relevant Works

  • Unlike prior work optimizing existing circuits [87] [89], this approach generates the quantum circuit from scratch.It also differs from [88] by not sampling directly from a circuit distribution and from [86] by growing the circuit incrementally rather than using batch generation or random search.Source: A. Relevant Works · p. 17

B. More Complex Problems

  • The approach can in principle approximate arbitrary quantum states with finitely many gates at arbitrary qubit sizes, but the complexity scales exponentially in qubit count.The finite-gate claim assumes a universal set of one- and two-qubit gates and an error tolerance.Source: B. More Complex Problems · p. 17
  • How training difficulty relates to target density-matrix complexity remains future work.The paper notes multiple possible definitions of density-matrix complexity [92, 93].Source: B. More Complex Problems · p. 17

C. Noisy Environments

  • The study investigates gate search under simple noisy configurations; the toolkit and framework may extend to more complex noise models.The results are presented as a testbed for future studies of different noise or error schemes.Source: C. Noisy Environments · p. 17
  • The framework may be combined with methods that learn circuit architecture under noise effects.The paper also identifies simultaneous optimization of architecture and parameters as a possible direction.Source: C. Noisy Environments · pp. 17–18, A. Relevant Works · p. 17
  • Real hardware topologies influence circuit design, and topology-aware quantum architecture search is left for future work.Source: C. Noisy Environments · p. 18

D. Real Quantum Computers

The conclusion reports near-optimal gate-sequence discovery with little physics knowledge encoded in the agent. The discussion covers resource constraints on real quantum computers, broader testing possibilities, environment extensions, and circuit optimization.

D. Real Quantum Computers

  • Real quantum computer experiments require thousands of training episodes, exceeding the resources currently available to the authors.They leave investigation with more accessible quantum computing resources as future work.Source: D. Real Quantum Computers · p. 18

E. Other Quantum States

  • The demonstrations use two-qubit Bell and three-qubit GHZ states, while the testing environment and RL agents are described as general enough to explore other states and noise configurations.The environment also supports testing different RL algorithms through the standard OpenAI Gym interface.Source: E. Other Quantum States · p. 18

F. Extension of the Environment

  • Predefining operations is not a framework limitation because the testing environment can be extended to fit devices of interest.The passage proposes customized environments using operations available on specific quantum hardware.Source: F. Extension of the Environment · p. 18

G. Circuit Optimization

  • This work builds circuits from scratch, whereas [95] optimizes the depth of a given circuit; the two tasks can be combined into a target-state-to-efficient-circuit pipeline.The passage notes that solving either task efficiently does not imply solving the other efficiently.Source: G. Circuit Optimization · p. 19

VI. CONCLUSION

  • Current DRL algorithms can discover near-optimal quantum gate sequences with very limited physics knowledge encoded in the agents.The framework generates gate sequences automatically from the density matrix.Source: §VI · p. 19
  • The authors present a customized OpenAI Gym environment as a tool for exploring related quantum applications.The framework is presented as useful for gate synthesis and compilation for many quantum states.Source: §VI · p. 19

Appendix A: RL Algorithms

The appendix outlines A2C and PPO procedures for quantum architecture search. Both interact with the testing environment and update agent parameters using trajectory data and calculated losses.

Advantage Actor-Critic (A2C)

  • A2C calculates value targets, state values, and a policy loss from trajectory states before updating policy and value parameters by gradient descent.The listed policy loss weights log action probability by the advantage.Source: Appendix A · pp. 20–21

Environment interaction

  • During interaction, the agent executes actions in the emulator and records observed rewards and subsequent states in a trajectory buffer.The loop continues until a terminal state or the maximum-step condition is reached.Source: Appendix A · pp. 20–21

PPO

  • PPO records state-action transitions, compares current and old policy probabilities, and updates parameters using a clipped surrogate objective with value and entropy terms.The algorithm also calculates discounted rewards, state values, and entropy from the trajectory buffer.Source: Appendix A · p. 21

Algorithm settings

  • The procedures define episode and step limits, with PPO additionally specifying update timesteps, update epochs, and an epsilon clipping value.Both procedures reset the environment and initialize a state; PPO also resets its trajectory buffer.Source: Appendix A · pp. 20–21

Appendix B: Code samples

The code samples initialize target states and quantum simulation environments, including noise-free and noisy two-qubit cases. The noisy example configures measurement and gate errors using Qiskit noise-model tools.

Environment setup

  • The examples set a target state and use Qiskit functions to define the noise model and quantum simulation backend before initializing the Gym environment.The noise-free example is specified for a two-qubit system.Source: Appendix B · pp. 22–24

Noisy environment

  • The noisy two-qubit example imports Qiskit Aer noise tools and defines a measurement error using a Pauli error parameterized by p_meas.The target is used to initialize the Gym environment.Source: Appendix B · p. 22

Noise configuration

  • The example applies measurement error to measurements, single-qubit error to X gates, and two-qubit error to CX gates.The backend configuration includes a noise model, coupling map, and basis gates.Source: Appendix B · p. 22

Related papers

Loading 2104.07715v1…