PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantum circuit optimization with deep reinforcement learning

Thomas Fösel, Murphy Yuezhen Niu, Florian Marquardt, Li Li

arXiv:2103.07585v1quant-ph

TL;DR

Quantum circuit optimization must account for hardware-specific details, while existing approaches focus on high-level structure. This paper formulates optimization as deep reinforcement learning, enabling agents to learn circuit-transformation strategies; across studied scenarios, RL tends to outperform simulated annealing and optimize faster after training.

  • Problem

    Existing optimization approaches do not account for hardware-specific quantum architecture details, while manual and hard-coded strategies require substantial platform-specific effort.

  • Method

    The paper uses reinforcement learning in which an agent applies circuit-modifying actions, receives rewards for improvements, and learns strategies through repeated optimization episodes.

  • Results

    RL tends to give overall slightly better results than simulated annealing, while trained agents can optimize arbitrary circuits faster than simulated annealing.

  • Takeaways & Limitations

    The approach can autonomously optimize local circuit structure for available hardware resources and supports user-chosen optimization targets and extrapolation to larger circuits.

  • Takeaways & Limitations

    The comparison of total computational expense remains unresolved because the approaches have different hyperparameters.

Abstract

from arXiv · show

A central aspect for operating future quantum computers is quantum circuit optimization, i.e., the search for efficient realizations of quantum algorithms given the device capabilities. In recent years, powerful approaches have been developed which focus on optimizing the high-level circuit structure. However, these approaches do not consider and thus cannot optimize for the hardware details of the quantum architecture, which is especially important for near-term devices. To address this point, we present an approach to quantum circuit optimization based on reinforcement learning. We demonstrate how an agent, realized by a deep convolutional neural network, can autonomously learn generic strategies to optimize arbitrary circuits on a specific architecture, where the optimization target can be chosen freely by the user. We demonstrate the feasibility of this approach by training agents on 12-qubit random circuits, where we find on average a depth reduction by 27% and a gate count reduction by 15%. We examine the extrapolation to larger circuits than used for training, and envision how this approach can be utilized for near-term quantum devices.

I. INTRODUCTION

Quantum circuit optimization is important for NISQ devices, but many existing methods operate globally and independently of hardware details. The paper introduces deep reinforcement learning to discover architecture-specific optimization strategies autonomously.

  • NISQ circuit optimization seeks logically equivalent circuits with shorter runtime and fewer gates to keep resources and errors manageable.
  • Existing approaches such as T-par, TOpt, T-Optimizer, and ZX-calculus methods are hardware-independent and operate on a global level.
  • Hardware-aware optimization must account for architecture-specific connectivity and gate sets because circuit changes have context-dependent benefits.
  • Manual and hard-coded optimization requires platform-specific effort, while genetic algorithms and simulated annealing lack computational efficiency and experience transfer.
  • Deep reinforcement learning autonomously discovers strategies for reducing circuit depth and gate count across arbitrary gate sets and connectivities, with user-selectable optimization targets.

II. TECHNIQUE

The paper formulates circuit optimization as reinforcement learning over complete, correct circuits rather than circuit synthesis from scratch. An agent selects equivalence-preserving transformations and can learn reusable strategies for circuits on a given architecture.

  • Building circuits gate by gate is problematic because positive rewards are unlikely and correctness verification can be difficult in the quantum supremacy regime.
  • The agent instead starts from a complete but inefficient circuit and chooses transformations that produce logically equivalent successor circuits.
  • The approach uses diagram-like circuit representations, with physical execution time treated as a spatial observation dimension separate from agent time.
  • The circuit is the RL state, each circuit transformation is an action, and restricting actions to equivalence transformations preserves correctness throughout optimization.
  • The intended outcome is a multi-purpose agent that optimizes a wide class of circuits for one hardware architecture without retraining for every instance.

B. Circuit transformations

The optimization process combines local circuit rules with reinforcement learning: the agent chooses context-dependent soft transformations, while automatically applicable hard transformations are pruned. Training uses episode-level rewards based on circuit improvement.

  • Local transformation rules include cancelling consecutive operations and exchanging commuting or anti-commuting operations.
  • Hard rules are always advantageous for the chosen objective, whereas soft-rule benefits depend on surrounding operations and the current circuit configuration.
  • The agent selects the next soft transformation, followed by automatic application of all permissible hard transformations through pruning.
  • The transformation-rule set is implemented by hand for each architecture, after which the approach can handle any circuit on that architecture.
  • Each training episode optimizes one circuit through T steps, with rewards indicating improvement over the previous circuit and an objective of maximizing cumulative reward.
  • The agent uses a parameterized neural network trained with PPO and advantage actor-critic quantities, including a policy and a state-value estimator.

D. Reward

The reward is designed to minimize a user-defined circuit cost through immediate improvements, with a specific error-related cost derived from circuit runtime and gates. Because the cost function can be replaced, the model-free RL algorithm remains unchanged.

  • q(s) quantifies a user-defined desirable circuit property that the RL agent minimizes.
  • The immediate reward reflects the change in q(s), providing better time resolution than a reward delivered only at the episode’s end.
  • The error-related cost estimates circuit error probability through q = −ln Psuccess = mΓT − Σ_k ln u_k.
  • Under equal operation costs and underperformance factors, q can be expressed using circuit depth and gate count.
  • q(s) may be replaced by any circuit function without changing the model-free RL learning algorithm.

E. Representation of observations and actions

The agent receives a complete circuit representation and uses a structured convolutional policy representation to map circuit transformations to actions. This design controls the action-space output size while supporting convolutional learning and extrapolation to different circuit sizes.

  • The complete circuit is provided as the observation, with neurons arranged by qubit index, moment, and gate class.
  • Continuous-parameter gates are grouped into classes while special parameter values receive separate classes for applicable transformation rules.
  • Combinatorial explosion makes reserving one output neuron for every possible action infeasible.
  • Circuit transformations are associated with locations that uniquely identify the gates they affect, rather than enumerating every possible action.
  • An action mask disables output neurons without an associated transformation, whose policy outputs are arranged by qubit index, moment, and transformation rule.
  • Convolutional layers process qubit and moment as spatial dimensions and gate classes or transformation rules as input channels.
  • Weight sharing supports efficient and robust learning, while the fully convolutional architecture permits extrapolation to different circuit sizes.

F. RL problem classification

The RL formulation is a deterministic Markov decision process with perfect information, in which circuits are states and circuit transformations are state-dependent discrete actions. Immediate rewards are possible because the optimized property can be evaluated for each circuit.

  • The problem is an MDP with perfect information because the circuit representation completely describes the environment state.
  • Circuits form the state space, while possible circuit transformations form a discrete action space whose size depends on the circuit.
  • The environment is deterministic: applying a fixed transformation to a fixed circuit always produces the same outcome.
  • Immediate rewards can be constructed because the target circuit property is evaluable for any individual circuit.

III. RESULTS

The simulations use a universal gate set chosen to provide varied transformation rules while resembling current quantum hardware. The selected CNOT-based architecture is related to hardware using Controlled-Z or fermionic simulation gates through local-gate relationships.

  • Gate set: The simulations use Z-Rotation, Phased-X, and CNOT gates, which together form a universal gate set.
  • Gate set: The chosen gate set provides a rich collection of relatively simple transformation rules and resembles current quantum hardware.
  • Hardware relation: CNOT differs from Controlled-Z only by local gates on the target qubit.
  • Hardware relation: The fermionic simulation gate is a generalization of the Controlled-Z gate.

A. Training on random circuits

The study trains reinforcement-learning agents on randomly expanded 12-qubit circuits, where transformations must handle strongly context-dependent optimization opportunities. The agent improves depth and gate count beyond pruning and simulated annealing, usually reducing both for individual circuits.

  • Dataset and training setup: Random circuits are generated on 12 nearest-neighbor-connected qubits, pruned, then expanded through 500 random transformations before training episodes.The initial circuits contain 150 randomly selected gates; pruning lowers mean depth to 37.15 and mean gate count to 115.27.
  • Dataset and training setup: The expanded circuits serve as episode starting points, exposing the agent to inefficient and efficient circuits encountered during optimization.This random expansion is intended to enhance generalization across a broad spectrum of circuit efficiencies.
  • Comparison with simulated annealing: Within 200000 steps, simulated annealing reaches mean depth 27.35 ± 0.08 and gate count 105.15 ± 0.32, while 10000 steps merely reach pruning-level performance.One complete RL learning process takes 6 to 7 days on a node with 32 CPU cores.
  • Learning results: Within roughly 50 epochs, the agent reaches mean depth ≈29 and gate count ≈100, already improving substantially over pruning.Training then continues with slower improvements toward mean depth 27.20 ± 0.07 and gate count ≈97.86 ± 0.33 around epoch 1000.
  • Optimization behavior: The agent rapidly improves circuits during the first 150–200 transformations, then sometimes crosses reward barriers by temporarily visiting lower-quality states.It was not given the ability to terminate an episode.
  • Circuit-level outcomes: For 3200 circuits, the agent reduces depth and gate count for the vast majority, sometimes by more than 50%, failing to improve only 12 cases.The 12 failures represent around 0.4% of the evaluated circuits.

B. Extrapolation to larger random circuits

A fully convolutional agent trained on 12-qubit random circuits is applied without further learning updates to larger random circuits. On 50-qubit circuits it substantially reduces depth and gate count, with simulated annealing requiring far more transformations for comparable optimization.

  • Generalization setup: The same fully convolutional agent trained on 12-qubit circuits can be directly tested on circuits of different, including larger, sizes.The experiment evaluates whether the learned strategy generalizes without additional learning updates.
  • 50-qubit circuits: For the 50-qubit test, circuits use 2500 initial gates, with mean depth 156.67 ± 0.07 and gate count 1940.2 ± 1.6 after pruning.Before pruning, the corresponding means are depth 199.25 ± 0.08 and gate count 2655.3 ± 1.2.
  • RL optimization: Within 2500 transformations, the RL agent reduces mean depth to 110.84 ± 0.07 and mean gate count to 1616.3 ± 2.0.Optimization starts directly from pruned circuits because these circuits are not used for further agent training.
  • Generalization outcome: The 50-qubit reduction ratios are comparable to those achieved on the smaller circuits used for training.This supports extrapolation of the learned optimization strategy to larger circuits.
  • Comparison with simulated annealing: Simulated annealing reaches depth 112.72 ± 0.24 and gate count 1583.0 ± 7.3 only after 93000 transformations, taking one week.The paper reports this runtime as comparable to training an RL agent and subsequently optimizing the circuit.

C. QAOA-MaxCut circuit

The paper applies reinforcement learning to optimize QAOA-MaxCut circuits, showing that circuit-specific training discovers additional improvements requiring long transformation sequences. A generic agent can sometimes improve the circuit, but reliable and highest-quality optimization depends on training on the target circuit.

  • Circuit setup: QAOA-MaxCut circuits use repeated ZZ gates and local X rotations with variable angles, with ZZ gates decomposed into CNOTs and local Z rotations.The considered gate set may differ from the algorithm’s native gates, reflecting compilation onto hardware-native operations.
  • Generic-agent optimization: The generic agent occasionally reduced the circuit from depth 75 to 68 and gate count 142 to 138.Across runs, the stochastic policy produced varying circuit quality and did not reliably optimize the circuit.
  • Generic-agent optimization: The generic agent’s observed optimizations occurred at interfaces between successive QAOA cycles and at the end of the final cycle.
  • Circuit-specific training: Circuit-specific training found two additional optimizations, each reducing depth by 1, producing depth 66 and gate count 138.These transformations required lookahead by 7 and 9 transformations, respectively, with search breadth around 350.
  • Scope and limitation: The QAOA-MaxCut example starts from an already efficient compilation, so its available optimization potential is limited.Reliable improvement and the best optimization quality were obtained only with agents trained on this circuit, emphasizing the training dataset’s importance.
  • Scope and limitation: Near-special rotation angles caused invalid transformations when a closeness criterion was too loose, although the behavior also revealed exploitable simplification opportunities.

D. Discussion

The discussion argues that RL can outperform simulated annealing in these scenarios and become faster after training, while emphasizing dataset requirements and computational trade-offs. It also identifies architecture-specific compilation, sufficient training data, training cost, and numerical parameter handling as important boundaries.

  • Dataset requirements: Real-world circuit performance is expected to improve with larger datasets of circuits compiled for specific hardware architectures.Fully training agents may require approximately 10000 to 100000 episodes.
  • Dataset requirements: The proposed future datasets would cover varied algorithms and many problem-specific circuit instances, with architecture-specific training for each agent.Data augmentation and circuit repetition are suggested when datasets are initially insufficient.
  • RL versus simulated annealing: RL tends to give slightly better overall results than simulated annealing across the discussed circuit scenarios.Relative to the pruning level, simulated annealing achieved 98.48 ± 1.06% and 109.67 ± 2.44% of RL’s gate-count reduction, but only 58.06 ± 2.15% and 95.88 ± 0.57% of its depth reduction.
  • RL versus simulated annealing: After training, an agent optimizes arbitrary circuits in its learned class considerably faster than simulated annealing.Reported comparisons include ca. 2 min versus 1–3 d and 3–5 h versus 7 d.
  • Computational cost: Fair computational-cost comparisons must include agent training time, which can reach 1 week, and hyperparameter-optimization time for both approaches.
  • Computational cost: Which method is less computationally expensive depends on the circuit class and previously solved instances; RL is cheaper only for a sufficient number of circuits.
  • Symbolic parameters: The current implementation represents gates numerically, requiring fixed gate-parameter choices and making optimization of parameterized circuit classes indirect.Extending the framework to symbolic variables would simplify optimization for circuits such as QAOA with varying angles.

IV. CONCLUSION

The paper presents reinforcement learning as a flexible approach to quantum circuit optimization, emphasizing hardware efficiency, portability, autonomy, and reliability. It addresses this approach through problem formulation, circuit representations, transformations, and automation infrastructure.

  • RL is proposed for quantum circuit optimization tailored to available hardware resources.
  • The approach aims to work across quantum architectures without large migration costs.
  • The optimization is designed to run autonomously after the user specifies the problem.
  • The authors formulate QCO so it can be efficiently addressed using reinforcement learning.
  • They develop circuit representations, output transformations, and a Python framework to automate circuit-identification processes.
Loading 2103.07585v1…