Source-linked AI summary
Reinforcement learning for optimization of variational quantum circuit architectures
Mateusz Ostaszewski, Lea M. Trenkwalder, Wojciech Masarczyk, Eleanor Scerri, Vedran Dunjko
TL;DR
VQE performance depends on selecting an ansatz that balances expressivity with circuit depth, while machine-learning support for this structure-optimization problem remains limited. The paper uses deep reinforcement learning with intrinsically motivated curriculum learning to construct VQE circuits, achieving chemical accuracy for LiH while maintaining low depth and state-of-the-art gate efficiency.
Problem
VQE ansatz optimization must balance expressive ground-state approximation against circuit depth and size, but machine-learning approaches to this problem remain incompletely investigated.
Method
The paper combines deep reinforcement learning with feedback-driven intrinsic curriculum learning to construct VQE circuit architectures, while independently optimizing rotation angles.
Results
The method achieves chemical accuracy for LiH while maintaining low circuit depth and state-of-the-art gate efficiency.
Takeaways & Limitations
The approach provides a general structure-learning method for VQE and other variational-circuit-based algorithms, including when only a lower-bound energy approximation is available.
Takeaways & Limitations
The study identifies finite measurement numbers on NISQ devices as an unresolved issue for similar reinforcement-learning methods.
Abstract
from arXiv · showhide
The study of Variational Quantum Eigensolvers (VQEs) has been in the spotlight in recent times as they may lead to real-world applications of near-term quantum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the corresponding circuit. In recent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem has not yet been fully investigated. In this work, we propose a reinforcement learning algorithm that autonomously explores the space of possible ans{ä}tze, identifying economic circuits which still yield accurate ground energy estimates. The algorithm is intrinsically motivated, and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of our algorithm on the problem of estimating the ground-state energy of lithium hydride (LiH). In this well-known benchmark problem, we achieve chemical accuracy, as well as state-of-the-art results in terms of circuit depth.
1 Introduction
VQE offers a NISQ-compatible route to estimating molecular ground-state energies, but its ansatz architecture must balance expressivity against shallow, gate-frugal circuits. This motivates machine-learning methods for autonomously optimizing VQE structures.
- NISQ applications must respect hardware size constraints while minimizing gate errors and decoherence through shallow, gate-frugal circuits.
- VQE prepares parametrized quantum states and uses classical optimization to estimate ground-state energies of Hamiltonians.
- Ansatz structure strongly influences VQE performance because it determines the accessible state manifold and the attainable ground-state approximation.
- VQE architecture optimization is combinatorial: circuits must be expressive enough for accurate energies while controlling depth and size for NISQ compatibility.
- The proposed deep-RL procedure uses feedback-driven curriculum learning to construct gate- and depth-efficient circuits, achieving chemical accuracy on LiH with state-of-the-art gate efficiency.
2 Related Work
Prior work has used machine learning, reinforcement learning, and evolutionary algorithms to automate quantum-circuit construction and optimize circuit structures for selected tasks.
- Related research applies heuristics and machine learning to subproblems including qubit routing, gate synthesis, and state preparation.
- Reinforcement learning has been used to construct parametrized circuits for data classification and to optimize square circuits by depth and gate count for QAOA MaxCut.
- Evolutionary algorithms have optimized VQE circuit structures for quantum chemistry and combinatorial optimization using mutation strategies over gate placement and layer optimization.
3 Methods
The method casts ansatz construction as sequential reinforcement learning: an agent appends gates, receives energy-based rewards, and uses intrinsically motivated curriculum learning to reach chemical accuracy without requiring exact ground-state energy.
- State and action representation: The agent represents each circuit as an ordered list of layers and iteratively appends CNOT or one-qubit rotation gates from a neural-network-friendly action space.
- Energy and reward: After each gate addition, the current circuit energy is evaluated from the parametrized state and Hamiltonian, then used to calculate the reward.
- Energy and reward: +5 is awarded when the energy falls below threshold ξ, −5 when the circuit exceeds maximum length L, and an intermediate reward otherwise.
- Energy and reward: The reward includes normalized energy improvement and caps substantially worse estimates at −1, while extreme rewards encourage discovery and reduce sparsity.
- Hybrid optimization: The hybrid approach lets reinforcement learning construct circuit structures while COBYLA or Rotosolve independently optimizes rotation angles after rotation-gate additions.
- Curriculum learning: Chemical accuracy is defined as an energy estimate approximately 0.001 Hartree from the ground-state energy, but exact minimum energy may be computationally infeasible.
- Curriculum learning: Intrinsic motivation dynamically adjusts task difficulty from agent performance, avoiding a human-defined schedule and gradually lowering the threshold toward chemical accuracy.
- Curriculum learning: The paper reports that tested manually scheduled curriculum variants did not pass the desired threshold.
4 Experiments
The experiments evaluate RL-based VQE circuit synthesis on LiH across reduced and higher-dimensional Hamiltonians, comparing optimization strategies and standard ansätze. The results show that adaptive thresholding and global optimization can produce chemically accurate circuits with reduced depth and gate count, under a proof-of-concept assumption of negligible sampling noise.
- Experimental setting: Experiments target LiH ground-state energies across intramolecular distances and Hamiltonians with different qubit counts.The study uses a 4-qubit reduced system and a higher-dimensional system, with negligible sampling noise assumed.
- Experimental setting: The RL approach is compared with Hardware Efficient and UCCSD ansätze using minimum circuit depth and gate count for chemically accurate circuits.RL trials use 10 random seeds; standard ansatz depth is fixed by the selected architecture.
- 4-qubit results: For 4-qubit LiH at 1.2 Å and 2.2 Å, RL circuits were shallower and used fewer gates than standard approaches in every trial.At 3.4 Å, local COBYLA and Rotosolve succeeded in 2/10 and 3/10 trials, respectively.
- 4-qubit results: Global optimization achieved chemical accuracy in every 4-qubit trial, regardless of whether COBYLA or Rotosolve was used.The results indicate that optimizing all angles improves the number of successful trials, with COBYLA outperforming Rotosolve for this task.
- 6-qubit results: The fixed-threshold approach failed on the 6-qubit LiH case, whereas intrinsically driven moving thresholds produced better results than standard approaches.The authors attribute the fixed-threshold failure to a sparse chemical-accuracy reward and report circuits roughly 5 times shallower than prior work, without quantitative comparison because of differing gate compilations.
- Threshold dynamics: Moving thresholds guide training toward chemical accuracy by decreasing the target error, with amortization occasionally increasing the threshold to help adaptation.Figure 4 tracks episode errors against the moving threshold and chemical-accuracy boundary.
5 Learning procedure with lower-bound approximation to the ground-state energy
The moving-threshold approach removes the need for exact ground-state energy while guiding reinforcement learning with a rough lower-bound approximation. On LiH, the agent reached chemical accuracy despite the proxy being below the exact energy.
- Exact-energy error is impractical because estimating that energy is itself a central VQE objective.
- The method replaces exact energy with the negative sum of absolute Hamiltonian Pauli coefficients as a rough, locally computable lower bound.For the LiH geometry, this bound evaluated to −10.0604Ha, compared with an exact energy of approximately −7.8448Ha.
- The agent reached the exact energy within chemical accuracy despite receiving a lower bound below the exact value.
- The moving threshold also removed the need for human input to set the threshold schedule, and similar results occurred at bond distances of 1.2 Å and 3.4 Å.
6 Discussion
The paper presents intrinsically motivated deep reinforcement learning with curriculum learning for shallow, accurate VQE circuit design. LiH experiments reached chemical accuracy while outperforming other approaches in circuit depth, though circuit structure and finite-measurement effects remain open questions.
- Discussion: VQE ansatz optimization must balance circuit depth and expressivity because the ansatz structure critically affects performance.
- Discussion: The proposed architecture autonomously adjusts task complexity through intrinsically motivated curriculum learning until chemical accuracy is achieved.
- Discussion: For LiH at several bond distances, the approach achieved chemical accuracy and consistently outperformed other VQE approaches in circuit depth.
- Discussion: The method also works with a lower-bound approximation to the ground-state energy rather than requiring the exact value.
- Discussion: The approach can be adapted to other VQE architectures and combined with other reinforcement-learning and angle-optimization algorithms.
- Discussion: The authors identify finite measurement numbers on NISQ devices as an important direction for evaluating similar reinforcement-learning methods.
- Discussion: Agent-constructed circuits remain redundant, with repeated gates that may require automated postprocessing before interpreting their physical structure.
- Discussion: The circuits are much shallower than UCCSD and unlike hardware-efficient architectures, making direct structural comparison difficult.