Source-linked AI summary
Variational Quantum Reinforcement Learning via Evolutionary Optimization
Samuel Yen-Chi Chen, Chih-Min Huang, Chia-Wei Hsing, Hsi-Sheng Goan, Ying-Jer Kao
TL;DR
Quantum RL on NISQ devices is constrained by limited qubit availability. This paper uses gradient-free evolutionary optimization with amplitude encoding for Cart-Pole and hybrid TN-VQC compression for larger inputs. It reports parameter savings and strong Cart-Pole performance, alongside learning on MiniGrid tasks of increasing difficulty.
Problem
Limited qubit counts on modern quantum devices constrain quantum RL applications, especially when environment inputs exceed available qubits.
Source: §I · p. 2, §VI · p. 10Method
The paper uses gradient-free evolutionary optimization for quantum RL, pairing amplitude encoding with Cart-Pole and a hybrid TN-VQC architecture with larger inputs.
Source: §I · pp. 2–3, A. Amplitude Encoding · p. 9Results
The authors report state-of-the-art Cart-Pole performance with few parameters and parameter savings from amplitude encoding; MiniGrid top-agent scores approach optimal values faster in smaller grids than in the hardest grid.
Source: §I · pp. 2–3, B. MiniGrid · p. 19Takeaways & Limitations
The hybrid TN-VQC model compresses large inputs into a small representation for quantum-circuit processing, supporting quantum RL on 147-dimensional MiniGrid observations.
Source: §VI · p. 10, C. VQC Processing · p. 14The hybrid model reaches sub-optimal results on harder problems, particularly when rewards are sparse.
Source: D. More from Classical Neuroevolution · p. 21
Abstract
from arXiv · showhide
Recent advance in classical reinforcement learning (RL) and quantum computation (QC) points to a promising direction of performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in the modern quantum devices. Here we present two frameworks of deep quantum RL tasks using a gradient-free evolution optimization: First, we apply the amplitude encoding scheme to the Cart-Pole problem; Second, we propose a hybrid framework where the quantum RL agents are equipped with hybrid tensor network-variational quantum circuit (TN-VQC) architecture to handle inputs with dimensions exceeding the number of qubits. This allows us to perform quantum RL on the MiniGrid environment with 147-dimensional inputs. We demonstrate the quantum advantage of parameter saving using the amplitude encoding. The hybrid TN-VQC architecture provides a natural way to perform efficient compression of the input dimension, enabling further quantum RL applications on noisy intermediate-scale quantum devices.
I. INTRODUCTION
The introduction motivates quantum reinforcement learning for resource-limited NISQ devices and presents evolutionary optimization with amplitude encoding and a hybrid TN-VQC approach. The section also reviews RL’s objective and introduces Cart-Pole as a task in which an agent learns actions to keep a pole upright.
I. INTRODUCTION
- NISQ devices lack error correction and cannot run large-qubit, deep-circuit algorithms, making near-term quantum algorithm development challenging.Variational quantum algorithms allocate parts of a task to a quantum computer and use classical processing to adjust circuit parameters from its outputs.Source: §I · p. 2
- The proposed gradient-free evolutionary method trains quantum RL agents, targeting state-of-the-art Cart-Pole performance with few parameters and end-to-end optimization of hybrid TN-VQC models.The paper presents amplitude encoding as a route to parameter savings and TN-VQC as a way to compress larger inputs for quantum processing.Source: §I · pp. 2–3
II. REINFORCEMENT LEARNING
- RL agents map observed states to action probabilities and seek to maximize expected return across an episode.The return is the total discounted reward from a time step onward.Source: §II · pp. 3–4
A. Cart-Pole
- In Cart-Pole, the agent pushes a cart left or right to keep an initially upright pole close to its starting state.An episode ends if the pole tilts more than 15 degrees or the cart moves over 2.4 units from the center.Source: A. Cart-Pole · p. 4
B. MiniGrid
MiniGrid tests a hybrid RL agent on 147-dimensional observations, with six actions and empty grids of different sizes. Its sparse reward is given for reaching the goal, making progress dependent on finding that goal within the step limit.
B. MiniGrid
- MiniGrid provides a 147-dimensional observation and six possible actions, and the agent is expected to find the shortest path from the start to the goal.The action set includes picking up or dropping an object and toggling doors or other objects.Source: B. MiniGrid · p. 5
- The experiments use MiniGrid-Empty environments with grid sizes 5x5, 6x6, and 8x8.The figure passage identifies the three environments; the text specifies the grid sizes and names.Source: B. MiniGrid · p. 6, Figure 2 · p. 6
- The agent receives a reward for reaching the goal, while the sparse scheme gives no reward along the way; the episode limit is 4 × n × n steps.The study considers n = 5, 6, and 8.Source: B. MiniGrid · p. 6
IV. VARIATIONAL QUANTUM CIRCUITS
Variational quantum circuits encode classical inputs into quantum states, apply trainable circuit parameters, and return measured outputs. The paper uses amplitude encoding for Cart-Pole, loading a four-dimensional observation into a two-qubit circuit.
IV. VARIATIONAL QUANTUM CIRCUITS
- VQCs encode classical data into quantum states, optimize a parameterized circuit block, and retrieve classical outputs through measurements.The data-encoding block is not optimized; the variational block contains trainable parameters.Source: §IV · pp. 6–7
A. Quantum Encoding
- Amplitude encoding maps a normalized m-dimensional vector to a quantum state using log2(m) qubits, while MiniGrid uses variational encoding instead.The paper selects the encoding scheme according to the problem; amplitude encoding is used for Cart-Pole and variational encoding for MiniGrid.Source: A. Quantum Encoding · p. 8
V. QUANTUM ARCHITECTURE FOR THE CART-POLE PROBLEM
- Amplitude encoding loads Cart-Pole’s four-dimensional observation into a two-qubit circuit, with four repeated variational blocks and two measured action outputs.The measurement expectations receive a two-dimensional classical bias.Source: §V · pp. 8–9, Figure 4 · p. 9
A. Amplitude Encoding
Amplitude encoding represents a normalized input vector in quantum-state amplitudes, requiring log2(m) qubits for an m-dimensional vector. The paper also describes tensor networks and MPS as a route to compact representations.
A. Amplitude Encoding
- A normalized m-dimensional vector can be amplitude-encoded in log2(m) qubits, and the paper applies this scheme to Cart-Pole.Amplitude encoding places the vector entries in the amplitudes of a quantum state.Source: A. Amplitude Encoding · p. 9
A. Tensor Network
- MPS decomposes a large tensor into matrices, with bond dimension m tuning trainable parameters and expressive power.A sufficiently large bond dimension can represent any tensor.Source: A. Tensor Network · pp. 10–11
- Tensor networks have been used in classification, generative modeling, and sequence modeling, and can be encoded into quantum circuits with single- and two-qubit gates.The cited applications include [97] [98], and the circuit-encoding result is cited as.Source: A. Tensor Network · pp. 11–12
B. MPS Operation
The MPS feature extractor maps each observation component through a feature map and contracts the resulting product state with trainable tensors. Its eight-dimensional output is encoded into a VQC, whose measured values are processed into action probabilities.
B. MPS Operation
- The input feature map uses local dimension d = 2, mapping each observation component to a two-dimensional vector.The mapped components form a tensor product state before contraction with the trainable MPS.Source: B. MPS Operation · p. 12
- For MiniGrid’s 147-dimensional observation, the MPS contracts 147 feature-mapped inputs into an eight-dimensional compressed representation.The architecture uses 147 input nodes and 148 MPS nodes.Source: B. MPS Operation · pp. 12–13
C. VQC Processing
- The VQC encodes the eight-dimensional MPS output, applies trainable entangling and rotation gates, then measures Pauli-Z expectations and classically converts them into action probabilities.The circuit measures the first k qubits, where k is the number of actions; MiniGrid uses one variational block.Source: C. VQC Processing · pp. 13–14
D. Action Selection
The action-selection procedure chooses the action associated with the largest processed circuit output. Quantum circuit parameters are optimized through an evolutionary population process that evaluates agents, selects parents, and produces successive generations.
VII. QUANTUM CIRCUIT EVOLUTION
- Quantum circuit evolution initializes agents with random parameters, evaluates their fitness, selects the top-scoring agents as parents, and evolves a new generation.Fitness is the average episode score across R1 trials; episode score is the sum of rewards.Source: §VII · p. 15, B. Running the Agents · p. 15
A. Initialization
- Initial agent parameters are sampled from N(0, I) and scaled by 0.01 to keep them near zero for a more stable training process.The population is initialized with N agents.Source: A. Initialization · p. 15
B. Running the Agents
- Each agent’s fitness is its average score over repeated game trials, and the top T agents are selected as parents for the next generation.An episode’s score is the sum of its rewards.Source: B. Running the Agents · p. 15
C. Mutation and the Next Generation
The evolutionary procedure mutates most children and preserves an elite selected by repeated average scores. In experiments, amplitude-encoded VQC reaches the Cart-Pole maximum, while hybrid TN-VQC handles 147-dimensional MiniGrid inputs with performance varying by environment size.
C. Mutation and the Next Generation
- Each generation creates N−1 children by mutating a randomly selected parent’s parameters, then preserves the highest-scoring parent as the unmutated elite.Parents are evaluated over R2 repetitions and ranked by average score to make selection more robust against noise.Source: C. Mutation and the Next Generation · p. 16
VIII. EXPERIMENTS AND RESULTS
- The experiments use the same evolutionary optimization procedure for Cart-Pole and MiniGrid.The Cart-Pole run uses 1,700 generations, population size 500, and mutation power 0.02; the MiniGrid runs use 500 generations, population size 500, and mutation power 0.02.Source: §VIII · p. 16, A. Cart-Pole · p. 16, B. MiniGrid · pp. 17, 19
A. Cart-Pole
- After about 1,300 generations, Cart-Pole’s top five agents reach the maximum score of 500 using 26 parameters.The paper reports this parameter count as at least one order of magnitude below typical classical neural networks.Source: A. Cart-Pole · p. 16, Figure 8 · p. 17
B. MiniGrid
- The hybrid TN-VQC compresses each 147-dimensional MiniGrid observation to an 8-dimensional vector for VQC processing.The MPS feature extractor and VQC are trained together; the authors present this compression as a route to processing larger inputs.Source: B. MiniGrid · p. 17
- MiniGrid-Empty-5x5-v0 reaches near-optimal scores in under 40 generations, while MiniGrid-Empty-8x8-v0 takes about 350 generations to stay above 0.9 and finishes worst.MiniGrid-Empty-6x6-v0 rises above 0.9 in 40 generations and reaches near-optimal value after around 120 generations.Source: Figure 9 · p. 18, B. MiniGrid · p. 19
IX. DISCUSSION
The discussion positions the work against gradient-based quantum RL and prior evolutionary quantum-circuit studies, highlighting evolutionary optimization and hybrid MPS–VQC training. The authors report limits on harder tasks and note that their results come from noise-free simulation rather than real quantum hardware.
A. Relevant Studies
- The Cart-Pole model reaches the optimal solution with 2 qubits, compared with 4-qubit VQCs in two prior studies; the authors leave optimizer effects for future investigation.They also report extending tests beyond discrete-state settings, while noting a continuing gap with classical RL in processing high-dimensional inputs.Source: A. Relevant Studies · pp. 19–20
B. Complex Benchmarks
- The hybrid TN-VQC extends quantum RL to more complex benchmarks, but a significant gap from classical RL remains for processing high-dimensional inputs.The conclusion describes a 147-dimensional MiniGrid input compressed for encoding on currently available quantum devices.Source: B. Complex Benchmarks · p. 20, §X · p. 22
C. Evolutionary Quantum Circuits
- The authors present their work as the first direct use of evolutionary optimization for quantum RL and as an end-to-end training demonstration for an MPS–VQC model.Earlier cited evolutionary quantum-circuit work addressed VQE and classification tasks.Source: C. Evolutionary Quantum Circuits · p. 20
D. More from Classical Neuroevolution
- The study evolves quantum-circuit parameters rather than model architecture; the authors identify topology evolution as a possible direction for quantum ML.They point to NEAT as an example of classical neural-network topology and weight evolution.Source: D. More from Classical Neuroevolution · pp. 20–21
- The hybrid model reaches only sub-optimal results on harder problems, particularly when sparse rewards make maximum scores challenging.Novelty search is suggested as a future direction, not reported as a tested solution.Source: D. More from Classical Neuroevolution · p. 21
E. Training on a Real Quantum Computer
- The reported experiments use noise-free simulation; limited cloud quantum-computing resources made full training of the customized TN-VQC impractical on real hardware.The authors expect more reliable and accessible commercial devices may resolve this constraint.Source: E. Training on a Real Quantum Computer · p. 21
X. CONCLUSION
- The conclusion presents amplitude encoding and tensor-network compression as two ways to reduce VQC qubit requirements, with the hybrid model trained end-to-end.For Cart-Pole, the paper reports poly(log n) parameter scaling with input dimension n; for MiniGrid, it demonstrates compression of 147-dimensional inputs.Source: §X · p. 22
Appendix A: Amplitude Encoding
Amplitude encoding maps a normalized real vector to the amplitudes of an n-qubit state. The paper describes constructing this encoding by reversing a cascade of multi-controlled rotations, using R_y rotations for real amplitudes.
Appendix A: Amplitude Encoding
- Amplitude encoding maps a normalized vector of 2^n real values to the amplitudes of an n-qubit state.The state contains one amplitude for each computational-basis state.Source: Appendix A · p. 23
- For real amplitudes, the routine uses a cascade of multi-controlled R_y rotations, beginning with an operation on q_n.The general routine uses rotations on the last qubit controlled by the other qubits.Source: Appendix A · p. 23
- To encode the vector, invert each operation in the cascade and apply them in reverse order to |00 ··· 0⟩.The paper illustrates the two-qubit version used in its Cart-Pole experiment in Figure 11.Source: Appendix A · p. 24, Figure 11 · p. 24
Appendix B: Quantum Circuit Evolution Algorithm
The algorithm evolves a population of quantum-circuit agents using repeated score evaluation, truncation selection, and parameter mutation. It evaluates selected agents again and retains top performers as parents for the next generation.
Appendix B: Quantum Circuit Evolution Algorithm
- Each generation evaluates N agents over R1 plays and records their average scores.The agents use quantum-circuit action-value functions initialized with random parameters.Source: Appendix B · p. 25
- The algorithm selects the top T agents by average score, then randomly selects from them and mutates parameters by θ ← θ + σϵ.T is the truncation-selection count, and the population is initialized with N agents.Source: Appendix B · p. 25
- Each selected agent is played R2 times, and the algorithm records its average score for comparison.The tested top agents are evaluated before the best are retained.Source: Appendix B · p. 25
- One fragment says to keep the best agent according to the average scores Eavg; another identifies j as the N-th children, and a third defines M as the number of generation.Source: Appendix B · p. 25