Source-linked AI summary
Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
Andrea Skolik, Sofiene Jerbi, Vedran Dunjko
TL;DR
The paper asks whether variational quantum algorithms can support competitive reinforcement-learning function approximation, particularly when quantum models have restricted output ranges. It develops a deep-Q-learning method for parametrized quantum circuits, evaluates architectural choices and classical DQN comparisons, and extends policy-gradient separation results to restricted Q-learning settings. The results emphasize that observables, hyperparameters, and architecture matter more than parameter count, while the theoretical extension applies only under stated environmental conditions.
Problem
Reinforcement learning is the least studied major QML setting, and it remains open whether VQAs can perform competitively with neural-network methods even on simple tasks.
Method
The paper develops parametrized quantum-circuit agents for deep Q-learning on discrete and continuous state spaces, studying encoding, readout, output-weight, and hyperparameter choices.
Results
Architectural choices and hyperparameters contribute more to reinforcement-learning success than parameter count, while suitable observables are crucial for representing optimal Q-value ranges.
Takeaways & Limitations
Quantum Q-learning should choose observables and output ranges according to the environment, and policy-gradient separation results can extend to optimal Q-value inference in restricted environment families.
Takeaways & Limitations
Frozen Lake lacks meaningful generalization because its small discrete state–action space exposes most pairs during training, while Cart Pole’s Q-value range depends on convergence speed.
Abstract
from arXiv · showhide
Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.
1 Introduction
The paper addresses the underexplored use of variational quantum algorithms for reinforcement learning by developing quantum Q-learning agents and studying the architectural choices needed across environments. It examines encoding, observables, trainable output ranges, benchmark performance, and extensions of classical–quantum separation results.
- Motivation: Reinforcement learning has received less attention in quantum machine learning than supervised and unsupervised learning, especially for variational quantum approaches.RL learns through interaction with an environment rather than from a fixed training set, aiming to maximize long-term reward.
- Contributions: The paper proposes a variational quantum algorithm that encodes states from discrete and continuous reinforcement-learning environments.The model is designed for deep Q-learning and directly handles the small state spaces studied in the benchmark environments.
- Contributions: Quantum Q-learning succeeds only when the quantum model’s readout observables satisfy requirements imposed by the environment and its Q-value range.The readout operators determine the range of values the model can produce, making them especially important for real-valued Q-functions.
- Contributions: The authors introduce trainable weights on model outputs so the attainable output range can match the range of optimal Q-values.They argue that the need for these weights can be inferred from the optimal Q-values of an environment.
- Evaluation: Ablation studies evaluate data encoding and readout strategies in Frozen Lake and Cart Pole, including data re-uploading and trainable input weights for continuous states.Both environments have sufficiently small state spaces for direct encoding without dimensionality reduction.
- Evaluation: The paper compares quantum agents with classical DQN models through extensive searches over PQC and neural-network hyperparameters and studies Q-learning extensions of policy-gradient separation results.The numerical evaluation and classical comparison are described as part of the paper’s experimental program.
2 Reinforcement learning
Reinforcement learning trains agents through interaction with environments to maximize expected cumulative reward. The section distinguishes policy-based and value-based methods, then introduces Q-learning and its deep neural-network implementation with replay and target-network stabilization.
- Reinforcement-learning setting: An environment provides states and actions, while the agent receives rewards after acting and seeks to maximize discounted cumulative return.States and actions may be discrete or continuous, and the return aggregates potentially future rewards using a discount factor.
- Policy-based and value-based learning: Policy-based methods learn an action distribution directly, whereas value-based methods learn a value function from which actions are selected.The two approaches use different figures of merit and have distinct disadvantages, so the suitable choice depends on the environment.
- Q-learning: Q-learning estimates action values Q(s, a), with higher values representing higher expected rewards and the highest-valued action defining the policy.The optimal Q-function is the maximum over policies of the action-value function.
- Q-learning: The tabular Q-learning update combines the observed reward with the discounted maximum estimated value of the next state.The learning rate controls the update, and tabular convergence is guaranteed when all state–action pairs are visited infinitely often.
- Q-learning objective: Q-learning need not recover exact optimal values for action selection, because solving an environment only requires the highest-valued action to remain highest.The paper therefore focuses its numerical studies on learning the correct ordering of Q-values.
- Deep Q-learning: Deep Q-learning replaces the intractable Q-table for large spaces with a neural-network function approximator that outputs values for all actions.The DQN uses experience replay and a periodically updated target network to stabilize training.
3 Quantum Q-learning
The paper replaces the DQN neural-network approximator with a PQC and studies how encoding, ansatz, and readout choices determine quantum Q-learning performance. In particular, the readout must produce a range compatible with the environment’s optimal Q-values.
- Quantum Q-learning: The quantum agent adapts DQN by using a PQC as its Q-function approximator instead of a neural network.The surrounding Q-learning procedure remains based on estimating action values and selecting actions from them.
- Quantum Q-learning: Discrete states are mapped to bitstrings, while continuous state components are transformed with arctan before variational X-rotation encoding.For continuous inputs, x is mapped to x′ = arctan(x) in [−π/2, π/2].
- Quantum Q-learning: Data re-uploading repeats data encoding between variational layers to increase the expressivity of PQCs beyond simple sine functions.Without re-uploading, only the variational portion is repeated; with re-uploading, the full pictured circuit is repeated in each layer.
- Quantum Q-learning: PQC Q-values are expectation values of action-specific observables, and the observable choice must match the environment’s required optimal-Q-value range.PQCs have a measurement-defined output range, unlike neural networks whose output range can change during training.
- Quantum Q-learning: A capped value range can saturate Q-values, preventing further learning when the optimal values exceed that range.In the example, the optimal value is 2 but the capped table stores 1, causing both action values to saturate at 1.
- Quantum Q-learning: Trainable action-specific output weights let the model adjust Q-value magnitudes to the environment instead of relying only on a fixed output scaling factor.The weights are multiplicative and add one parameter per action.
4 Separation between quantum and classical Q-learning in restricted environments
The paper extends quantum–classical separation results from DLP-based policy learning to optimal Q-values in restricted environments, under conditions that make Q-values inferable from optimal policies. These constructions rely on DLP hardness for classical learners and efficient quantum solvability.
- Scope of the separation: The separation concerns Q-learning in restricted environments, where optimal Q-values can be inferred from optimal policies.Q-values contain information over the whole state-action space, so this extension is not automatic.
- Environment construction: Classification tasks become RL environments by treating data points as states and assigning rewards according to the agent’s predicted labels.Correct and incorrect classifications receive rewards of 1 and −1, respectively.
- DLP-based hardness: The underlying DLP classification task is classically hard unless an efficient classical DLP algorithm exists, while quantum learners can solve it in polynomial time.The quantum advantage relies on Shor’s algorithm for efficiently solving the discrete logarithm problem.
- Quantum learning construction: The DLP-based quantum learner can use either an implicit SVM or an explicit SVM implemented with a feature-encoding unitary, variational circuit, and observable.The explicit SVM’s PQC learns a separating hyperplane from quantum-encoded feature states.
- Environment construction: SL-DLP environments use one-step classification episodes, whereas Cliffwalk-DLP environments impose ordered state transitions whose continuation depends on correct actions.An incorrect classification in Cliffwalk-DLP terminates the episode, while a correct one advances to the next state.
5 Numerical results
Experiments on Frozen Lake and Cart Pole show that quantum Q-learning depends strongly on task-matched encoding, readout, output range, and hyperparameter choices. Increasing parameters alone does not reliably improve performance, and the best PQC remains slower than the best NN on Cart Pole.
- 5.1 Frozen Lake: Optimal Q-values are directly computable for Frozen Lake but not straightforwardly for continuous-state Cart Pole, limiting direct Q-value comparisons there.For Cart Pole, Q-value magnitudes can vary substantially even among agents that solve the environment.
- 5.1 Frozen Lake: Frozen Lake Q-values were represented by scaled Z-basis expectation values because the optimal values need not exceed γ and the model outputs are bounded by 1.The four action values use one observable Zi per qubit, scaled from [−1, 1] to [0, 1].
- 5.2 Cart Pole: Combining data re-uploading with trainable input weights enabled Cart Pole solving, whereas either enhancement alone failed within 5000 episodes under both hyperparameter settings.The combined strategy solved Cart Pole in at most 3000 and 600 episodes on average for the optimal and sub-optimal settings, respectively.
- 5.2.1 Comparison of data encoding and readout strategies: Trainable output weights solved Cart Pole robustly, fixed outputs in [0, 1] failed, and a fixed maximum range of 90 was more sensitive to initialization and other hyperparameters.Late-converging agents could reach final Q-values near 176, while expanding a fixed range to [0, 180] could cause complete failure.
- 5.2.2 Comparison to the classical DQN algorithm: The best PQC used 25 layers and 302 parameters and solved Cart Pole in 500 episodes on average, while the best NN used 562 parameters and solved it in 250 episodes.The best NN therefore took roughly half as many episodes on average, despite using roughly twice as many parameters.
6 Conclusion
The paper concludes that quantum deep Q-learning can handle discrete and continuous state spaces, but success depends strongly on output observables, trainable output ranges, and task-specific hyperparameters. It also extends policy-gradient separation results to optimal Q-value inference in restricted environment families.
- Quantum deep Q-learning can encode environments with discrete and continuous state spaces.
- The observable-defined output range can impede Q-learning when it does not match the environment’s optimal Q-values.
- Trainable output weights improve performance and reduce sensitivity to hyperparameters and random initialization in Cart Pole.
- Hyperparameter settings and data-encoding or readout strategies determine success more than the number of model parameters.
- Optimal Q-values can be efficiently inferred from optimal policies in the SL-DLP family, with additional state-structure requirements for more general environments.
A Visualization of a learned Q-function
Figure 8 visualizes the learned left- and right-action Q-values across three subspaces of Cart Pole’s four-dimensional state space. The approximate Q-values appear well-behaved compared with policy-gradient PQC outputs.
- Figure 8 plots left and right action Q-values across three Cart Pole state-space subspaces.The unrepresented fourth state dimension is fixed to 0 in each plot.
- The approximate Q-values appear well-behaved, unlike the analogous policy-gradient PQC outputs.The passage attributes this likely to stronger Q-learning constraints on effective function approximations.
B Model hyperparameters
The appendix lists the hyperparameters searched for the quantum and classical models and identifies tables containing their descriptions and configurations.
- The hyperparameter search included separate settings for quantum and classical Cart Pole models.Quantum settings included batch size, model-update schedules, learning rate, weight decay, and output-weight parameters; classical settings included layer width.
- Table 1 describes the hyperparameters considered in the study.
- Tables 2 and 3 provide PQC hyperparameter settings for the reported figures.
- Table 4 provides NN hyperparameter settings for Figure 6b.