PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling quantum neural network gradient with reinforcement learning

Nhan Trong Luu, Duong Trung Luu, Nam Ngoc Pham, Thang Cong Truong

arXiv:2609.31066v1quant-phcs.ETcs.NE

TL;DR

QNN training faces barren-plateau gradient decay and costly circuit differentiation, while reinforcement learning for supervised QNN optimization remains underexplored. RLQ-Grad uses a classical policy to propose parameter updates from the training state. It reports near-flat gradient variance, lower computational costs, and accuracy gains over gradient-based baselines across the reported benchmarks.

  • Problem

    Supervised QNN optimization remains difficult because of barren-plateau gradient decay and high circuit-evaluation costs, while reinforcement-learning approaches to supervised QNNs remain underexplored.

    Source: §1 · pp. 1–2

  • Method

    RLQ-Grad uses a reinforcement-learning policy to propose QNN parameter updates conditioned on the current training state, including parameters, loss, accuracy, and the previous update.

    Source: §2.3 · p. 4, §2.3.1 · p. 4

  • Results

    RLQ-Grad maintains near-constant gradient variance, gains 2% to 10% over gradient-based baselines across four supervised benchmarks, and matches dedicated mitigation methods on CIFAR-10 up to 20 qubits.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The classical policy avoids differentiating through the quantum unitary, so its training follows classical concentration behavior independent of the number of qubits.

    Source: §5 · p. 10

    Experiments use shot-noise-free statevector simulators and do not report physical hardware, realistic noise channels, or non-classification tasks.

    Source: §4 · p. 10

Abstract

from arXiv · show

Training quantum neural networks (QNNs) on near-term hardware remains hampered by two compounding difficulties: the exponential vanishing of gradient variance known as the barren plateau, and the $\mathcal{O}(L \cdot 2^n)$ time and memory cost of differentiating through an $n$-qubit, $L$-layer circuit. We propose RLQ-Grad, a reinforcement-learning-based optimizer in which a classical policy $\pi_\phi$ (a spectrally-normalized PPO agent) learns to propose parameter updates directly, conditioned on the QNN's current parameters, loss, accuracy, and previous update. Because the surrogate gradient is emitted by a classical network rather than obtained by differentiating through the unitary $U(\theta)$, its variance is not constrained by the barren plateau concentration bound, and its cost scales with the number of trainable parameters rather than the Hilbert-space dimension. We prove these properties formally and verify them on a hardware-efficient ansatz across four supervised benchmarks with up to $n=20$ qubits. RLQ-Grad preserves a near-flat gradient-variance curve where backpropagation, parameter-shift, and adjoint differentiation decay by 1 to 2 orders of magnitude. Accounting for the full training pipeline (PPO rollouts, actor-critic updates, and optimizer states), RLQ-Grad needs under 2 MB of memory and runs $2490\times$, $7876\times$, and $673\times$ faster per iteration than these three methods at $n=20$. It improves top-1 accuracy by up to $+10\%$ over gradient-based baselines on circuits of up to 12 qubits, and matches dedicated barren plateau mitigation methods on CIFAR-10 at 14 to 20 qubits, where evolutionary and gradient-free optimizers collapse to chance.

1 Introduction

QNN training combines difficult optimization landscapes with high circuit-evaluation costs, while supervised-learning applications of reinforcement learning remain underexplored. RLQ-Grad uses an RL agent to propose QNN updates, with reported accuracy and computational advantages over conventional gradient-based methods.

1 Introduction

  • Prior RL research spans parameter optimization for fixed circuits, quantum architecture search, and methods aimed at improving scalability and robustness.Examples include QAOA and VQE parameter learning, circuit-design agents, and approaches combining RL with tensor-network approximations or training-stabilization techniques [1] [5] [12].Source: §1 · pp. 1–2
  • RLQ-Grad learns QNN parameter updates rather than conducting exhaustive parameter search, addressing a gap in supervised QNN learning with reinforcement learning.Earlier RL parameter-search approaches are described as ill-suited to larger problems [12].Source: §1 · p. 2
  • RLQ-Grad improves top-1 accuracy by up to +10% over the strongest gradient-based baseline from 2 to 12 qubits; on CIFAR-10 extended to 20 qubits (PQNN = 600), it matches dedicated barren plateau mitigation methods.It clearly outperforms CMA-ES, sep-CMA-ES, and SPSA, which collapse to chance.Source: §1 · p. 2
  • At 20 qubits, RLQ-Grad uses 1.95 MB versus 6174 MB for backpropagation and runs approximately 2490×, 7876×, and 673× faster per iteration than backpropagation, parameter-shift, and adjoint differentiation, respectively.The memory comparison accounts for the full pipeline, including the actor, critic, gradient buffers, and Adam states.Source: §1 · p. 2

2 Proposed method

The method formulates supervised QNN training as learning a gradient generator through reinforcement learning. Its policy proposes updates from the training state, while the QNN uses a hardware-efficient circuit and a reward based on loss and accuracy.

2.1 Problem Formulation

  • QNN training aims to minimize empirical risk, and RLQ-Grad learns a gradient generator instead of explicitly computing the QNN gradient.The passage contrasts this approach with backpropagation, adjoint differentiation [18], and the parameter-shift rule.Source: §2.1 · p. 3

2.2 QNN selection

  • The selected HEA-like circuit has n qubits and L layers, with parameterized rotations and entangling operations in each variational layer.The circuit prepares the state using its variational unitary.Source: §2.2 · p. 3
  • The model measures a Pauli Z observable and passes the measurements through a fully connected output layer to produce class predictions.The QNN framework uses PennyLane and TorchQuantum.Source: §2.2 · p. 3

2.3 RLQ-Grad algorithm

  • An RL policy interacts with QNN training to propose parameter updates; the experiments mainly use PPO with spectral normalization.The proposed update is assigned to the QNN parameter gradient, while the remaining gradient-graph components are updated with backpropagation.Source: §2.3 · p. 4, §2.2 · p. 4

2.3.1 State representation

  • The state contains current QNN parameters, mean training loss and accuracy, and the agent’s previous proposed gradient.The agent’s output is interpreted as a gradient estimate for the trainable parameters.Source: §2.3.1 · p. 4

2.3.2 Reward function

  • The reward combines training accuracy with the reciprocal of training loss plus a small constant, encouraging updates that improve accuracy and reduce loss.The constant prevents division by zero.Source: §2.3.2 · p. 5

3 Results

RLQ-Grad replaces quantum-circuit gradient computation with learned classical updates, bypassing gradient-variance decay while reducing resource costs. Across the reported benchmarks it achieves strong accuracy, remains stable at larger scales, and matches dedicated mitigation methods on large CIFAR-10 circuits.

3.1.1 Resource advantages

  • At 20 qubits, RLQ-Grad uses 1.95 MB and is 2490x, 7876x, and 673x faster than backpropagation, parameter-shift, and adjoint differentiation.The measurements include the full RLQ-Grad pipeline, including PPO rollouts, actor-critic updates, gradient buffers, and Adam states.Source: Figure 3 · p. 6, §3.1.1 · p. 7

3.1 Surrogate gradient advantages of RLQ-Grad

  • RLQ-Grad's surrogate updates bypass circuit-gradient variance decay; measured HEA variance remains robust against decay across the reported comparisons.The agent's own parameter gradient uses classical backpropagation and is not subject to the same variance-decay phenomenon.Source: §3.1 · pp. 5–6, Figure 2 · p. 5

3.1.1 Resource advantages

  • RLQ-Grad's proposal cost and full-step memory scale linearly with P_QNN and are independent of 2^n, unlike circuit-gradient costs.The reported complexity accounts for actor and critic parameters, gradient buffers, and Adam moment estimates.Source: §3.1.1 · pp. 6–7, Table 1 · p. 6

3.2 Benchmarking against well-known quantum optimization algorithms

  • RLQ-Grad leads across datasets and configurations; on BC, it reaches 96.58% with a 2%-10% margin over the second-best method.It also achieves the highest reported accuracy on F-MNIST and CIFAR-10, with gains on CIFAR-10 of about 2%-4%.Source: §3.2 · pp. 7–8
  • Across four datasets, accuracy generally improves with circuit size, while conventional gradient methods show diminishing returns or instability at higher configurations.The 2-12-qubit results include up to a 3% RLQ-Grad advantage over competitive backpropagation and adjoint differentiation on MNIST at 12q6d.Source: §3.2 · pp. 7–8

3.3.1 Action-space scalability and non-stationarity

  • RLQ-Grad remains stable and improves monotonically through P_QNN=600, with standard deviations of 2.46-3.84 comparable to backpropagation and mitigation baselines.On-policy rollouts, a clipped objective, and spectral normalization of the critic address environment non-stationarity and bound policy or TD-error changes.Source: §3.3.1 · p. 8

3.3 Scaling to 20 qubits and comparison with stronger baselines

  • At 14-20 qubits, RLQ-Grad matches layerwise learning and Gaussian initialization, while CMA-ES, sep-CMA-ES, and SPSA collapse to chance (~10%).The mitigation methods remain backpropagation variants and retain its time and memory costs.Source: §3.3.2 · pp. 8–9

3.3.3 Are the benchmarks in the barren plateau limited regime?

  • The 2-12-qubit benchmarks are early-onset, not saturated; from 12 to 20 qubits, backpropagation gains about 2 pp while RLQ-Grad follows mitigation methods' about 8.5 pp trajectory.The comparison suggests gradient-freeness is necessary but not sufficient: CMA-ES and SPSA fail, while RLQ-Grad combines it with amortized policy learning for a high-dimensional action space.Source: §3.3.3 · p. 9

4 Limitations and discussions

The discussion identifies three regimes where RLQ-Grad lacks the advantages suggested by its theorems, alongside practical caveats that bound the reported gains.

4 Limitations and discussions

  • In truly exponentially flat cost landscapes, reward differences also vanish exponentially, leaving RLQ-Grad only its computational advantage.Its landscape advantage requires the Jordan algebra to leave the cost non-flat.Source: §4 · p. 9
  • Underparameterized circuits can still converge to poor local minima, with sharp loss basins appearing for every method on the cited F-MNIST configuration.The obstruction applies when the stated overparameterization condition fails.Source: §4 · p. 9
  • When the circuit forms at least a 2-design, input-space Jacobians collapse exponentially, starving classical layers upstream of quantum encoding of gradients.Source: §4 · p. 9
  • Experiments use shot-noise-free statevector simulators and classification tasks; the four-dataset benchmark covers 2–12 qubits, while CIFAR-10 and resource tests extend to 20.At 14–20 qubits RLQ-Grad matches, but does not exceed, dedicated mitigation accuracy; on CPU it is slower than analytic differentiation below 6–8 qubits, and adjoint differentiation uses less memory at every tested scale.Source: §4 · p. 10
  • Agents are trained separately for each circuit-parameter, depth, and dataset configuration, and their training cost is included but not amortized across configurations.Meta-training across scales and tasks is outside the present scope.Source: §4 · p. 10

5 Conclusion

The conclusion presents RLQ-Grad as a classical-policy optimizer with favorable variance, resource, and accuracy results, while the appendices state HEA universality and define barren-plateau behavior. The reported scope includes matching mitigation accuracy on larger CIFAR-10 circuits and leaves several limitations unresolved.

5 Conclusion

  • RLQ-Grad replaces explicit circuit differentiation with a classical policy that directly proposes parameter updates.The conclusion attributes avoidance of analogous concentration pathologies to the finite-width policy MLP's classical training behavior, independent of qubit count.Source: §5 · p. 10
  • Across four supervised benchmarks, RLQ-Grad gains 2%–10% accuracy over gradient-based baselines and matches dedicated mitigation methods on CIFAR-10 up to 20 qubits.The conclusion also reports evolutionary and gradient-free optimizers fail in that CIFAR-10 setting.Source: §5 · p. 10
  • The authors frame decoupling the optimization signal from circuit geometry as complementary to landscape-shaping approaches, matching their accuracy at a fraction of simulation cost.They identify unresolved loss concentration, underparameterization, and classical-preprocessing gradient collapse.Source: §5 · p. 10

A Universal Approximation Capability of HEA

  • With sufficient depth and data re-uploading, the HEA can approximate arbitrary continuous classification functions to any prescribed precision.The passage grounds this capability in universality results for PQCs, including single-qubit QNNs over compact domains.Source: A Universal Approximation Capability… · p. 13

B Quantum Barren Plateaus

  • Barren plateaus are exponential concentration of PQC loss landscapes, where output and gradient variances vanish exponentially with qubit count.For global observables, deep HEA circuits approaching a 2-design are sufficient for exponentially small gradient variance; this raises circuit-evaluation costs for fixed-precision estimates.Source: B Quantum Barren Plateaus · p. 14

C.1 Proof of Theorem 1

The proof contrasts conventional circuit-gradient updates with RLQ-Grad's policy-generated update, which is classical and not subject to the circuit-gradient variance bound. The theorem is structural: it does not establish that the policy's estimates are useful.

C.1 Proof of Theorem 1

  • RLQ-Grad updates parameters using a policy-sampled classical quantity rather than a gradient obtained by differentiating through the quantum circuit.The update distribution depends on agent parameters and state, so the barren-plateau variance bound does not constrain it.Source: §C.1 · p. 14
  • The update's independence from circuit differentiation means it proceeds regardless of circuit depth or qubit count, without inheriting the stated exponential variance collapse.The proof describes the policy output as statistically independent of the quantum loss gradient.Source: §C.1 · p. 14
  • The theorem is structural only: it holds for any policy and circuit, and does not show that the agent produces useful gradient estimates.Source: §C.1 · p. 14

C.2 Proof of Theorem 2

The proof traces the policy-loss gradient through a classical MLP and invokes finite-width network concentration bounds rather than quantum-circuit gradient bounds. The preliminary agent comparison used dataset-specific rollout counts.

C.2 Proof of Theorem 2

  • Differentiating the policy loss with respect to agent weights uses the MLP-output Jacobian and classical backpropagation.Source: §C.2 · p. 14
  • The preliminary agent-type study used 400 rollouts for BC, 46,800 for MNIST and FMNIST, and 39,000 for CIFAR10.Source: Table 4 · p. 15
  • The resulting variance is governed by finite-width MLP bounds, polynomial in network width and depth rather than exponential in qubit count, with no quantum circuit evaluation.Source: §C.2 · p. 15

D.1 Preliminary studies and agent training settings

The preliminary study compares policy-gradient agents and rollout settings, then motivates PPO as the on-policy agent used in the main experiments. Across datasets, replay-based agents generally outperform rollout-based agents, while PPO is competitive with TRPO at lower parameter and MAC counts.

D.1 Preliminary studies and agent training settings

  • Main experiments use PPO with the default Stable-Baselines3 MLP policy, fixed agent initialization seeds, and default SB3 hyperparameters unless specified otherwise.Source: §D.1 · pp. 15–16
  • Replay-based SAC, TD3, and DDPG outperform rollout-based TRPO and PPO across datasets; TD3 has the best overall accuracy.TD3 also has lower variance than SAC despite comparable computational costs.Source: §D.1 · p. 15
  • On CIFAR10, TD3 and SAC reach approximately 28.7% accuracy, versus 26.46% for TRPO and 25.12% for PPO.Source: §D.1 · p. 15
  • PPO is competitive with TRPO despite nearly half the parameters and MACs (5,575 vs. 10,695), while replay-buffer agents consume more memory.Source: §D.1 · p. 15
  • Larger rollout lengths are reported to yield more stable training and improved convergence; Table 5 compares PPO and TRPO across rollout lengths on MNIST.Source: §D.1 · p. 16, Table 5 · p. 16

D.2 Stablization tricks with spectral normalization

This section examines spectral normalization (SN) in PPO, its effects on training stability and accuracy, and its associated trade-offs. It also describes PCA-based input reduction, accuracy-convergence comparisons, and benchmark hardware.

D.2 Stablization tricks with spectral normalization

  • SN caps layer spectral norms to bound the network's Lipschitz constant, limiting error propagation through bootstrapped TD targets and promoting smoother value landscapes.Source: §D.2 · p. 16
  • SN also rescales gradients in a way that inflates Adam's effective damping over training, shifting updates from adaptive behavior toward momentum-based updates with a decaying effective learning rate.Source: §D.2 · p. 17
  • With SN on both policy and value networks, RLQ-Grad reaches near-optimal accuracy (>93%) in about 50 epochs with minimal run-to-run variance.Value-network-only SN reaches comparable final accuracy somewhat later; no SN gives slower, less reliable convergence.Source: §D.2 · p. 18, Figure 4 · p. 18
  • Value-network SN accounts for most stability gains because the critic participates in TD targets, where approximation errors can compound.The policy SN is not required for convergence but measurably speeds it, consistent with its effect on policy-update magnitude.Source: §D.2 · pp. 18–19
  • PPO (SN) outperforms baseline PPO on all datasets, with gains of 5.47% on BC, 5.12% on F-MNIST, and 2.05% on CIFAR10.The parameter count and MACs approximately double, from 5,575 to 11,847 and from 5,440 to 10,432, respectively.Source: Table 6 · p. 19, §D.2 · p. 19

D.3 Input dimensionality reduction

  • PCA projects inputs into a lower-dimensional subspace, with the selected feature count chosen to capture approximately 95% of total variance.Source: §D.3 · p. 19

D.4 Accuracy convergence curves

  • Figure 5 plots validation-accuracy convergence for RLQ-Grad on four datasets using a 2-qubit HEA, against the maximum accuracy of other gradient-computation methods.Source: §D.4 · p. 19, Figure 5 · p. 19

D.5 Hardware specifications

  • Resource benchmarks use CPU measurements for comparisons involving adjoint and parameter-shift methods, and an NVIDIA RTX 3090 GPU for RLQ-Grad versus backpropagation.The CPU comparison averages 12 timed trials after four warm-up iterations; the GPU comparison averages 15 trials.Source: §D.5 · pp. 19–20

D.6 Full resource benchmark values

The resource benchmarks report end-to-end memory and per-iteration timing for gradient-computation methods, with RLQ-Grad measurements covering its full training pipeline.

D.6 Full resource benchmark values

  • RLQ-Grad resource entries account for PPO rollouts, policy and value updates, gradient buffers, and Adam states—not only the actor forward pass.Source: §D.6 · p. 20
  • Table 7 reports end-to-end CPU RAM usage for gradient-computation methods across systems with 2 to 20 qubits.Source: Table 7 · p. 20
  • Tables 8 and 9 report per-iteration CPU wall-clock time and GPU wall-clock time, respectively.Table 9 specifies NVIDIA RTX 3090 measurements and omits parameter-shift and adjoint differentiation because current QNN frameworks lack efficient GPU kernels for them.Source: Table 8 · p. 20, Table 9 · p. 20

E Derivation process of quantum gradient calculation methods

The appendix derives the costs of several quantum gradient methods and of RLQ-Grad, then states conditions for RLQ-Grad's efficient trainability. The analyses distinguish circuit-size costs from policy-network costs and identify underparameterization as an obstruction.

E.1 Forward pass complexity

E.2.1 Backpropagation

  • Backpropagation costs O(L · 2^n) time and memory because it stores all intermediate circuit states.Source: §E.2.1 · p. 20

E.2.2 Parameter-Shift Rule

  • Parameter-shift computes each parameter gradient using full forward passes, giving O(P · L · 2^n) time and theoretical O(2^n) memory.Some frameworks accumulate all parameter gradients, yielding practical memory complexity O(P_QNN · 2^n).Source: §E.2.2 · p. 21

E.2.3 Adjoint differentiation

  • Adjoint differentiation reconstructs intermediate states during the backward sweep, obtaining all P gradients in O(P_QNN · 2^n) time with O(1) additional memory.Source: §E.2.3 · p. 21

E.3 RLQ-Grad

  • RLQ-Grad's actor emits updates through an MLP, with computation and activation memory determined by its width, depth, and QNN parameter count.Including the critic, PPO backpropagation, and Adam states, the full pipeline scales linearly with P_QNN and independently of 2^n.Source: §E.3 · p. 21

F.1 Optimal conditions for RLQ-Grad trainability

  • Efficient RLQ-Grad training requires an informative loss signal and QNN overparameterization, which the theorem links to good local minima being dense near the global minimum.Source: §F.1 · p. 22
  • If either condition fails, an uninformative reward can prevent policy improvement, while underparameterization can leave any convergent optimizer—including RLQ-Grad—at a poor local minimum.Source: §F.1 · p. 22

F.2 Under Wishart process framework, how does this affect agent reward informativeness?

Under the Wishart-process account, loss-based rewards can carry information about gradient magnitude while loss remains distinguishable from zero. Barren plateaus and underparameterized poor local minima impose separate limits on what RLQ-Grad can achieve.

F.2 Under Wishart process framework, how does this affect agent reward informativeness?

  • While loss remains distinguishable from zero, the loss-based reward carries information about gradient magnitude without direct gradient computation.The analysis says reward differences become unresolvable as the loss concentrates in the barren-plateau regime.Source: §F.2 · p. 23
  • In a barren plateau, loss and its correlation with reward vanish together, leaving the agent without a resolvable directional signal.The conditional gradient distribution also concentrates at zero as the loss approaches zero exponentially.Source: §F.2 · p. 23

G.1 Poor local minima obstruction

  • Underparameterized networks have exponentially few local minima near the global minimum, so optimizers converge to poor minima with overwhelming probability.The stated obstruction applies even when gradients are not exponentially suppressed.Source: §G.1 · pp. 23–24

1. Barren plateau affected and underparameterized (doubly obstructed)

  • RLQ-Grad bypasses barren-plateau obstruction 3 but not underparameterized obstruction 2, which is independent of gradient quality.When the loss is truly exponentially flat, its primary advantage is computational rather than landscape-related.Source: §F.2 · p. 23, §1 · p. 24

G.2 Barren plateau induced input gradient decay

For a fixed input, the paper shows that input gradients can concentrate near zero over random circuit initializations, with consequences for hybrid-network backpropagation. It also discusses computational benefits, broader impacts, and deployment precautions for RLQ-Grad.

G.2 Barren plateau induced input gradient decay

  • For a fixed input, an L-layer trainable block forming a unitary 2-design has input-gradient variance bounded by C/2^n.The paper describes concentration over parameter initializations, not over the data distribution.Source: §G.2 · pp. 24–26
  • The full input Jacobian collapses exponentially, so classical layers preceding a quantum encoding also receive exponentially vanishing gradients.The stated bound for the expected squared Jacobian norm is Cn/2^n.Source: §G.2 · p. 26
  • Depolarizing noise doubly exponentially suppresses the input-gradient bound in circuit depth, while local observables reduce variance to O(2^-k).Using k much smaller than n may destroy the expressiveness needed for the learning task, creating an expressiveness–trainability trade-off.Source: §G.2 · pp. 26–27

Negative and dual-use considerations

  • RLQ-Grad does not target a specific deployed application, and its non-analytic update rule may hinder auditing in safety- or compliance-critical pipelines.The paper also notes that the optimizer could be applied to supervised objectives with negative social externalities.Source: H Broader Impact · p. 27, Negative and dual-use considerations · p. 27

Positive impacts

  • At 20 qubits, RLQ-Grad reduces simulation-based QNN training CPU cost by 2 to 4 orders of magnitude after accounting for agent training.The passage reports a small fixed overhead on circuits below roughly 8 qubits.Source: Positive impacts · p. 27

Mitigations

  • The authors recommend logging surrogate and analytic gradients on held-out probes, retaining analytic verification where tractable, and reporting agent-training costs.The cost reporting is intended to support fair comparisons.Source: Mitigations · p. 27
Loading 2609.31066v1…